Skip to content

fix(cluster): handle "Dnode is starting up" error gracefully in checkDnodes method - #34783

Merged
guanshengliang merged 2 commits into
mainfrom
fix/check-dnode-error
Mar 15, 2026
Merged

guanshengliang merged 2 commits into
mainfrom
fix/check-dnode-error

Conversation

@tomchon

@tomchon tomchon commented Mar 15, 2026

Copy link
Copy Markdown
Contributor

No description provided.

@tomchon
tomchon requested a review from a team as a code owner March 15, 2026 02:48
Copilot AI review requested due to automatic review settings March 15, 2026 02:48
@gemini-code-assist

Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request improves the resilience of the cluster checking utility by introducing a mechanism to gracefully handle Dnodes that are still in the process of starting up. Instead of failing immediately, the system will now patiently wait and retry status checks, leading to more stable and reliable test execution during cluster initialization.

Highlights

  • Graceful Dnode Startup Handling: Modified the checkDnodes method in clusterCommonCheck.py to prevent premature exit when Dnodes are still starting up. This is achieved by using exit=False with tdSql.query and implementing a retry mechanism with a timeout.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog
  • test/new_test_framework/utils/clusterCommonCheck.py
    • Implemented a retry mechanism for tdSql.query within the checkDnodes function to handle "Dnode is starting up" errors, preventing immediate exit and allowing the system to wait for Dnodes to become ready.
Activity
  • No human activity has been recorded on this pull request yet.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Handles the "Dnode is starting up" error in checkDnodes by using exit=False on the query call and retrying instead of crashing.

Changes:

  • Modified checkDnodes to gracefully handle query failures during dnode startup by continuing the retry loop

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request aims to handle the 'Dnode is starting up' error more gracefully in the checkDnodes method by preventing the test from exiting prematurely. The change correctly uses the exit=False parameter in tdSql.query. However, this introduces a side effect: the actual timeout becomes significantly longer than intended because of nested retry loops in checkDnodes and tdSql.query. My review includes a suggestion to correct the timeout behavior by disabling the inner retry mechanism, ensuring the method waits for the intended duration.

Comment thread test/new_test_framework/utils/clusterCommonCheck.py Outdated
@guanshengliang
guanshengliang merged commit 8f865da into main Mar 15, 2026
6 of 7 checks passed
@guanshengliang
guanshengliang deleted the fix/check-dnode-error branch March 15, 2026 07:39
jiajingbin pushed a commit that referenced this pull request Mar 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants