enh(test): improve random database creation with customizable row count and batch inserts

[tomchon]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request refines the test suite's data generation capabilities by introducing a configurable row count for random database creation and optimizing data insertion through batching. These improvements enhance the efficiency and adaptability of test data setup for a wide range of test scenarios.

Highlights

• Customizable Random Data Generation: The dropandcreateDB_random helper function has been updated across multiple test files to accept a num_random parameter. This allows test cases to specify the number of random rows to insert into databases, providing greater flexibility and control over test data volume.
• Batch Insert Optimization: The dropandcreateDB_random function in test_nestedQuery.py (and its uncataloged counterpart) has been refactored to utilize batch inserts. This significantly improves the performance of test data generation by reducing the number of individual SQL INSERT statements executed.
• Standardized num_random Usage: Existing calls to dropandcreateDB_random in various test files have been updated to explicitly pass a num_random value, often setting it to 50, ensuring consistent and explicit control over the generated data size.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• test/cases/06-DataIngestion/01-SQL/test_write_basic.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 100.
  • Modified the call to dropandcreateDB_random in do_insert_select to explicitly set num_random=50.
• test/cases/06-DataIngestion/01-SQL/test_write_null_none.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 10.
  • Removed the hardcoded num_random = 10 assignment within the dropandcreateDB_random function.
  • Modified the call to dropandcreateDB_random in test_insert_null_none to explicitly set num_random=10.
• test/cases/09-DataQuerying/03-GroupBy/test_query_groupby_alwaysreturn.py
  • Modified the call to dropandcreateDB_random to explicitly set num_random=50.
• test/cases/09-DataQuerying/03-GroupBy/test_query_groupby_noreturn.py
  • Modified the call to dropandcreateDB_random to explicitly set num_random=50.
• test/cases/09-DataQuerying/08-SubQuery/test_nestedQuery.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 100.
  • Removed the hardcoded num_random = 100 assignment within the dropandcreateDB_random function.
  • Refactored dropandcreateDB_random to use batch inserts for populating tables, improving performance.
  • Modified calls to dropandcreateDB_random in case_init, math_nest, str_nest, time_nest, base_nest, and function_before_26 to explicitly set num_random=50.
• test/cases/09-DataQuerying/08-SubQuery/test_nestedQuery_time.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 100.
  • Removed the hardcoded num_random = 100 assignment within the dropandcreateDB_random function.
  • Modified calls to dropandcreateDB_random in case_init and time_nest to explicitly set num_random=50.
• test/cases/09-DataQuerying/13-CaseWhen/test_query_case_when.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 10.
  • Removed the hardcoded num_random = 10 assignment within the dropandcreateDB_random function.
  • Modified the call to dropandcreateDB_random in do_system_test_case_when to explicitly set num_random=10.
• test/cases/09-DataQuerying/18-Stability/test_query_stability.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 100.
  • Removed the hardcoded num_random = 100 assignment within the dropandcreateDB_random function.
  • Modified calls to dropandcreateDB_random in case_init, function_before_26, and test_stablity to explicitly set num_random=50.
  • Updated commented-out calls to dropandcreateDB_random in math_nest, str_nest, and time_nest to include num_random=50.
• test/cases/09-DataQuerying/18-Stability/test_query_stability26.py
  • Modified the call to dropandcreateDB_random to explicitly set num_random=50.
• test/cases/11-Functions/02-Aggregate/test_agg_bugs.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 10.
  • Removed the hardcoded num_random = 10 assignment within the dropandcreateDB_random function.
  • Modified the call to dropandcreateDB_random in do_td_21561 to explicitly set num_random=50.
• test/cases/11-Functions/03-Selection/test_select_last_model.py
  • Modified the call to dropandcreateDB_random to explicitly set num_random=50.
• test/cases/14-JoinQueries/test_join_func.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 100.
  • Removed the hardcoded num_random = 100 assignment within the dropandcreateDB_random function.
  • Modified calls to dropandcreateDB_random in function_before_26 and test_para_tms to explicitly set num_random=50.
  • Updated commented-out calls to dropandcreateDB_random in math_nest, str_nest, and time_nest to include num_random=50.
• test/cases/14-JoinQueries/test_join_func_cursor.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 100.
  • Removed the hardcoded num_random = 100 assignment within the dropandcreateDB_random function.
  • Modified calls to dropandcreateDB_random in case_init, math_nest, str_nest, time_nest, base_nest, and function_before_26 to explicitly set num_random=50.
• test/cases/uncatalog/system-test/2-query/test_max_min_data.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 5.
  • Removed the hardcoded num_random = 5 assignment within the dropandcreateDB_random function.
  • Modified calls to dropandcreateDB_random in test_max_min_data to explicitly set num_random=50.
• test/cases/uncatalog/system-test/2-query/test_nestedQuery.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 100.
  • Removed the hardcoded num_random = 100 assignment within the dropandcreateDB_random function.
  • Refactored dropandcreateDB_random to use batch inserts for populating tables, improving performance.
  • Modified calls to dropandcreateDB_random in case_init, math_nest, str_nest, time_nest, base_nest, and function_before_26 to explicitly set num_random=50.
• test/cases/uncatalog/system-test/2-query/test_para_tms.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 100.
  • Removed the hardcoded num_random = 100 assignment within the dropandcreateDB_random function.
  • Modified calls to dropandcreateDB_random in function_before_26 and test_para_tms to explicitly set num_random=50.
  • Updated commented-out calls to dropandcreateDB_random in math_nest, str_nest, and time_nest to include num_random=50.
• test/cases/uncatalog/system-test/2-query/test_para_tms2.py
  • Updated dropandcreateDB_random function signature to include a num_random parameter with a default value of 100.
  • Removed the hardcoded num_random = 100 assignment within the dropandcreateDB_random function.
  • Modified calls to dropandcreateDB_random in case_init, math_nest, str_nest, time_nest, base_nest, and function_before_26 to explicitly set num_random=50.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request enhances the test data generation by making the number of random rows configurable in dropandcreateDB_random and by introducing batch inserts for better performance. The changes are applied across numerous test files. While the batch insertion is a good improvement, the implementation in test_nestedQuery.py contains significant code duplication that should be refactored for better maintainability. My review includes a suggestion to address this.

[gemini-code-assist]

The logic to generate values for stable_1_1 and regular_table_1 is identical, leading to code duplication. You can generate the value string once and append it to both lists.

This pattern of duplication also appears for other tables (stable_1_2 vs regular_table_2, and within stable_2_1 variants) and in the final block of tdSql.execute calls. Consider refactoring the entire data generation and insertion logic to be more modular, for example by using helper functions. This will improve readability and maintainability.

Suggested change

	# stable_1_1 values
	values_stable_1_1.append('''(%d, %d, %d, %d, %d, %f, %f, 0, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s',
	'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s')'''
	% (ts + i*1000, fake.random_int(min=-2147483647, max=2147483647, step=1),
	fake.random_int(min=-9223372036854775807, max=9223372036854775807, step=1),
	fake.random_int(min=-32767, max=32767, step=1) , fake.random_int(min=-127, max=127, step=1) ,
	fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() ,
	fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr()))
	tdSql.execute('''insert into regular_table_1 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double, q_bool , q_binary , q_nchar, q_ts,\
	q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8) \
	values(%d, %d, %d, %d, %d, %f, %f, 0, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \
	'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;'''
	# regular_table_1 values
	values_regular_table_1.append('''(%d, %d, %d, %d, %d, %f, %f, 0, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s',
	'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s')'''
	% (ts + i*1000, fake.random_int(min=-2147483647, max=2147483647, step=1) ,
	fake.random_int(min=-9223372036854775807, max=9223372036854775807, step=1) ,
	fake.random_int(min=-32767, max=32767, step=1) , fake.random_int(min=-127, max=127, step=1) ,
	fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() ,
	fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr()))
	# stable_1_1 and regular_table_1 values
	row_values = '''(%d, %d, %d, %d, %d, %f, %f, 0, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s',
	'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s')''' % (ts + i*1000, fake.random_int(min=-2147483647, max=2147483647, step=1) ,
	fake.random_int(min=-9223372036854775807, max=9223372036854775807, step=1) ,
	fake.random_int(min=-32767, max=32767, step=1) , fake.random_int(min=-127, max=127, step=1) ,
	fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() ,
	fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())
	values_stable_1_1.append(row_values)
	values_regular_table_1.append(row_values)

[Copilot]

Pull request overview

This PR standardizes/extends several test helpers that create random databases by adding a num_random parameter (to control generated table/row volume), updates many tests to call these helpers with num_random=50, and refactors one nested-query data generator to use batched inserts for better performance.

Changes:

• Add num_random parameter to multiple dropandcreateDB_random() helpers and update callers to pass num_random=50.
• Implement batched insert generation in test/cases/09-DataQuerying/08-SubQuery/test_nestedQuery.py to reduce per-row execute overhead.
• Adjust a handful of existing test cases to use the new helper signature.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
test/cases/uncatalog/system-test/2-query/test_para_tms2.py	Adds num_random to DB generator and updates calls to generate fewer rows.
test/cases/uncatalog/system-test/2-query/test_para_tms.py	Adds num_random to DB generator and updates call sites (including commented examples).
test/cases/uncatalog/system-test/2-query/test_nestedQuery.py	Adds num_random and updates call sites to generate fewer rows.
test/cases/uncatalog/system-test/2-query/test_max_min_data.py	Adds num_random and updates calls (but currently conflicts with hard-coded expectations).
test/cases/14-JoinQueries/test_join_func_cursor.py	Adds num_random and updates call sites to generate fewer rows.
test/cases/14-JoinQueries/test_join_func.py	Adds num_random and updates call sites to generate fewer rows.
test/cases/11-Functions/03-Selection/test_select_last_model.py	Updates NestedQueryHelper.dropandcreateDB_random() call to pass num_random=50.
test/cases/11-Functions/02-Aggregate/test_agg_bugs.py	Adds num_random and updates call site to use num_random=50.
test/cases/09-DataQuerying/18-Stability/test_query_stability26.py	Updates helper call to pass num_random=50.
test/cases/09-DataQuerying/18-Stability/test_query_stability.py	Adds num_random and updates call sites to pass num_random=50.
test/cases/09-DataQuerying/13-CaseWhen/test_query_case_when.py	Adds num_random and updates call to pass num_random explicitly.
test/cases/09-DataQuerying/08-SubQuery/test_nestedQuery_time.py	Adds num_random and updates call sites to pass num_random=50.
test/cases/09-DataQuerying/08-SubQuery/test_nestedQuery.py	Adds num_random, switches inserts to batching, and updates call sites to pass num_random=50.
test/cases/09-DataQuerying/03-GroupBy/test_query_groupby_noreturn.py	Updates helper call to pass num_random=50.
test/cases/09-DataQuerying/03-GroupBy/test_query_groupby_alwaysreturn.py	Updates helper call to pass num_random=50.
test/cases/06-DataIngestion/01-SQL/test_write_null_none.py	Adds num_random and updates call to pass num_random explicitly.
test/cases/06-DataIngestion/01-SQL/test_write_basic.py	Adds num_random parameter and updates a call site (but helper does not currently use num_random).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.