[COR-4] Cleanup generate_config

[mpoeter]

Scope & Purpose

Previously we had a special add-yaml attribute in the tests/kafka-test-definitions.yml and the generate_config would create a new job based on it. This PR instead adds an explicit kafka-driver-tests with all the containers and has the test-definition refer to that job. This way we avoid the whole magic of creating new jobs with some local yaml snippets.

• 🔨 Refactoring/simplification

---

Note

Replaces add-yaml logic with explicit kafka-driver-tests job and refactors CI config/scripts to use shared test params and new driver/Kafka setup.

• CI config (.circleci/base_config.yml)
  • Introduces shared test parameters (.shared-test-job-parameters) and reusable commands (clone-driver-repo, run-test).
  • Adds jobs: run-driver-tests and kafka-driver-tests (spins up Kafka brokers and schema registry; waits for readiness; sets KAFKA_HOST_LIST/KAFKA_SCHEMA_HOST).
  • Refactors run-linux-tests to use shared params; adds kafkaHostList pipeline parameter.
• Config generator (.circleci/generate_config.py)
  • Removes add-yaml handling; tests now specify target job via job key.
  • Simplifies YAML readers and returns; propagates driver repo/branch and optional init_command.
  • Cleans up unused fields and wiring; supports multi-bucket suites with explicit job.
• Test definitions (tests/kafka-test-definitions.yml)
  • Drops add-yaml; points suite to kafka-driver-tests.
  • Uses env vars for kafkaHost/kafkaSchemaHost instead of hardcoded hosts.

^{Written by Cursor Bugbot for commit f43e88a. This will update automatically on new commits. Configure here.}

[Copilot]

Pull Request Overview

This PR refactors the test configuration system by removing the dynamic job generation mechanism (add-yaml) and replacing it with an explicit kafka-driver-tests job definition. The changes simplify the configuration by eliminating magic job creation and moving Kafka-specific setup from the test definition file into the base CircleCI config.

Key changes:

• Removed add-yaml dynamic job generation logic from generate_config.py
• Added explicit kafka-driver-tests job in base_config.yml with Kafka containers
• Updated test definitions to reference the new job directly and use environment variables

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File	Description
tests/kafka-test-definitions.yml	Removed add-yaml section and inline Kafka container definitions; updated to reference kafka-driver-tests job and use environment variables for hosts
.circleci/generate_config.py	Removed all add-yaml parsing and dynamic job generation logic; simplified function signatures by removing yaml_struct parameter
.circleci/base_config.yml	Added explicit kafka-driver-tests job with Kafka containers; extracted run-test command and refactored run-linux-tests to use it

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

This logic sets init_driver_repo_command to an empty string when not present, but the new kafka-driver-tests job hardcodes the command at line 727. Consider whether this empty string default is still needed or if it should be removed for consistency.

[dothebart]

I also dislike having this set/configured in two places.

[dothebart]

why isn't <<: *test-job-parameters used here as well?

[mpoeter]

Because the parameter list is slightly different. We could factor out the shared parameters and put them into a separate achor though.

[dothebart]

I also dislike having this set/configured in two places.