run a 6000 node scale test

[upodroid]

5k kops scale test has been passing green for a while, lets push the limits further.

https://testgrid.k8s.io/sig-scalability-gce#gce-master-scale-performance-5000

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: upodroid

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• config/jobs/kubernetes/sig-scalability/OWNERS [upodroid]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[serathius]

I'm against merging this.

The goal of our scalability tests is to protect our current 5k node goal and prevent regressions, not to push the limits. Increasing the node count comes with performance trade-offs and adds complexity that would slow down project velocity, which we can't do lightly.

Let's stick to the current 5k node goal.

[aojea]

@upodroid is there some strategic intention to increase the limit or just to see what happens?

[upodroid]

We have plenty of compute credits on GCP to spend over the next 60 days (about 500k+ USD spare money projected) so I'm using it to push scale limits further as well as running more scale scenarios such as DRA at 5k + kops at 5k using nftables/kindnet, etc + large resource size.

[serathius]

/cc @wojtek-t @liggitt @thockin

[thockin]

I love the idea of pushing a little, but we need to consider the cost increase. Tagging the TLs of this group.

[ameukam]

I'm against merging this.

The goal of our scalability tests is to protect our current 5k node goal and prevent regressions, not to push the limits. Increasing the node count comes with performance trade-offs and adds complexity that would slow down project velocity, which we can't do lightly.

Let's stick to the current 5k node goal.

@upodroid is it a permanent experiment or something bound to our 2025 budget (which means in 2026 we can drop it) ?

[serathius]

I don't think there was any experiments planned in scalability. Instead of burning our budget on a 6k test that won't give us a new signal, let's use these resources to address our known blind spots, as outlined in #134375.

[upodroid]

It's temporary till the end of the year.

Marek, we are running other scenarios such as DRA, large resource sizes and kubeup to kops migration and we still have money left over to use for experiments like these.