Pod eviction timeout is too aggressive on certain configurations

[norbertcyran]

In scale-down, there's a timeout on initiating eviction of a pod. It's controlled by max-pod-eviction-time flag, which defaults to 2 minutes.

In some scenarios this is too aggressive. Recreating a pod protected by PDB can take much longer than that, especially if things like termination grace period, startup probe or readiness probe are configured.

The user can just increase that timeout with the flag, but that's not perfect either. Eviction can fail due to other issues, like misconfiguration of the PDB or the workload.

Ideally, there would be a smarter mechanism of evicting the pods, possible improvements could include:

• having an overall node drain timeout instead of a timeout for a single pod eviction
• differentiating between 429 and 500 errors from eviction API, retrying only on 429
• making the timeout dynamic based on the workload's termination grace period, readiness probe's initialDelaySeconds, and possibly other configurations

/kind feature
/area cluster-autoscaler

[x13n]

• having an overall node drain timeout instead of a timeout for a single pod eviction

+1 to this one. We already have drain config, it should be updated to work that way.

• differentiating between 429 and 500 errors from eviction API, retrying only on 429

500 errors are internal (hopefully transient) errors, why wouldn't we retry on them?

• making the timeout dynamic based on the workload's termination grace period, readiness probe's initialDelaySeconds, and possibly other configurations

This will be tricky and brittle. With multiple pods targeting the same PDB on a single node (forcing them to be drained one by one), even random delays on apiserver can add up.

[norbertcyran]

500 errors are internal (hopefully transient) errors, why wouldn't we retry on them?

Apparently not in this case. From the docs:

500 Internal Server Error: the eviction is not allowed because there is a misconfiguration, like if multiple PodDisruptionBudgets reference the same Pod.

Even if those were internal transient errors, it would probably make sense to retry on them for a shorter period than in case of waiting for PDBs

[x13n]

This is very unfortunate, since we cannot distinguish real server errors from PDB preventing pods from getting evicted. Anyway, I'd keep retry behavior simple here and if we want to reduce load on the apiserver, a better solution would be to keep a PDB informer and track allowed disruptions in the node drain process to queue pods targeting the same PDB.