Pod restart strategy

When using immediate mode, you can optionally enable one-by-one pod restart to minimize disruption during resource adjustments. This is useful for workloads without Pod Disruption Budgets or with aggressive rollout strategies that might cause multiple pods to restart simultaneously.

How it works

When enabled, the one-by-one restart changes how immediate mode applies recommendations:

1. Sequential processing: The system applies the recommendation to a single pod at a time
2. Health verification: Waits for each pod to become healthy and ready before proceeding
3. Gradual rollout: Moves to the next pod only after the previous one is successfully running
4. Automatic recovery: If a pod fails to start, the rollout pauses until the workload recovers

This ensures gradual rollouts and maintains service availability throughout the resource adjustment process.

Enabling one-by-one restart

You can enable this feature through the Cast AI console:

1. Navigate to Workload Autoscaler → Scaling policies
2. Select or create a policy with When to apply changes set to immediately
3. Under Advanced settings, locate Rollout strategy
4. Check the Restart pods one by one option
5. Save the policy

You can also configure this setting using annotations:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
  annotations:
    workloads.cast.ai/configuration: |
      vertical:
        optimization: on
        applyType: immediate
      rolloutBehavior:
        type: NoDisruption
        preferOneByOne: true
spec:
  # deployment spec...

For more information on configuring workloads via annotations, see Annotations reference.

📘

Note

The one-by-one restart feature is currently opt-in. Based on customer feedback and continued improvements, it may become the default behavior in system policies in future releases.

Supported workload types

The one-by-one restart feature works with the following workload types:

• Deployments
• StatefulSets
• ReplicaSets
• ArgoCD Rollouts

Not supported: Bare pods, CronJobs, Jobs, and other non-replicated workloads.

Version requirements

To use the one-by-one restart feature, ensure you have the following minimum versions:

Considerations and limitations

When using one-by-one restart, be aware of these considerations:

Rollout duration: For workloads with many replicas (100+), the gradual rollout process may take longer depending on pod startup time. This is expected behavior as each pod must become healthy before the next one is processed.

Mid-rollout recommendations: If a new recommendation is generated while a rollout is in progress, it takes precedence. For example, if half the pods have been updated with an immediate recommendation and a deferred recommendation is then issued, the rollout will stop. The new deferred recommendation will be applied to the remaining pods only during their natural restarts.

Failed pod scenarios: If a pod fails to start during the rollout (for example, due to image pull errors or application crashes), the entire rollout pauses until the workload recovers. This prevents further disruption but requires manual intervention to resolve the underlying issue.

OOM kills and surge handling: When enabled, OOM kill responses and surge-triggered adjustments also follow the one-by-one pattern, which may delay the system's response to these events.