Taking control of flagger canary pods

Photo by Juan Emilio

As part of continuous deployment you want to be assured that a new release won’t break anything when it’s deployed. To that end you can use canaries — a fraction of the traffic will be directed to pods running the new version of the application. If certain thresholds aren’t met, e.g. a percentage of successful requests or an average maximum request duration, then the canary pods will be rolled back and the previous deployment remains active.

Flagger and Istio can be used to accomplish this type of canary deployment. Flagger makes a copy of the deployment and HPA, which it then uses to scale replicas up and down as required. At a certain point there could be twice the number of pods running as usual. The issue with this is that quotas can block scaling up canary pods for HPAs with a large number of replicas, halting the deployment.

If you want to use your own HPAs for complete control of the canary pods then you need to:

1. Remove the autoscalerRef block in your canary resource definition
2. Adjust the existing “primary” HPA to reference the deployment that flagger creates automatically. This is done by changing scaleTargetRef to point to the deployment with the “-primary” suffix
3. Create a new HPA to control the canary pods. It should reference the deployment that the “primary” HPA was referencing before carrying out step 2. i.e. without the “-primary” suffix. Also give it a name to distinguish it from the primary HPA, such as with a prefix “-customcanary”
4. This step is only needed if the application was already deployed to kubernetes with the canary enabled prior to making these changes. There will be three HPAs after deployment —two referencing the primary deployment and one referencing the canary. Delete the one referencing the primary that was automatically created by flagger, i.e. that doesn’t match your own HPA definition