Upgrade Machine Pools

Revisions

Most changes to a Machine Pool (i.e. Kubernetes version upgrades or changes to network or disk settings) require a reboot of the assigned machines to adopt the new settings.

To allow for a controlled rollout that doesn't disrupt the workloads, Revisions were introduced:

• For each change on machine pools that require a reboot, a new Revision containing the new settings is created
• The revision can either be rolled out to the pool instantly (by saving the changes with Save Machine Pool & Apply) or applied manually (by saving the changes with Save Machine Pool and applying the revision in the Revisions tab later)

Revision Rollout

When a revision is being applied, the following happens:

1. Perform sanity checks that the cluster, the machine pools and the machines are in a healthy state
2. In a rolling-upgrade strategy, one machine after another...
   • receives a new Machine Revision that references the Machine Pool Revision with the new config
   • is cordoned and drained on the Kubernetes API
   • is gracefully stopped, boots into the new revision and joins the cluster again
   • is uncordened on the Kubernetes API
   • receives workload again.
3. After all machines in the pool are on the new revision, the rollout is finished.

INFO

In order for the rolling-upgrade to happen without disruptions, ensure to have spare capacity to tolerate one node being unavailable. Also, as with any Kubernetes setup, ensure that critical applications that can't tolerate downtimes properly have their Pod Disruption Budget configured.

The rollout operation is fully transparent and can be observed in the operations log. If the operation fails or times out, it will transition to the failed state and can be retried afterward.

Rolling out manually

If you are rolling out the changes to the pool for a first time and want to be extra careful that the upgrade works gracefully, you can roll out the upgrade manually:

• Save your changes with Save Machine Pool
• In the Revisions tab, hit Apply on the desired revision
• Select Yes, but apply manually when asked for the apply.

This will create new Machine Revisions on each Machine, but not apply them yet.

One by one, apply the revision of the selected machine (via Machines → Machine Name → Revisions Tab → Apply) and watch that the roll out doesn't cause disruptions.

Shared Responsibility Model

• The user is responsible for setting network and disk configuration and triggering the upgrade of the machine pool to newer Kubernetes minor versions.
• The meltcloud platform is responsible for upgrading the machine pool (i.e. the Kubelet) to the latest Kubernetes patch version. To achieve this, the platform will periodically create new revisions with the new patch versions which can be applied by the user (and, in the future, automatically in a configurable maintenance window)

Upgrade a Cluster to a newer minor version

1. Navigate to the Machine Pool → Actions → Edit Machine Pool
2. Select the desired Kubernetes minor version.
3. Hit Save Machine Pool and apply the revision later, or use Save Machine Pool & Apply for immediate rollout (see details on Revisions above)