Machines

Machines are bare-metal or virtualized computers designated as worker nodes for the Kubernetes Clusters provided by the meltcloud platform.

Everything that can boot from iPXE can be turned into a Machine: a physical server, a virtual machine or even a Raspberry PI.

Architecture

A Machine runs the following software:

• At its heart, it uses the Flatcar Container Linux Operating System, a minimal OS designed to run container workloads.
• It is immutable and stateless by running completely from memory (no disk install)
• It is made up of the following components:
  • the Linux Kernel, glibc and other foundational services
  • systemd to run services
  • melt-agent which interacts with the meltcloud platform to send diagnostics and receive instructions
  • containerd to run containers
  • kubeadm to join the cluster
  • kubelet to run Kubernetes workloads
  • disk configuration (inherited from Machine Pool) to cache docker images on an ephemeral disk
  • network configuration (inherited from Machine Pool) for advanced networking like bonds

Due to its immutable nature, all lifecycle operations (patching, assigning to a pool, upgrading) are done via a machine reboot.

The platform ensures that the OS is kept up to date – roughly every 2-4 weeks new Flatcar Linux versions are released and made available to Machines.

How to add Machines

Machines can be added to your organization in one of two ways:

Self-Register

If a Machine boots from iPXE Boot Artifact or a iPXE Chain URL, it automatically registers itself with its unique BIOS UUID in the associated organization in meltcloud.

If a Machine with the UUID already exists (i.e. on subsequent boot or if it was pre-registered), then the configuration for the Machine with this UUID is applied.

Once the Machine has registered itself, it will be in status ready_unassigned. From this point onwards, it can be managed in the Console or API, for example assigned to a Machine Pool.

The Tutorials in our Guides (Provision on Bare Metal, Provision on Equinix Metal, Provision on QEMU) use this approach to add a Machine.

Pre-Register

If you have a fleet of servers and want to configure them in meltcloud before boot (i.e. already assign them to pools, give them names etc.), you can pre-register the machines in advance:

• Go To Machines → Register Machine
• Enter the UUID of the future Machine, set Hostnames and other settings (i.e. assignments to machine pools)
• Hit Save Machine

Once your machines boot, they will automatically take over the configuration prepared for the specific UUID.

Revisions

Most changes to a Machine (i.e. a hostname change or changes in the assigned Machine Pools like Kubernetes Version Upgrades) require a reboot to adopt the new settings.

To allow for a controlled rollout that doesn't disrupt the workloads, Revisions were introduced:

• For each change on machines that require a reboot, a new Revision containing the new settings is created
• The revision can either be rolled out to the Machine instantly (by saving the changes with Save Machine & Apply) or applied manually (by saving the changes with Save Machine and applying the revision in the Revisions tab later)

Revision Rollout

When a revision is being applied, the following happens:

• The Machine is cordoned and drained on the Kubernetes API
• The Machine is gracefully stopped, boots into the new revision and joins the cluster again
• The Machine is uncordened on the Kubernetes API
• receives workload again.

The rollout operation is fully transparent and can be observed in the operations log. If the operation fails or times out, it will transition to the failed state and can be retried afterward.

Troubleshooting

If the Machine has trouble booting, is unresponsive or otherwise stuck, consider the following steps:

Operations

The Operations tab shows recent or running operations accompanied with detailed logs what's happening. Check them first to see if something is going on or went wrong.

You can also apply an older, working revision (this should trigger a reboot if the melt-agent is still reachable) or reapply the current one.

Timestamps

In the Info sections are various timestamps which hint at the current state of the machine:

• Last Boot at: When the Machine had its last successful iPXE Boot reaching meltcloud
• Last Ignition at: When the Machine had its last successful Ignition (Flatcar booting)
• Agent Hearbeat: When the Machine last fully booted, was able to start the melt-agent and reached meltcloud

Startup Logs

If the Machine booted successfully, the startup logs (basically all systemd logs) can be found in the Console:

Serial console

If the Machine is stuck in the earlier stages of boot (i.e. during iPXE or Flatcar Ignition), you can only find the error by looking at the serial console.

Force unassign / delete

When all else fails and you consider your Machine lost, you can Force Unassign and delete the Machine to start from scratch.