🔄 Kubernetes Home Cluster Automation: Rejoining Worker Nodes After Reboot

Hello, Kubernetes home lab enthusiasts! 🏠

Have you ever experienced this? You’ve beautifully set up your Kubernetes cluster, but after rebooting your host PC or recovering from a power outage, you find all your worker nodes missing from the cluster! 😱 Today, I’ll show you an effective way to solve this annoying problem.

📋 The Problem: Worker Nodes Disconnect After Reboot

When running Kubernetes in a home environment, you occasionally need to reboot your system for various reasons. The issue is that after rebooting, while the master node operates normally, worker nodes don’t automatically reconnect to the cluster.

Common symptoms:

• Running kubectl get nodes shows worker nodes are missing or in NotReady state
• The kubelet service on worker nodes repeatedly restarts and fails
• Worker node logs show authentication-related errors

Solving this manually every time is too cumbersome. Fortunately, there’s an excellent way to automate this using Ansible! 🛠️

🧩 Root Cause Analysis: Why Does This Happen?

The main reasons why worker nodes disconnect after a reboot include:

1. Certificate and Token Issues: Kubernetes uses certificates for inter-node communication, and these authentication credentials might not be properly restored after a reboot.

2. Kubelet Configuration Changes: During the reboot process, kubelet settings might get corrupted or modified.

3. Network Initialization Timing: Container networking might initialize after kubelet, causing connectivity issues.

To resolve these problems, you need to rejoin nodes to the cluster.

🚀 Solution: Automated Rejoining with Ansible

Using an Ansible playbook to automate this process is the most effective approach. This is particularly convenient if you’ve already built your cluster using Vagrant and Ansible.

Creating a Dedicated Ansible Playbook for Rejoining

Here’s a simple Ansible playbook to reconnect worker nodes to the cluster:

---
# rejoin-worker.yml - Worker Node Rejoining Playbook
- name: Rejoin worker nodes to Kubernetes cluster
  hosts: workers
  become: yes
  tasks:
    - name: Check if node is already joined
      shell: kubectl get nodes | grep -w $(hostname)
      register: node_joined
      delegate_to: k8s-master
      become: yes
      become_user: vagrant
      environment:
        KUBECONFIG: /home/vagrant/.kube/config
      failed_when: false
      changed_when: false
      
    - name: Reset kubeadm if needed
      shell: kubeadm reset -f
      when: node_joined.rc != 0
      
    - name: Get new join command from master
      shell: kubeadm token create --print-join-command
      register: join_command
      delegate_to: k8s-master
      become: yes
      when: node_joined.rc != 0
      
    - name: Join the cluster
      shell: "{{ join_command.stdout }}"
      when: node_joined.rc != 0
      
    - name: Verify worker joined successfully
      shell: kubectl get nodes
      delegate_to: k8s-master
      become: yes
      become_user: vagrant
      environment:
        KUBECONFIG: /home/vagrant/.kube/config
      register: nodes_status
      
    - name: Show nodes status
      debug:
        var: nodes_status.stdout_lines

This playbook works with a simple logic:

1. Check if the worker node is already connected to the cluster ✅
2. If not connected, reset kubeadm ♻️
3. Generate a new join command from the master node 🔑
4. Reconnect the worker node to the cluster with that command 🔗

Using the Rejoining Playbook

To reconnect worker nodes after a reboot, simply run this command:

ansible-playbook -i inventory.yml rejoin-worker.yml

Simple, right? Now you can reconnect worker nodes to the cluster with just this one-line command after rebooting your host PC or VMs! 🎉

🔄 Advanced Automation Methods

You can implement more advanced automation using this playbook:

1. Create a Script and Register an Alias

Create a shell script to run the command more easily:

#!/bin/bash
cd /path/to/ansible/files
ansible-playbook -i inventory.yml rejoin-worker.yml

Save this script to /usr/local/bin/k8s-rejoin and make it executable (chmod +x /usr/local/bin/k8s-rejoin), then you can reconnect anytime with the k8s-rejoin command.

2. Automatic Execution at System Startup

To automatically reconnect worker nodes after system reboot, use cron’s @reboot feature:

# Edit with crontab -e
@reboot sleep 120 && /usr/local/bin/k8s-rejoin >> /var/log/k8s-rejoin.log 2>&1

This sets up the rejoining script to run after waiting 2 minutes post-boot. The delay ensures network and basic services are fully initialized.

3. Set Up Periodic Checks

You can also set up periodic checks to verify and restore connections:

# Check and reconnect every hour
0 * * * * /usr/local/bin/k8s-rejoin >> /var/log/k8s-rejoin.log 2>&1

💡 Additional Tips: Advice for More Effective Cluster Management

1. Monitor Kubelet Logs: Regularly check kubelet logs to understand the root cause of reconnection issues.

   sudo journalctl -u kubelet -n 100
   

2. Drain Nodes Before Planned Reboots: Before planned reboots, drain nodes to safely move workloads to other nodes.

   kubectl drain k8s-worker --ignore-daemonsets
   

3. Consider High Availability Setup: If running important workloads, consider configuring multiple master nodes for high availability.

🎯 Conclusion

When operating Kubernetes in a home environment, unexpected situations can always arise. The issue of worker nodes disconnecting after a reboot is just one of them. However, by using automation tools like Ansible, these problems can be easily resolved.

Now you can reconnect worker nodes with a single command, and with automation settings, you can even skip this step entirely! Stop wasting unnecessary time on cluster management. 🕒

Have you applied this method? Or do you have other good solutions? Please share your experiences in the comments! I hope we can become a Kubernetes community that learns and grows together. 😊