🚀 Building a Kubernetes Home Cluster: A Problem-Solving Guide

Hello, Kubernetes enthusiasts! Today I’m sharing the challenges I faced while building a Kubernetes cluster at home and the solutions I discovered along the way. In my previous post, we set up a simple cluster using k3s, but this time we’ll install standard Kubernetes (kubeadm) and automate everything with Vagrant and Ansible. Ready to dive in? 😄

🌐 Overall Network Architecture

First, let’s look at the big picture of our system:

moolmeow.com domain
├── Frontend: Static site deployed on Vercel
└── Backend API: api.moolmeow.com → Home public IP → Home cluster

Detailed Network Flow:

1. User → Domain: User accesses api.moolmeow.com
2. Domain → Public IP: DDNS service connects domain to current home public IP
3. Public IP → Mikrotik Router: Ports 80/443 forwarding
4. Router → Mac Mini → VM: Host-only network communication
5. VM → Kubernetes: Ingress and MetalLB route traffic

With this setup, you can run a home Kubernetes cluster that’s accessible from the internet!

🛠️ VM Setup with Vagrant: Problems and Solutions

Compared to our previous k3s setup, I encountered several issues when installing standard Kubernetes.

Problem 1: VirtualBox Guest Additions Installation Failure

Symptom: VMs were created but shared folders weren’t mounting

Solution:

# Connect to VM and install manually
sudo apt-get install -y dkms build-essential linux-headers-$(uname -r)
sudo /opt/VBoxGuestAdditions*/init/vboxadd setup

I automated this in the Ansible playbook by:

1. Installing required packages
2. Running the Guest Additions setup script
3. Rebooting to apply changes

Problem 2: Ansible SSH Connection Issues

Symptom:

ansible-playbook -i inventory.yml main.yml
SSH Error: Permission denied

Root Cause Analysis: The inventory file had incorrect SSH settings. Vagrant uses port forwarding for SSH, but I was trying to connect directly to the VM’s IP.

Solution:

# Modify inventory.yml
k8s-master:
  ansible_host: 127.0.0.1  # Use localhost
  ansible_port: 2222       # Vagrant's forwarded port
  ansible_user: vagrant
  ansible_ssh_private_key_file: ./.vagrant/machines/k8s-master/virtualbox/private_key

Using localhost with the forwarded port resolved the SSH connection issues!

Problem 3: Environment-specific Storage Path Configuration

Issue: Using hardcoded paths reduces portability

Solution: Modified Vagrant configuration to reference environment variables and home directory:

home_dir = ENV['HOME']
mount_base_path = "#{home_dir}/Desktop/moolmeow"

This way, the correct paths are referenced regardless of which user’s system it runs on!

🤖 Key Points in Ansible Automation

Here are the problems I encountered with Ansible automation and their solutions:

Problem: Playbook Structure and Dependency Management

Initially started with a single playbook but ran into issues as complexity grew.

Solution: Split playbooks by functionality:

• common.yml: Basic setup for all nodes (packages, kernel settings, etc.)
• master.yml: Master node configuration (cluster init, network plugins, etc.)
• worker.yml: Worker node joining
• storage-volumes.yml: Environment-specific storage volume setup

Each playbook handles a specific role, imported in sequence by main.yml.

Problem: YAML Syntax Errors

Frequent syntax errors related to YAML indentation.

Error Example:

ERROR! Syntax Error while loading YAML.
  found character that cannot start any token

Solution: Carefully check indentation and use YAML validation tools:

yamllint common.yml

Problem: Conflicts on Repeated Execution

Errors like “already exists” when running playbooks multiple times.

Solution: Add conditional execution and error handling logic:

failed_when: 
  - result.rc != 0 
  - "'already exists' not in result.stderr"

This prevents the playbook from failing when resources already exist!

💾 Key Points in Storage Configuration

Lessons learned while configuring storage for development, staging, and production environments:

Differentiating Storage Capacity by Environment

Set development and staging to 5GB, production to 200GB to reflect real-world requirements.

Issue: Simply setting different sizes doesn’t limit actual usage.

Solution: Implemented ResourceQuota and LimitRange to enforce namespace-specific resource limits.

Host Path Mounting Issues

Problem: VirtualBox shared folder setup occasionally failing

Solution:

1. Verify Guest Additions is properly installed
2. Set mount point ownership
3. Run mount commands manually if needed:

   sudo mount -t vboxsf host-images-dev /mnt/host-images-dev
   

🌟 The Importance of MetalLB

Used MetalLB to assign external IPs in an on-premises Kubernetes environment.

Learning Points:

1. MetalLB assigns real IPs to LoadBalancer services in local environments
2. Understanding differences between Layer 2 and BGP modes
3. Importance of configuring IP ranges that don’t conflict with local network

After setting up MetalLB, I assigned a fixed IP to the ingress controller for consistent external access.

🚢 Cluster Building Journey: Failures and Success

Not everything went smoothly from the start. I experienced several iterations:

1. First Attempt: Started simple with k3s → encountered network plugin issues
2. Second Attempt: Switched to kubeadm → Guest Additions problems
3. Third Attempt: Introduced Ansible automation → SSH connection issues
4. Final Success: Resolved all issues and built a stable cluster!

The biggest lesson learned was “solve one problem at a time.” Trying to change too many things at once makes it difficult to identify the root cause of issues.

🔍 Solving Dynamic IP Problems with DDNS

When running servers from home, dynamic IP is a major concern. I solved this by configuring a DDNS client on my Mikrotik router.

Setup Process:

1. Create a script on the Mikrotik router
2. Write the script according to your DDNS provider (DuckDNS, Namecheap, etc.)
3. Configure scheduler to run periodically (every 5 minutes)

This ensures your domain always points to your current public IP, even when it changes!

🌟 Conclusion and Next Steps

Throughout this journey of building a Kubernetes cluster with Vagrant and Ansible, I’ve learned:

1. The importance of automation: saving time and preventing mistakes
2. Modular approach: breaking down big problems into smaller units
3. Testing and validation: verifying each step works correctly

Consider these next steps:

• Automating SSL certificates with Let’s Encrypt
• Building a monitoring system (Prometheus + Grafana)
• Integrating CI/CD pipelines

Running Kubernetes in a home lab is a great learning experience. Don’t fear failure and enjoy the problem-solving process! 🚀

If you have any questions or feedback, please leave them in the comments. In the next post, we’ll look more closely at application deployment and management. 😊