Troubleshooting Kubernetes Environment Setup: Challenges and Solutions

Hello everyone! Today I’d like to share the various issues I encountered while setting up a Kubernetes environment for my MoolMeow project, along with the solutions I found. + = MoolMeow!

Project Introduction

MoolMeow is an integrated online community service centered around swimming (Mool, 물) and cats (Meow, 야옹). It’s an exciting project that operates two concept-based communities under a single domain (moolmeow.com)!

I built a Kubernetes cluster in an on-premises environment for this project, and along the way, I faced several challenging situations. Let’s explore this journey together!

Infrastructure Setup

• Hardware: Mac Mini Pro (M2, 16GB RAM, 512GB SSD)
• Virtualization: VM configuration using Vagrant + VMware Fusion
• Kubernetes: Installed with kubeadm (1 master node, 1 worker node)
• Automation: Installation automated with Ansible
• Storage: Distributed storage using Rook-Ceph

Issues Encountered and Solutions

1. Architecture Mismatch Problem

Issue: Exec format error occurred during Helm installation

/bin/sh: 1: helm: Exec format error

Cause: Helm binary didn’t match the VM architecture.

Solution: Modified the script to check the target system’s architecture and download the appropriate binary.

# Download Helm for the appropriate architecture
- name: Download Helm for the appropriate architecture
  get_url:
    url: "https://get.helm.sh/helm-v3.12.3-linux-{{ 'arm64' if 'aarch64' in vm_arch.stdout else 'amd64' }}.tar.gz"
    dest: /tmp/helm.tar.gz

2. Network Plugin Installation Failure

Issue: Invalid URL for Calico network plugin installation

error: error validating "calico.yaml": error validating data: invalid object to validate

Solution: Modified the approach to use a locally stored manifest file.

- name: Copy local Calico manifest to master node
  copy:
    src: ./calico.yml
    dest: /home/vagrant/calico.yml

- name: Apply Calico network plugin
  shell: |
    kubectl apply -f /home/vagrant/calico.yml

3. Idempotency Issues

Issue: Error when Helm chart was already installed

Error: INSTALLATION FAILED: cannot re-use a name that is still in use

Solution: Added logic to check first and then either install or upgrade based on the current state.

- name: Check if Rook-Ceph is already installed
  shell: helm list -n rook-ceph | grep rook-ceph
  register: rook_ceph_check
  failed_when: false

- name: Install Rook-Ceph Operator
  shell: helm install --create-namespace --namespace rook-ceph rook-ceph rook-release/rook-ceph
  when: rook_ceph_check.rc != 0  # Only run if not already installed

- name: Upgrade Rook-Ceph if already installed
  shell: helm upgrade rook-ceph rook-release/rook-ceph --namespace rook-ceph
  when: rook_ceph_check.rc == 0  # Only run if already installed

4. Pod Readiness Timeout

Issue: MetalLB pods took too long to become ready, causing timeouts

timed out waiting for the condition on pods/controller-5f56cd6f78-fghps

Solution: Added longer timeout and periodic status check logic.

- name: Wait for MetalLB pods with extended timeout
  shell: |
    # Check every minute for up to 10 minutes
    timeout=600
    elapsed=0
    interval=60

    while [ $elapsed -lt $timeout ]; do
      echo "Checking MetalLB pods status (elapsed time: ${elapsed}s / ${timeout}s)"
      kubectl get pods -n metallb-system
      
      if kubectl get pods -n metallb-system -o jsonpath='{.items[*].status.conditions[?(@.type=="Ready")].status}' | grep -v False; then
        echo "All MetalLB pods are ready!"
        break
      fi
      
      sleep $interval
      elapsed=$((elapsed + interval))
    done

Lessons Learned

1. Always Consider Architecture Differences: Especially when working on ARM-based systems like Apple Silicon, make it a habit to verify your VM’s architecture!

2. Idempotency is Critical: Ansible playbooks should run multiple times without issues. Always design them to check the current state and act accordingly.

3. Provide Sufficient Wait Times: Kubernetes components, especially load balancers or storage solutions, may need more initialization time than expected.

4. Implement Error Handling Carefully: Prepare handling methods for all foreseeable error situations so your playbook doesn’t stop in the middle of execution.

5. Collect Ample Debugging Information: Configure your playbook to automatically collect information needed for troubleshooting (pod status, logs, etc.), which makes resolving issues much easier.

Conclusion

Building a Kubernetes cluster in an on-premises environment is certainly not an easy task, but I learned a lot by solving these problems one by one. In particular, automation with Ansible significantly reduces repetitive work and helps maintain a consistent environment.

I plan to continue improving this infrastructure for the MoolMeow project. If you’re working on a similar project, I hope this post helps you in some way!

Happy Kuberneting!

#Kubernetes #Ansible #DevOps #Vagrant #RookCeph #MetalLB #MoolMeow