Common Issues and Solutions When Setting Up Kubernetes with Ansible 🚀

Hello everyone! Today I’d like to share some of the problems I encountered while setting up a Kubernetes environment using Ansible, and how I solved them. We’ll focus particularly on pod scheduling in single-node clusters, resource management, and YAML syntax issues! 😊

Understanding Taints and Tolerations 🧩

First, let’s understand two important Kubernetes concepts: taints and tolerations.

What are Taints? 🚫

Taints are like “No Entry” signs that you place on nodes to prevent certain pods from being scheduled on them. They’re primarily used to protect specialized nodes (like GPU nodes or master nodes).

# Applying a taint to a node
kubectl taint nodes node1 key=value:NoSchedule

# Checking taints on a node
kubectl describe node node1 | grep Taints

Taints can have three different effects:

• NoSchedule: Pods without matching tolerations will never be scheduled
• PreferNoSchedule: The system will try to avoid scheduling pods without matching tolerations
• NoExecute: Pods without matching tolerations will be evicted if already running

What are Tolerations? 🛡️

Tolerations are like “Entry Permits” that you give to pods, allowing them to be scheduled on nodes with matching taints. If a taint is the “No Entry” sign on a node, a toleration is the special pass that lets a pod ignore that sign.

# Adding tolerations to a pod
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
  tolerations:
  - key: "key"
    operator: "Equal"
    value: "value"
    effect: "NoSchedule"

Master Node Taints 🎮

By default, Kubernetes applies the following taints to master nodes (control plane nodes):

node-role.kubernetes.io/control-plane:NoSchedule
node-role.kubernetes.io/master:NoSchedule (older versions)

These taints mean “Don’t schedule regular workloads on the master node!” However, in single-node clusters or development environments, it makes sense to either remove these taints or add tolerations to all pods to make use of the master node!

Problem 1: Pods Stuck in Pending State! 🕐

The first problem I encountered when setting up the Kubernetes environment for the MoolMeow project was that almost all pods were stuck in the Pending state.

$ kubectl get pods --all-namespaces
NAMESPACE        NAME                                       READY   STATUS    RESTARTS   AGE
ingress-nginx    ingress-nginx-admission-create-qpm8v       0/1     Pending   0          33m
metallb-system   controller-5f56cd6f78-fghps                0/1     Pending   0          22h
metallb-system   speaker-plpxf                              0/1     Pending   0          22h
rook-ceph        rook-ceph-operator-6dd77b4466-kv7pq        0/1     Pending   0          22h

Checking the event logs showed this message:

0/1 nodes are available: 1 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }

Cause Analysis 🔍

This problem occurred because our cluster had a single-node configuration (just the master node), and since master nodes have taints by default, regular workloads weren’t being scheduled on it.

Solution 1: Remove the Taints ✂️

The simplest solution is to remove the taints from the master node:

# Ansible task
- name: Remove control-plane taint to allow pods to be scheduled on master
  become: yes
  become_user: vagrant
  shell: |
    kubectl taint nodes --all node-role.kubernetes.io/control-plane- || true
    kubectl taint nodes --all node-role.kubernetes.io/master- || true
  environment:
    KUBECONFIG: /home/vagrant/.kube/config

Note that adding a - after the taint name means we’re removing that taint!

Solution 2: Add Tolerations 🛠️

Another approach is to add tolerations to all components. If you’re using Helm charts, you can do this:

# Ansible task (Nginx Ingress installation example)
- name: Install Nginx Ingress Controller
  become: yes
  become_user: vagrant
  shell: |
    helm install ingress-nginx ingress-nginx/ingress-nginx \
      --namespace ingress-nginx \
      --create-namespace \
      --set controller.service.type=LoadBalancer \
      --set controller.publishService.enabled=true \
      --set controller.tolerations[0].key=node-role.kubernetes.io/control-plane \
      --set controller.tolerations[0].operator=Exists \
      --set controller.tolerations[0].effect=NoSchedule \
      --set controller.tolerations[1].key=node-role.kubernetes.io/master \
      --set controller.tolerations[1].operator=Exists \
      --set controller.tolerations[1].effect=NoSchedule
  environment:
    KUBECONFIG: /home/vagrant/.kube/config

Solution 3: Apply Patches to Already Deployed Resources 🩹

For resources that are already deployed, you can apply patches:

# Ansible task
- name: Apply tolerations patch to MetalLB components
  become: yes
  become_user: vagrant
  shell: |
    kubectl -n metallb-system patch deployment controller --type=json -p='[{"op":"add","path":"/spec/template/spec/tolerations","value":[{"key":"node-role.kubernetes.io/control-plane","operator":"Exists","effect":"NoSchedule"}]}]' || true
  environment:
    KUBECONFIG: /home/vagrant/.kube/config

Problem 2: Resource Shortage Errors! 📉

After solving the pod scheduling problem, a new issue appeared:

Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
timed out waiting for the condition

Cause Analysis 🔍

These timeout errors usually occur for the following reasons:

1. Node CPU/memory resource shortage
2. Image download delays
3. Network issues

Solution: Set Resource Limits and Check Status 💡

I added a task to check the cluster status:

- name: Check system resources and cluster status
  become: yes
  become_user: vagrant
  shell: |
    echo "=== Node Status ==="
    kubectl get nodes
    
    echo "=== System Resources ==="
    free -h
    df -h
    
    echo "=== Pod Status ==="
    kubectl get pods --all-namespaces
  environment:
    KUBECONFIG: /home/vagrant/.kube/config
  register: system_resources

And set more conservative resource limits:

- name: Install Nginx Ingress Controller
  # ...
  shell: |
    helm install ingress-nginx ingress-nginx/ingress-nginx \
      # ...
      --set controller.resources.requests.cpu=100m \
      --set controller.resources.requests.memory=128Mi \
      --set controller.resources.limits.cpu=200m \
      --set controller.resources.limits.memory=256Mi

Problem 3: YAML Syntax and Backslash Issues 😱

The most annoying problem when writing Ansible playbooks was dealing with backslashes (\) for multi-line commands:

/bin/sh: 2: --set: not found

Cause Analysis 🔍

This error occurs when YAML multiline strings and shell command line breaks aren’t properly handled. It’s especially common when there are spaces after backslashes or when the next line’s indentation is improper.

Solution 1: Be Careful with Backslash Usage ⚠️

When using backslashes, follow these rules:

1. No spaces or comments immediately after the backslash
2. Proper indentation on the next line

# Correct method
- name: Helm command with backslashes
  shell: |
    helm upgrade nginx nginx/nginx \
      --set key1=value1 \
      --set key2=value2

Solution 2: Specify the Shell 🐚

Explicitly specifying /bin/bash instead of the default /bin/sh can help:

- name: Helm command with explicit bash
  shell: |
    helm upgrade nginx nginx/nginx \
      --set key1=value1 \
      --set key2=value2
  args:
    executable: /bin/bash

Solution 3: Write Commands in a Single Line 📝

The most reliable method is to write all commands in a single line:

- name: Helm command in one line
  shell: helm upgrade nginx nginx/nginx --set key1=value1 --set key2=value2

But this approach reduces readability, so it’s only recommended for short commands.

Summary: Integrated Solutions 🎯

Here’s an example of Ansible tasks that address all the issues we’ve discussed:

# 1. Remove taints
- name: Remove control-plane taint
  shell: |
    kubectl taint nodes --all node-role.kubernetes.io/control-plane- || true
    kubectl taint nodes --all node-role.kubernetes.io/master- || true
  args:
    executable: /bin/bash

# 2. Check resource status
- name: Check system resources
  shell: |
    echo "=== System Resources ==="
    free -h
    df -h
    echo "=== Pod Status ==="
    kubectl get pods --all-namespaces
  register: system_resources

# 3. Use tolerations and proper backslashes
- name: Install component with tolerations
  shell: |
    helm install component repo/component \
      --namespace example \
      --set tolerations[0].key=node-role.kubernetes.io/control-plane \
      --set tolerations[0].operator=Exists \
      --set tolerations[0].effect=NoSchedule \
      --set resources.requests.cpu=100m \
      --set resources.requests.memory=128Mi
  args:
    executable: /bin/bash

Conclusion 🌟

When building a Kubernetes environment with Ansible, keep these points in mind:

1. Remove taints in single-node clusters: Allow the master node to run workloads.
2. Make resource monitoring a habit: Check resource status first when problems occur.
3. Be careful with YAML syntax and backslashes: Pay attention to spaces and indentation when using backslashes.
4. Specify the shell explicitly: When possible, add args: executable: /bin/bash.

By being aware of these issues and preparing for them, you can build a Kubernetes environment more efficiently with Ansible! 🚀

Enjoy your Kubernetes journey! 🌈

#Kubernetes #Ansible #DevOps #Taint #Toleration