Battling NFS and MinIO in Kubernetes: A Survival Tale 🥲

Hello fellow developers! Today I’d like to share my adventure (or rather, misadventure) setting up NFS and MinIO in a Kubernetes environment. Let me tell you, there was blood, sweat, and tears involved… 😭

🌱 Why Choose MinIO?

Object storage plays a crucial role in modern cloud-native applications. While commercial services like S3 exist, on-premises environments often require self-hosted object storage solutions.

Here’s why I chose MinIO:

• S3 Compatibility 💯 – Compatible with AWS S3 API, making application integration seamless
• Kubernetes-Friendly 🤗 – Optimized deployment options for Kubernetes environments
• Lightweight & Fast ⚡ – Written in Go, efficient resource utilization
• Scalability 📈 – Scales from terabytes to exabytes
• Open Source 🔓 – Community support and free to use

🚀 Kubernetes Environment Setup

Our environment consists of:

• Host Machine: macOS (Mac Mini)
• Kubernetes Nodes: Ubuntu VM created with Vagrant
• Storage Method: NFS sharing directories from Mac to VM

😱 The Problem: “Protocol not supported”

When trying to install MinIO, I encountered this error:

MountVolume.SetUp failed for volume "minio-pv" : mount failed: exit status 32
Mounting command: mount
Mounting arguments: -t nfs 192.168.100.2:/Users/kimvanye/Desktop/moolmeow/services/minio /var/lib/kubelet/pods/.../volumes/kubernetes.io~nfs/minio-pv
Output: mount.nfs: Protocol not supported

What on earth does this mean? 🤔

🔍 Root Cause Analysis

After hours of debugging, I discovered several causes:

1. NFS Client Issues: NFS client not properly installed on the VM
2. NFS Protocol Version Issues: macOS supports NFSv3/v4 by default, but compatibility issues arose
3. NFS Share Configuration Issues: Incorrect /etc/exports file settings
4. Permission Issues: UID/GID mapping needed

🛠️ The Resolution Journey

1. Verifying NFS Client Installation on VM

sudo apt update
sudo apt install -y nfs-common
sudo modprobe nfs
sudo modprobe nfsd

2. Testing NFS Mounts Directly

I tested NFS mounts directly on the VM:

sudo mkdir -p /mnt/services
sudo mount -t nfs -o vers=3,rw,soft 192.168.x.2:/Users/path/services /mnt/services

This command succeeded! So the problem must be elsewhere… 🧐

3. Modifying NFS Settings on macOS

The core issue was in the /etc/exports file configuration. macOS NFS server requires a different syntax for network specification, not CIDR notation:

# Wrong way ❌
/Users/path/services/minio -alldirs -mapall=501:20 192.168.x.0/24

# Correct way ✅
/Users/path/services/minio -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0

I edited the /etc/exports file:

# Edit the /etc/exports file
sudo vi /etc/exports

Then I added the following content to the file:

# NFS share settings
/Users/path/services/jenkins -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/services/registry -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/services/minio -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/services/argocd -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/storage -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/shared -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/configs -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/k8s -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/databases -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/backups -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0
/Users/path/logs -mapall=501:20 -network 192.168.x.0 -mask 255.255.255.0

After saving the file, I restarted the NFS server:

sudo nfsd restart

Note that the -mapall=501:20 values refer to macOS user UID and GID. You should check your own values using the id command and adjust accordingly.

### 4. Creating Kubernetes PV

Now that NFS was properly configured, I created the PV:

```bash
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolume
metadata:
  name: minio-pv
spec:
  capacity:
    storage: 20Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: minio-nfs
  nfs:
    server: 192.168.x.2  # NFS server IP address
    path: /Users/path/services/minio
EOF

5. Creating PVC and Installing MinIO

kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: minio-pvc
  namespace: minio
spec:
  storageClassName: minio-nfs
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
EOF

helm install minio minio/minio \
  --namespace minio \
  --set replicas=1 \
  --set mode=standalone \
  --set service.type=ClusterIP \
  --set persistence.enabled=true \
  --set persistence.existingClaim=minio-pvc \
  --set resources.requests.memory=256Mi \
  --set resources.requests.cpu=100m \
  --set ingress.enabled=true \
  --set ingress.host=minio.moolmeow.local \
  --set rootUser=minioadmin \
  --set rootPassword=minioadmin

🎉 Success at Last!

After numerous attempts, the MinIO pod finally started running normally:

kubectl get pods -n minio
NAME                    READY   STATUS    RESTARTS   AGE
minio-75d5b4d78-jx9p7   1/1     Running   0          2m

🧩 Understanding Helm Chart Values

When installing MinIO, we used a Helm chart with several --set options. This is where the concept of Helm Values comes into play! 😃

What are Values?

Helm charts are essentially collections of Kubernetes resource templates. These templates contain variables in the form of {{ .Values.xxx }}, and we can provide actual values for these variables.

For example, in the MinIO chart:

# Template has something like this
service:
  type: {{ .Values.service.type }}

# And we set it with --set service.type=ClusterIP

How to Set Values

1. Using the --set option on the command line:

   helm install minio minio/minio --set key1=value1 --set key2=value2
   

2. Using a values.yaml file:

   # Create a values.yaml file
   cat <<EOF > minio-values.yaml
   replicas: 1
   mode: standalone
   persistence:
     enabled: true
     existingClaim: minio-pvc
   ingress:
     enabled: true
     host: minio.example.com
   EOF
   
   # Apply the values.yaml file
   helm install minio minio/minio -f minio-values.yaml
   

Why Use values.yaml?

In production environments, using a values.yaml file is preferred over --set options:

• Readability: Easier to manage many configurations
• Version Control: Track configuration changes with Git
• Environment Separation: Use different values files for dev, staging, production
• Reusability: Reuse the same configurations across deployments

As this project evolves, we plan to use values.yaml files to manage MinIO and other services more systematically. This will make it easier to change and manage configurations for different environments! 👍

🧠 Lessons and Tips

1. macOS NFS Configuration Differs from Linux – Use -network and -mask instead of CIDR notation.
2. UID/GID Mapping is Crucial – Use the -mapall=UID:GID option for permission mapping on macOS.
3. Specify NFS Version – It’s best to explicitly specify the NFS version with the vers=3 option.
4. Consider Host Path as an Alternative – If NFS continues to be problematic, hostPath is an option.
5. Test Step by Step – Test mounting directly on the VM before Kubernetes configuration.

📝 Conclusion

NFS and Kubernetes make a powerful combination, but configuration can be trickier than expected, especially in a macOS environment. I hope this post helps others who face similar issues! 💪

If you have any questions or additional tips, please leave them in the comments. Let’s grow together! 🌱