The Evolution of Kubernetes Storage: From Rook-Ceph to NFS

Hello everyone! Today I want to share my journey with Kubernetes storage systems in the MoolMeow project. This journey was like a roller coaster!  

First Choice: Rook-Ceph – “Enterprise-grade Storage Solution”

Initially, I needed a storage solution that was proven in production environments. Rook-Ceph was a logical choice as it’s widely used in professional settings.

Practical Reasons for Choosing Rook-Ceph

• Proven reliability and stability in enterprise environments
• Strong data protection features and consistency guarantees
• Advanced features like snapshots and replication built-in
• Excellent security features with audit capabilities
• Self-healing mechanisms for data safety during failures

Reality Check

$ helm install rook-ceph rook-release/rook-ceph
Error: failed to download "rook-release/rook-ceph"... resources exhausted...

When I tried to install Rook-Ceph on my Mac Mini (M2, 16GB RAM), there weren’t enough resources. Memory usage exploded, and CPU constantly hit 100%. Plus, the configuration was incredibly complex!

# Dozens of lines of complex YAML configuration...
rook-ceph:
  resources:
    requests:
      memory: "2Gi" # This alone uses 1/8 of my Mac Mini's memory!

The conclusion: “This is meant for data centers, not for my Mac Mini…”

Second Choice: hostPath – “Let’s Keep It Simple!”

Since Rook-Ceph was too heavyweight, I switched to the simplest approach: hostPath.

Benefits of hostPath

• Extremely simple configuration
• Almost no additional resources required
• Uses host system paths directly

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  hostPath:
    path: /mnt/services/jenkins
    type: DirectoryOrCreate

It was so simple and easy to set up! But then…

Problems Emerged

While using Vagrant and VirtualBox, I used VirtualBox shared folders (vboxsf) to share directories between the host (Mac) and VM, and this became the root of the problem!

Error: jenkins-0 pod failed: CrashLoopBackOff
$ kubectl logs jenkins-0
java.io.IOException: Invalid argument
  at java.base/sun.nio.ch.FileDispatcherImpl.force0(Native Method)
  at java.base/sun.nio.ch.FileDispatcherImpl.force(FileDispatcherImpl.java:77)
  at java.base/sun.nio.ch.FileChannelImpl.force(FileChannelImpl.java:376)

Jenkins was trying to use the fsync system call, but VirtualBox shared folders don’t properly support this feature!

Third Choice: local-storage – “Let’s Store Inside the VM”

Since shared folders between host and VM were causing problems, I switched to local-storage using VM internal paths.

local-storage Configuration

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  storageClassName: local-storage
  hostPath:
    path: /var/lib/jenkins-data  # Changed to VM internal path!
    type: DirectoryOrCreate
  nodeAffinity:  # Binding to specific node
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-master

Jenkins finally worked properly! But a new problem emerged…

New Problem

By storing inside the VM, I couldn’t see the data from the host (Mac). Our project often needed to check and modify data directly from the host.

# On the host
$ ls ~/Desktop/moolmeow/services/jenkins
# Nothing there! The data is only inside the VM 

Final Choice: NFS – “The Best Compromise”

I needed a way to share data between the host and VM in real-time while supporting all filesystem features, and NFS was the answer!

What is NFS?

NFS (Network File System) is a distributed file system protocol developed by Sun Microsystems in 1984. Despite being almost 40 years old, there are good reasons it’s still widely used!

• Basic Concept: It allows you to mount remote computer file systems locally, accessing them as if they were local files.
• How It Works: It operates on a client-server model based on RPC (Remote Procedure Call).
• Versions: Currently, NFSv3 and NFSv4 are commonly used, with NFSv4.2 being the latest.

Advantages of NFS

• Standard Linux Filesystem Compatibility: Properly supports all system calls like fsync, flock.
• Network Efficiency: Optimized protocol for efficient file access over networks.
• Security Options: Since NFSv4, Kerberos authentication is supported for enhanced security.
• Mature Technology: A time-tested and stable solution.

Disadvantages of NFS

• Network Dependency: File system access is affected if the network is unstable.
• Performance Limitations: Performance may degrade when handling large numbers of small files.
• Default Security: Default configuration may be less secure, requiring additional setup.

Setting up NFS Server on Mac

# Edit /etc/exports file
sudo vi /etc/exports

# Add shared directory
/Users/myname/Desktop/moolmeow -network 192.168.56.0 -mask 255.255.255.0 -mapall=501:20

# Restart NFS server
sudo nfsd restart

Mounting NFS on VM

# Install NFS client
sudo apt-get install -y nfs-common

# Mount NFS
sudo mount -t nfs 192.168.56.1:/Users/myname/Desktop/moolmeow/services /mnt/nfs/services

Modifying Kubernetes PV

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: nfs-storage
  nfs:
    server: 192.168.56.1
    path: /Users/myname/Desktop/moolmeow/services/jenkins

Conclusion: Lessons Learned

Here are the lessons I learned from this journey:

1. Avoid Overengineering

   • Rook-Ceph on a home Mac Mini was overengineering
   • Choose technology appropriate for your environment

2. Beware of Virtualization Pitfalls

   • VirtualBox shared folders are convenient but don’t support all standard Linux filesystem features
   • Applications like Jenkins and MinIO need system calls like fsync

3. Practical Approaches Matter

   • Choose solutions that actually work reliably over cutting-edge technology
   • NFS is an older technology but still a reliable storage solution for Kubernetes

4. Testing is Essential

   • Problems are hard to detect until you actually run applications
   • Without Jenkins, we might have discovered filesystem compatibility issues much later

The storage journey of the MoolMeow project evolved from Rook-Ceph → hostPath → local-storage → NFS. It feels like we started with trendy cloud technology and returned to a traditional but reliable solution. In the end, the most important thing isn’t fancy technology, but a system that actually works.

Hope your Kubernetes journey goes smoothly! If you have any questions or comments, please leave them below!

#Kubernetes #Storage #HomeLabK8s #DevOps #NFS