Why I Chose HostPath Over Rook-Ceph in Kubernetes 🤔

April 8, 2025 Dev Journal: A Full Day Wrestling with Rook-Ceph and What I Learned

Hello there! Today I want to share my journey of attempting to set up Rook-Ceph as a storage system in a Kubernetes environment, and why I eventually switched to HostPath. I hope documenting these common development environment issues and their solutions might help someone else out there! 😊

🔍 The Problem: Rook-Ceph Cluster Not Working Properly

In my MoolMeow project, all CI/CD tools like Jenkins and Harbor were stuck in a Pending state. When I investigated, I found that the issue was with the Rook-Ceph cluster.

Current Infrastructure Environment

I’m running my Kubernetes cluster on VirtualBox VMs hosted on a Mac Mini M2 Pro:

Master Node:

• RAM: 8GB
• CPU: 4 cores
• Disk: 80GB

Worker Node:

• RAM: 4GB
• CPU: 2 cores
• Disk: 40GB

# Checking the problem
kubectl -n rook-ceph get cephclusters
kubectl -n rook-ceph get pods | grep osd

🧐 Root Cause Analysis

1. Missing OSDs (Object Storage Daemons)

The Ceph cluster was created, but there were no OSDs:

Reduced data availability: 128 pgs inactive
4 pool(s) have no replicas configured
usage: 0 B used, 0 B / 0 B avail

Essentially, Ceph had no actual storage to use! 💾

2. Loop Device Attempts

I tried creating loop devices to solve the problem:

mkdir -p /var/lib/rook/osd-loop
truncate -s 10G /var/lib/rook/osd-loop/osd.img
losetup -fP /var/lib/rook/osd-loop/osd.img
losetup -a  # Verify connection

However, Rook-Ceph couldn’t recognize these devices:

skipping OSD configuration as no devices matched the storage settings for this node "k8s-master"

3. Disk Pressure Issues

Further log inspection revealed a disk-pressure taint on the node:

kubectl get node k8s-master -o yaml | grep -A5 "taints:"
# Result:
taints:
- effect: NoSchedule
  key: node.kubernetes.io/disk-pressure
  timeAdded: "2025-04-08T10:16:01Z"

This is a taint that Kubernetes automatically adds when it detects low disk space. 😱

4. Resource Constraints

Finally, I discovered the most critical issue:

Resource           Requests      Limits
--------           --------      ------
cpu                3 (75%)       3800m (95%)
memory             3568Mi (46%)  6996Mi (90%)

The virtual machine was nearly at its resource limit! Not enough to run a heavy distributed storage system like Rook-Ceph. 🚫

🛠️ Solution: Switch to HostPath

In situations like this, it’s wise to choose a lighter storage solution than Rook-Ceph. For development environments, HostPath can be a good alternative.

Benefits of HostPath:

• 🚀 Simple and quick to set up
• 🪶 Uses minimal resources
• 🧩 Built into Kubernetes, no additional installation required
• 🔧 Suitable for development environments

Example HostPath PV:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  hostPath:
    path: /mnt/moolmeow-dev/jenkins
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: jenkins-pvc
  namespace: jenkins
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  volumeName: jenkins-pv

💻 Rook-Ceph Recommended Specifications

Running Rook-Ceph properly requires more substantial resources than you might expect:

Production Environment Recommendations

• CPU: Minimum 4-8 cores per node (about 1 core per OSD)
• Memory: Minimum 8-16GB per node (about 2GB per OSD + additional for MON/MGR)
• Disk:
  • Minimum 50GB SSD for system
  • Dedicated disks for data
  • Separate SSD for metadata recommended
• Node count: Minimum 3 nodes for high availability

Development Environment Minimum Requirements

• CPU: Minimum 2-4 cores
• Memory: Minimum 4-8GB
• Disk: Minimum 20-40GB free space

My current setup (Master: 8GB RAM/4 cores, Worker: 4GB RAM/2 cores) seems to meet the minimum requirements on paper, but in reality, Kubernetes itself and other workloads were already consuming significant resources, making it insufficient for stable Rook-Ceph operation.

🌟 Conclusion

In development environments, it’s important to choose the appropriate storage solution based on your resource constraints. While Rook-Ceph is an excellent distributed storage solution, using lighter alternatives like HostPath can be more efficient when sufficient resources aren’t available.

This experience reminded me of the importance of storage selection and resource management in Kubernetes environments. I hope you find this helpful when setting up your own development environment! 💪

📚 Reference Commands

# Check Ceph status
kubectl -n rook-ceph exec -it deploy/rook-ceph-tools -- ceph status

# Check OSD preparation and creation status
kubectl -n rook-ceph get pods | grep osd

# Check logs
kubectl -n rook-ceph logs deploy/rook-ceph-operator | grep "osd"

# Check disk usage
df -h

# Clean up container images
docker system prune -a
# Or for containerd:
ctr image rm $(ctr image ls -q)

# Remove taint
kubectl taint nodes <node-name> node.kubernetes.io/disk-pressure:NoSchedule-