Introduction 👋

Hello everyone! Today I’d like to share my experience setting up a distributed storage system using Rook-Ceph in a Kubernetes environment. While considering storage solutions for our CI/CD pipeline and application data, I found myself torn between simple HostPath storage and the more advanced Rook-Ceph distributed storage. I eventually chose Rook-Ceph for the experience and scalability it offers, and I want to share what I learned along the way.

What is Rook-Ceph? 🤔

Let’s start with a brief overview of Rook and Ceph:

Ceph is an open-source distributed storage system that offers:

• Object, block, and file system interfaces
• Distributed architecture with no single point of failure
• Data replication and self-healing capabilities
• Scalability to petabyte scale

Rook is a storage orchestrator for Kubernetes that:

• Integrates distributed storage systems like Ceph with Kubernetes
• Automates deployment, bootstrapping, configuration, scaling, and upgrading of storage resources
• Provides self-managing, self-scaling, and self-healing storage services

In simple terms, Rook makes the complex Ceph distributed storage easy to use within Kubernetes. Together, they form “Rook-Ceph,” a powerful Kubernetes-based storage solution.

HostPath vs Rook-Ceph: The Dilemma 🤷‍♂️

When choosing a storage solution, I considered two main approaches:

1. HostPath (or Local Path Provisioner)

Pros:

• Simple and quick to set up
• Minimal resource usage
• Can use already mounted host paths directly

Cons:

• Data stored on a single node, risking data loss in case of node failure
• Limited storage management features
• Poor scalability

2. Rook-Ceph

Pros:

• High availability and data replication
• Excellent scalability
• Advanced storage features like volume snapshots and expansion
• Supports block, file, and object storage

Cons:

• Complex setup
• Requires more system resources
• Steeper learning curve

Why I Chose Rook-Ceph 💡

After careful consideration, I chose Rook-Ceph for these reasons:

1. Future Scalability – Even if starting small, as the service grows, storage needs to scale accordingly. Rook-Ceph makes it easy to expand storage by simply adding nodes to the cluster.

2. Experience and Learning – Rook-Ceph is an important technology in cloud-native environments. Hands-on experience with such technology is invaluable in a professional setting.

3. Variety of Storage Interfaces – Different applications may have different storage requirements. Rook-Ceph provides block, file, and object storage, supporting a wide range of use cases.

4. Data Reliability – Data loss can be a significant issue even in development environments. Rook-Ceph’s data replication features reduce this risk.

Considerations for Rook-Ceph Setup ⚙️

When setting up Rook-Ceph, several elements need consideration:

1. Storage Architecture Design

When designing a Rook-Ceph cluster, you need to understand these components:

• Monitor (MON) – Daemons that monitor cluster state
• Manager (MGR) – Provides monitoring features and dashboard
• Object Storage Device (OSD) – Daemons that store actual data
• Metadata Server (MDS) – Manages metadata for CephFS

In development environments, you can configure lightweight versions of these components. For example, reducing the number of MONs from 3 to 1, or reducing the replication factor from 3 to 1.

2. Storage Pools and Storage Classes

To use Rook-Ceph in Kubernetes, you need:

1. Create Storage Pools – Logical spaces within Ceph for storing data
2. Create Storage Classes – Kubernetes resources used when requesting storage via PVCs

You can create multiple storage pools and storage classes for different purposes. For example:

• A storage pool for Harbor container registry
• A storage pool for application user upload images
• A storage pool for database backups

Each pool can have different replication factors, failure domains, etc., allowing customization to fit specific requirements.

3. Resource Requirements

Rook-Ceph demands significant system resources:

• Minimum 2GB RAM recommended per OSD daemon
• Additional memory for monitoring and management services
• CPU resources (especially with many OSD daemons)
• Disk space for storing actual data

In development environments, you can use a lightweight configuration to minimize these requirements, but you still need to meet the basic resource demands.

Practical Implementation Experience 🛠️

During the implementation of Rook-Ceph in our project, I encountered several challenges:

First Challenge: Disk Preparation

Initially, I found it challenging that Rook-Ceph requires actual physical disks or partitions. In a virtual environment, we considered:

• Setting up additional virtual disks
• Utilizing loop devices
• Configuring directory-based OSDs

We eventually opted for directory-based OSD configuration, which isn’t performance-optimized but was sufficient for our development environment.

Second Challenge: Resource Allocation

To meet Rook-Ceph’s resource requirements, we had to adjust the VM resources in our development environment. We increased memory allocation and relaxed CPU limits.

Third Challenge: Separating Storage for Different Uses

We configured separate storage pools for our CI/CD pipeline (Harbor) and application (Moolmeow) data:

1. Pool for Harbor image registry
2. Pool for Moolmeow post images
3. Pool for Moolmeow market images

We created separate storage classes for each pool to ensure applications use the appropriate storage.

Lessons Learned and Advice 🎓

Here are some key lessons I learned while implementing Rook-Ceph:

1. Thorough Planning is Essential – Rook-Ceph is a complex system, so plan your architecture and resource requirements thoroughly before implementation.

2. Use Lightweight Configurations for Development – If you don’t need production-level high availability, reduce the number of monitors and replication factor to save resources.

3. Take an Incremental Approach – Rather than aiming for a perfect configuration from the start, begin with basic functionality and expand gradually.

4. Leverage Documentation and Community – Both Rook and Ceph have active communities and extensive documentation. Use these resources when encountering issues.

Conclusion 🎉

Rook-Ceph may be complex, but it’s a powerful storage solution. While the initial learning curve is steep, once configured, it provides a reliable and scalable storage system.

Whether in development or production environments, data is your most valuable asset. Rook-Ceph offers the protection this data deserves while providing the flexibility to scale as needed.

I hope this post helps those considering Rook-Ceph for their Kubernetes environments. If you have any questions or comments, please feel free to leave them below! 😊

In the next post, we’ll explore how to set up Harbor container registry, the first component of our CI/CD pipeline, with Rook-Ceph storage. Stay tuned!