Ceph, as a large-scale distributed storage system, has established itself as a critical component of cloud infrastructure. Specifically, the Object Storage Daemon (OSD) is a core daemon responsible for actually storing and managing data within a Ceph cluster, and its stability directly correlates with the overall cluster's stability. OSD failures can occur at unpredictable moments, and particularly, concurrent failures pose a severe risk that can lead to service disruptions.
Scenario Introduction
In the cloud infrastructure environment operated by the organization, a Ceph Luminous-based distributed block storage platform utilizes 140 OSDs distributed across 7 dedicated storage nodes. This cluster comprises high-performance SSD pools and large-capacity HDD pools, meeting the storage requirements for mission-critical production workloads. The paramount objective is to maintain system availability and Service Level Objectives (SLOs) at their peak.
Recently, a significant incident occurred. On a specific storage node, storage004, 4 SSDs and 1 HDD failed sequentially, resulting in a total of 5 OSDs becoming down and out of the cluster. To compound the situation, 1 and 2 HDD OSDs also failed on other nodes, storage001 and storage002, respectively. This led to a total of 8 OSDs out of 140 being in a down/out state. Consequently, an effective capacity loss of 15.13 TiB was observed in the SSD pool. Fortunately, due to the CRUSH map design and data replication strategy, no data loss occurred. However, this experience served as a renewed reminder of the critical importance of proactive detection and rapid response to OSD failures.
Challenges
Effectively managing OSD failures in such a large-scale Ceph cluster environment presents the following multifaceted challenges:
- Scale and Complexity: Manually monitoring 140 OSDs is impractical; an automated system capable of tracking the status and performance of all OSDs in real-time is essential.
- Silent Failures: Subtle failures, such as performance degradation (slow OSDs) or intermittent response delays, rather than complete OSD process termination, are difficult to detect with traditional simple 'UP/DOWN' monitoring. Such silent failures can negatively impact service performance in the long term.
- Alert Fatigue: Overly sensitive alerts can increase alert fatigue for SRE teams, potentially causing them to miss genuinely critical failure signals. Conversely, insufficient alerts can lead to a situation where potential failures go unnoticed until they escalate into actual service disruptions.
- Operational Overhead: The manual process of diagnosing and recovering from failures consumes significant time and effort, diverting focus from other important engineering tasks.
- Trade-off between Data Durability and Availability: Settings such as
min_size=1contribute to increased availability but simultaneously present a trade-off by increasing the risk of data loss. Understanding and managing this balance appropriately is crucial.
Existing monitoring systems relied on basic ceph health status checks but had limitations in proactively detecting individual OSD performance metrics or signs of potential performance degradation.
Technology Selection Process
To address these challenges and maximize the stability of the Ceph cluster, the organization sought a new monitoring solution based on the following criteria:
- Real-time, Advanced Metric Collection: The solution must be capable of collecting and analyzing granular metrics in real-time, including OSD status, performance (latency), and Placement Group (PG) status.
- Powerful Query and Visualization Capabilities: It should enable flexible querying of collected metrics and their visualization on meaningful dashboards to intuitively grasp the overall cluster status.
- Flexible and Scalable Alerting System: The system must promptly and accurately send alerts to the SRE team upon exceeding specific thresholds or detecting patterns, and integrate with various alerting channels (e.g., Slack, PagerDuty).
- Ease of Integration with Existing Observability Stack: Seamless integration with existing monitoring and logging infrastructure is required.
Based on these requirements, several candidate technologies were comparatively analyzed.
| Technology/Solution | Advantages | Disadvantages | Suitability |
|---|---|---|---|
| Ceph Dashboard | Provides an at-a-glance view of basic Ceph cluster status. | Limitations in advanced queries and custom alert configurations, lack of historical data analysis capabilities. | Good for overall status overview, but unsuitable for in-depth monitoring and alerting. |
| ELK Stack | Strong in log aggregation and searching, capable of handling large-scale data. | Absence of time-series metric processing and powerful query languages like PromQL. | Primarily specialized for log analysis, requiring additional effort for metric-based real-time monitoring. |
| Prometheus + Grafana | Optimized design for time-series data, powerful PromQL, rich Exporter ecosystem, flexible alerting via Alertmanager. | Requires resource management for long-term data retention in large-scale clusters. | Optimal for precise metric-based monitoring, real-time alerting, and custom dashboard creation. |
After a comprehensive comparison, the combination of Prometheus and Grafana was chosen as the solution that best met the organization's requirements, owing to its real-time metric collection and analysis capabilities, powerful query language, and sophisticated alerting system via Alertmanager. Specifically, the flexibility to precisely define and monitor SLOs/SLIs from an SRE perspective was a key selection criterion for this solution.
Implementation Process
1. Ceph Exporter Deployment and Prometheus Integration
The initial step involves deploying ceph-exporter to enable Prometheus to collect metrics from the Ceph cluster. The organization configured ceph-exporter to be deployed on each Ceph Manager (Mgr) node to collect cluster status and OSD-specific metrics. This approach minimizes overhead on the monitored Ceph cluster while ensuring stable metric collection.
# prometheus.yml 설정 예시
scrape_configs:
- job_name: 'ceph'
static_configs:
- targets:
- 'ceph-mgr01:9283' # Ceph Mgr 노드에 배포된 ceph-exporter 엔드포인트
- 'ceph-mgr02:9283'
# ... 다른 Mgr 노드들
relabel_configs:
- source_labels: [__address__]
regex: '(.*):9283'
target_label: instance
replacement: '$1'
This configuration instructs Prometheus to periodically scrape metrics from the HTTP endpoints exposed by ceph-exporter. The collected metrics are stored in Prometheus's time-series database for use in queries and alerts.
2. Core OSD Metric-Based Monitoring Design
To accurately assess the health and performance of Ceph OSDs, monitoring should be designed around the following core Prometheus metrics:
ceph_osd_up: This metric indicates whether the OSD daemon is currently running.0signifies down, and1signifies up.ceph_osd_in: This metric indicates whether the OSD is currently participating in the Ceph cluster and serving data.0signifies out, and1signifies in. If an OSD isupbut notin, it may be undergoing recovery or have been manually excluded.ceph_osd_apply_latency_ms: This is the time (in milliseconds) it takes for an OSD to apply a client's write request to the journal or data disk. Abnormally high values for this metric may suggest OSD disk I/O performance issues or system load.ceph_osd_commit_latency_ms: This is the time (in milliseconds) it takes for an OSD to commit a write request to all replicas and respond to the client. This metric can be associated with network latency or performance issues on other OSDs.ceph_pg_degraded: This is the number of Ceph Placement Groups (PGs) that are in adegradedstate. AdegradedPG implies that data is stored on fewer OSDs than the configured replica count, indicating an increased risk of data loss.ceph_health_status: This represents the overall health status of the Ceph cluster.0signifies OK,1signifies WARNING, and2signifies ERROR.
In addition, deeper performance analysis can be conducted using metrics such as ceph_osd_op_r_bytes, ceph_osd_op_w_bytes (OSD read/write bytes), and ceph_osd_op_out_op_per_sec (OSD throughput).
3. PromQL Query and Alertmanager Rule Definition
Based on the collected metrics, PromQL queries are formulated to detect specific situations, which are then defined as alerting rules in Alertmanager. The following are examples of key PromQL queries and Alertmanager rules.
3.1. OSD Status Detection
# 특정 OSD가 down 상태인지 감지
ceph_osd_up{job="ceph"} == 0
# OSD가 up 상태이지만 cluster에서 out 상태인지 감지 (수동 제외 또는 복구 중이 아님을 전제)
ceph_osd_in{job="ceph"} != on(instance) ceph_osd_up{job="ceph"}
3.2. OSD Latency Anomaly Detection
# OSD apply latency가 100ms를 초과하는 OSD 감지 (임계값은 환경에 따라 조정 필요)
ceph_osd_apply_latency_ms{job="ceph"} > 100
# OSD commit latency가 50ms를 초과하는 OSD 감지
ceph_osd_commit_latency_ms{job="ceph"} > 50
3.3. PG Degraded Ratio and Ceph Health Status Detection
# 전체 PG 중 degraded 상태인 PG의 비율이 10%를 초과하는 경우 감지
(sum(ceph_pg_degraded{job="ceph"}) / sum(ceph_pg_total{job="ceph"})) * 100 > 10
# Ceph 클러스터 health status가 ERROR (2)인 경우 감지
ceph_health_status{job="ceph"} == 2
# Ceph 클러스터 health status가 WARNING (1)인 경우 감지
ceph_health_status{job="ceph"} == 1
3.4. Alertmanager Alert Rules
# alert.rules.yml 예시
groups:
- name: ceph.rules
rules:
- alert: CephOSDDown
expr: ceph_osd_up{job="ceph"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Ceph OSD {{ $labels.instance }} is down"
description: "Ceph OSD {{ $labels.instance }} (ID: {{ $labels.osd_id }}) is not up for more than 1 minute. Immediate investigation required."
- alert: CephOSDHighLatency
expr: ceph_osd_apply_latency_ms{job="ceph"} > 100 or ceph_osd_commit_latency_ms{job="ceph"} > 50
for: 5m
labels:
severity: warning
annotations:
summary: "Ceph OSD {{ $labels.instance }} reports high latency"
description: "Ceph OSD {{ $labels.instance }} (ID: {{ $labels.osd_id }}) has high apply/commit latency for more than 5 minutes. Current apply: {{ $value | humanize }}ms, commit: {{ $labels.commit_latency | humanize }}ms. Investigate performance degradation."
- alert: CephOSDNearFull
expr: (ceph_osd_pool_used_ratio{job="ceph", name="ssd_pool"} * 100) > 69 # 69%는 현재 상태에 맞춘 예시, nearfull 75% 임계값을 고려
for: 5m
labels:
severity: warning
annotations:
summary: "Ceph SSD pool is near full"
description: "Ceph SSD pool usage is at {{ $value | humanize }}%. Nearfull threshold is 75%. Consider adding capacity."
- alert: CephHealthError
expr: ceph_health_status{job="ceph"} == 2
for: 1m
labels:
severity: critical
annotations:
summary: "Ceph Cluster Health is in ERROR state"
description: "The overall Ceph cluster health is in ERROR state for more than 1 minute. Immediate investigation required."
- alert: CephPGDegradedRatioHigh
expr: (sum(ceph_pg_degraded{job="ceph"}) / sum(ceph_pg_total{job="ceph"})) * 100 > 10
for: 5m
labels:
severity: critical
annotations:
summary: "High ratio of degraded Placement Groups in Ceph cluster"
description: "{{ $value | humanize }}% of Ceph PGs are in degraded state. This indicates potential data redundancy loss or slow recovery. Investigate OSDs or network issues."
These rules assist SRE teams in proactively responding not only to OSD failures but also to signs of potential performance degradation and capacity exhaustion risks.
4. CRUSH Host-Level Failure Domains and min_size Trade-off
Ceph's CRUSH (Controlled Replication Under Scalable Hashing) map defines how data is distributed across OSDs and how replicas are placed. The organization designed the CRUSH map with host-level failure domains to prevent data loss and maintain service availability even during single-node failures. This was a critical factor enabling the cluster to recover without data loss in situations such as the complete failure of the storage004 node.
# CRUSH map 예시 스니펫 (부분 발췌)
# rule replicated_rule {
# ruleset 0
# type replicated
# min_size 1
# max_size 10
# step take default
# step chooseleaf firstn 0 type host
# step emit
# }
# (실제 환경에서는 더 복잡한 구성이 적용됩니다)
However, caution is required with the min_size=1 setting. This configuration maximizes availability by allowing I/O operations to continue even if a PG is stored on only a single OSD. For instance, in a PG with 3 replicas (size=3), if two OSDs fail, service disruption can be prevented as long as the remaining OSD is operational. Concurrently, if that last remaining OSD also fails, data loss becomes inevitable.
In the organization's case, this setting was chosen for rapid recovery and high availability, but it was clearly recognized that it introduced a trade-off: an increased risk of data loss during simultaneous failures of multiple OSDs. To mitigate this, a process for ultra-fast detection and recovery from OSD failures is essential. The organization is actively considering integrating automated response playbooks triggered by failure alerts using Seekurity SIEM/SOAR. This forms part of a strategy to minimize the potential risks of min_size=1 while maintaining high availability.
Results and Achievements
Following the establishment of a Prometheus-based Ceph monitoring system, storage operations achieved the following quantitative and qualitative results:
Quantitative Achievements
Qualitative Achievements
- Reduced Operational Burden and Enhanced Psychological Stability for SRE Teams: Anxiety regarding unpredictable storage failures has been alleviated, enabling teams to focus on more strategic tasks.
- Established Systematic and Rapid Response Process for Failures: A system has been implemented to prioritize and respond to failures according to alerting rules and standardized procedures.
- Significantly Enhanced Storage Cluster Observability: Grafana dashboards enable real-time monitoring of the overall Ceph cluster status and individual OSD performance, improving visibility.
- Validation of CRUSH Map Design Effectiveness: In a real-world large-scale failure scenario, the CRUSH host-level failure domain design was confirmed to be effective in preventing data loss during single-node failures.
Lessons Learned and Retrospection
Several significant lessons were learned from the experience of encountering Ceph OSD failures and building a Prometheus-based monitoring system.
- Dual Nature of the
min_size=1Setting: It was reconfirmed that themin_size=1setting, intended for high availability, inherently carries significant risks concerning data durability. The delicate balance between the number of replicas (replica size) andmin_sizewas deeply understood not merely as a technical choice but as a policy decision directly tied to Business Continuity. - Necessity for Expanding Initial Monitoring Scope: While the initial focus was on core OSD metrics, it became evident that deeper monitoring, including OSD-specific I/O patterns,
scrubscheduling, and the status and performance metrics ofceph-mgrmodules, would be far more effective in early identification of overall cluster health and potential issues. - Unexpected Ancillary Benefits: During the development of the monitoring system, the SRE team significantly enhanced their understanding of Ceph's internal operating principles, particularly the behavior of CRUSH maps and PGs. This fostered a positive feedback loop that extended beyond merely responding to failures, leading to proactive performance optimization and architectural improvements. Furthermore, a deeper insight was gained that
nearfullalerts might be an early indicator not just of capacity issues but of data imbalance (skew) or data distribution bias among OSDs, contributing to the formulation of operational strategies to address these.
Application Guide
Organizations operating similar Ceph cluster environments and seeking to enhance OSD failure detection and response systems can quantitatively ensure system reliability through the following guidelines:
ceph-exporterDeployment Strategy: In large-scale Ceph clusters, deployingceph-exporterindependently on dedicated monitoring nodes or each Manager node minimizes the resource utilization of the Ceph cluster itself and efficiently manages workload impact.- Optimizing Alertmanager Routing: Meticulously design the Alertmanager routing tree to ensure that alerts, categorized by severity (e.g., Critical, Warning), are accurately and promptly delivered to the appropriate teams or individuals within the minimum latency that does not violate SLOs. Regular tuning to prevent 'alert fatigue' from exceeding a specific threshold is a critical element of system reliability.
- Grafana Dashboard Visualization: Building Grafana dashboards that provide an at-a-glance view of key Ceph cluster status (health, PG count, OSD count) and OSD-specific performance (latency, I/O throughput, utilization) metrics is crucial for proactively detecting failure signs and ensuring visual operational visibility.
- Phased Implementation Roadmap:
- Deploy
ceph-exporterand integrate it with Prometheus to commence core metric collection. - Construct Grafana dashboards based on key indicators such as
ceph_osd_upandceph_health_statusfor intuitive cluster status monitoring. - Prioritize the configuration of OSD down/out alert rules through Alertmanager to clearly define the conditions that trigger critical alerts.
- Extend monitoring with advanced metrics such as OSD latency, PG status, and pool utilization, and refine associated alert rules to detect potential failure signals at an early stage.
- Integrate with Seekurity SIEM/SOAR to immediately execute automated response playbooks when Ceph-related failure alerts are triggered. For example, configure workflows to run automatic recovery scripts if an OSD remains down for a certain period, violating SLOs, or to provide detailed diagnostic information for post-incident analysis to responsible personnel. Furthermore, leverage FRIIM CNAPP to secure comprehensive security visibility across the cloud infrastructure and to holistically manage potential security threats that may arise at the storage layer, thereby enhancing the overall system reliability.
- Deploy
By establishing a proactive detection and efficient response system for Ceph OSD failures through these practical guidelines, organizations can quantitatively ensure the stability of cloud storage services by reducing Mean Time To Recovery (MTTR) and minimizing service downtime.







