The establishment of a sophisticated and integrated monitoring system is essential for the stable operation of complex and large-scale cloud environments. Private clouds, especially those based on OpenStack and Ceph, inherently involve various interconnected components. Therefore, the ability to grasp the real-time status of each component and proactively detect potential issues is crucial. In such environments, the design of an integrated control system that secures visibility across distributed infrastructure, maximizes operational efficiency, and identifies potential security threats early on has become a core element of cloud architecture.
This post shares the experience of designing and implementing an integrated monitoring system utilizing Prometheus, Grafana, and Alertmanager within a large-scale private cloud environment built on OpenStack and Ceph. It presents a practical architecture where all functionalities are deployed using Docker Compose on a single monitoring VM, even considering network routing constraints, for a real-world environment operating 10 compute nodes providing 480 vCPUs and 1,784GB of memory, 7 storage nodes with 140 Ceph OSDs and 178TiB of RAW storage, and 639 virtual machines (VMs). This contributes to ensuring infrastructure stability from a Cloud Security Architecture perspective and establishes a robust foundation for future integration with security solutions such as Seekurity SIEM/SOAR.
Technology Overview
OpenStack and Ceph form the core of private cloud deployments, serving as an Infrastructure as a Service (IaaS) platform and a distributed storage system, respectively. OpenStack abstracts compute, network, and storage resources to provide a virtualized environment, while Ceph supports highly available and scalable object, block, and file storage. These two technology stacks operate complementarily, forming the foundation of modern data centers.
To efficiently manage and operate such complex distributed environments, a monitoring system capable of real-time collection, analysis, and visualization of each component's status, performance metrics, and potential events is essential. Prometheus, a Cloud-Native Computing Foundation (CNCF) project, is a monitoring system that collects and stores time-series data and provides a powerful query language (PromQL). Grafana is a tool for visualizing time-series data from various data sources, including Prometheus, into intuitive and visually appealing dashboards. Alertmanager centralizes the management of alerts generated by Prometheus, performing deduplication, grouping, and routing to deliver them to end-users.
This technology stack overcomes the limitations of traditional monitoring solutions regarding scalability and dynamic environment responsiveness, offering a flexible monitoring framework optimized for Cloud-Native environments such as microservice architectures. This minimizes single points of failure and enables rapid root cause analysis and recovery in the event of an outage, significantly enhancing the overall stability of cloud infrastructure. SeekersLab's FRIIM CNAPP/CSPM/CWPP solution utilizes this infrastructure monitoring data to more precisely analyze cloud security posture and provide crucial foundational information for threat detection.
Architecture Analysis
The monitoring architecture for the large-scale OpenStack + Ceph private cloud environment was designed for integrated deployment of Prometheus, Grafana, and Alertmanager via Docker Compose on a single monitoring VM (IP: 172.19.0.201). This monitoring VM collects metrics in a pull-based manner from OpenStack infrastructure nodes (IP: 10.20.0.x range), taking into account an environment where network routing constraints only permit one-way communication from the monitoring VM to OpenStack nodes.
Key components include Prometheus, responsible for metric collection and storage; Grafana, for visualizing collected data; and Alertmanager, for handling alerts. All these operate as isolated Docker containers within a single VM. Metric sources are Exporters deployed on various nodes in the OpenStack environment. Specifically, these include node_exporter deployed on all compute and storage nodes, the ceph_exporter embedded within the Ceph Manager (MGR), and a custom OpenStack exporter for collecting dynamic hypervisor and VM metrics from OpenStack. Additionally, haproxy_exporter for monitoring the status of load balancers like HAProxy and process_exporter for tracking the status of specific processes are also utilized.
The data flow is as follows: Exporters deployed on each OpenStack infrastructure and compute node expose metrics via specific ports. Prometheus, within the monitoring VM, periodically pulls metrics by connecting to these Exporter endpoints according to the configured scrape_interval (15 seconds). Collected metrics are stored in Prometheus's time-series database, and Grafana designates Prometheus as a data source to visualize this data through dashboards. When an event exceeding a threshold defined in Prometheus's Alerting Rules occurs, Prometheus sends it to Alertmanager. Alertmanager then groups alerts according to predefined rules and forwards them to a Mattermost channel. This structure provides a foundation for quickly recognizing and responding to potential issues centrally within a large-scale OpenStack + Ceph environment.
Core Mechanisms
Prometheus-based Metric Collection and Processing
Prometheus is a pull-based monitoring system optimized for Cloud-Native environments. It operates by deploying lightweight Exporters on each monitoring target, with the Prometheus server periodically accessing the HTTP endpoint of these Exporters to retrieve metric data. In this architecture, a scrape_interval of 15 seconds was configured to achieve near real-time monitoring.
node_exporter: Deployed on all compute and storage nodes to collect operating system and hardware-level metrics such as CPU utilization, memory usage, disk I/O, and network traffic.ceph_exporter: Utilizes the Exporter embedded in the Ceph Manager (MGR) daemon to provide detailed Ceph-related metrics, including Ceph cluster status, OSD status, Pool usage, PG (Placement Group) status, and I/O latency. Thehonor_labels: truesetting preserves Ceph's native labels to maintain rich context.- Custom
OpenStack exporter: Developed to collect key metrics such as the number of VMs per hypervisor, CPU/memory/disk/network usage per VM, and OpenStack API service response times, due to OpenStack's dynamic nature. It extracts metrics using OpenStack Horizon API and Nova API, among others, and exposes them in Prometheus format. haproxy_exporter,process_exporter: Monitor HAProxy load balancer metrics such as session count and error rates, and track the status and resource usage of specific critical processes (e.g., MySQL, RabbitMQ) to ascertain the overall health of the infrastructure.
To efficiently manage dynamically changing monitoring targets across 639 VMs and numerous OpenStack nodes, Prometheus's file_sd_configs feature was actively utilized. By placing files defining target lists in JSON or YAML format in a specific directory, Prometheus periodically scans these files to automatically add or remove targets. This significantly reduces the manual configuration overhead in large-scale virtualized environments.
# prometheus.yml 예시
scrape_configs:
- job_name: 'openstack-nodes'
scrape_interval: 15s
static_configs:
- targets: ['10.20.0.10:9100', '10.20.0.11:9100', ...]
labels:
instance_type: 'controller'
- targets: ['10.20.0.20:9100', '10.20.0.21:9100', ...]
labels:
instance_type: 'compute'
- job_name: 'ceph-mgr'
scrape_interval: 15s
static_configs:
- targets: ['10.20.0.30:9283'] # Ceph MGR 노드
metric_relabel_configs:
- source_labels: [__name__]
regex: 'ceph_mgr_.*'
action: keep
honor_labels: true
- job_name: 'openstack-vms'
file_sd_configs:
- names: ['/etc/prometheus/file_sd/openstack_vms.json']
refresh_interval: 30s
- job_name: 'custom-openstack-exporter'
static_configs:
- targets: ['10.20.0.10:9xxx'] # 커스텀 exporter 실행 노드
As demonstrated in the example configuration above, file_sd_configs is used to manage dynamically changing VM lists, and honor_labels preserves Ceph's unique labels to maintain rich metric context.
Building Integrated Dashboards with Grafana
Grafana visualizes numerous collected time-series metrics into intuitive dashboards, enabling operators to grasp the system's status at a glance. In this private cloud environment, the following types of dashboards were established to maximize operational efficiency.
- Overall Infrastructure Overview Dashboard: Comprehensively displays CPU, memory, disk, and network utilization for all compute and storage nodes, along with the overall status of the Ceph cluster (cluster health, number of OSDs, data usage, etc.).
- OpenStack Detailed Dashboard: A dashboard focused on OpenStack services and virtual resources, including Nova API response times, Glance/Cinder/Neutron service status, VM availability per hypervisor, and resource usage per VM (CPU, memory, network I/O, disk I/O). Notably, for VM-specific metrics, dashboard panels can be designed to be automatically generated for dynamically added VMs via
file_sd_configs. - Ceph Cluster Detailed Dashboard: Provides in-depth information essential for Ceph operations, such as OSD I/O, latency, status changes, PG status, Monitor/Manager daemon status, and storage pool usage and performance metrics.
- Network/Load Balancer Dashboard: Monitors HAProxy traffic, session counts, backend server status, OpenStack network port usage, and error rates, among other metrics.
Leveraging Grafana's Template Variables feature allows users to select specific nodes or VMs within a single dashboard to filter detailed information, enabling efficient drill-down analysis even in large-scale environments. This visualization plays a crucial role in rapidly identifying infrastructure performance bottlenecks and proactively responding to potential failures.
Intelligent Alerting and Integration via Alertmanager
Alerting Rules defined based on metrics collected by Prometheus generate alerts upon threshold exceedance or detection of specific patterns. These generated alerts are forwarded to Alertmanager for centralized processing. Alertmanager provides various functionalities for efficient alert management.
- Grouping: Combines similar alerts that occur simultaneously to reduce alert fatigue. For example, if CPU utilization exceeds a threshold on multiple servers within the same server farm, these are grouped into a single alert for transmission.
- Inhibition: Suppresses the transmission of subordinate alerts when a higher-level alert occurs. For instance, if an alert indicating a complete datacenter power outage is triggered, individual server power failure alerts are redundant and therefore inhibited.
- Silencing: Temporarily suspends specific alerts during planned maintenance periods.
- Routing: Directs alerts to specific receivers based on their labels.
In this architecture, Mattermost was configured as the alert receiving channel, enabling the operations team to review and respond to alerts in real-time. Mattermost integration is implemented through Alertmanager's Webhook functionality.
# alertmanager.yml 예시
global:
resolve_timeout: 5m
route:
receiver: 'default-receiver'
group_by: ['alertname', 'instance']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receivers:
- name: 'default-receiver'
webhook_configs:
- url: 'http://mattermost.example.com/hooks/xxxxxxxxxxxxxxxxxxxxxxxxxx'
send_resolved: true
# Prometheus alerting rule 예시 (rules.yml)
# groups:
# - name: 'instance_alerts'
# rules:
# - alert: HighCPULoad
# expr: node_cpu_usage_total{mode!="idle"} > 80
# for: 5m
# labels:
# severity: warning
# annotations:
# summary: 'Instance {{ $labels.instance }} has high CPU load'
# description: 'CPU usage is above 80% for 5 minutes. Current value: {{ $value }}%'
Alertmanager's robust features significantly contribute to enabling operations teams to focus on critical issues amidst numerous alerts, reduce unnecessary alert fatigue, and shorten incident response times. Furthermore, these real-time infrastructure alerts can be integrated into SeekersLab's Seekurity SIEM/SOAR solution to analyze correlations between infrastructure-level anomalies and security events, providing crucial information for triggering automated security response playbooks.
Performance Comparison
The choice of a solution for private cloud monitoring can vary depending on the characteristics of the operating environment, scale, budget, and the team's expertise. When compared to other popular monitoring tools such as the ELK Stack (Elasticsearch, Logstash, Kibana) or Zabbix, Prometheus exhibits particular strengths in Cloud-Native environments and time-series metric monitoring. The following is a comparison table of major monitoring solutions.
| Characteristic | Prometheus | ELK Stack | Zabbix |
|---|---|---|---|
| Primary Purpose | Time-series metric monitoring and alerting | Log collection, search, analysis, visualization | Network device, server monitoring and alerting |
| Data Model | Key-Value label-based time-series data | JSON document-based logs/events | Item, Trigger-based |
| Metric Collection Method | Pull model (HTTP scraping) | Push model (Beats/Logstash) | Push/Pull model (Agent/SNMP) |
| Scalability | Horizontal scaling (Federation, Remote Storage) | Horizontal scaling (Clustering, Sharding) | Distributed monitoring (Proxy) |
| Dynamic Target Discovery | Excellent (Service Discovery integration) | Limited (Configuration-based) | Limited (Host registration) |
| Query Language | PromQL (Powerful time-series query) | Lucene Query Syntax | Zabbix Macro/Function |
| Resource Efficiency | Relatively high (Metric storage optimization) | Very high depending on data volume (when processing logs) | Moderate |
| Cloud-Native Suitability | Very high | High | Moderate |
Prometheus excels particularly in cloud environments where numerous instances are dynamically created and destroyed. Its pull-based collection method reduces dependency on monitoring targets, and the Service Discovery feature allows dynamic resources such as OpenStack VMs to be flexibly added as monitoring targets. In contrast, the ELK Stack is strong in processing and analyzing log data, while Zabbix is suitable for traditional infrastructure monitoring. For this project, real-time, metric-based monitoring in a large-scale OpenStack + Ceph environment was a core requirement, making the Prometheus + Grafana + Alertmanager stack the most appropriate choice.
Practical Configuration
All monitoring components were deployed on a single VM (172.19.0.201) utilizing Docker Compose. This approach reduced initial deployment complexity, enhanced operational convenience, and aligned with a centralized single-VM strategy necessitated by network routing constraints. Docker Compose facilitates easy definition and management of Prometheus, Grafana, and Alertmanager containers.
# docker-compose.yml 예시
version: '3.8'
services:
prometheus:
image: prom/prometheus:v2.47.0
container_name: prometheus
volumes:
- ./prometheus:/etc/prometheus
- prometheus_data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=30d'
ports:
- "9090:9090"
restart: always
grafana:
image: grafana/grafana:10.1.5
container_name: grafana
volumes:
- grafana_data:/var/lib/grafana
- ./grafana/provisioning:/etc/grafana/provisioning
environment:
- GF_SECURITY_ADMIN_USER=admin
- GF_SECURITY_ADMIN_PASSWORD=your_secure_password
ports:
- "3000:3000"
restart: always
alertmanager:
image: prom/alertmanager:v0.26.0
container_name: alertmanager
volumes:
- ./alertmanager:/etc/alertmanager
- alertmanager_data:/alertmanager
command:
- '--config.file=/etc/alertmanager/alertmanager.yml'
- '--storage.path=/alertmanager'
ports:
- "9093:9093"
restart: always
volumes:
prometheus_data:
grafana_data:
alertmanager_data:
This docker-compose.yml file defines all components and specifies volumes for persistently storing each component's configuration files (`prometheus.yml`, `alertmanager.yml`) and data. Deployment is performed simply with the `docker compose up -d` command.
Resource allocation for the monitoring VM is critically important. Collecting metrics at 15-second intervals from 10 compute nodes, 7 storage nodes, and 639 VMs demands substantial I/O and CPU resources. Experience indicates that for this scale, allocating a minimum of 8 vCPUs, 32GB RAM, and sufficient SSD-based storage contributes to stable operation. The data retention period (`--storage.tsdb.retention.time`) setting for Prometheus directly impacts storage usage; therefore, it should be adjusted to an appropriate duration (e.g., 30 days) according to operational policies. For long-term data preservation, utilizing Prometheus's Remote Storage feature to offload data to Object Storage or similar solutions can be considered.
Due to network routing constraints, the environment permitted only one-way pulling from the monitoring VM to OpenStack nodes. Consequently, it was essential to add rules to the firewall of each OpenStack node, allowing Exporter ports (e.g., 9100, 9283, custom Exporter ports) for the monitoring VM's IP address. This aspect can be enhanced by integrating with SeekersLab's FRIIM CWPP (Cloud Workload Protection Platform) to enable more granular control and auditing of network security policies per workload.
Monitoring and Operations
The established integrated monitoring system requires continuous operation and management. Periodically checking core monitoring metrics and establishing a rapid response system for alerts generated by Alertmanager are essential.
- OpenStack Key Metrics:
- Hypervisor Resources: CPU, memory, disk utilization, and load average for each compute node.
- VM Resources: CPU utilization, memory usage, disk I/O (read/write), and network I/O (receive/transmit) for individual VMs.
- API Service Response Time: Response latency and error rates for key OpenStack API services such as Nova, Glance, Cinder, and Neutron.
- Network Flow: OpenStack network traffic patterns, packet drop rates, and error rates.
- Ceph Key Metrics:
- Cluster Status: Overall cluster health, including
HEALTH_OK,HEALTH_WARN,HEALTH_ERR. - OSD Status: Activation status (up/in), I/O performance, latency, and storage usage for each OSD.
- PG Status: Placement Group status (e.g., active+clean, degraded, stale) and Recovering/Backfilling progress.
- Storage Pool Usage: Utilization and remaining space for each storage pool.
- Cluster Status: Overall cluster health, including
A cautionary aspect during operations is the management of 'Alert Fatigue'. Unnecessary or duplicate alerts can diminish sensitivity to genuinely critical alerts. Therefore, it is crucial to actively utilize Alertmanager's grouping, inhibition, and silencing features, and to periodically tune the thresholds of alerting rules. Furthermore, monitoring the health of the monitoring system itself (Monitoring the Monitor) must not be overlooked. Disk usage, memory usage, and Exporter scraping success rates of the Prometheus server should be monitored to immediately identify any issues that arise.
From the perspective of incident response scenarios, establishing Standard Operating Procedures (SOPs) for diagnosing and resolving issues upon the occurrence of alerts exceeding specific thresholds is effective. For instance, upon a Ceph OSD failure alert, one would follow the procedure for restarting or replacing that OSD. For high CPU utilization alerts on an OpenStack VM, the processes on that VM would be examined, and measures such as resource augmentation or workload distribution would be taken. Such operational experience and monitoring data can be leveraged through SeekersLab's KYRA AI Guardrail to train AI-based anomaly detection and predictive models, thereby contributing to proactive infrastructure management and enhanced security.
Conclusion
From an architectural perspective, an integrated monitoring system utilizing Prometheus, Grafana, and Alertmanager within a large-scale OpenStack and Ceph private cloud environment serves as an essential foundation for stable infrastructure operation and efficient management. The core of this architecture lies in its ability to collect, analyze, and visualize vast amounts of metrics generated from numerous compute and storage nodes and hundreds of virtual machines in real-time, while providing immediate alerts upon critical events. Even in complex cloud environments, the essence ultimately rests on visibility and prompt response.
From a practical standpoint, the key strengths of this system are clear. Firstly, its Cloud-Native oriented design offers flexible responsiveness to dynamic cloud environments and provides excellent scalability, which is a prerequisite for modern cloud infrastructure. Secondly, the pull-based metric collection method combined with file_sd_configs is crucial for minimizing management overhead even in large-scale VM environments. Thirdly, intuitive visualization through Grafana and intelligent alert management by Alertmanager are essential for maximizing operational efficiency. Finally, its open-source foundation makes it cost-effective and grants access to active community support, an undeniable advantage.
Based on operational experience, several limitations must also be acknowledged. It is an undeniable fact that a learning curve may exist for teams unfamiliar with Prometheus. Furthermore, concentrating all components on a single monitoring VM inherently carries the clear risk of a single point of failure, where an outage of that VM could bring down the entire monitoring system. Ensuring high availability for the monitoring VM itself and establishing backup strategies are indispensable measures to mitigate this. Ultimately, the infrastructure data collected from such an integrated monitoring system will serve as a core asset for SeekersLab's FRIIM CNAPP/CSPM/CWPP solution and Seekurity SIEM/SOAR, becoming a crucial foundation for further strengthening cloud security posture management, threat detection, and response capabilities beyond mere infrastructure operations.







