The dynamic nature of cloud infrastructure in modern enterprise environments necessitates fundamental changes in operational and monitoring approaches. Particularly in private cloud environments like OpenStack, hundreds or thousands of virtual machine (VM) instances are frequently created and deleted in real-time. Effectively tracking and monitoring the status of all VMs in such an environment is nearly impossible with traditional static methods. Manually registering or deleting monitoring targets increases the risk of human error, significantly impedes operational efficiency, and ultimately hinders rapid detection and response during outages.
This article addresses practical methods for dynamically monitoring OpenStack VM instances using Prometheus's file_sd feature. Based on a real-world environment where over 600 VMs are operated dynamically, it provides detailed guidance on the rationale for choosing file_sd, specific implementation scripts, Prometheus configuration, and alert setup. This aims to offer practical insights that organizations can immediately apply to their operational environments.
Monitoring Challenges in Dynamic Cloud Environments
The proliferation of cloud computing and the establishment of DevOps culture have maximized the flexibility and scalability of infrastructure resources. OpenStack, as a key driver of these changes, supports enterprises in building cloud environments within their own data centers. However, this flexibility concurrently introduces new operational challenges. VM instances are created or deleted as needed across development, testing, and production environments, and their numbers constantly fluctuate according to scale-out/in policies. For instance, in large-scale web services or batch processing systems, numerous VMs may be created for a short period and then promptly deleted after completing their tasks.
In such dynamic environments, conventional monitoring systems encounter limitations. Manually registering monitoring targets for each VM is impractical, and even automated scripts require manual updates whenever a specific VM's IP address changes or it is recreated. To address these issues, the monitoring system itself must incorporate Dynamic Service Discovery capabilities to recognize infrastructure changes and automatically update the target list. Prometheus is designed to effectively respond to these dynamic environments by offering various service discovery mechanisms.
Comparison of Prometheus Service Discovery Methods: Why file_sd Was Chosen
Prometheus supports several Service Discovery (SD) mechanisms for identifying dynamic targets, including file_sd, consul_sd, kubernetes_sd, ec2_sd, and openstack_sd. Each method possesses unique advantages, disadvantages, and suitability for specific environments.
Considering a scenario where node_exporter instances across 639 VMs in an OpenStack environment require monitoring, a review of various service discovery methods led to the ultimate selection of file_sd. The following outlines a comparison of the primary service discovery methods and the specific reasons for choosing file_sd.
| Method | Description | Advantages | Disadvantages | Suitability for OpenStack Environment |
|---|---|---|---|---|
file_sd | Prometheus periodically reads YAML/JSON files to update target lists | No external dependencies, simple operation, high flexibility, robust against network constraints | Requires separate script, direct implementation of file generation logic | Very High: Can be optimized through integration scripts with OpenStack API |
consul_sd | Prometheus discovers services registered in HashiCorp Consul | Easy service registration/management in distributed systems, integrated health checks | Requires Consul cluster setup/operation, introduces additional external dependencies | Medium: Feasible if Consul infrastructure is already established |
openstack_sd | Prometheus directly calls OpenStack API to discover VM lists | Native Prometheus feature, no separate script required | Network constraints (Prometheus -> OpenStack API), configuration complexity, exposure of OpenStack authentication information in Prometheus | Low: May be difficult to implement due to network firewalls and authentication issues |
kubernetes_sd | Discovers Pods, Services, etc., via Kubernetes API server | Kubernetes native integration, powerful label filtering | Applicable only to Kubernetes environments | Not Applicable: Cannot be directly applied as the target is OpenStack VMs |
In the described environment, file_sd was the most suitable choice for the following reasons:
- No External Dependencies: The aim was to avoid the burden of building and operating additional distributed systems, such as Consul.
file_sddoes not require separate service discovery infrastructure beyond Prometheus. - Network Constraints: Network connectivity from the monitoring VM (Prometheus server) to OpenStack nodes was restricted to one-way pull. Specifically, directly calling the OpenStack API was constrained by security policies or network configurations.
file_sdcircumvents these restrictions because a Python script calls the OpenStack API to generate a file, which Prometheus then reads locally. - Simple Operation: A significant advantage was the ability to retrieve all VM information and convert it into the desired format using a simple Python script, without complex configurations or debugging. This approach offers high operational simplicity and ease of debugging should issues arise.
OpenStack VM Dynamic Monitoring Architecture and Implementation
The file_sd-based OpenStack VM dynamic monitoring architecture is structured as follows. A scheduler (e.g., cron) is configured to periodically execute a Python script for file_sd on a separate host from the Prometheus server or on the same host. This script calls the OpenStack API to query the list of currently active VMs and extracts each VM's IP address and other metadata. The extracted information is then converted into a YAML or JSON file, adhering to the Prometheus file_sd format, and stored in a path accessible to the Prometheus server. Prometheus reloads this file at a configured refresh interval (e.g., every 5 minutes) to automatically update the monitoring target list.
Python Script Implementation
The following is an example Python script that queries the list of active VMs using the OpenStack API (nova client) and automatically generates it into a Prometheus file_sd formatted YAML file. This script is scheduled to run every 5 minutes to maintain an up-to-date VM list.
#!/usr/bin/env python3
import os
import yaml
from novaclient import client as nova_client
import logging
# 로깅 설정
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s',
handlers=[
logging.FileHandler("/var/log/prometheus_sd_openstack.log"),
logging.StreamHandler()
]
)
# OpenStack 인증 정보 설정 (환경 변수 또는 직접 설정)
OS_AUTH_URL = os.environ.get('OS_AUTH_URL')
OS_USERNAME = os.environ.get('OS_USERNAME')
OS_PASSWORD = os.environ.get('OS_PASSWORD')
OS_PROJECT_NAME = os.environ.get('OS_PROJECT_NAME')
OS_USER_DOMAIN_NAME = os.environ.get('OS_USER_DOMAIN_NAME')
OS_PROJECT_DOMAIN_NAME = os.environ.get('OS_PROJECT_DOMAIN_NAME')
# Nova 클라이언트 초기화
def get_nova_client():
try:
return nova_client.Client(
'2',
os_auth_url=OS_AUTH_URL,
os_username=OS_USERNAME,
os_password=OS_PASSWORD,
os_project_name=OS_PROJECT_NAME,
os_user_domain_name=OS_USER_DOMAIN_NAME,
os_project_domain_name=OS_PROJECT_DOMAIN_NAME
)
except Exception as e:
logging.error(f"Failed to initialize Nova client: {e}")
exit(1)
# Prometheus file_sd 타겟 파일 경로
PROMETHEUS_SD_FILE = '/etc/prometheus/file_sd/instances.yml'
# Node Exporter 포트
NODE_EXPORTER_PORT = 9100
def generate_sd_config():
nova = get_nova_client()
targets = []
try:
# Active 상태의 VM만 조회
servers = nova.servers.list(search_opts={'status': 'ACTIVE'})
for server in servers:
if not server.addresses:
logging.warning(f"VM '{server.name}' has no IP addresses. Skipping.")
continue
# 네트워크 인터페이스에서 IP 주소 추출
# 환경에 따라 'private', 'public' 등 네트워크 이름을 확인해야 합니다.
ip_address = None
for network_name, network_info in server.addresses.items():
for addr_info in network_info:
if addr_info['OS-EXT-IPS:type'] == 'fixed': # fixed IP 우선
ip_address = addr_info['addr']
break
if ip_address: # 고정 IP를 찾았으면 더 이상 검색하지 않음
break
if not ip_address:
logging.warning(f"Could not find a fixed IP for VM '{server.name}'. Skipping.")
continue
target = {
'targets': [f"{ip_address}:{NODE_EXPORTER_PORT}"],
'labels': {
'instance': server.name, # instance 레이블에 VM 이름 사용
'vm_id': server.id,
'az': getattr(server, 'OS-EXT-AZ:availability_zone', 'unknown'), # 가용성 존
# 추가적인 OpenStack 메타데이터를 레이블로 추가 가능
# 'flavor': server.flavor['original_name'],
# 'image': server.image else 'unknown',
}
}
targets.append(target)
# YAML 파일로 저장
with open(PROMETHEUS_SD_FILE, 'w') as f:
yaml.dump(targets, f, default_flow_style=False)
logging.info(f"Successfully generated {len(targets)} targets to {PROMETHEUS_SD_FILE}")
except Exception as e:
logging.error(f"Error during SD config generation: {e}")
if __name__ == '__main__':
generate_sd_config()
This script authenticates with the OpenStack environment using nova client and retrieves a list of all 'ACTIVE' status servers. It extracts the IP address of each server and combines it with node_exporter's default port, 9100, to create targets. Notably, it includes the VM name in the instance label and the Availability Zone in the az label, enabling simultaneous verification of VM name and IP address in Prometheus and facilitating their use in alert routing. The script execution environment must have the python-novaclient library installed. OpenStack authentication information should be managed via environment variables or appropriate security practices.
cron Scheduling
The generated Python script is registered with cron to execute every 5 minutes. This ensures that Prometheus consistently monitors based on the most current VM list.
# crontab -e
*/5 * * * * /usr/bin/python3 /path/to/your/openstack_sd_script.py >> /var/log/openstack_sd_cron.log 2>&1
Prometheus Configuration and Target Management
Once the Python script generates the /etc/prometheus/file_sd/instances.yml file, Prometheus must be configured to read this file. A file_sd_configs entry is added to Prometheus's scrape_configs section.
Prometheus file_sd Configuration Example
# /etc/prometheus/prometheus.yml
scrape_configs:
- job_name: 'openstack-vms'
# Prometheus가 file_sd 파일을 재로드하는 주기. 기본 5분.
# Python 스크립트 실행 주기와 일치시키거나 더 짧게 설정.
scrape_interval: 1m
file_sd_configs:
- names:
- /etc/prometheus/file_sd/instances.yml
refresh_interval: 1m # Prometheus가 파일을 재로드하는 주기 (스크립트 실행 주기와 독립적)
# relabel_configs를 사용하여 레이블을 재정의하거나 필터링할 수 있습니다.
# 예를 들어, '__address__' 레이블에서 IP 주소를 추출하여 'ip_address' 레이블로 추가할 수 있습니다.
relabel_configs:
- source_labels: ['instance']
target_label: 'hostname'
- source_labels: ['__address__']
regex: '(.*):9100'
target_label: 'ip_address'
replacement: '$1'
In the configuration above, refresh_interval defines the frequency at which Prometheus checks the instances.yml file for changes and reloads it. It is recommended to set this value to be equal to or shorter than the Python script's execution interval. For instance, if the script runs every 5 minutes and the refresh_interval is 1 minute, Prometheus will detect and reflect changes in the target list within a maximum of 1 minute after the script updates the file. This mechanism enables Prometheus to automatically add or remove targets when VMs are created or deleted, thereby minimizing monitoring gaps.
file_sd YAML File Format Example
The format of the instances.yml file generated by the Python script is as follows.
- targets:
- 192.168.10.101:9100
labels:
instance: web-server-01
vm_id: a1b2c3d4-e5f6-7890-1234-567890abcdef
az: dmz-az-01
- targets:
- 192.168.10.102:9100
labels:
instance: api-server-02
vm_id: b2c3d4e5-f6a7-8901-2345-67890abcdef1
az: dmz-az-01
- targets:
- 10.0.0.10:9100
labels:
instance: db-server-01
vm_id: c3d4e5f6-a7b8-9012-3456-7890abcdef12
az: internal-az-01
Each entry consists of targets and labels. targets include the address and port to be monitored, while labels provide additional metadata for the target, which is utilized for data querying using Prometheus Query Language (PromQL) and alert routing. The use of the VM name in the instance label, in particular, allows for simultaneous viewing of the VM's IP address and name in the Prometheus UI, significantly enhancing operational convenience.
Alert Configuration and DMZ/INTERNAL AZ Segregation
In dynamic VM environments, prompt and accurate alerting is essential. It is particularly important to differentiate scenarios where a VM temporarily restarts or experiences network issues that prevent monitoring, and to generate appropriate alerts accordingly.
InstanceUnreachable Alert Configuration
Prometheus's up metric indicates the availability of a target. up == 0 signifies that the target is unreachable. However, a brief period of up == 0 might be due to a temporary VM restart or transient network issues. Therefore, to distinguish from actual severe outages, it is effective to generate an alert only if the up == 0 state persists for a defined duration.
# /etc/prometheus/alert.rules
groups:
- name: openstack-vm-alerts
rules:
- alert: InstanceUnreachable
expr: up{job="openstack-vms"} == 0
for: 10m # 10분 동안 'up == 0' 상태 지속 시 알림 발생
labels:
severity: critical
annotations:
summary: 'VM Instance {{ $labels.instance }} ({{ $labels.ip_address }}) is unreachable.'
description: 'OpenStack VM {{ $labels.instance }} (ID: {{ $labels.vm_id }}) in AZ {{ $labels.az }} has been unreachable for 10 minutes. Please investigate network connectivity or VM status.'
In the configuration above, for: 10m causes the InstanceUnreachable alert to be triggered only when the up == 0 state persists for 10 minutes. This helps prevent unnecessary alerts when a VM temporarily restarts due to OS updates or configuration changes, as it typically returns to a normal state within a few minutes, allowing focus on actual outages. The alert message includes label information such as instance (VM name), ip_address, vm_id, and az (Availability Zone) to facilitate rapid situation assessment during an incident.
Segregated Alert Routing by DMZ/INTERNAL AZ
In OpenStack environments, it is common to operate with Demilitarized Zone (DMZ) and INTERNAL Availability Zones segregated according to security and network policies. In such environments, applying alert routing policies tailored to the characteristics of each AZ is efficient. Prometheus Alertmanager enables routing alerts to specific recipient groups based on labels.
# /etc/prometheus/alertmanager.yml
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'job', 'az']
group_wait: 30s
group_interval: 5m
repeat_interval: 1h
receiver: 'default-receiver'
routes:
- match:
az: 'dmz-az-01'
receiver: 'dmz-ops-team'
continue: true
- match:
az: 'internal-az-01'
receiver: 'internal-ops-team'
continue: true
receivers:
- name: 'default-receiver'
# ... 기본 알림 채널 설정 (예: Slack, Email)
- name: 'dmz-ops-team'
# ... DMZ 담당팀 알림 채널 설정
- name: 'internal-ops-team'
# ... 내부망 담당팀 알림 채널 설정
The Alertmanager configuration above routes alerts to dmz-ops-team and internal-ops-team, respectively, based on the az label. This ensures that alerts originating from VMs located in the DMZ are delivered only to DMZ security and operations personnel, and internal network VM alerts are sent only to internal network personnel. This enhances alert efficiency and reduces unnecessary information dissemination. Furthermore, the group_by: ['alertname', 'job', 'az'] setting prevents alert storms by grouping alerts of the same type within the same AZ before transmission.
Troubleshooting and Operational Tips
This section introduces common issues and their resolutions that may arise while operating a file_sd-based dynamic monitoring system, along with tips for effective operation.
Common Issues and Resolutions
- Python script execution errors:
- Problem: The script does not execute as expected, or the
instances.ymlfile is not generated. - Resolution: Examine the cron logs (
/var/log/openstack_sd_cron.log) or the script's own logs to diagnose Python errors, OpenStack authentication errors, ornova clientAPI call issues. Verify environment variable settings and confirm library installations (pip install python-novaclient pyyaml).
- Problem: The script does not execute as expected, or the
- OpenStack API Rate Limit:
- Problem: In large-scale OpenStack environments, API Rate Limits may be reached if the script runs too frequently or queries too many VMs at once.
- Resolution: Adjust the script's execution frequency or check the Rate Limit settings on the OpenStack API server. If necessary, consult with OpenStack administrators to increase API quotas for the monitoring system.
- Prometheus not reloading
file_sdfiles:- Problem: The script successfully generated the
instances.ymlfile, but new targets are not visible in the Prometheus UI, or deleted targets remain. - Resolution: Verify the
file_sd_configspath andrefresh_intervalin the Prometheus configuration (prometheus.yml). Check Prometheus logs for anyfile_sd-related errors and confirm file permissions. The Prometheus server must have the necessary permissions to read the file.
- Problem: The script successfully generated the
node_exporterconnection issues:- Problem: Prometheus recognizes the target, but the
upmetric consistently displays as 0. - Resolution: Confirm that the
node_exporteron the respective VM is running correctly, that port 9100 is open in the firewall (including OpenStack Security Groups), and that there are no network path issues. Test connectivity from the Prometheus server to the VM's IP address on port 9100 usingtelnetorcurl.
- Problem: Prometheus recognizes the target, but the
Operational Tips
- Log Management: Record the execution results and errors of the Python script to a file to facilitate tracing in case of issues.
- Minimize Privileges: The account calling the OpenStack API should be granted only the minimum necessary permissions required for VM list retrieval. VM creation/deletion privileges are unnecessary.
- Configuration Management: Prometheus configuration files and Python scripts should be managed with a version control system (e.g., Git) to track change history and automate deployment.
- Status Validation: Implement logic to validate the content of the generated
instances.ymlfile after script execution to prevent the creation of incorrectly formatted files. An example would be a step to check for YAML parsing errors.
Practical Application and Improvement Effects
This file_sd-based dynamic monitoring system has been successfully implemented on hundreds of OpenStack-based VMs in an operational environment. In an environment where approximately 639 VM instances were dynamically created and deleted, significant resources were expended on manually managing VM monitoring targets prior to the adoption of this system. When new VMs were deployed, Prometheus configuration files required manual updates, and when VMs were deleted, they had to be manually removed—a repetitive process. This increased the workload for operations teams and was a primary cause of monitoring gaps, especially when VMs were urgently deployed during nights or weekends.
After implementing the file_sd-based automated monitoring system, the following improvements were observed:
- Maximized Operational Efficiency: Prometheus targets are automatically updated in alignment with the VM lifecycle, eliminating the need for manual monitoring target management. This significantly reduces the operational burden on teams, allowing them to focus on core tasks.
- Eliminated Monitoring Blind Spots: All active VMs are included in the monitoring scope without omissions, enabling prompt detection of unexpected outages or performance degradations.
- Reduced Mean Time To Detect (MTTD) / Mean Time To Resolve (MTTR): Through
InstanceUnreachablealerts and AZ-specific routing, accurate information is conveyed to relevant personnel during incidents, significantly shortening the time required for problem resolution. - Supported Data-Driven Decision Making: Consistent monitoring data for all OpenStack VMs enables data-driven decision-making, including resource utilization efficiency analysis and capacity planning.
Integrating this monitoring data with Seekurity SIEM enables the establishment of a more comprehensive threat detection and response framework by consolidating VM performance and availability data with security events. For example, if a specific VM's CPU utilization spikes simultaneously with abnormal login attempts on that VM, it can be considered a security threat rather than a mere performance issue, and automated responses can be initiated through Seekurity SIEM/SOAR. Furthermore, the security posture and compliance of the OpenStack environment can be continuously managed and assessed through FRIIM CNAPP, contributing to the maintenance of an all-encompassing Cloud Security Posture beyond operational monitoring.
Future Outlook
Private cloud environments like OpenStack are continuously evolving, and monitoring and operational methods are expected to become more sophisticated. Prometheus's service discovery capabilities will enhance integration with a wider range of cloud and container orchestration platforms in the future. For instance, in hybrid cloud environments where Kubernetes clusters are built on top of OpenStack, specialized SD mechanisms such as kubernetes_sd may become even more critical. To adapt to these changes, it is essential to possess the capability to flexibly apply service discovery strategies optimized for each environment.
Furthermore, monitoring data will evolve beyond mere collection and visualization, progressing towards proactively predicting and responding to potential issues through AI/ML-based Anomaly Detection. For example, by learning historical VM usage patterns, abnormal resource consumption can be automatically detected, and events can be forwarded to Seekurity SIEM for security-centric analysis. Such intelligent monitoring systems will reduce the incident response burden on operations teams and further enhance service stability.
Conclusion
Efficient monitoring is a critical factor for ensuring service stability and enhancing operational efficiency in dynamic cloud environments like OpenStack VMs. A Prometheus file_sd-based dynamic monitoring system offers a robust and practical solution for these complex environments. The key takeaways from this article are summarized as follows.
- Service Discovery Essential for Dynamic Environments: Manual target management is inefficient for responding to the dynamic creation/deletion lifecycle of OpenStack VMs, making Prometheus's dynamic service discovery feature indispensable.
- Practical Value of
file_sd: Due to its advantages of minimized external dependencies, overcoming network constraints, and simple operation,file_sdcan be a highly effective choice in large-scale private cloud environments. - Automation through Python Script: By leveraging the OpenStack API to query
activeVM lists and periodically generating Prometheusfile_sdformatted YAML files, monitoring target lists can be automatically updated. - Sophisticated Alerting and Routing: Conditional alert settings, such as
up == 0 for 10m, reduce unnecessary notifications, while AZ-specific label-based routing enhances the efficiency of alert delivery, enabling prompt incident response.
Establishing such a monitoring system is a significant step towards elevating service stability and security levels beyond operational efficiency. Dynamic monitoring via Prometheus file_sd in an OpenStack environment can alleviate the burden on operations teams, further strengthen security threat detection capabilities through integration with Seekurity SIEM, and establish a foundation for optimizing cloud security posture with FRIIM CNAPP. Organizations are encouraged to adopt these practical strategies to enhance their OpenStack operational environments.







