Kubernetes Cost Optimization: A Comprehensive Guide to Dynamic Autoscaling with Karpenter and Prometheus

This is a common scenario in practical operations. Development and operations teams managing large-scale SaaS services leverage the scalability and flexibility of Kubernetes clusters to provide stable applications. However, they frequently encounter cluster resource shortages during unexpected traffic surges or service deployments, or conversely, face situations where abundant idle resources lead to unnecessary costs. Such resource inefficiencies directly result in increased cloud infrastructure expenses, posing a significant burden on development and operational budgets. Ensuring cost efficiency alongside stable service operation is a core objective in all cloud-native environments.

Our team also faced these challenges. The objective was clear: to optimize Kubernetes cluster resource utilization to reduce unnecessary expenditures, while simultaneously building a mechanism that allows for rapid scaling without compromising application performance and availability. This process is crucial, not merely for infrastructure cost reduction, but for simultaneously enhancing service operational agility and stability.

Challenges: Inefficient Resource Management and High Cloud Costs

Previously, cluster nodes were primarily managed using Kubernetes' native feature, Cluster Autoscaler. While Cluster Autoscaler is a valuable tool, it possesses certain limitations. The most significant issue is the node provisioning speed. When new nodes are required, initiating instances from a pre-defined EC2 Auto Scaling Group often incurs considerable time, making it challenging to respond promptly to changes in workload demand. Particularly during traffic surges, Pods would frequently remain in a Pending state, leading to service delays or outages.

Furthermore, there were significant constraints on the utilization of Spot Instances. Cluster Autoscaler primarily operates with On-Demand Instances, and the flexible use of Spot Instances involved complex configurations and management overhead. This represented a substantial loss in terms of cost optimization, as Spot Instances are a powerful means to utilize computing resources at up to 90% lower cost compared to On-Demand Instances. One of the most frequently asked questions in the field concerns how to reliably leverage Spot Instances.

Moreover, the fixed node types within the cluster made it difficult to accommodate the diverse requirements of various workloads. In environments with a mix of memory-intensive and CPU-intensive workloads, achieving optimal resource utilization with a single node type was challenging. Consequently, inefficient situations repeatedly arose, either with excessive idle resources or resource exhaustion on specific nodes. Addressing these issues and securing improved cost efficiency and operational stability constituted our core requirements.

Technology Selection Process: The Combination of Karpenter and Prometheus

To address these challenges, several solutions were evaluated. Candidates ranged from enhancements to the existing Cluster Autoscaler to various third-party node management solutions. Among these, Karpenter, a Kubernetes node provisioner developed by Amazon, was determined to best meet our requirements.

Candidate Technology Comparison: Cluster Autoscaler vs. Karpenter

Karpenter utilizes a Just-in-Time provisioning approach, immediately creating the most suitable node for a Pod's requirements via the EC2 API when a Pod is detected in a Pending state. This offers significantly faster node readiness times compared to Cluster Autoscaler's Auto Scaling Group-based method. Furthermore, its ability to flexibly utilize Spot Instances, On-Demand Instances, and various instance families (e.g., C, M, R types) was appealing, as it enables maximum cost efficiency. Karpenter actively pursues cost savings through its node consolidation feature, which reduces idle nodes by rescheduling Pods distributed across existing nodes to a smaller number of nodes, even when no unschedulable Pods exist.

Additionally, it was deemed critical to accurately monitor cluster resource utilization and Pod statuses. For this purpose, Prometheus, the de facto standard for Kubernetes monitoring, was selected. Prometheus, through various Exporters, can collect detailed metrics at the Kubernetes internal, node, Pod, and container levels, and store them in a time-series database. These metrics provide crucial insights for Karpenter's node provisioning and scale-down decisions, and when visualized with Grafana, allow for an at-a-glance understanding of the cluster's operational status. The conclusion was reached that integrating Karpenter with Prometheus would enable real-time comprehension of cluster states and automated optimal node management based on this information.

Implementation Process: Dynamic Autoscaling with Karpenter and Prometheus

This section describes the specific process for implementing dynamic autoscaling in a Kubernetes cluster using Karpenter and Prometheus. This guide is YAML/code-centric and directly applicable in practical scenarios.

Karpenter Installation and Initial Setup

Karpenter can be easily installed via Helm Chart. Prior to installation, appropriate IAM permissions must be granted to allow Karpenter to provision and manage EC2 instances. This necessitates the creation of a Service Account and an IAM Role for accessing the AWS EC2 API.

First, configure the OIDC provider and create the IAM Policy to be used by Karpenter.

aws iam create-policy \
    --policy-name KarpenterControllerPolicy-${CLUSTER_NAME} \
    --policy-document '{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:CreateLaunchTemplate", "ec2:CreateFleet", "ec2:RunInstances", "ec2:CreateTags", "ec2:TerminateInstances", "ec2:DeleteLaunchTemplate", "ec2:DescribeLaunchTemplates", "ec2:DescribeInstances", "ec2:DescribeImages", "ec2:DescribeSubnets", "ec2:DescribeSecurityGroups", "ec2:DescribeInstanceTypes", "ec2:DescribeInstanceTypeOfferings", "ec2:DescribeAvailabilityZones", "ec2:DeleteTags", "ec2:AssociateAddress", "ec2:DisassociateAddress" ], "Resource": "*" }, { "Effect": "Allow", "Action": "iam:PassRole", "Resource": "arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}" }, { "Effect": "Allow", "Action": "ssm:GetParameter", "Resource": "arn:aws:ssm:${AWS_REGION}::parameter/aws/service/ami-amazon-linux-2-recommended/recommendation/image_id" } ] }'

Karpenter 컨트롤러를 위한 IAM Role과 Service Account 생성


(IRSA: IAM Roles for Service Accounts)



TEMP_FILE=$(mktemp)
curl -fsSL https://raw.githubusercontent.com/kubernetes-sigs/karpenter/main/website/content/en/preview/getting-started/getting-started-with-karpenter/cloudformation.yaml > $TEMP_FILE




aws cloudformation deploy 

--stack-name Karpenter-${CLUSTER_NAME} 

--template-file ${TEMP_FILE} 

--capabilities CAPABILITY_NAMED_IAM 

--parameter-overrides ClusterName=${CLUSTER_NAME} ClusterEndpoint=${CLUSTER_ENDPOINT}

Subsequently, install Karpenter using Helm.

helm upgrade --install karpenter karpenter/karpenter --namespace karpenter --create-namespace \
  --set serviceAccount.create=false \
  --set serviceAccount.name=karpenter \
  --set clusterName=${CLUSTER_NAME} \
  --set clusterEndpoint=${CLUSTER_ENDPOINT} \
  --set defaultProvisioner.enablePodDeletion=false # 필요에 따라 Pod Deletion 활성화

Defining NodePool and EC2NodeClass

The core of Karpenter lies in its NodePool and EC2NodeClass resources. NodePool defines node provisioning policies, while EC2NodeClass defines the detailed attributes of EC2 instances (e.g., AMI, security groups, tags).

The example below presents a NodePool definition that prioritizes Spot Instances and includes various instance types.

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: KarpenterNodeRole-${CLUSTER_NAME} # 이전 단계에서 생성한 노드용 IAM Role
  securityGroupSelector:
    karpenter.sh/discovery: ${CLUSTER_NAME}
  subnetSelector:
    karpenter.sh/discovery: ${CLUSTER_NAME}
  tags:
    karpenter.sh/discovery: ${CLUSTER_NAME}
    environment: production
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      nodeClassRef:
        name: default
      requirements:
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: [c, m, r]
        - key: karpenter.k8s.aws/instance-cpu
          operator: Gt
          values: ['2']
        - key: karpenter.k8s.aws/instance-memory
          operator: Gt
          values: ['4Gi']
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: [c5, c5a, m5, m5a, r5, r5a] # 사용할 인스턴스 패밀리 지정
        - key: karpenter.sh/capacity-type # 스팟 인스턴스 우선 사용
          operator: In
          values: [spot, on-demand]
      taints:
        - key: dedicated
          value: critical-workloads
          effect: NoSchedule
      startupTaints:
        - key: karpenter.sh/provisioning
          effect: NoSchedule
  limits:
    cpu: "1000" # 클러스터가 최대로 사용할 CPU 코어 수 제한
    memory: "1000Gi" # 클러스터가 최대로 사용할 메모리 제한
  disruption:
    consolidationPolicy: WhenUnderutilized # 자원 효율이 떨어지면 노드 통합 시도
    expireAfter: 720h # 30일 후 노드 만료 및 재생성
    budgets:
      - nodes: 1 # 한 번에 최대 1개의 노드만 삭제 (안정성을 위해)

This NodePool definition instructs Karpenter to primarily utilize Spot Instances within the C, M, and R instance families, with a minimum of 2 CPU cores and 4GiB of memory. To isolate nodes for specific workloads, taints can be leveraged.

Metric Collection Using Prometheus and Kube-state-metrics

For Karpenter to operate efficiently and accurately ascertain cluster status, monitoring via Prometheus is essential. Kube-state-metrics, in particular, exposes the status of various resources from the Kubernetes API server as metrics, enabling Prometheus to collect them. This provides critical information, such as Pod Pending status, node resource utilization, and node status, which Karpenter requires for decision-making.

If Prometheus Operator is utilized, Kube-state-metrics can be easily deployed, and metrics collected via ServiceMonitor.

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
  namespace: monitoring
  labels:
    app.kubernetes.io/name: kube-state-metrics
spec:
  endpoints:
  - port: http-metrics
    interval: 30s
  selector:
    matchLabels:
      app.kubernetes.io/name: kube-state-metrics
  namespaceSelector:
    matchNames:
    - kube-system

This ServiceMonitor instructs Prometheus to collect metrics every 30 seconds from the http-metrics port of the kube-state-metrics Pod deployed in the kube-system namespace. Based on the collected metrics, a Grafana dashboard can be configured to track Karpenter's node provisioning and scale-down events, and to monitor the overall cluster status in real time. As a tip, Karpenter itself also exposes controller metrics in Prometheus format, so collecting these is recommended for monitoring Karpenter's internal operations.

Results and Achievements: Cost Reduction and Operational Efficiency Improvement

Following the implementation of dynamic autoscaling based on Karpenter and Prometheus, our team achieved results that exceeded expectations. Significant improvements were observed in both quantitative and qualitative metrics.

Summary of Key Achievements:

• Cloud Cost Reduction: An average of 25% reduction in cloud infrastructure costs was observed compared to previous operations. This was primarily due to increased Spot Instance utilization and the node consolidation feature. The reduction in unnecessary On-Demand Instance over-provisioning during specific peak times was a key contributing factor.
• Resource Utilization Improvement: The overall CPU and memory resource utilization across the cluster improved from an average of 60% to over 85%. This was attributed to Karpenter dynamically selecting node types optimized for workloads and promptly reclaiming unnecessary idle nodes.
• Provisioning Speed Enhancement: The time required for provisioning new nodes was reduced from several minutes to within tens of seconds. This significantly decreased Pod Pending states, thereby resolving service delay issues.

From a qualitative perspective, the operational burden on the operations team significantly decreased. Manual adjustments of node groups or changes in instance types were eliminated, and emergency response situations due to cluster resource shortages were substantially reduced. The development team was able to focus on application development and deployment without concerns about infrastructure resources. The ability to visualize cluster status and cost metrics in real time via Prometheus and Grafana enabled more data-driven decision-making.

Comparison Before and After Karpenter Implementation:

Lessons Learned and Reflection: Managing Unpredictability and Continuous Tuning

The implementation of Karpenter was highly successful; however, several lessons were also learned. The most crucial aspect was managing the unpredictability of Spot Instances. While cost-effective, Spot Instances possess the characteristic of being reclaimable at any time. To mitigate this, it is vital to employ a strategy of combining nodeSelector or nodeAffinity for critical workloads, fixing them to On-Demand Instances or specific Availability Zones. Furthermore, Karpenter's disruption policy must be configured carefully to ensure Pod service continuity during node reclamation. It proved effective to start conservatively with consolidationPolicy and expireAfter settings and then progressively optimize them.

Moreover, fine-tuning Karpenter's NodePool definitions was necessary to satisfy diverse workload requirements. For instance, optimization tasks such as provisioning GPU-enabled instance types for specific GPU workloads or designating instance types with ample storage for storage-intensive workloads proved crucial. Throughout this process, meticulous analysis of metrics collected by Prometheus and accurate identification of application-specific resource demands were essential. Contrary to initial expectations, operating multiple NodePools specialized for particular workload groups often proved more efficient than consolidating all Pods into a single NodePool.

An unexpected ancillary benefit was the increased understanding of cloud resources among team members. Learning about Karpenter's criteria for provisioning and reclaiming nodes, as well as the advantages and disadvantages of Spot Instances, enhanced the overall knowledge level of cloud infrastructure. This also had a positive impact on the establishment of a DevSecOps culture.

Application Guide: Phased Adoption and Enhanced Monitoring

Kubernetes cost optimization using Karpenter and Prometheus is applicable to all cloud-native environments. Organizations considering its adoption in similar environments are advised to proceed with the following steps.

1. Prior Establishment of a Monitoring Environment: Before implementing Karpenter, it is imperative to establish an environment capable of sufficiently collecting and analyzing metrics such as current cluster resource utilization, Pod status, and node count, using tools like Prometheus and Grafana. Accurately identifying current issues is the starting point for optimization.
2. Development of a Phased Adoption Roadmap: Rather than applying Karpenter to all workloads from the outset, it is safer to begin with less critical development/test environments and progressively expand to production environments. Consider first defining specific NodePools and EC2NodeClasses, then gradually migrating workloads using specific taints and tolerations.
3. Formulation of a Spot Instance Utilization Strategy: A strategy for leveraging Spot Instances must be established in advance, including setting Pod Disruption Budgets (PDBs) to prepare for Spot Instance reclamation, adjusting Pods' terminationGracePeriodSeconds, and identifying and isolating workloads for which On-Demand Instance usage is essential.
4. Continuous Tuning and Optimization: Karpenter's NodePool definitions are not a one-time setup. Continuous tuning of instance-type requirements, CPU/Memory constraints, disruption policies, and other settings is necessary as workloads evolve. Metrics collected via Prometheus will serve as crucial guidelines for this tuning process.

Karpenter is a powerful solution that can transform the paradigm of Kubernetes cluster operations, extending beyond a mere autoscaling tool. By integrating it with Prometheus to establish real-time monitoring and feedback loops, organizations can achieve both cost efficiency and operational stability. It is recommended to commence the cloud cost optimization journey by implementing Karpenter without delay.