Executive Summary & Background
When high-growth B2B SaaS platforms scale past $1M ARR, monthly AWS cloud compute bills frequently double faster than subscription revenue.
During an infrastructure diagnostic audit for a multi-tenant SaaS client, the monthly AWS EC2 bill had reached $11,000/month. The cluster was running 8 t3.2xlarge EC2 worker nodes (16 vCPU, 32GB RAM each), yet overall node CPU utilization averaged less than 6.5%.
This case study demonstrates how we conducted an empirical 30-day Prometheus metrics audit, right-sized pod resource requests to match actual P95 consumption, and deployed Horizontal Pod Autoscaling (HPA) alongside Cluster Autoscaler — reducing the required worker node count from 8 instances to 4 instances, saving $3,850/month ($46,200/year) with zero downtime.
Financial Impact & Metrics Comparison
| Optimization Metric | Pre-Audit Cluster State | Post-Optimization State | Financial Impact / Benefit |
|---|---|---|---|
| EC2 Worker Node Count | 8 x t3.2xlarge Instances | 4 x t3.2xlarge Instances | 50% Node Count Reduction |
| Monthly AWS Compute Bill | $11,000 / month | $7,150 / month | $3,850/mo ($46,200/yr) Saved |
| Average Node CPU Utilization | 6.2% (Massive Waste) | 58.4% (Optimal Efficiency) | 9x Resource Utilization Increase |
| Pod OOMKilled Incidents | 12 per month | 0 per month | 100% Application Stability |
Root Cause Analysis: The Idle Resource Trap
In Kubernetes, worker node capacity is allocated based on Resource Requests, NOT actual CPU/RAM usage.
Developers had configured default pod requests of cpu: "2000m" (2 vCPU cores) and memory: "4Gi" for every microservice container to prevent Out-Of-Memory (OOM) crashes during initial staging tests.
When scaled across 20+ microservice replicas, Kubernetes reserved 32 vCPU cores on node capacity. The Kubernetes scheduler refused to schedule new pods on existing worker nodes, forcing AWS Autoscaling Groups to spin up 8 expensive EC2 instances even though actual CPU usage across the entire cluster was under 6%.
[ EC2 Worker Node: 16 vCPU / 32GB RAM ]
├── Pod A: Requested 4 CPU (Actual Avg: 0.1 CPU) ──▶ Reserved
├── Pod B: Requested 4 CPU (Actual Avg: 0.15 CPU) ──▶ Reserved
├── Pod C: Requested 4 CPU (Actual Avg: 0.08 CPU) ──▶ Reserved
└── Node Capacity Full! (Kubernetes demands Node 2 even though CPU is 95% idle!)
3-Step Remediation Engineering
Step 1: Scraping 30-Day P95/P99 Metrics with PromQL
We deployed Prometheus and scraped container-level metrics across peak business hours to identify actual resource usage patterns:
# 1. Scrape P95 CPU consumption per microservice container over 30 days
histogram_quantile(0.95, sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod, container, le))
# 2. Scrape P99 Memory working set bytes per container
quantile_over_time(0.99, container_memory_working_set_bytes{container!=""}[30d])
# 3. Detect pods with excessive CPU request-to-usage ratio (> 10x gap)
(sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod)
/
sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)) > 10
Step 2: Right-Sizing Pod Requests & Limits
Based on empirical PromQL data, we updated deployment manifests. We aligned Requests to actual P95 consumption and set Limits to P99 plus a 25% safety margin:
# k8s/deployments/backend-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: saas-api-backend
namespace: production
spec:
replicas: 4
template:
spec:
containers:
- name: api-server
image: registry.company.com/api-server:v2.4.1
resources:
requests:
cpu: "150m" # Reduced from 2000m (P95 actual: 110m)
memory: "256Mi" # Reduced from 4096Mi (P95 actual: 180Mi)
limits:
cpu: "500m" # Allows bursts during request spikes
memory: "512Mi" # Prevents runaway memory leaks
Step 3: Configuring Horizontal Pod Autoscaling (HPA)
Instead of running fixed replica counts, we configured HPA to scale dynamically based on custom HTTP request throughput metrics scraped from Traefik Ingress:
# k8s/autoscaling/hpa-backend.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: saas-api-backend-hpa
namespace: production
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: saas-api-backend
minReplicas: 3
maxReplicas: 16
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
- type: External
external:
metric:
name: traefik_entrypoint_requests_per_second
target:
type: AverageValue
averageValue: "150"
behavior:
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 25
periodSeconds: 60
Final Results & Recommendations
- ▸Immediate Compute Savings: Safely drained and decommissioned 4 EC2 worker nodes, saving $3,850/month ($46,200/year).
- ▸Zero Performance Loss: Passed synthetic 5x traffic load tests with zero HTTP 5xx errors and sub-200ms P95 API latencies.
- ▸Automated Scalability: During marketing email blasts, HPA automatically scales backend pods from 3 to 12 replicas in under 20 seconds, scaling back down smoothly after traffic subsides.