Cutting AWS Compute Bills by 35% with K3s Pod Request Tuning

Executive Summary & Background

When high-growth B2B SaaS platforms scale past $1M ARR, monthly AWS cloud compute bills frequently double faster than subscription revenue.

During an infrastructure diagnostic audit for a multi-tenant SaaS client, the monthly AWS EC2 bill had reached $11,000/month. The cluster was running 8 t3.2xlarge EC2 worker nodes (16 vCPU, 32GB RAM each), yet overall node CPU utilization averaged less than 6.5%.

This case study demonstrates how we conducted an empirical 30-day Prometheus metrics audit, right-sized pod resource requests to match actual P95 consumption, and deployed Horizontal Pod Autoscaling (HPA) alongside Cluster Autoscaler — reducing the required worker node count from 8 instances to 4 instances, saving $3,850/month ($46,200/year) with zero downtime.

Financial Impact & Metrics Comparison

Root Cause Analysis: The Idle Resource Trap

In Kubernetes, worker node capacity is allocated based on Resource Requests, NOT actual CPU/RAM usage.

Developers had configured default pod requests of cpu: "2000m" (2 vCPU cores) and memory: "4Gi" for every microservice container to prevent Out-Of-Memory (OOM) crashes during initial staging tests.

When scaled across 20+ microservice replicas, Kubernetes reserved 32 vCPU cores on node capacity. The Kubernetes scheduler refused to schedule new pods on existing worker nodes, forcing AWS Autoscaling Groups to spin up 8 expensive EC2 instances even though actual CPU usage across the entire cluster was under 6%.

[ EC2 Worker Node: 16 vCPU / 32GB RAM ]
 ├── Pod A: Requested 4 CPU (Actual Avg: 0.1 CPU) ──▶ Reserved
 ├── Pod B: Requested 4 CPU (Actual Avg: 0.15 CPU) ──▶ Reserved
 ├── Pod C: Requested 4 CPU (Actual Avg: 0.08 CPU) ──▶ Reserved
 └── Node Capacity Full! (Kubernetes demands Node 2 even though CPU is 95% idle!)

3-Step Remediation Engineering

Step 1: Scraping 30-Day P95/P99 Metrics with PromQL

We deployed Prometheus and scraped container-level metrics across peak business hours to identify actual resource usage patterns:

# 1. Scrape P95 CPU consumption per microservice container over 30 days
histogram_quantile(0.95, sum(rate(container_cpu_usage_seconds_total{container!=""}[5m])) by (pod, container, le))

# 2. Scrape P99 Memory working set bytes per container
quantile_over_time(0.99, container_memory_working_set_bytes{container!=""}[30d])

# 3. Detect pods with excessive CPU request-to-usage ratio (> 10x gap)
(sum(kube_pod_container_resource_requests{resource="cpu"}) by (pod)
  /
 sum(rate(container_cpu_usage_seconds_total[5m])) by (pod)) > 10

Step 2: Right-Sizing Pod Requests & Limits

Based on empirical PromQL data, we updated deployment manifests. We aligned Requests to actual P95 consumption and set Limits to P99 plus a 25% safety margin:

# k8s/deployments/backend-service.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: saas-api-backend
  namespace: production
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: api-server
          image: registry.company.com/api-server:v2.4.1
          resources:
            requests:
              cpu: "150m"      # Reduced from 2000m (P95 actual: 110m)
              memory: "256Mi"  # Reduced from 4096Mi (P95 actual: 180Mi)
            limits:
              cpu: "500m"      # Allows bursts during request spikes
              memory: "512Mi"  # Prevents runaway memory leaks

Step 3: Configuring Horizontal Pod Autoscaling (HPA)

Instead of running fixed replica counts, we configured HPA to scale dynamically based on custom HTTP request throughput metrics scraped from Traefik Ingress:

# k8s/autoscaling/hpa-backend.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: saas-api-backend-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: saas-api-backend
  minReplicas: 3
  maxReplicas: 16
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: External
      external:
        metric:
          name: traefik_entrypoint_requests_per_second
        target:
          type: AverageValue
          averageValue: "150"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

Final Results & Recommendations

1. ▸
   Immediate Compute Savings: Safely drained and decommissioned 4 EC2 worker nodes, saving $3,850/month ($46,200/year).
2. ▸
   Zero Performance Loss: Passed synthetic 5x traffic load tests with zero HTTP 5xx errors and sub-200ms P95 API latencies.
3. ▸
   Automated Scalability: During marketing email blasts, HPA automatically scales backend pods from 3 to 12 replicas in under 20 seconds, scaling back down smoothly after traffic subsides.