# Kubernetes Cost Optimization

Source: https://docs.ops0.com/docs/how-to/kubernetes-cost-optimization

Use ops0 Kubernetes cost analysis to identify over-provisioned workloads, idle namespaces, and orphaned resources — then right-size and cut spending.

---

## Scenario

Your production Kubernetes cluster bill jumped 40% last quarter. Engineering says they didn't deploy anything major. You need to find where the money is going and cut waste without impacting production workloads.

**What you'll accomplish:**
- Install OpenCost to get per-namespace cost visibility
- Identify the top 3 cost drivers
- Right-size over-provisioned workloads
- Clean up idle namespaces and orphaned resources
- Set up ongoing cost monitoring

---

## Prerequisites

- Kubernetes cluster connected to ops0

- Cluster deploy permissions (required for OpenCost installation)

- kubectl access (for applying resource limit changes)

---

## Step 1: Install OpenCost

1. Go to **Kubernetes → [your cluster] → Cost Analysis** tab

2. Click **Install OpenCost**

3. Wait 2–3 minutes for installation to complete — status changes from **Installing** to **Running**

4. Cost data begins appearing within 15 minutes as OpenCost collects metrics

**OpenCost uses resource requests, not actual usage**

OpenCost calculates cost based on resource requests (not actual usage). Pods without requests set will show $0 — fix this first if you see missing data.

---

## Step 2: Find Your Top Cost Drivers

1. Go to **Cost Analysis → view by Namespace** → sort by **Total/month** descending

2. Note the top 5 namespaces by cost

3. Click each namespace → **view by Workload** → identify which Deployments/StatefulSets are most expensive

### Example cost breakdown

```
Namespace           CPU/mo    Mem/mo   Storage/mo   Total/mo
──────────────────────────────────────────────────────────────
ml-training         $520      $210     $65          $795  ← investigate
api-services        $185      $90      $20          $295
monitoring          $75       $145     $60          $280
dev-sandbox         $310      $125     $0           $435  ← investigate
frontend            $40       $25      $5           $70
```

Two namespaces stand out: `ml-training` (expensive — is it justified?) and `dev-sandbox` (high compute cost, no storage — likely idle).

---

## Step 3: Identify Over-Provisioned Workloads

In **Cost Analysis → select a namespace → Workload view**, look for workloads where Requested >> Actual usage. ops0 shows both columns when metrics are available.

**Signs of over-provisioning:**

| Signal | Example |
|--------|---------|
| CPU over-requested | CPU requested: 4 cores, CPU used: 0.3 cores consistently |
| Memory over-requested | Memory requested: 8Gi, Memory used: 1.2Gi |

**Steps to right-size:**

1. Note the workload name and current requests/limits from the Cost Analysis view

2. Observe actual usage in **Kubernetes → [cluster] → Pods → select the pod → Metrics tab** (shows CPU/memory graphs)

3. Set new requests to ~120% of p99 actual usage — leave headroom for spikes

4. Update your Terraform or manifest with the new values (see example below)

5. Apply via IaC project (preferred) or `kubectl apply`

6. Monitor for 24 hours — confirm no OOMKilled events or CPU throttling

**Before (over-provisioned):**

```yaml
resources:
  requests:
    cpu: "4"
    memory: "8Gi"
  limits:
    cpu: "8"
    memory: "16Gi"
```

**After (right-sized):**

```yaml
resources:
  requests:
    cpu: "500m"
    memory: "1.5Gi"
  limits:
    cpu: "2"
    memory: "3Gi"
```

**Estimated savings**

Right-sizing one workload from 4 CPU to 0.5 CPU on a 3-replica deployment saves approximately $180/month on a standard cloud instance type.

---

## Step 4: Clean Up Idle Namespaces

The `dev-sandbox` namespace costs $435/month with no persistent storage — it's burning compute outside business hours.

**Option A — Add resource quotas to cap dev spending:**

```yaml
apiVersion: v1
kind: ResourceQuota
metadata:
  name: dev-sandbox-quota
  namespace: dev-sandbox
spec:
  hard:
    requests.cpu: "4"
    requests.memory: 8Gi
    limits.cpu: "8"
    limits.memory: 16Gi
```

**Option B — Schedule shutdown outside business hours using a Workflow:**

1. Create a workflow in **ops0 → Workflows** with two steps

2. Step 1 (cron: `0 20 * * 1-5`): `kubectl scale deployments --replicas=0 -n dev-sandbox`

3. Step 2 (cron: `0 8 * * 1-5`): `kubectl scale deployments --replicas=1 -n dev-sandbox`

This stops dev workloads at 8 PM and restarts at 8 AM on weekdays — approximately 65% cost reduction for the namespace.

---

## Step 5: Remove Orphaned Resources

Go to **Kubernetes → [cluster] → Resources → Orphan Resources** tab.

**Look for:**

| Resource type | Why it costs money |
|--------------|-------------------|
| PersistentVolumeClaims not attached to any pod | Charged even when unused |
| LoadBalancer Services with no active backends | Cloud LB hourly charges still apply |
| Completed Jobs that were never cleaned up | Linger as clutter, may hold PVCs |

**For each orphan:**

1. Click **Adopt** if it should be managed by an IaC project

2. Click **Delete** if it's genuinely unused — confirm with the owning team first

3. Click **Ignore** if it's intentionally standalone and should be excluded from future scans

**Typical orphan savings**

Orphaned PVCs at 100 GB SSD cost approximately $12/month each. Orphaned cloud LoadBalancers cost approximately $18/month each. These accumulate silently over time.

---

## Step 6: Set Up Ongoing Cost Monitoring

1. Go to **Kubernetes → [cluster] → Cost Analysis → Alerts** → set a monthly budget → ops0 notifies when spending exceeds the threshold

2. Bookmark **Cost Analysis → Namespace view → sort by Total/month** — review the top 5 namespaces for 15 minutes each month

---

## Verification

After completing the steps above, confirm:

| Check | Expected result |
|-------|----------------|
| OpenCost status | Shows **Running** in the Cost Analysis tab |
| Cost data | All namespaces show cost data (not $0 across the board) |
| Right-sized workload | CPU usage stays below new limits with no throttling events after 24 hours |
| Idle namespace shutdown | Cost Analysis shows reduced spend the following day |

---

## Troubleshooting

Cost shows $0 for all workloads

Pods don't have resource requests set. Add requests to all deployments — even `requests: {cpu: "100m", memory: "128Mi"}` enables cost tracking.

OpenCost stuck in Installing

Check the OpenCost pod in the `opencost` namespace. It may be failing due to insufficient RBAC permissions — the cluster connection needs cluster-admin or a custom ClusterRole that grants the required access.

Right-sized workload keeps OOMKilling

Actual memory usage spiked above the p99 you measured. Increase the memory limit by 50% and re-observe for another week before reducing further.

Orphan Resources tab is empty

The cluster connection may lack list permissions for all resource types. Check that the ops0 service account ClusterRole includes `get`, `list`, and `watch` on all API groups.

---

## Next Steps

[Cost Optimization → Broader cloud cost controls including policies, budget alerts, and discovery](https://docs.ops0.com/docs/how-to/cost-optimization)
[Kubernetes Cost Analysis → Reference docs for the Cost Analysis tab, metrics, and OpenCost configuration](https://docs.ops0.com/docs/kubernetes/cost-analysis)
[Workflows → Schedule namespace shutdowns and other automated cost-saving actions](https://docs.ops0.com/docs/how-to/first-workflow)
[Connect a Cluster → Add your Kubernetes cluster to ops0 to enable cost analysis and monitoring](https://docs.ops0.com/docs/how-to/connect-kubernetes-cluster)