Skip to content

Lab: Kubernetes Metrics Pipeline

Goal: Deploy metrics-server → Scrape with Prometheus → Visualize in Grafana

Prerequisites

  • A running Kubernetes cluster (Minikube, Kind, or similar)
  • kubectl and optionally helm installed

Overview

+-------------------+
| metrics-server    |  <-- collects resource metrics (CPU/mem) from nodes and pods
+-------------------+
         ↓
+-------------------+
| Prometheus        |  <-- scrapes metrics-server via HTTP
+-------------------+
         ↓
+-------------------+
| Grafana           |  <-- visualizes Prometheus metrics
+-------------------+

Step-by-Step Instructions

Deploy metrics-server

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

⚠️ If you're using Minikube, patch the deployment:

kubectl patch deployment metrics-server -n kube-system \
  --type=json -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

Confirm it's working:

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes"

Install Prometheus using Helm

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install prometheus prometheus-community/prometheus \
  --namespace monitoring --create-namespace

Verify installation:

kubectl get pods -n monitoring
kubectl port-forward svc/prometheus-server -n monitoring 9090:80

Visit: http://localhost:9090

Configure Prometheus to Scrape Metrics Server

Add a new scrape config

Create a configmap to add custom scrape configs:

# metrics-scrape-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: additional-scrape-configs
  namespace: monitoring
data:
  scrape-configs.yaml: |
    - job_name: 'metrics-server'
      metrics_path: /metrics
      scheme: https
      static_configs:
        - targets: ['metrics-server.kube-system.svc:443']
      tls_config:
        insecure_skip_verify: true

Apply it:

kubectl apply -f metrics-scrape-config.yaml

Now update the Prometheus release to use it:

helm upgrade prometheus prometheus-community/prometheus \
  --namespace monitoring \
  --set server.extraScrapeConfigsSecret.enabled=true \
  --set server.extraScrapeConfigsSecret.name=additional-scrape-configs \
  --set-file server.extraScrapeConfigsSecret.data.scrape-configs.yaml=metrics-scrape-config.yaml

Install Grafana with Helm

helm install grafana grafana/grafana \
  --namespace monitoring \
  --set adminPassword='admin' \
  --set service.type=NodePort \
  --create-namespace

Forward port:

kubectl port-forward svc/grafana -n monitoring 3000:80

Visit: http://localhost:3000 Login with: admin / admin

Configure Grafana

  • Add Prometheus as a data source:

  • URL: http://prometheus-server.monitoring.svc.cluster.local

  • Import dashboard (ID: 6417 for Kubernetes resource metrics)

Done

Now you're visualizing real-time Kubernetes metrics via metrics-server → Prometheus → Grafana.

Cleanup

helm uninstall prometheus -n monitoring
helm uninstall grafana -n monitoring
kubectl delete ns monitoring
kubectl delete -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml