Prometheus & Thanos on OpenShift

// 01

Why Do We Monitor?

Before touching any YAML, it's worth understanding what problem monitoring solves. In a production platform like OpenShift, you're running tens or hundreds of microservices simultaneously. Without monitoring you're flying blind — you don't know if a model serving pod is OOMKilling, if inference latency is spiking, or if GPU utilisation is wasted.

🔍

Visibility

See what's happening inside every pod, deployment, and node in real time.

🚨

Alerting

Get notified before users feel the pain — high error rates, memory leaks, pod restarts.

📈

Capacity Planning

Understand resource trends to scale workloads before bottlenecks hit.

🔎

Debugging

Correlate a timeline of metrics, logs and traces during an incident.

// 02

Core Concepts — Plain English

🔥

Prometheus

A time-series database that periodically scrapes (pulls) metrics from your apps and stores them with timestamps.

📦

Metrics

Numerical measurements exposed by your app at an HTTP endpoint (usually /metrics). E.g. request count, memory used.

🏷️

Labels

Key=value pairs attached to every metric. They let you slice data by namespace, pod, version, endpoint, etc.

📡

Scrape

Prometheus calls your app's /metrics endpoint every N seconds and stores the result. This is a "scrape".

📋

ServiceMonitor

A Kubernetes CRD that tells Prometheus which Services to scrape, on which port and path.

🔭

Thanos

An extension on top of Prometheus that adds long-term storage and multi-cluster querying.

The 4 Metric Types

Type	Description	Example
Counter	Always increases. Resets on restart. Measures totals.	`http_requests_total`
Gauge	Can go up or down. Current value snapshot.	`memory_usage_bytes`
Histogram	Tracks value distribution in configurable buckets.	`request_duration_seconds`
Summary	Like histogram but pre-calculates quantiles at the app.	`rpc_duration_seconds{quantile="0.99"}`

ℹ️

Rule of thumb: Use Counter for "how many times did X happen", Gauge for "what is the current value of X", Histogram for "how long did X take".

// 03

Architecture Overview

Here's how all the components connect on OpenShift, from your application all the way up to Thanos and Grafana.

Full monitoring data flow

Your App Pod
/metrics endpoint

exposes metrics

→

ServiceMonitor
CRD (YAML)

defines scrape target

→

Prometheus
scrapes & stores

short-term (15d)

↓ ↓

Thanos Layer

Thanos Sidecar
beside Prometheus

uploads TSDB blocks

→

Object Store
Azure Blob / S3

long-term storage

←

Thanos Store
queries object store

historical data

↑

Thanos Querier
unified query API

deduplicates + merges

↓ ↓

Grafana
dashboards

Alertmanager
routes alerts

💡

Key insight: Prometheus is the workhorse for recent data. Thanos is a long-term extension — it sits alongside Prometheus and takes care of archiving and multi-cluster querying without replacing Prometheus.

// 04

OpenShift's Built-in Monitoring Stack

OpenShift ships with a pre-installed Prometheus-based monitoring stack managed by the Cluster Monitoring Operator (CMO). There are two distinct stacks you need to know about:

🔒

Platform Monitoring

Monitors cluster infrastructure (nodes, etcd, API server). Lives in openshift-monitoring namespace. Read-only — you cannot add your own apps here.

🧑‍💻

User Workload Monitoring

Monitors your application namespaces. Lives in openshift-cma-monitoring. Must be explicitly enabled. This is where your ServiceMonitors go.

Enabling User Workload Monitoring

A cluster-admin must patch the ConfigMap once:

    // kubectl/oc — enable user workload monitoring
    oc -n openshift-monitoring patch configmap cluster-monitoring-config \

      --type merge \

      -p '{"data":{"config.yaml":"enableUserWorkload: true\n"}}'

⚠️

Without this step, ServiceMonitors you create in your namespace will be silently ignored — no error, just no data. Always confirm this is enabled first.

Verifying It's Running

    // check pods in user workload namespace
    oc get pods -n openshift-cma-monitoring

    # Expected output includes:

    prometheus-user-workload-0           Running

    prometheus-operator-xxxxx            Running

    thanos-ruler-user-workload-0         Running

// 05

ServiceMonitor — Telling Prometheus What to Scrape

A ServiceMonitor is the most important YAML you'll write. It's a Kubernetes Custom Resource that instructs the Prometheus Operator which Service to scrape, at which port, and at which path.

Prerequisite: Your App Must Expose `/metrics`

Your application (or sidecar) needs to expose metrics in Prometheus text format at an HTTP endpoint. Most frameworks have libraries for this (e.g. prometheus-client for Python, prom-client for Node.js).

    // servicemonitor.yaml — annotated
    apiVersion: monitoring.coreos.com/v1

    kind: ServiceMonitor

    metadata:

      name: my-triton-server

      namespace: jsvtriton-dev  # same ns as your app

      labels:

        release: prometheus  # must match Prometheus selector

    spec:

      selector:

        matchLabels:

          app: triton-inference-server  # must match your Service labels

      endpoints:

      - port: metrics        # named port on your Service

        path: /metrics       # default, but explicit is better

        interval: 30s       # scrape every 30 seconds

        scheme: http        # or https if TLS

      namespaceSelector:

        matchNames:

        - jsvtriton-dev

The Matching Chain

The ServiceMonitor won't work unless these three things align:

Your Kubernetes Service has a named port

The port field in ServiceMonitor references a name (not a number). Your Service must have name: metrics on the port.
Your Service labels match ServiceMonitor's matchLabels

The selector on the ServiceMonitor finds Services using label matching. If the labels don't match, Prometheus won't discover the target.
Prometheus is configured to watch your namespace

The Prometheus CR must have a serviceMonitorNamespaceSelector that includes your app namespace, or be set to {} to match all.

Companion Service YAML (for reference)

    // service.yaml — note the named port "metrics"
    apiVersion: v1

    kind: Service

    metadata:

      name: triton-inference-server

      labels:

        app: triton-inference-server  # ← ServiceMonitor selects this

    spec:

      selector:

        app: triton-inference-server

      ports:

      - name: metrics    # ← ServiceMonitor port: references this name

        port: 8002

        targetPort: 8002

// 06

PromQL — Querying Your Metrics

PromQL (Prometheus Query Language) is how you ask questions of your metrics data. You use it in Grafana dashboards and alert rules.

The 3 Core Query Patterns

Pattern	When to use	Example
Instant vector	Current value right now	`up{namespace="jsvtriton-dev"}`
Rate over time	Per-second rate of a counter	`rate(http_requests_total[5m])`
Aggregation	Sum/avg across pods or namespaces	`sum by(pod) (rate(...))`

Practical Queries for Triton / OpenShift

    // PromQL examples
    # 1 — Is my target being scraped? (1=up, 0=down)

    up{namespace="jsvtriton-dev"}

    # 2 — Inference request rate per model

    rate(nv_inference_request_success{namespace="jsvtriton-dev"}[5m])

    # 3 — Memory usage of all Triton pods

    sum by(pod) (

      container_memory_working_set_bytes{namespace="jsvtriton-dev"}

    )

    # 4 — P99 inference latency (histogram)

    histogram_quantile(0.99,

      rate(nv_inference_request_duration_us_bucket[5m])

    )

💡

Beginner tip: Always start with up before writing complex queries. If your target isn't being scraped at all, no other query will return data — and you'll waste time debugging PromQL when the real issue is the ServiceMonitor.

Label Filtering Syntax

    // Label matchers
    # Exact match

    metric_name{label="value"}

    # Regex match

    metric_name{pod=~"triton-.*"}

    # Negative match

    metric_name{namespace!="kube-system"}

// 07

Thanos — Why and What

Prometheus alone has two limitations that Thanos solves:

⏱️

Short Retention

Prometheus defaults to ~15 days of local storage. For compliance, capacity planning, or SLA reporting you need months or years of history.

🌐

Single Cluster Only

Each Prometheus only knows its own cluster. If you have multiple OpenShift clusters, you can't query across them from one place.

Thanos Components Demystified

Component	Where it runs	What it does
Sidecar	Beside each Prometheus pod	Exposes Prometheus data via gRPC; uploads 2h TSDB blocks to object store
Store Gateway	Standalone deployment	Reads historical blocks from object store and serves queries
Querier	Standalone deployment	Fan-out queries to Sidecar + Store; deduplicates replica data
Ruler	Optional deployment	Evaluates alerting/recording rules across long-term data
Compactor	Single instance job	Downsamples and compacts old blocks in object store
Receive	Optional	Accepts remote-write from Prometheus (push model alternative)

On OpenShift: What's Already There

OpenShift's user workload monitoring stack already ships a Thanos Ruler (thanos-ruler-user-workload) and the platform Prometheus communicates via Thanos Querier internally. For your custom long-term storage, you'd add a Thanos Sidecar + object store config to your own Prometheus instance.

    // Thanos object store secret (Azure Blob example)
    apiVersion: v1

    kind: Secret

    metadata:

      name: thanos-objstore-config

    stringData:

      objstore.yml: |

        type: AZURE

        config:

          storage_account: "your-storage-account"

          storage_account_key: "your-key"

          container: "thanos-metrics"

ℹ️

In your test stack on Azure-hosted OpenShift, Thanos naturally pairs with Azure Blob Storage for long-term metric retention — keeping everything in the same cloud ecosystem.

// 07B

gRPC Monitoring — How Prometheus Scrapes It

The core rule: Prometheus cannot scrape gRPC directly. Every workable pattern ultimately exposes an HTTP metrics endpoint that Prometheus can poll. The difference between approaches is simply where that HTTP endpoint is created and how it is wired into your deployment.

⚠️

Important: Even in a gRPC-first architecture, Prometheus still expects an HTTP /metrics endpoint. gRPC traffic and metrics scraping are separate concerns.

Option 1 — Embed an HTTP Metrics Server in the App

This is the recommended pattern for custom applications. Your app runs two servers at the same time: the main gRPC server on one port, and a lightweight HTTP metrics server on another. Prometheus scrapes only the metrics port.

    // Python example — gRPC app + separate HTTP metrics endpoint
    import grpc

    from prometheus_client import start_http_server, Counter

    REQUEST_COUNT = Counter('grpc_requests_total', 'Total gRPC requests', ['method'])

    # Start HTTP metrics server on a separate port

    start_http_server(8080)  # ← Prometheus scrapes this

    # Your gRPC server runs normally on 50051

    server = grpc.server(...)

    server.add_insecure_port('[::]:50051')

    server.start()

What this means for the blueprint:

Expose two named ports

One port is for gRPC traffic, and one is dedicated to HTTP metrics.
Point ServiceMonitor only to the metrics port

Your monitoring stack never needs to scrape the gRPC port itself.
Keep the pod simple

No sidecar is needed, so this is usually the cleanest and lowest-overhead option.

    // Service + ServiceMonitor — two-port pattern
    # Service — two named ports

    ports:

    - name: grpc # inference traffic

      port: 50051

    - name: http-metrics # ← Prometheus scrapes this only

      port: 8080

    # ServiceMonitor — only cares about http-metrics

    endpoints:

    - port: http-metrics

      path: /metrics

      interval: 30s

Option 2 — Prometheus Exporter Sidecar

Use this when the application image is third-party, gRPC-only, or cannot be modified. A sidecar container in the same pod acts as a translator: it gathers application stats and exposes them over HTTP for Prometheus.

    // Deployment — sidecar exporter pattern
    containers:

    - name: my-grpc-app

      image: my-grpc-app:latest

      ports:

      - name: grpc

        containerPort: 50051

    - name: metrics-exporter # ← sidecar

      image: prom/statsd-exporter # or a custom exporter

      ports:

      - name: http-metrics

        containerPort: 9102

      args:

      - "--statsd.listen-udp=:9125"

The ServiceMonitor stays the same as Option 1. It still points to an HTTP metrics port; the only difference is that the port now belongs to the exporter sidecar instead of the main app container.

Option 3 — Triton gRPC-Only Exposure

This is a special case. If Triton is started with --allow-http=false, then the metrics endpoint on port 8002 disappears too, because Triton metrics are HTTP-based.

ℹ️

Best practice: do not disable HTTP entirely for Triton. Instead, keep the HTTP metrics port open internally for Prometheus, while exposing gRPC externally through an OpenShift passthrough route and limiting HTTP access with NetworkPolicy.

    // NetworkPolicy — allow internal metrics, keep gRPC externally usable
    apiVersion: networking.k8s.io/v1

    kind: NetworkPolicy

    metadata:

      name: restrict-triton-http

    spec:

      podSelector:

        matchLabels:

          app: triton-inference-server

      ingress:

      - ports:

        - port: 8002 # allow metrics scraping

        from:

        - namespaceSelector:

          matchLabels:

            name: openshift-cma-monitoring

      - ports:

        - port: 8001 # allow gRPC from anywhere

This gives you the best of both worlds: gRPC is the externally reachable protocol, but Prometheus can still scrape Triton metrics internally on 8002.

Blueprint Decision Tree

    // Choosing the right pattern
    App team wants gRPC-only exposure?

    │

    ├── Custom app (own source code)?

    │   └── → Option 1: embed start_http_server() on a separate port

    │

    ├── Third-party image / can't modify code?

    │   └── → Option 2: sidecar exporter container

    │

    └── Triton with --allow-http=false?

        └── → Option 3: keep 8002 open internally,

             lock it down via NetworkPolicy

// 08

Alerting Rules — PrometheusRule

Alerts are defined using the PrometheusRule CRD. Prometheus evaluates these rules on a schedule and fires alerts to Alertmanager when the condition is true for a given duration.

    // prometheusrule.yaml — annotated example
    apiVersion: monitoring.coreos.com/v1

    kind: PrometheusRule

    metadata:

      name: triton-alerts

      namespace: jsvtriton-dev

      labels:

        role: alert-rules  # must match Prometheus ruleSelector

    spec:

      groups:

      - name: triton.rules

        rules:

        # Alert if Triton pod is down for 2+ minutes

        - alert: TritonDown

          expr: up{job="triton-inference-server"} == 0

          for: 2m

          labels:

            severity: critical

          annotations:

            summary: "Triton inference server is unreachable"

            description: "Pod {{ $labels.pod }} has been down for 2m"

        # Alert on high inference error rate

        - alert: TritonHighErrorRate

          expr: |

            rate(nv_inference_request_failure_total[5m])

              / rate(nv_inference_request_success_total[5m]) > 0.05

          for: 5m

          labels:

            severity: warning

// 09

Beginner Checklist — Where to Start

Follow this order when setting up monitoring for the first time on OpenShift:

Confirm User Workload Monitoring is enabled

Run oc get pods -n openshift-cma-monitoring. If empty, ask your cluster-admin to enable it.
Verify your app exposes /metrics

Port-forward to your pod and curl localhost:PORT/metrics. You should see text like # HELP and # TYPE lines.
Create a Service with a named port

Make sure the port has name: metrics — this is what your ServiceMonitor will reference.
Deploy the ServiceMonitor YAML

Apply it to the same namespace as your app. Double-check labels match your Service.
Check Prometheus Targets UI

In OCP console → Observe → Targets. Your app should appear with state UP (green). If it's missing, the ServiceMonitor selector doesn't match.
Write your first PromQL query

Start with up{namespace="your-ns"} in the Metrics Explorer, then explore with rate() and sum by().
Build a Grafana dashboard

Use the Thanos Querier endpoint as the Grafana data source — this gives you both recent Prometheus data and long-term Thanos data in one pane.
Add a PrometheusRule for alerting

Start simple: alert when up == 0 for your service. Build up to latency and error-rate rules once you understand your baseline.

// 10

Quick Glossary

CMO

Cluster Monitoring Operator — manages the full Prometheus stack lifecycle on OpenShift

CRD

Custom Resource Definition — extends Kubernetes API with new resource types like ServiceMonitor

TSDB

Time Series Database — Prometheus's internal storage format, also what Thanos archives to object store

Sidecar

Container running alongside Prometheus in the same pod; used by Thanos to bridge Prometheus and object storage

Scrape interval

How often Prometheus calls your /metrics endpoint. Typically 15s–60s.

Recording rule

Pre-computed PromQL query result stored as a new metric — speeds up expensive dashboard queries

Deduplication

Thanos Querier removes duplicate data when multiple Prometheus replicas scrape the same targets

remote_write

Prometheus feature to push metrics to an external system (e.g. Thanos Receive, Cortex)

Why Do We Monitor?

Visibility

Alerting

Capacity Planning

Debugging

Core Concepts — Plain English

Prometheus

Metrics

Labels

Scrape

ServiceMonitor

Thanos

The 4 Metric Types

Architecture Overview

OpenShift's Built-in Monitoring Stack

Platform Monitoring

User Workload Monitoring

Enabling User Workload Monitoring

Verifying It's Running

ServiceMonitor — Telling Prometheus What to Scrape

Prerequisite: Your App Must Expose /metrics

The Matching Chain

Your Kubernetes Service has a named port

Your Service labels match ServiceMonitor's matchLabels

Prometheus is configured to watch your namespace

Companion Service YAML (for reference)

PromQL — Querying Your Metrics

The 3 Core Query Patterns

Practical Queries for Triton / OpenShift

Label Filtering Syntax

Thanos — Why and What

Short Retention

Single Cluster Only

Thanos Components Demystified

On OpenShift: What's Already There

gRPC Monitoring — How Prometheus Scrapes It

Option 1 — Embed an HTTP Metrics Server in the App

Expose two named ports

Point ServiceMonitor only to the metrics port

Keep the pod simple

Option 2 — Prometheus Exporter Sidecar

Option 3 — Triton gRPC-Only Exposure

Blueprint Decision Tree

Alerting Rules — PrometheusRule

Beginner Checklist — Where to Start

Confirm User Workload Monitoring is enabled

Verify your app exposes /metrics

Create a Service with a named port

Deploy the ServiceMonitor YAML

Check Prometheus Targets UI

Write your first PromQL query

Build a Grafana dashboard

Add a PrometheusRule for alerting

Quick Glossary

Prerequisite: Your App Must Expose `/metrics`

Your Service labels match ServiceMonitor's `matchLabels`

Verify your app exposes `/metrics`