Production ML Inference Server on Kubernetes — Architecture Guide

MLOps Architecture · Production Guide

Production ML Inference Server on Kubernetes

After building and operating inference infrastructure across multiple teams, I wrote this as the guide I wish existed when I started. It covers every decision that matters — from where you store model weights to how you roll out without dropping a single request.

Platform Engineering Kubernetes · GitOps tool · Helm CPU & GPU

01 — Foundation

Model storage strategy

This is the one decision that shapes everything else. Get it wrong and you're fighting your own infrastructure for months. Get it right and deployments, rollbacks, and GitOps all just... work. The question: where do your model weights live?

I've seen three patterns in the wild. Most teams start with the first one, hit the race conditions, and eventually migrate to the third. Save yourself that journey — here's what each one actually looks like in production:

Pattern	How it works	Race condition	GitOps fidelity	Rollback	Ops complexity
Shared PVC	All replicas mount one volume; CI writes new weights in-place	High	Partial	Manual copy	High
Sidecar polling	Sidecar watches object store, downloads new weights, calls inference server model control API	Medium	Broken	Re-pull	High
OCI-baked (recommended)	Model + server config baked into container image; initContainer populates emptyDir at startup	None	Full	Git revert + sync	Low

Key insight With OCI-baked models, your model version is your image tag. This makes your GitOps tool the single source of truth for every aspect of the running system — infrastructure and model weights alike. The version that Git says is deployed is exactly what is running.

Why shared PVC is dangerous

When multiple inference server replicas mount the same PVC and a CI job writes new weights in-place, you create a window where the server reads partially-written files. The result is non-deterministic: the pod may crash, silently serve corrupted predictions, or deadlock during model loading. This is difficult to reproduce and nearly impossible to detect without purpose-built inference correctness monitoring.

Why sidecar polling breaks GitOps

The sidecar pattern has five compounding problems beyond the race condition: version drift across replicas during the polling interval; the GitOps tool losing visibility because the actual model version is set at runtime not by Git; silent failure when the sidecar crashes with no GitOps trail; inability to answer "what model was running at 14:32?" during an incident; and the need to maintain and monitor an additional container per pod.

02 — Architecture

End-to-end architecture

The full pipeline has four logical lanes: data science, CI/registry, GitOps, and the cluster runtime. Each lane has clear ownership and a clean handoff point.

fig 1 — end-to-end flow from model training to serving pod. git commit is the deployment unit.

The initContainer pattern

Each inference pod starts with an initContainer that copies the model repository from the OCI image layer into a shared emptyDir volume. The inference server then starts with --model-repository=/models pointing at that directory. When the pod terminates, the data goes with it — there is no shared state between pods.

init container spec — values.yaml yaml

initContainers:
  - name: model-loader
    image: "{{ .Values.model.image }}:{{ .Values.model.tag }}"
    command: ["sh", "-c"]
    args:
      - "cp -r /model_repo/. /models/"
    volumeMounts:
      - name: model-store
        mountPath: /models

containers:
  - name: inference-server
    image: your-registry/inference-server:24.01
    args: ["inference-server", "--model-repository=/models",
            "--model-control-mode=none"]  # disable hot-reload
    volumeMounts:
      - name: model-store
        mountPath: /models

volumes:
  - name: model-store
    emptyDir: {}  # isolated per pod — no sharing

03 — Trade-offs

The real drawback of OCI-baked models

OCI-baked models win on every operational dimension — except one: pod startup time. When a new pod starts, it must pull the full OCI image before the initContainer can run. For models in the 500MB–2GB range, this can add 30 seconds to 3 minutes of latency before the pod is ready to serve.

This affects three real production scenarios:

Canary rollout speed New pods in the canary tier come up slowly, delaying the point at which you have enough canary traffic to make a statistically valid promotion decision.

HPA scale-out lag Under a traffic spike, the Horizontal Pod Autoscaler triggers new pods but they take minutes to become ready — defeating the purpose of autoscaling for latency-sensitive workloads.

Node failure recovery When a node is lost and pods reschedule, the cold pull time extends the outage window.

Mitigation 1 — Image pre-pull DaemonSet

Before triggering a rollout, force the new image onto every node's container runtime cache with a temporary DaemonSet. By the time the real rollout begins, the image pull is a no-op.

image-prepull-daemonset.yaml yaml

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: model-prepull-v1-2-0
  annotations:
    gitops.io/hook: PreSync
    gitops.io/hook-delete-policy: HookSucceeded
spec:
  selector:
    matchLabels: { app: model-prepull }
  template:
    spec:
      initContainers:
        - name: pull
          image: "your-registry/model-name:v1.2.0"
          command: ["true"]  # just pulls and exits
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9

Mitigation 2 — Hybrid: OCI for metadata, CSI for weights

For very large models (multi-GB or foundation models), use OCI image for server config only. Mount model weights from a cloud file store (e.g. NFS, Azure Files, EFS) via CSI driver — a pre-populated, read-only mount that does not introduce the shared PVC race condition because it is never written to during a deployment.

When to use the hybrid approach Use OCI-baked for models under ~1GB. Switch to the hybrid pattern when image pull time exceeds your HPA readiness budget — typically when model weights are larger than 1.5GB and you need sub-60s pod startup.

04 — Deployment strategies

Deployment strategies for app teams

The platform blueprint should expose three strategies via a single chart value. App teams select the appropriate strategy based on their model's criticality and downtime tolerance.

Strategy A

Rolling update

Kubernetes replaces pods one at a time. Old pods serve traffic until new pods pass readiness. Brief window where two versions run simultaneously.

Downtime: ~0–30s Requires: Helm + GitOps tool only Recommended default

Strategy B

Canary rollout

A progressive delivery controller shifts traffic in steps (10% → 30% → 60% → 100%). Pauses between steps allow metric validation. Auto-rollback on error rate breach.

Downtime: zero Requires: delivery controller + Prometheus Recommended for ML models

Strategy C

Blue-Green

Full parallel deployment. Smoke test the new (green) stack via preview service before flipping the load balancer. Instant rollback by re-pointing traffic.

Downtime: zero Requires: 2× resources + delivery controller Critical / regulated models

GitOps tool + progressive delivery controller integration Configure your GitOps tool so it does not auto-heal a manually paused rollout. This prevents it from reverting a canary mid-flight — a subtle but critical setting. The /spec/paused field on the Rollout resource must be listed in ignored differences so the GitOps tool treats a mid-rollout pause as intentional, not drift.

gitops-application.yaml — critical sync settings yaml

syncPolicy:
  automated:
    prune: true
    selfHeal: false  # never auto-revert a paused canary
  syncOptions:
    - RespectIgnoreDifferences=true
ignoreDifferences:
  - group: delivery.k8s.io
    kind: Rollout
    jsonPointers:
      - /spec/paused  # GitOps tool ignores mid-rollout pause state

05 — Model lifecycle

Model lifecycle management

Each model version passes through five states. The key design principle: only one actor owns each state transition, and the Git commit is the authoritative record of every transition.

Built

Data science team trains the model, exports to a standard format (e.g. ONNX, SavedModel, PMML), and pushes the OCI image to the container registry with a semver + SHA tag (e.g. v1.2.0-abc1234). The image contains the model weights, server config, and any pre/post-processing scripts.

owner: data science CI pipeline

Staged

CI pipeline opens a Pull Request updating values.yaml with the new image tag. The PR is the change review artefact — app team reviews model card, performance benchmarks, and the tag itself before approving.

owner: CI pipeline raises · app team approves

Canary

After merge, the GitOps tool syncs and the progressive rollout begins. Traffic shifts in configured steps. Prometheus metrics are evaluated at each pause — error rate, P99 latency, and optionally business-level accuracy metrics via a custom analysis template. The rollout auto-aborts if thresholds are breached.

owner: delivery controller (automated) · app team (manual gates)

Active

100% of traffic on the new version. SLO dashboards monitor inference server metrics (e.g. request success rate, queue duration, GPU/CPU utilization) exposed via Prometheus. Previous version pods have terminated — their weights are gone from the cluster.

owner: platform (SLO monitoring)

Deprecated

When the next version is promoted to Active, the previous version is marked deprecated. The OCI image is retained in the registry for 30 days minimum to allow emergency rollback via a one-line Git revert. After 30 days, registry garbage collection removes it.

owner: registry retention policy

06 — Chart design

Deployment chart design for app teams

The chart should be a contract between the platform team (who maintains it) and app teams (who configure it). App teams should only need to touch four values to perform a full model update and strategy selection.

values.yaml — app team surface yaml

# ── The four values app teams control ──────────────────

model:
  image: your-registry/pricing-model
  tag:   "v1.2.0-abc1234"   # bumped per release via CI PR

deployment:
  strategy: canary           # rolling | canary | bluegreen

# ── Platform defaults (app teams can override if needed) ─

rollout:
  canary:
    steps:
      - setWeight: 10
      - pause: { duration: 5m }
      - setWeight: 30
      - pause: { duration: 10m }
      - setWeight: 60
      - pause: {}        # manual gate — requires explicit promote
    analysis:
      errorRateThreshold:  0.05
      latencyP99Threshold: 500   # ms

inferenceServer:
  resources:
    requests: { cpu: "2", memory: "4Gi" }
    limits:   { cpu: "4", memory: "8Gi" }
  replicaCount: 2

Multi-model migration path Start with single-model-per-deployment. When you move to multiple models in one inference server instance, the only change is in the OCI image build convention — the model repository uses subdirectories (model_repo/model_name/1/model.onnx) and the server auto-discovers them. No chart or GitOps tool changes are needed. Design the image build convention now so the migration is seamless.

07 — Principles

Key architectural principles

Git is the single source of truth — always

Every model version, every config change, every rollback must be traceable to a Git commit. If a change happens outside Git (sidecar hot-swap, manual kubectl patch, inference server model control API), it creates an invisible state that makes incident investigation nearly impossible and breaks compliance audit trails.

Model version = image tag — immutable

Never mutate a running deployment's model in-place. A new model version must produce a new image tag and trigger a new deployment. This gives you clean rollback (revert the tag), clean canary (old tag vs new tag), and clean audit (tag appears in cluster event logs).

Isolate per-pod state with emptyDir

Use emptyDir volumes for model data, populated at pod start by the initContainer. This eliminates all shared-volume race conditions between replicas and between deployments. When the pod dies, the data dies with it — no cleanup, no lingering state.

Disable the inference server's model control API in production

Run the inference server with model control mode disabled (e.g. --model-control-mode=none). This prevents any runtime mutation of the model repository — accidental or intentional. If model hot-reload is needed for development environments, gate it behind a chart flag and never enable it in production namespaces.

Surface inference metrics to Prometheus before going live

Your inference server must expose Prometheus metrics before canary analysis can function. Scrape at minimum: request success rate, request failure rate, queue duration, and GPU/CPU utilisation. Without these, canary promotion is a manual guess rather than a data-driven decision. Confirm scraping is working in staging before any production rollout.

Architecture summary OCI-baked model images + GitOps-driven deployments + progressive delivery canary analysis is the right architecture for production ML inference on Kubernetes. It trades slightly longer pod startup time for full Git auditability, zero race conditions, clean rollback, and an app team experience reducible to a single chart value change.