Contents
In brief
Habr published a hands-on blueprint for a minimal MLOps platform in a closed network: no public cloud, private GitLab, image registry, and internal Ingress. The author writes for DevOps engineers—not another Kubeflow tour, but a path from a bare cluster to a reproducible model pipeline.
The stack is familiar: Kubernetes, ArgoCD, MLflow, DVC, MinIO, OpenBao, Prometheus. A dedicated GPU host for training sits outside the cluster. The hard part is not picking trendy tools; it is moving artifacts and secrets inside the perimeter.
What happened
Industry estimates still put a large share of ML work stuck as Jupyter experiments: no disciplined way to version data, compare runs, and ship a model like a service. Data scientists think in experiments; DevOps thinks in services; without a shared process you get pain—and secrets baked into images.
An ML pipeline differs from classic CI/CD. Artifacts are not only code but data and hyperparameters. Git cannot hold multi-gigabyte datasets—the article uses DVC pointers in Git and files in MinIO. Dozens of runs are compared in MLflow (PostgreSQL instead of SQLite). Production models can degrade without a code change because of data drift—the lifecycle is cyclic. A Model Registry with aliases such as @champion enables rollback without editing application code.
The platform splits into two contours: Kubernetes for inference, tracking, and secrets; a separate GPU machine for training via a Celery + Redis queue and a meta-worker that runs Docker images with the NVIDIA runtime. Contours talk over the network through Ingress, so compromising the GPU host is not the same as owning the cluster control plane.
Why it matters
Air-gapped environments break cloud shortcuts. Helm charts are vendored into Git, public images are mirrored into a private registry, and cloud KMS is replaced by SOPS with age keys plus an ArgoCD sidecar. After HashiCorp’s license change, Vault is swapped for OpenBao (compatible API, MPL 2.0) plus External Secrets Operator.
The author skips Kubeflow for a prototype (heavy and cloud-shaped) and avoids Airflow as the only orchestrator (schedules fit poorly with “train now from the notebook”). GitOps uses Apps-of-Apps with sync waves so secrets land before apps.
Security is not an afterthought: Bandit and Trivy in CI, non-root API images, strict SecurityContext, and a Docker Socket Proxy on the GPU host instead of raw /var/run/docker.sock. Auto-Unseal for OpenBao and pickle model risk remain open—safer formats such as ONNX are recommended for production.
In practice
- Version data apart from code (DVC or similar) and keep heavy files in an internal object store.
- Do not run multi-worker MLflow on SQLite—use a real backend store.
- Put training on a dedicated GPU contour with a job queue and least-privilege access to the Docker API.
- In air-gapped setups, mirror images and charts inside the perimeter; encrypt GitOps secrets before commit.
- Monitor more than API latency—watch for drift, or a green code deploy will not save model quality.
The article’s prototype is a starting point, not a scale-invariant design. Growth forces revisiting modularity, secret unseal, and model serialization formats.
Takeaway
MLOps for DevOps means extending familiar practices—versions, CI/CD, secrets, observability—to data and models, which hurts most behind a closed perimeter. A minimal open-source Kubernetes stack can carry work from experiment to inference if you solve artifact logistics, not only “give them a GPU Jupyter box.”