Quick Navigation
- The implementation gap
- MLOps — the operational backbone
- Model serving — making models accessible
- RAG architecture — grounding AI in your data
- Infrastructure design
- Monitoring — watching for silent degradation
- Scaling
- The production pipeline in one diagram
- Cheat sheet
The implementation gap
There's a moment I've seen in almost every AI team.
They've built something impressive. A model in a Jupyter notebook that achieves 95% accuracy on a test set. The demo works. Leadership is excited. And then comes the question nobody has a good answer for: how do we put this in production?
The gap between "it works on my laptop" and "it works at scale, reliably, with monitoring and rollback" is where most AI projects fail technically. Not because the model is bad. Because the skills that make a good prototype are different from the skills that make a production system.
A prototype proves the model works. Implementation proves the system works. And implementation is a different discipline entirely.
flowchart TD
GAP["The Implementation Gap"]
GAP --> G1["Data\nSample → Production pipelines"]
GAP --> G2["Code\nNotebook → Tested, versioned, modular"]
GAP --> G3["Deployment\nLocal → Scalable API"]
GAP --> G4["Monitoring\nNone → Comprehensive observability"]
GAP --> G5["Lifecycle\nBuild once → Continuous retraining"]
| Dimension | Prototype | Production |
|---|---|---|
| Data | Sample or synthetic | Real pipelines, governed, monitored |
| Code | Notebook, exploratory | Tested, versioned, modular, reviewed |
| Deployment | Local machine | Scalable API with versioning |
| Monitoring | Manual checks | Automated alerting on drift and performance |
| Lifecycle | Build once | Continuous retraining and improvement |
The challenges multiply in production. Data drift — the real world changes, the training data doesn't. Scale — hundreds of examples become millions of requests. Latency — response time doesn't matter in a notebook; it's everything in production. Reliability — restart if it crashes becomes 99.9% uptime required. Governance — ignored becomes audit trail, versioning, compliance.
This guide walks through each piece of the production puzzle.
Try it yourself — The implementation gap audit
For your AI project (or planned one), rate each dimension. Your largest gap is your biggest implementation risk.
| Dimension | Current state | Target state | Gap size |
|---|---|---|---|
| Data | Sample / Production pipeline | Small / Medium / Large | |
| Code | Notebook / Tested, versioned | ||
| Deployment | Local / Scalable API | ||
| Monitoring | Manual / Automated | ||
| Lifecycle | Build once / Continuous |
MLOps — the operational backbone
Traditional software has DevOps — the practices that bridge development and operations. AI needs MLOps. The equivalent practices for machine learning systems, extended to address the unique challenges of ML: data versioning, model versioning, experiment tracking, and automated retraining.
flowchart TD
MLOPS["MLOps"]
MLOPS --> D1["Data Management\nVersioning, quality, pipelines"]
MLOPS --> D2["Experiment Tracking\nReproduce results, compare models"]
MLOPS --> D3["Model Registry\nVersion, stage, deploy models"]
MLOPS --> D4["CI/CD for ML\nAutomated testing and deployment"]
MLOPS --> D5["Monitoring\nDrift detection, performance tracking"]
MLOps maturity tends to follow a predictable curve:
| Level | Description | Practices |
|---|---|---|
| Level 0 | Manual | Notebooks, manual deployment, no monitoring |
| Level 1 | Pipeline automation | Automated training, manual deployment |
| Level 2 | CI/CD pipeline | Automated training and deployment |
| Level 3 | Full MLOps | Automated retraining, monitoring, rollback |
Most organisations start at Level 0. The goal isn't to jump to Level 3 — it's to move one level at a time, with each level producing measurable operational improvement.
The tool landscape is large — MLflow, Weights & Biases, Neptune for experiment tracking. Feast, Tecton for feature stores. Kubeflow, Airflow, Prefect for orchestration. Evidently, Whylabs, Arize for monitoring. The right choice depends on your stack, your team, and your maturity. The wrong choice is no choice at all — picking nothing and staying manual.
Model serving — making models accessible
A trained model is useless if applications can't access it. Model serving is the infrastructure that makes models available to the systems and people that need them.
flowchart LR
APP["Application"] --> API["API Endpoint\n/model/predict"]
API --> ROUTE["Version Router\nWhich model version?"]
ROUTE --> MODEL["Model\nLoaded and ready"]
MODEL --> RESULT["Prediction\nReturned to application"]
| Pattern | How it works | Best for |
|---|---|---|
| Real-time API | Synchronous request-response | Low-latency predictions |
| Batch | Process large volumes on schedule | Reports, bulk processing |
| Streaming | Process data as it arrives | Real-time analytics, anomaly detection |
| Edge | Model runs on device | Offline, low-latency, privacy |
The serving decisions matter: latency requirements determine infrastructure choice. Throughput requirements determine scaling strategy. Versioning enables safe rollouts and rollbacks. And cost — GPU time is expensive — determines optimisation priority.
Try it yourself — Your serving requirements
For your AI model. If you can't answer these, you can't choose the right serving pattern.
| Requirement | Your answer | Implication |
|---|---|---|
| Latency — how fast must response be? | Determines infrastructure | |
| Throughput — requests per second? | Determines scaling | |
| Versioning — need to rollback? | Determines deployment pattern | |
| Budget — what can you spend on GPU? | Determines optimisation need |
RAG architecture — grounding AI in your data
LLMs have a knowledge cutoff and they hallucinate. RAG — Retrieval-Augmented Generation — solves both by grounding AI responses in your actual data.
flowchart LR
DOCS["Your Data\nDocuments · Wikis · Databases"] --> CHUNK["Chunk\nSplit into segments"]
CHUNK --> EMBED["Embed\nConvert to vectors"]
EMBED --> STORE["Vector DB\nPinecone · Weaviate · pgvector"]
QUERY["User Query"] --> EMBED_Q["Embed Query"]
EMBED_Q --> SEARCH["Similarity Search"]
STORE --> SEARCH
SEARCH --> RETRIEVE["Retrieve Top-K Chunks"]
RETRIEVE --> PROMPT["Inject into LLM Context"]
PROMPT --> GENERATE["Generate Grounded Response"]
The pipeline: your documents get split into chunks, converted to vectors, and stored in a vector database. When a user asks a question, the query gets embedded the same way, the vector database finds the most similar chunks, and those chunks get injected into the LLM's context before it generates a response. The model answers from your data, not from its training memory.
| Component | Purpose | Options |
|---|---|---|
| Embedding model | Converts text to vectors | OpenAI Ada, Cohere, Sentence Transformers |
| Vector database | Stores and searches embeddings | Pinecone, Weaviate, Chroma, pgvector, Qdrant |
| Chunking strategy | How documents are split | Fixed-size, semantic, recursive, document-aware |
| Retriever | Finds relevant chunks | Similarity search, hybrid search, reranking |
RAG vs fine-tuning is a common question. RAG adds knowledge at inference time — always current, can cite sources, lower cost. Fine-tuning embeds knowledge in model weights — frozen at training time, can't explain where knowledge came from, higher cost. RAG is better for knowledge Q&A and document search. Fine-tuning is better for style, format, and domain-specific behaviour. They're not alternatives. They're complementary.
Advanced RAG patterns worth knowing: hybrid search combines semantic and keyword search for when exact terms matter. Reranking reorders results by relevance. Multi-hop RAG retrieves, reasons, retrieves again for complex questions. Agentic RAG lets an agent decide when and what to retrieve. Graph RAG uses knowledge graphs for structured retrieval of complex relationships.
Try it yourself — Your RAG vs fine-tuning decision
For your use case. Count each column. The winner is your primary approach. The other may complement it.
| Factor | Points to RAG | Points to fine-tuning |
|---|---|---|
| Does knowledge change frequently? | ||
| Do you need to cite sources? | ||
| Is cost a major constraint? | ||
| Do you need specific style/format? | ||
| Do you have labelled training data? |
Infrastructure design
AI workloads have different infrastructure requirements than traditional applications. They need GPU compute, high-bandwidth networking, large storage, and specialised software stacks.
flowchart TD
INFRA["AI Infrastructure"]
INFRA --> COMP["Compute\nGPUs · TPUs · CPUs"]
INFRA --> STORE["Storage\nData · Models · Artifacts"]
INFRA --> NET["Networking\nBandwidth · Latency"]
INFRA --> SW["Software\nFrameworks · Libraries · Runtimes"]
| Choice | Options | Trade-offs |
|---|---|---|
| Cloud vs on-premise | AWS, GCP, Azure vs own hardware | Flexibility vs control, OpEx vs CapEx |
| GPU type | NVIDIA A100, H100, T4 | Performance vs cost |
| Managed vs self-managed | SageMaker, Vertex AI vs Kubernetes | Ease vs flexibility |
| Multi-cloud | Single vs multiple providers | Resilience vs complexity |
GPU costs are often the largest line item in an AI budget. Cost optimisation isn't optional — it's a survival skill. Cloud vs on-premise decisions affect data governance and compliance. Managed services reduce operational overhead but reduce flexibility. The right choice depends on your team's skills, your compliance requirements, and your scale.
Try it yourself — Your infrastructure decision
Document your reasoning. You'll need it when requirements change.
| Decision | Option A | Option B | Your choice | Why? |
|---|---|---|---|---|
| Cloud vs on-premise | Flexibility | Control | ||
| GPU type | Performance | Cost | ||
| Managed vs self-managed | Ease | Flexibility | ||
| Multi-cloud | Resilience | Simplicity |
Monitoring — watching for silent degradation
AI systems degrade silently. Unlike traditional software that either works or crashes, AI systems can produce increasingly wrong outputs without any errors being thrown.
flowchart TD
MON["AI Monitoring"]
MON --> M1["Data Drift\nInput data changed\nfrom training distribution"]
MON --> M2["Model Drift\nModel accuracy\ndecreasing over time"]
MON --> M3["Performance\nLatency · Throughput\nErrors · Uptime"]
MON --> M4["Cost\nCompute spend\nGPU utilisation"]
| Layer | What to monitor | Alert when |
|---|---|---|
| Data | Input distribution, quality, volume | Distribution shifts significantly |
| Model | Accuracy, precision, recall, confidence | Metrics drop below threshold |
| System | Latency, throughput, errors, uptime | SLA breach or errors spike |
| Cost | Compute spend, GPU utilisation | Spend exceeds budget |
Drift detection is the core challenge. Statistical tests compare current input distributions against training distributions. Performance metrics track prediction quality against ground truth when it becomes available. Feature importance monitoring catches when the relationships the model learned start to break down. Prediction analysis flags unusual output patterns.
When drift is detected, it should trigger retraining — which connects monitoring back to the MLOps lifecycle. This is the feedback loop that keeps production AI systems healthy.
Try it yourself — Your monitoring plan
For your AI system. If any row is blank, you have a blind spot.
| Layer | What to monitor | Alert threshold | Who responds? |
|---|---|---|---|
| Data | |||
| Model | |||
| System | |||
| Cost |
Scaling
AI systems that work for 100 users may collapse at 10,000. Scaling is not just about adding more hardware — it requires architectural decisions that enable growth.
flowchart TD
SCALE["Scaling AI"]
SCALE --> S1["User Scaling\nMore requests\nHigher throughput"]
SCALE --> S2["Data Scaling\nLarger datasets\nMore features"]
SCALE --> S3["Model Scaling\nLarger models\nMore models"]
| Strategy | What it does | When to use |
|---|---|---|
| Horizontal scaling | Add more instances | Stateless workloads |
| Vertical scaling | Bigger instances | Single-model performance needs |
| Caching | Store frequent results | Repeated predictions |
| Batching | Process multiple requests together | Throughput optimisation |
| Model optimisation | Reduce model size | Latency and cost constraints |
The scaling questions: can each request be handled independently (statelessness)? Is data close to compute (data locality)? Can the model fit on available hardware? How long to spin up new instances (cold start)?
Try it yourself — Your scaling plan
For your AI system. If you don't have a scaling action defined, you'll hit a wall when growth comes.
| Scaling dimension | Current capacity | Growth trigger | Scaling action |
|---|---|---|---|
| Users | |||
| Data | |||
| Models |
The production pipeline in one diagram
flowchart TD
PROTO["Prototype\nModel works"] --> MLOPS["MLOps\nVersioning · Automation"]
MLOPS --> SERVE["Serving\nAPIs · Deployment"]
SERVE --> INFRA["Infrastructure\nCompute · Storage"]
INFRA --> MON["Monitoring\nDrift · Performance"]
MON --> SCALE["Scaling\nUsers · Data · Models"]
SCALE --> PROD["Production\nReliable, scalable, monitored"]
The essentials:
The prototype-to-production gap is where most AI projects fail technically. Bridging it requires engineering rigour across data, code, deployment, monitoring, and lifecycle — the model working in a notebook is a research result, the model working in production is a system. Everything else follows from that distinction.
Cheat sheet — all the key terms
| Term | Plain English | Where it fits |
|---|---|---|
| Implementation gap | Data, code, deployment, monitoring, lifecycle — each needs engineering rigour | Prototype to production |
| MLOps | DevOps for machine learning | Operational backbone |
| Experiment tracking | Reproduce results, compare models | MLOps component |
| Model registry | Version, stage, deploy models | MLOps component |
| Model serving | Making models accessible through APIs | Deployment layer |
| Real-time API | Synchronous request-response | Serving pattern |
| Batch | Process large volumes on schedule | Serving pattern |
| RAG | Retrieve relevant data, inject into context, generate grounded response | Grounding AI in your data |
| Vector database | Stores embeddings for similarity search | RAG infrastructure |
| Chunking strategy | How documents are split for retrieval | RAG quality |
| Infrastructure | Compute, storage, networking, software for ML workloads | Foundation layer |
| Data drift | Input data changed from training distribution | Monitoring layer |
| Model drift | Model accuracy decreasing over time | Monitoring layer |
| Horizontal scaling | Add more instances | Scaling strategy |
| Model optimisation | Reduce model size for latency and cost | Scaling strategy |
How to know if this landed
You'll know this has landed when someone designs the production architecture before they build the prototype — not after. When their models are versioned in a registry and every production model can be traced to its training data and code. When deployment is automated and manual deployment is the exception, not the rule. When they have monitoring in place for data drift, model performance, and cost — with alerts that trigger before users notice degradation. When they can explain why RAG is better than fine-tuning for their use case — and when they'd use both. When they've tested their rollback path and can revert to a previous model version within minutes. And when they know their GPU costs and have a plan for optimising them.
What teams discover in the workshop
The moment that shifts most minds in this workshop is the deployment timeline exercise.
I ask teams to estimate how long it takes to move a model from notebook to production. The answers are consistently measured in weeks. Then I ask how long it should take with a proper MLOps pipeline. The answers drop to hours. The gap between those two numbers — weeks vs hours — is the value of the operational backbone.
The monitoring conversation produces the most immediate behaviour change. I show what data drift looks like — the same model, the same code, silently producing worse outputs because the world changed and the training data didn't. The room goes quiet. People who've been treating monitoring as a "nice to have" start asking what tools they need to implement it by Monday.
The RAG architecture section tends to resolve a conversation that's been happening for months in most organisations: should we fine-tune or should we use RAG? After walking through the trade-offs — data freshness, cost, transparency, use case fit — the answer is usually clear. And when it's both, the conversation shifts to sequencing: RAG first for knowledge grounding, fine-tuning later for behaviour shaping.
The infrastructure planning exercise surfaces the GPU cost conversation. Teams that have been running models on whatever GPU was available suddenly see the spend numbers and start asking about model optimisation, autoscaling, and caching. That's a far better problem to have than the alternative — nobody asking because nobody is tracking.
Book a Workshop
Ready to turn your AI prototypes into production systems that deliver value reliably?
or
2-day workshop + coaching includes MLOps architecture design for your organisation, model serving pattern selection for your use cases, infrastructure planning for cloud, on-premise, or hybrid, monitoring framework with drift detection and alerting, scaling strategy for your expected growth, deployment playbook with rollback procedures, and MLOps tool evaluation and selection.