Quick Navigation
- The data problem AI exposes
- Data quality — the six dimensions
- Data governance — who owns and protects it
- Data architecture for AI
- Feature engineering
- Data pipelines
- Data for Generative AI — what's different
- Common failure patterns
- The data pipeline in one diagram
- Cheat sheet
The data problem AI exposes
There's a saying in the AI world: garbage in, garbage out. It's true, but it doesn't go far enough.
The quality of your AI is capped by the quality of your data. No algorithm can compensate for bad data. And most organisational data is not AI-ready.
Organisations often invest millions in AI models and platforms while neglecting the data that feeds them. They build sophisticated systems on top of messy, siloed, ungoverned data — and wonder why the results are unreliable.
The hard truth: most of the work in a successful AI project is data work. Data discovery, data cleaning, data integration, data governance. The model itself is often the easy part.
flowchart TD
DATA["Data Foundation"] --> QUALITY["Data Quality\nHow good is it?"]
DATA --> GOVERN["Data Governance\nWho controls it?"]
DATA --> ARCH["Data Architecture\nHow is it organised?"]
QUALITY --> AI["AI System"]
GOVERN --> AI
ARCH --> AI
AI --> VALUE["Business Value"]
| Challenge | What it means for AI |
|---|---|
| Data is siloed | AI can't see the full picture |
| Data is inconsistent | Same entity, different records across systems |
| Data is incomplete | Missing values lead to biased or unreliable models |
| Data is unlabelled | Supervised learning requires labelled examples |
| Data is not governed | Privacy, compliance, and access issues block deployment |
Most organisations don't have an AI problem — they have a data problem that AI exposes.
Try it yourself — The data reality check
Think of the data your AI project needs. Answer honestly. How many "No" answers? Each one is a blocker to fix before building.
| Question | Yes / No / Unsure | What this tells you |
|---|---|---|
| Is the data accessible without manual extraction? | If no: availability blocker | |
| Is the data quality monitored? | If no: quality blocker | |
| Do you have enough data for production? | If no: volume blocker | |
| Is the data current enough for your use case? | If no: timeliness blocker | |
| Has legal approved this data for AI use? | If no: governance blocker |
Data quality — the six dimensions
"Bad data" is too vague. Data quality has specific, measurable dimensions — and failure in any one can undermine your AI.
| Dimension | Question | Example issue |
|---|---|---|
| Accuracy | Does the data correctly represent what it describes? | Customer address is wrong |
| Completeness | Are all required fields populated? | Missing dates, null values in key fields |
| Consistency | Same data, same value across all systems? | Customer name spelled differently in two systems |
| Timeliness | Is the data current enough for its purpose? | Sales data updated weekly but AI needs daily |
| Validity | Does it conform to rules and formats? | Phone number in email field |
| Uniqueness | Are there duplicate records? | Same customer entered twice |
The data readiness assessment that works best covers eight dimensions: availability, quality, volume, timeliness, labelling, governance, unstructured readiness (for GenAI), and privacy clearance. Score each 1–5 for the specific data domain your AI project will use. Any dimension scoring 1 or 2 is a blocker — fix it before you build.
Try it yourself — Your data readiness scorecard
Rate your data 1-5 on each dimension. Your lowest score is your biggest data risk. That's where to invest first.
| Dimension | Score (1-5) | Evidence | Action if below 3 |
|---|---|---|---|
| Availability | |||
| Quality | |||
| Volume | |||
| Timeliness | |||
| Labelling | |||
| Governance | |||
| Unstructured readiness | |||
| Privacy clearance |
Data governance — who owns and protects it
Data without governance is a liability. Data with governance is an asset. The difference is who controls it, who can access it, and how it's protected.
| Component | Key question | AI implication |
|---|---|---|
| Data ownership | Who is accountable for this dataset? | Someone must approve AI use of each dataset |
| Data quality | Who monitors and improves quality? | AI models need quality guarantees |
| Data access | Who can access what, under what conditions? | AI teams need clear access policies |
| Data privacy | How is personal/sensitive data protected? | AI training on personal data has legal implications |
| Regulatory compliance | What laws apply? | GDPR, DPDP, industry-specific regulations |
The governance roles that matter: Data Owner (accountable for a data domain — approves access, defines quality standards), Data Steward (manages day-to-day quality and metadata), Data Engineer (builds and maintains infrastructure and pipelines), Data Consumer (uses data for analysis or AI — must follow governance policies).
Data governance is the data-specific expression of the governance principles covered in the Responsible AI training. Data access policies determine what AI teams can build — overly restrictive means no AI, overly permissive means risk.
Try it yourself — Your governance gap
For your AI project's data needs. Any blank answer is a governance gap. Fill it before building.
| Question | Answer | Risk if unclear |
|---|---|---|
| Who owns each dataset you need? | ||
| Who can approve access? | ||
| What data classification applies? | ||
| Has privacy reviewed this use? |
Data architecture for AI
Traditional data architectures were built for reporting and analytics. AI workloads have different requirements — larger volumes, real-time processing, and different access patterns.
| Component | Purpose | Why AI needs it |
|---|---|---|
| Data Lake | Store raw data at scale | AI needs access to unstructured data (text, images, logs) |
| Feature Store | Pre-computed, reusable features | Consistency between training and production |
| Streaming Platform | Real-time data processing | AI applications often need live data |
| Data Catalogue | Discoverable data assets | AI teams need to find the right data quickly |
| ML Data Pipeline | Automated data preparation | Repeatable, versioned data transformations |
Architecture patterns to know: Data Warehouse for structured, curated, SQL-based BI. Data Lake for raw, unstructured, file-based AI/ML exploration. Data Lakehouse combining both. Data Mesh for domain-owned, decentralised data in large organisations.
Architecture decisions cascade into AI Implementation & Technical Architecture. Feature stores bridge data engineering and ML engineering — a critical integration point.
Try it yourself — Your architecture gap
For your AI project. Each "no" is an architecture gap. The workaround tells you how much manual effort you're carrying.
| Component | Do you have it? | If not, what's the workaround? |
|---|---|---|
| Data Lake for raw, unstructured data | ||
| Feature Store for reusable features | ||
| Streaming Platform for real-time data | ||
| Data Catalogue for discoverability | ||
| ML Data Pipeline for automated prep |
Feature engineering
Raw data is not what AI models consume. Features — the meaningful, transformed inputs derived from raw data — are what models actually learn from.
| Technique | What it does | Example |
|---|---|---|
| Aggregation | Summarises data over time or groups | Average spend per customer per month |
| Transformation | Converts data into useful formats | Date → day of week, hour of day |
| Encoding | Converts categories to numbers | Country name → one-hot encoding |
| Derivation | Creates new features from existing ones | BMI from height and weight |
| Selection | Removes irrelevant or redundant features | Dropping columns that add noise |
Feature engineering quality directly determines model performance — more than algorithm choice. Feature stores enable consistency between training and serving, which prevents training-serving skew. Domain expertise is critical — the best features come from understanding the business problem, not from the data itself.
Try it yourself — Your feature inventory
For your AI project, what features would you need? If you can't list at least 5 features, you haven't thought deeply enough about what your model actually needs.
| Feature | Source data | How you'd create it | Who owns it? |
|---|---|---|---|
Data pipelines
Data doesn't magically appear where AI needs it. It must flow — from source systems, through transformations, to the AI model, and ultimately to the decision or action.
| Pipeline type | When | Use case |
|---|---|---|
| Batch | Scheduled (hourly, daily) | Model training, reporting, historical analysis |
| Streaming | Real-time | Live predictions, anomaly detection, recommendations |
| Hybrid | Both | Training on batch, serving on streaming |
| Requirement | Why it matters |
|---|---|
| Reliability | Broken pipeline = no data = no AI |
| Monitoring | You need to know when data is late or wrong |
| Versioning | Reproducibility — know exactly what data trained each model |
| Scalability | Data volumes grow — pipelines must handle it |
Pipeline reliability directly affects AI reliability — stale data produces stale predictions. Pipeline monitoring is part of MLOps. Streaming pipelines enable the real-time AI applications discussed in the AI Agents training.
Try it yourself — Your pipeline map
Map the data flow for your AI project. If "what happens if it breaks" is "nobody notices until the AI gives wrong answers," that's your biggest pipeline risk.
| Stage | Current process | Frequency | Who monitors? | What happens if it breaks? |
|---|---|---|---|---|
| Extract | ||||
| Transform | ||||
| Load | ||||
| Serve to AI |
Data for Generative AI — what's different
Everything covered so far — data quality, feature engineering, pipelines — assumes traditional AI: structured data, labelled examples, tabular features. Generative AI works differently. It ingests unstructured data (documents, text, code), uses embeddings instead of features, and connects to organisational knowledge through retrieval — not training.
If you apply traditional data thinking to GenAI projects, you will prepare the wrong data, build the wrong architecture, and miss the patterns that actually make LLM applications work.
| Dimension | Traditional ML | Generative AI |
|---|---|---|
| Data type | Structured (tables, logs) | Unstructured (text, documents, images) |
| Data preparation | Clean, label, feature-engineer | Chunk, embed, index |
| How data reaches the model | Training (model learns from data) | Retrieval (data injected at query time) |
| When your data is used | Before deployment (training) | During use (every query) |
With traditional ML, you train once and deploy. With GenAI, your data is used on every interaction — which means data freshness, access control, and retrieval quality matter continuously, not just at build time.
Most organisational knowledge is unstructured: documents, emails, code repositories, meeting transcripts, web content, images. Traditional AI could not use this data effectively. LLMs can — but only with the right architecture.
Embeddings are numerical representations of meaning. When text is embedded, it becomes a vector — a list of numbers that captures semantic similarity. Similar meanings produce nearby vectors. This is what makes semantic search possible — searching by meaning, not keywords.
Vector databases store embeddings and retrieve the most similar ones when queried. This is the infrastructure layer that powers RAG. Options include Pinecone (managed SaaS, fastest to production), Weaviate (open source, good for hybrid search), pgvector (PostgreSQL extension, keep it simple), Chroma (lightweight, good for prototyping), and Qdrant (high-performance, Rust-based).
RAG — Retrieval-Augmented Generation — is the primary pattern for connecting organisational data to LLMs. Your documents get chunked, embedded, and stored. When a user asks a question, the query gets embedded, the vector database finds the most similar chunks, and those chunks get injected into the LLM's context before it generates a response. RAG solves the two biggest GenAI problems simultaneously: knowledge cutoff (the model answers from your documents, not its training memory) and hallucination (grounded in real retrieved content, not pure generation).
RAG vs fine-tuning: RAG injects your data at query time — always current, lower cost, medium complexity. Fine-tuning teaches the model your style or domain — snapshot at training time, higher cost, higher complexity. Rule of thumb: start with RAG. It handles 80%+ of enterprise GenAI use cases. Fine-tune only when you need the model to consistently produce a specific style or format that prompting cannot achieve.
Common failure patterns
Most AI project post-mortems point to the same root causes — and they are almost always data problems, not model problems. Knowing the patterns in advance lets you prevent them.
| # | Pattern | What happens | Preventable with |
|---|---|---|---|
| 1 | Data leakage | Training data contains information that won't exist at prediction time — model looks great in testing, fails in production | Proper train/test splitting; temporal separation |
| 2 | Label bias | Human annotators label data inconsistently or with systematic bias — model learns the bias as signal | Label guidelines, inter-rater agreement, diverse annotator pools |
| 3 | Stale data | Model trained on historical data that no longer reflects current reality — predictions drift silently | Data freshness monitoring; retraining triggers |
| 4 | Governance block | AI team builds a working prototype, then privacy or compliance blocks deployment because data was never assessed for legal use | Data governance assessment before building, not after |
| 5 | RAG retrieval failure | GenAI system retrieves irrelevant or outdated chunks — answers look confident but are wrong | Chunking strategy tuning; retrieval evaluation; freshness monitoring |
These are not edge cases. They are the most common reasons AI projects fail to reach production or fail after deployment. Every one of them is a data problem, not a model problem.
The data pipeline in one diagram
flowchart TD
RAW["Raw Data\nFrom source systems"] --> ASSESS["Data Readiness Assessment\nScore 8 dimensions 1–5\nFix blockers before building"]
ASSESS --> QUAL["Data Quality\nIs it good enough?"]
QUAL --> GOV["Data Governance\nIs it protected and controlled?"]
GOV --> BRANCH{"Traditional ML\nor GenAI?"}
BRANCH -->|"Traditional ML"| ARCH["Data Architecture\nData lake, feature store"]
ARCH --> FE["Feature Engineering\nPrepare model inputs"]
FE --> PIPE["Data Pipelines\nSource → Model → Decision"]
BRANCH -->|"GenAI / LLM"| UNSTRUCT["Unstructured Data\nDocuments, text, code"]
UNSTRUCT --> CHUNK["Chunk & Embed\nSemantic meaning → vectors"]
CHUNK --> VECTOR["Vector Database\nIndex for retrieval"]
VECTOR --> RAG["RAG\nRetrieve → Inject → Generate"]
PIPE --> FAILCHECK["Failure Patterns\nLeakage · Bias · Stale · Block · Bad retrieval"]
RAG --> FAILCHECK
FAILCHECK --> AI["Reliable AI\nBuilt on solid data foundations"]
The data discipline:
Data is not the fuel for AI — it is the foundation. Get it wrong and nothing else matters. Most AI failures are data failures, and the organisations that invest in data quality before investing in models are the ones that get results.
Cheat sheet — all the key terms
| Term | Plain English | Where it fits |
|---|---|---|
| Data Readiness | Eight dimensions, score 1–5 — any 1 or 2 is a blocker | Before every AI project |
| Data Quality | Six dimensions — accuracy, completeness, consistency, timeliness, validity, uniqueness | Foundation layer |
| Data Governance | Rules that turn data from liability to asset | Ownership, access, privacy |
| Data Architecture | Scale, speed, variety, discoverability | How data is organised for AI |
| Feature Engineering | Bridge between raw data and model performance | Traditional ML preparation |
| Data Pipelines | Plumbing that makes data flow reliably | Source → Model → Decision |
| Unstructured Data | Text, documents, code — the dominant GenAI data type | What LLMs consume |
| Embeddings | Semantic meaning converted to vectors | How LLMs "understand" text |
| Vector Database | Stores embeddings for semantic search | RAG infrastructure |
| RAG | Retrieve your data, inject into context, generate grounded answers | Connecting documents to LLMs |
| Fine-tuning vs RAG | RAG for knowledge, fine-tuning for style/domain | Approach selection |
| Data leakage | Training data contains future information | Failure pattern 1 |
| Label bias | Human annotators introduce systematic bias | Failure pattern 2 |
| Stale data | Model trained on data that no longer reflects reality | Failure pattern 3 |
| Governance block | Privacy/compliance blocks deployment after prototype | Failure pattern 4 |
| RAG retrieval failure | Irrelevant or outdated chunks produce wrong answers | Failure pattern 5 |
How to know if this landed
You'll know this has landed when someone completes a data readiness assessment before every AI project — not after the prototype. When data quality metrics exist for all critical data domains and are monitored. When a data governance framework is documented with owners, stewards, and access policies. When AI teams can discover available data assets through a catalogue. When data pipelines are monitored with alerting for failures or quality issues. When a vector database exists with an embedding pipeline for key document sets for GenAI use cases. When RAG retrieval quality is evaluated separately from generation quality. And when common failure patterns are monitored for — not discovered after deployment.
What the data assessment always uncovers
The data readiness assessment is the exercise that shifts behaviour most immediately.
I ask teams to score their own data domains across the eight dimensions. The scores are consistently lower than anyone expected. Teams that believed their data was "good enough" discover 1s and 2s in availability, labelling, and governance. That's not a failure — it's clarity. And clarity is what lets you plan honestly.
The governance conversation tends to surface tensions that were already there but unnamed. AI teams that have been accessing production databases directly suddenly see why that's a risk. Privacy teams that have been blocking AI projects suddenly have a framework for saying "here's what we need to approve this" instead of just "no." The conversation changes from obstruction to enablement.
The failure pattern review is where the most uncomfortable moments happen. Teams recognise patterns they've lived through — models that looked great in testing and failed in production, prototypes that were blocked by legal after weeks of work, RAG systems that retrieved irrelevant chunks and produced confident wrong answers. Recognising the pattern is the first step to preventing it.
The GenAI data session tends to resolve a conversation that's been happening for months: should we fine-tune or should we use RAG? After walking through the trade-offs — data freshness, cost, complexity, use case fit — the answer is usually clear. And when it's both, the conversation shifts to sequencing: RAG first for knowledge grounding, fine-tuning later for behaviour shaping.
Book a Workshop
Ready to build the data foundation your AI needs — before investing in more models?
or
1.5-day workshop includes data readiness assessment using the eight-dimension scorecard applied to your data domains, governance framework design tailored to your organisation, data architecture review for AI readiness covering both traditional ML and GenAI, feature engineering workshop with your real data, pipeline design for batch and streaming AI workloads, GenAI data session covering RAG, vector databases, and embedding strategies, failure pattern workshop to identify and prevent the five common data failures, and data readiness scorecard as a takeaway.