The 10‑20‑70 Rule for AI: What It Means and Why Data Dominates

A few years ago, I watched a team of engineers spend six months optimizing a neural network architecture for a supply‑chain forecasting model. They tried different layer depths, adjusted learning rates through dozens of experiments, and squeezed out incremental improvements in validation accuracy. The model never reached production. Not because the architecture was wrong — because the training data had a 30% sensor drift rate that no one had checked. The entire investment in algorithm tuning was wasted on corrupted inputs.

That scenario plays out across AI teams more often than most people realize. The 10‑20‑70 rule exists to prevent exactly this kind of waste. It states that roughly 10% of machine learning project effort should go toward algorithms, 20% toward infrastructure, and 70% toward data work — labeling, cleaning, validation, augmentation, and versioning. The imbalance is not accidental; it reflects where real bottlenecks live, and teams that ignore it usually discover the hard way that no amount of model tweaking compensates for bad data.

The Origin of the 10‑20‑70 Rule

The rule emerged from observations inside Google and from Andrew Ng’s work on practical machine learning. Google’s internal analysis of ML projects found that roughly 70% of total effort went into data preparation and cleaning, even as teams believed they were spending most of their time on model development. The number was counterintuitive enough that it became a reference point for how teams misallocate attention.

Andrew Ng articulated the pattern more directly: in many applied ML projects, organizing and validating data is the dominant cost, yet project plans consistently underweight it. The rule was never published as a formal paper. It spread through conference talks, engineering blogs, and project post‑mortems — a heuristic that kept resurfacing because it described an uncomfortable truth that most teams learned too late.

An early example that made this painfully clear was a 2015 image‑recognition project I encountered. The team had acquired a large dataset and spent months tuning a convolutional architecture. Accuracy plateaued at 82% and refused to improve. When they finally inspected the labels, they discovered that 15% of training images were misclassified. Some had multiple conflicting labels. Others were completely irrelevant to the task. The algorithmic work was sound. The data was not. The project was abandoned after the relabeling cost exceeded the remaining budget.

Breaking Down the Numbers: Algorithms, Infrastructure, Data

The 10‑20‑70 split is not a budget template. It is a diagnostic lens for where your project’s actual resistance lives.

10% for algorithms covers model architecture selection, hyperparameter tuning, and the experimentation loop. For most teams using modern frameworks, this is the smallest chunk. Pre‑trained models, transfer learning, and AutoML have reduced the algorithmic workload substantially. If your team is spending 50% of its time here, it is worth asking whether you have actually validated the data distribution first.

20% for infrastructure includes data pipelines, compute provisioning, storage orchestration, and deployment tooling. This is the layer that keeps the rest running. A typical enterprise AI project that spends $1M on model development often ends up spending $3M+ on data engineering before the model ever sees a production environment. The infrastructure number hides the data cost because most of the pipeline work is data management disguised as engineering.

70% for data is where the majority of real work lives. Labeling, cleaning, deduplication, schema alignment, augmentation, and versioning are not glamorous tasks, but they determine whether the model succeeds or fails. A 2017 supply‑chain AI project I tracked spent 80% of its budget building a custom LSTM architecture for demand forecasting. The sensor data feeding the model had a 30% drift rate, and the training pipeline had no drift detection. The model performed well in offline validation on clean holdout data. In production, it failed within two weeks. The team had optimized the algorithm for a distribution that no longer existed.

Why the Rule Still Matters in the Age of Large Language Models

Foundation models shift the emphasis away from custom algorithms and toward the data that shapes retrieval and generation. For teams using large language models, the 70% data effort now applies to curation, prompt crafting, and knowledge‑base structuring. Building a retrieval‑augmented generation pipeline is almost entirely a data problem: chunking strategy, embedding selection, relevance ranking, and content freshness all depend on how you prepare the underlying data.

The same imbalance holds for AI search engines. Systems like ChatGPT, Perplexity, and Google AI Overview depend on structured, semantic content. Raw product pages with minimal descriptive depth do not provide the entity coverage these systems need. 73% of AI search queries never reach traditional ecommerce pages because the pages lack the discussion‑driven, comparison‑oriented content that AI retrieval prefers.

The algorithmic work of fine‑tuning a model is small compared to the effort of building a corpus that an AI system can actually use. Teams that spend months drafting perfect few‑shot prompts while ignoring signal degradation in their embedding corpus make the same mistake as the 2015 image‑recognition team: they optimize the wrong bottleneck.

AI visibility tools are essentially data‑gap detectors. They reveal which semantic content your catalog is missing — comparison frameworks, FAQ intents, structured attribute discussions — and make the 70% data effort quantifiable rather than abstract. This is not a coincidence. The 10‑20‑70 rule has re‑emerged in a new domain because the underlying dynamic has not changed: technical infrastructure and algorithmic sophistication matter much less than whether the data supports the task.

Practical Implications for Teams and Tools

If you want to apply the 10‑20‑70 lens to your own project, start by auditing where your team’s calendar time actually goes. Track hours spent on architecture discussions versus hours spent on data pipelines, label review, and validation failures. Most teams I have worked with overestimate algorithmic time by a factor of two and underestimate data time by a similar margin.

One concrete step is investing in data versioning. Tools like DVC and Pachyderm make it feasible to track dataset changes, roll back corrupted versions, and reproduce experiments. Another is automated validation: write assertions for distribution shifts, missing values, and label consistency before running a single training loop. The cost of catching a data problem early is orders of magnitude lower than the cost of debugging a model that trained on corrupted inputs.

For ecommerce teams, the 70% data problem shows up in a specific form: most product pages lack the comparison content and FAQ coverage that AI search engines actually retrieve. Standard product descriptions optimized for human shoppers leave semantic gaps that AI systems cannot fill. I wrote about this dynamic in detail in a post about the gap between Shopify stores and AI search engines, where I traced how AI retrieval prefers structured comparisons over typical PDP content.

The infrastructure side of the equation — platforms like Shopify, WooCommerce, and custom CMS setups — handles delivery but not semantic coverage. 91% of stores lack AI‑search‑ready comparison content, which means most product catalogs are invisible to retrieval systems regardless of how well the infrastructure runs. Tools such as VISNIB specifically analyze which semantic content is missing from your product catalog, directly reducing the 70% data‑gap effort. VISNIB generates AI‑ready articles, comparison frameworks, and FAQ content to fill those gaps, making the data work automated rather than manual.

The point is not that every team needs a specific tool. It is that the 70% should receive 70% of your attention, and most teams allocate far less. I have seen more projects fail because of bad data than because of bad algorithms, and the failures share a pattern: the team noticed the data problem late, estimated the cleanup cost incorrectly, and had no systemic way to measure coverage. A deeper breakdown of which specific semantic gaps most ecommerce brands miss is available in a separate analysis of semantic gaps ecommerce brands overlook in AI search.

If your team is spending 90% of its time on architecture, stop and validate the data distribution first. Run a distribution audit. Count label conflicts. Measure drift. The algorithmic work can wait.

FAQ

What is the 10 20 70 rule in AI? It is a heuristic describing how machine learning project effort typically distributes: 10% for algorithms, 20% for infrastructure, and 70% for data work. The rule is not a budget prescription but a diagnostic that reveals where real bottlenecks live.

Who coined the 10 20 70 rule? Andrew Ng popularized the framing during his talks on applied machine learning, drawing on Google’s internal analyses that showed data preparation consuming the majority of project resources. The numbers percolated through the ML community and became a widely shared reference point.

Does the rule apply to generative AI and large language models? Yes. For LLM projects, the 70% data effort shifts to corpus curation, prompt engineering, embedding selection, and retrieval pipeline tuning. The algorithmic fine‑tuning is typically the smallest component, while data structure and coverage dominate the work.

How can a team start applying the 10 20 70 rule today? Track actual hours spent on algorithm work versus data work for two weeks. If your algorithmic time exceeds data time, audit your data pipeline for drift, label conflicts, and missing coverage. Invest in automated validation and versioning before tuning the model further.

Are the percentages fixed or flexible? They are flexible diagnostics, not fixed budgets. Some projects with clean, curated datasets may shift effort toward algorithms. The rule is valuable as a detection mechanism: if you are spending 50% on algorithms, question whether the data side has been validated. The imbalance signals where your team should look for hidden waste.