The Overfit Prompt: Spoon-Feeding the Pipeline

This is the third in a series on the quiet failure modes of AI pipelines: the ones that don’t throw, don’t fail a test, and don’t surface until someone runs the system on data it has never seen. The first was On Messy Data. The second was The Confused Deputy Problem in Agent Systems. This one is about a failure you introduce yourself, on purpose, while being careful.

Assumptions: you are building a pipeline that is meant to be domain-general (one engine, many verticals) and its behaviour is driven by prompt templates. You configure it per domain with data, not by editing the templates. The templates are supposed to know nothing about any particular vertical. If that describes your system, you have almost certainly already leaked, and you probably can’t point to where.

What the problem is

A domain-general pipeline has exactly one asset: its ignorance of any particular domain. That ignorance is what lets the same code serve a new vertical on Monday that it has never seen before. Every concrete detail you bake into a “general” template spends a little of that asset.

The leak is easy to describe and hard to notice. You are debugging the pipeline against a real corpus. The model gets an instruction slightly wrong. So you reach for the fastest fix that works: you add a concrete example, or you enumerate the specific cases the model keeps missing. The instruction now lands. The bug is closed. Tests are green.

What you have actually done is teach the general engine the shape of the one domain you happened to be looking at. The template is no longer domain-blind; it carries a fingerprint of your test corpus. The next vertical inherits that fingerprint as a bias, and nothing in your pipeline will tell you it’s there.

I call this spoon-feeding, because that is what it feels like in the moment: the model is struggling, so you feed it the answer. It works every time, which is the problem.

The demo is the degenerate case

There is an extreme version of this leak, and you may have watched it on a conference stage. The demo runs flawlessly because the entire solution is encoded in the prompt. The task, the format, the edge cases, the worked answer: all of it sits in the template, and the model’s job is reduced to reciting it back. The system appears to reason. It is replaying. This is spoon-feeding taken to its limit, done on purpose to impress.

It is worth naming because it sets the scale. The rigged demo is one end of a continuous line, the point where the prompt contains 100 percent of the answer and the engine contributes nothing. Every worked example you add to a “general” template moves you a step along that line. You are not building the demo, but you are borrowing its trick in miniature, and the metric that looks good on your corpus is the same metric that looked good on stage. The distance between a real pipeline and a magic trick is how much of the answer lives in the prompt.

The stage version is easy to dismiss, because someone rigged it on purpose. The version that bites is the internal one. You run a demo for your own stakeholders on the handful of inputs you happened to test, and it sends the wrong signal to the room: it looks production-ready, and everyone now believes you are a month from launch when you are a year from it. No one lied. You picked the inputs that worked, because those were the ones in front of you, and cherry-picking is what optimism looks like before a pipeline has been broadly exercised. The corpus you demoed on is the corpus the prompts have overfit to, and the gap only shows when someone feeds it the fourth input.

It is overfitting, in prose

This is a cousin of a problem you already know. Train a model on data that overlaps its test set and the scores look sweet while it collapses on anything new; machine learning has a whole discipline of hygiene around that, held-out sets, leakage audits, the rule that the test set never touches training. Quant traders hit the same wall from the other side: a strategy tuned until the backtest prints money, then bleeding it the moment it meets a market it did not fit. Same class of failure. Fit the knobs to the data you can see, and the fit becomes a lie about the data you cannot.

Prompt templates are the inductive bias of an LLM pipeline. They are where you encode what the system should assume before it sees any input. When you hand-tune a template against a specific corpus until the output looks right, you are fitting that bias to your evaluation data: the same overfitting, expressed in prose instead of gradients. The difference is that no one has trained you to be paranoid about it. There is no train_test_split for a paragraph of instructions, so the leakage goes uncaught.

The mental correction is to treat every worked example and every enumerated case in a shared template as a sample from your test distribution that you have pasted into the prompt. Stated that way, the discipline is obvious. It is just not obvious while you are typing.

It enters exactly when you are being careful

The leak does not arrive through laziness. It arrives through diligence, through the specific act of staring at real data and trying to make an instruction precise. It comes in two flavours.

Worked-example leakage. You illustrate a format with a specimen, and the specimen is real.

(* Leaky: a real-looking identifier anchors the model to one corpus *)
let citation_hint =
  {|Emit the reference as {"source": ID, "clause": S},
    for example clause "14.2(1)(b)(ii)".|}

(* Structural: describe the shape, never show a specimen *)
let citation_hint =
  {|Emit the reference as {"source": ID, "clause": S},
    where clause is the pinpoint within the source,
    a section, paragraph, or sub-clause reference.|}

The first version is faster to write and reads more helpfully. It also quietly tells the model what your sources look like. Swap the example for one from a different corpus and the model’s behaviour shifts, which is proof the example was doing work it was never supposed to do.

Enumeration leakage. You list the cases instead of stating the rule.

(* Leaky: the enumeration IS the domain's object model, smuggled into a general instruction *)
let out_of_scope =
  "Ignore obligations attaching to widgets, depots, couriers, filed complaints, or the auditor."

(* Structural: the same rule as a relation, teaching nothing about any one domain *)
let out_of_scope =
  "Ignore every obligation whose SUBJECT is not the entity you were asked about, \
   whether borne by another party or attaching to a thing that is not that entity."

The enumerated version feels safer because it is concrete. It is the more dangerous of the two. A list of nouns is a compressed description of a specific domain’s ontology, and the model reads it as such. The structural version says less and generalises more: it names a relation (subject-is-the-thing-asked-about) that holds in every domain, and lists no nouns for any of them.

Notice that the fix is never weaker. “The subject must be the entity asked about” is a sharper statement of the rule than any list of examples. The examples were only ever a crutch. Removing them removes the leak and tightens the instruction at the same time.

Why it is worse in a pipeline than in a single prompt

If you have one prompt, a leak is contained and you will probably feel it. In a multi-stage pipeline the blast radius is larger and the signal is weaker.

Stages share context. A single briefing (the schema, the vocabulary, the ground rules) is usually threaded into several call sites at once. One leaked example in that shared text contaminates every stage that embeds it. You fixed one bug; you biased five agents.

And the failure is silent by construction. A leaked template produces better output on the corpus you are testing, because that is where the leak came from. Every metric you have improves. The regression is entirely in the counterfactual (the vertical you have not run yet), and counterfactuals do not appear in a test report. This is the same shape as the failures in the earlier posts: the system is confidently wrong in a direction your tests are structurally unable to see.

You already run this pass on your code

If you build with domain-driven design or a hexagonal architecture, you should already be running anti-leak passes. You just run them on code. The point of ports and adapters is that the domain core knows nothing about any particular adapter, and infrastructure concerns do not bleed into the domain. You lint for it, you review for it, and an import of a database driver inside the domain layer is a leak you catch on sight, because keeping the boundary clean is a habit rather than an afterthought.

Domain-driven design makes the same demand one level up. A bounded context owns its model and its ubiquitous language, and it does not reach into another context’s internals; where two contexts meet they integrate through a published contract or an anti-corruption layer, never by one helping itself to the other’s vocabulary. That is the rule a leak breaks. A prompt is a domain artifact, so it belongs inside a bounded context and should speak only that context’s language. When a general-engine prompt starts naming a vertical’s entities, one context has reached into another’s internals, which is the precise coupling DDD exists to forbid. You would not let the billing context import the shipping context’s aggregates. The prompt is the same kind of artifact, and it takes the same rule.

A shared prompt template is a boundary of exactly the same kind. The domain-general engine is the core; each vertical is an adapter, supplied as data. The rule is identical: the core must not name the adapter. The only thing that differs is what leaks. In code you scan for a forbidden import or a concrete type. In a prompt you scan for a forbidden vocabulary or a smuggled concept. It is the same discipline pointed at a softer artifact, and the reason it feels unfamiliar is only that no one told you the prompt was part of the domain boundary. It is.

Mitigations

There is no single guard, because the leak has two different natures: it is a human habit at authoring time and a fuzzy property of text at detection time. You defend it in layers.

1. Author structurally, not by example. The primary defence is the writing itself. State relations, not instances. Use synthetic placeholders when you must show a shape. Render every template against neutral, obviously-fake fixtures, never a slice of real data. This is the cheapest fix and the one that stops mattering the moment you are tired or shipping under a deadline, which is why it cannot be the only one.

Structured outputs do a lot of this work for you. When the model must emit JSON against a schema, or text against a grammar, the shape is enforced by the decoder rather than taught by a specimen, so the usual reason to paste a real example, that you need to show the format, goes away. Let the schema carry the structure and you have one less place to leak. The invariants a schema cannot express (a cross-field rule, an ordering constraint, a value that must reference another) still go in the prose, stated as relations rather than shown as instances. Check those where checks are exact: validate the output after generation, and when it fails, repair or retry deterministically with the specific violation fed back, instead of reaching for another worked example that happens to satisfy the rule. If your domain calls for it, that same post-generation gate is where your other eval-time safety checks live.

2. Check the templates against the corpus, with an agent. The right check is not a list you keep in the codebase. It is an agent that reads your rendered templates alongside the vertical’s corpus and flags anything that reads as domain-specific, because deciding “does this belong to one domain” is a judgement, not a match. The check runs, reports, and leaves nothing behind but the fix it prompts.

You can drop a cheap deterministic step in the middle to make it efficient. Derive a throwaway blacklist of distinctive terms from the corpus, string-search the rendered templates for them first, and spend the agent’s judgement only on what the search cannot see: a concept smuggled in using ordinary words, a piece of vocabulary the corpus never contained. The blacklist is generated for the run and discarded after it. It is an optimisation inside the loop, not a fixture you maintain.

(* Nothing here is committed. Each run regenerates a throwaway blacklist
   from the corpus to narrow the work, then a model judges whatever the
   cheap string-search cannot see. *)
let cheap_hits ~corpus (rendered : string) : string list =
  distinctive_terms corpus              (* generated per run, then discarded *)
  |> List.filter (contains_word rendered)

let check ~(corpus : Corpus.t) ~(llm : (module Llm.S)) (rendered : string)
    : finding list =
  match cheap_hits ~corpus rendered with
  | _ :: _ as hits -> List.map exact_violation hits   (* fast path: obvious leaks *)
  | []             -> judge llm ~corpus rendered       (* slow path: concepts in plain words *)

The deterministic pass is not a guarantee, it is a filter that makes the expensive judgement rarer. It catches the obvious leaks for nothing and hands the agent a smaller problem. What it cannot catch, a domain’s shape expressed entirely in general words, is exactly what the agent is there for.

3. Close the loop, do not just report. A check that prints a warning is a suggestion, and suggestions are ignored under deadline. Make it a loop instead. It detects each violation, proposes a structural redraft that removes it, re-renders the templates, and checks again, measuring the leak count on every pass. It stops when the count converges to zero and the other quality gates stay green. The redraft is the abstraction move from earlier in this post, done automatically: replace the instance with the relation it stood for, then confirm the replacement cost you nothing the metrics can see.

(* The check is a loop, not a report. Detect, redraft the template that
   leaked, re-render, and measure. Stop when the leaks converge to zero
   and the other gates stay green. *)
let rec converge ~fuel ~llm ~corpus (t : template) : template =
  match check ~corpus ~llm (render t) with
  | []              -> t                        (* clean, and the gates are green *)
  | _ when fuel = 0 -> failwith "leak did not converge"
  | findings        ->
      let t' = redraft ~llm t findings in       (* fix the template, never the render *)
      converge ~fuel:(fuel - 1) ~llm ~corpus t'

The loop measures, it does not only detect. Each pass records how many leaks remain, so “converged” is a number you can gate on, and a redraft that trades a vocabulary leak for a concept leak shows up as a count that refuses to fall.

Step	Catches	Misses
Structural authoring	the leak at its source	lapses when you are tired or rushed
Deterministic pre-filter	corpus terms, instantly and for nothing	concepts in general words; vocabulary the corpus lacks
Agent judgement	the novel and the borderline, read in context	probabilistic; needs the loop to converge

4. Screen what the model sees, not what you wrote. Judge the rendered bytes (templates interpolated with fixtures, exactly as they reach the model), not the source with its placeholders. A leak can hide in a value that only appears after interpolation. If your guard reads the source and the model reads the render, your guard is checking the wrong artifact.

5. Make the guard automatic and gating. This is the one that matters most. The leak enters precisely when you are not thinking about leakage; you are thinking about the bug in front of you. A defence that depends on remembering to look is a defence that fails in the exact conditions that produce the failure. The check must run without being asked, ahead of any expensive pipeline work, and must be able to stop the build. Discipline you have to invoke is not a control; it is a hope.

Fix the pipeline, not the artifact

There is a way to lose all of this at the last step, and it shows up the moment you build the pipeline with an AI coding harness of your own. The harness runs the pipeline, the check flags a leak, and the quickest thing the coding agent can do is reach into the generated output and edit the offending line by hand. The leak is gone from the artifact. The metric is green. You have learned nothing, because the template that produced the leak is untouched, and it will produce the leak again on the next run and in every vertical you have not looked at.

This is the same spoon-feeding, moved up a level. The first version was you hand-patching a prompt against a corpus. This version is your coding agent hand-patching the pipeline’s output against a check. Both close the visible bug by editing the symptom, and both leave the generator carrying the fault.

The rule is to fix the generator, never the generated. When a leak is found, the harness edits the template or the pipeline code that emitted it, re-runs the pipeline from clean, and checks the fresh output, and it repeats until the leak count converges to zero and the other quality gates stay green across the whole run rather than on one patched file. A manual edit to a generated artifact is forbidden, because it decouples what you shipped from what your pipeline produces, and a pipeline whose output no longer matches its code is not a pipeline. It is a pile of hand-edited files wearing one.

This is production AI reliability engineering, and it is the rule everything above rests on. Structural authoring, the corpus check, the convergence loop: all of it assumes the thing you fix is the thing that runs. The moment a human or an agent patches the output instead of the producer, a green metric describes a file and not a system, and the leak you believed you closed is still in the machine, waiting for the vertical that reveals it.

The language grows, so prune it

The leak is not the only thing that accretes. Exercise the pipeline against more examples and the prompts grow, because every example that stumps the model tempts one more clause, one more caveat, one more clarifying line. Each addition helped the case in front of you. Together they silt up the template until it is long, redundant, and quietly self-contradicting. A bloated prompt is its own quality problem: it costs tokens, it buries the instruction that matters under ten that do not, and it makes the next edit harder to reason about.

So growth needs a countervailing phase of its own. Alongside the passes that add precision, run passes that trim and consolidate: merge two clauses that say the same thing, cut the caveat that no longer earns its place, fold three near-duplicate instructions into the one relation they were all circling. It is the same abstraction move the leak fix uses, pointed at redundancy instead of specificity.

And it wants the same machinery. The leak check is one member of a class of pipeline-quality agents, not a special case. A consolidation agent belongs to that class: it reads the rendered templates, measures a quality (here, redundancy and length against a budget, rather than domain leak), proposes a redraft, re-runs, and converges. Build the harness once, as a loop that detects, redrafts, and measures to green, and every quality you care about, leak, bloat, contradiction, drift, becomes another agent plugged into it. The leak agent taught you the shape. The shape is what you reuse.

Generalisation has a ceiling

Push far enough the other way and there is a symmetric failure. The guards in this post all pull toward generality: strip the specifics, state the relation, serve more verticals with one template. But generality is not free, and a template is not infinitely elastic. Exercise the same engine against too many wildly different corpuses and it stops generalising and starts underfitting, the template accumulating so many hedges and cross-cutting caveats that it models each domain weakly and none of them well. Overfitting memorises one domain; underfitting smears across so many that it commits to nothing.

This is the bias-variance tradeoff in prose, and it means there is a sweet spot for your problem rather than a direction you ride to the limit. A prompt has finite capacity, the way a small model does. Spend it all on one vertical and you have the rigged demo. Spread it across twenty unrelated ones and you have mush. The right number of verticals to generalise across is the largest set that still shares enough structure for one relation to hold cleanly over all of them, and no larger. When adding a vertical starts making every prompt vaguer, you have found the edge, and the answer is another engine, a separate bounded context of its own, not another caveat crammed into this one.

This is eval-driven development

Step back and every guard in this post is the same kind of thing: an eval. A leak check is an assertion about a rendered prompt. A convergence loop is a red-green cycle. Structured-output validation is a post-condition. None of it is new in spirit. It is test-driven development with the determinism removed, because the unit under test is a model and the oracle is sometimes another model. Call it eval-driven development: a stochastic subset of TDD, where a passing eval is a distribution of outcomes over a threshold rather than a single green tick.

The discipline has two homes, and you want it in both. Offline, in your evaluation framework, the evals gate change: no template ships until the leak count is zero, the consolidation budget is met, and the quality metrics hold on a held-out set of verticals the prompts never saw. Online, in the pipeline itself, the same checks run as runtime guards: validate the structured output, screen the rendered template before an expensive call, block the run when a leak shows up in production that your offline set did not contain. The offline evals stop you shipping the failure you know about. The runtime evals catch the one you could not have known. Write the check once and run it in both places.

In practice

• Name the asset. A general pipeline’s only moat is that it knows nothing about any one domain. Treat every concrete detail in a shared template as a withdrawal from that account.
• Treat the prompt as a domain artifact. You already keep your domain core free of infrastructure and one bounded context out of another’s internals. A prompt lives inside a context and should speak only its language.
• State the rule; let the schema carry the shape. Replace the worked example with the relation it stood for, and lean on structured outputs for format so you never paste a real specimen to show one. Invariants a schema cannot hold go in the prose and get checked after generation.
• Assume you have leaked, then prove where. You have been debugging against real data, so the prior is high. Run an agent that reads the rendered templates against the corpus, with a throwaway deterministic pre-filter for speed, and redraft and re-run until the leak count is zero.
• Fix the generator, not the output. Never let a human or a coding agent hand-patch a pipeline artifact to make a check pass. Fix the template or the pipeline and re-run to green. A green metric on a patched file is a lie about the system.
• Trim as deliberately as you tighten. Prompts accrete clauses as you exercise more examples, so run consolidation as its own quality agent in the same converge-to-green loop. Bloat, contradiction, and drift are each another agent of that class.
• Find the sweet spot, do not ride the limit. Generalising is the goal, but a template has finite capacity, and too many unrelated verticals underfit as surely as one overfits. When a new vertical makes every prompt vaguer, the fix is another engine, not another caveat.
• Make the evals gate, offline and at runtime. This is eval-driven development: run the checks in your evaluation framework so nothing ships dirty, and as runtime guards so the leak your offline set missed still stops the run.

The through-line across all three posts is the same. The failures that hurt are not the ones that crash. They are the ones that make the numbers look sweet while narrowing what the system can actually do. A rigged demo hides the whole answer in the prompt to win a room. A spoon-fed pipeline hides a sliver of it to close a bug, and fools only the person who wrote it. The defence in both cases is the same: fix the generator, not the output, and let an eval decide when it is fixed. Overfitting was a training-time worry. In a prompt-driven pipeline it is an authoring-time one, and it looks exactly like good engineering.