When a contractor builds a house on a cracked foundation, no amount of premium finishing work fixes the underlying problem. The floors might look level. The walls might be straight. But over time, the structure shifts in ways that the surface work never anticipated. Calibrating a marketing mix model with incrementality test data works the same way. If the foundation of the model is wrong, feeding it test data doesn’t correct it. It just gives the wrong model something new to conform to.
For brands investing in both marketing mix modeling and incrementality testing, understanding what test-calibrated MMM actually does—and what it can’t do—is one of the most consequential measurement decisions you can make.
Key takeaways
- Test-calibrated MMM is the practice of feeding incrementality experiment results into a marketing mix model as anchors that constrain how the model attributes revenue across channels.
- The general process involves running experiments like geo tests or lift tests, then using those results as reference points to which the model is expected to conform.
- Incrementality tests are locally accurate but globally inaccurate: they capture a specific moment in time under specific conditions and cannot represent how marketing works across your full business over time. This makes them an unreliable foundation for model calibration.
- Even solid test data cannot fix a poorly constructed MMM. Calibration adjusts parameter values within a model; it cannot repair flawed structural assumptions. The danger is that a badly built model can appear to be working after calibration, which is a more costly problem than one that visibly fails.
- “Causal MMM” is often positioned as a more rigorous evolution of test-calibrated MMM, but no MMM on the market establishes true causality. The distinction is largely a marketing one.
- Prescient’s approach runs parallel model versions—one with your test data incorporated, one without—and scores both for accuracy. Clients can then choose if they want to proceed with a model calibration using the test data.
What test-calibrated MMM is and how it works
The appeal of test-calibrated MMM starts with a reasonable frustration. Traditional marketing mix models are built on historical, observational data. They track what happened across channels over time and use that history to attribute revenue and project future performance. The limitation is that correlation-heavy historical data can produce attribution outputs that feel off, like a channel that a marketer knows drove results gets undervalued, or spend that clearly wasn’t working looks more efficient than it was.
Incrementality experiments—geo tests, holdout tests, lift tests—offer something that feels more concrete. You split your audience or geography into two groups, expose one to a marketing campaign and withhold it from the other (the control group), and measure the difference in outcomes between them. The result is a number that represents the incremental or true contribution of that channel or campaign under those conditions.
Test calibration takes those experiment results and uses them as ground truth inside the model. The general process works like this:
- A brand runs experiments on one or more marketing channels, collects the results, and feeds them into the MMM as fixed reference points.
- The model then adjusts its coefficients—the internal weights that determine how much revenue it attributes to each channel—so that its outputs align with what the tests found.
- Some providers run this calibration work continuously, updating the model as new experiments come in. Others treat it as a periodic process tied to major campaigns or budget cycles.
The underlying logic is that experimental data is more reliable than observational data, so anchoring the model to experiment results should improve accuracy. That logic is worth examining carefully. We’ll get to the two major problems after a brief explanation about calibration vs validation.
Calibration vs. validation: what’s the difference?
These two terms get used interchangeably in this space, but they describe meaningfully different relationships between your MMM and your test data.
Calibrating an MMM with test data means the experiments come first. Their results are fed into the model as inputs that constrain what it can output. The test is the authority. The model conforms to it. Validating an MMM using test data means something different: you run your model independently and then check whether your experiment results agree with what the model already concluded. Here, the test results are doing the assessing; they’re being used to judge the model rather than shape it.
Both approaches treat incrementality tests as the more trustworthy input. That’s where we disagree with both. We’d rather see the model as a check on the test results. But it gets more complicated from there.
Some providers do use their MMM to evaluate incrementality test results rather than the other way around. That’s closer to the right direction. But an MMM evaluating test data isn’t automatically a neutral evaluator. A number of marketing measurement vendors offer both MMM and incrementality testing, which means the model doing the evaluating belongs to the same company that sold you the test. Even with the best intentions, that’s a structural incentive to find that the tests check out.
Prescient doesn’t offer incrementality testing. When our model evaluates whether your test data improves or degrades accuracy, there’s no commercial stake in the outcome either way. That’s the distinction that actually makes the evaluation trustworthy.
Problem one: Incrementality tests aren’t reliable enough to be ground truth
Treating experiment results as ground truth for model calibration assumes those results are accurate enough to anchor a model to. That assumption deserves scrutiny.
Incrementality tests are point-in-time measurements. They tell you something real about a specific channel, in a specific geography, during a specific window of time. That’s the definition of locally accurate. The problem is that local accuracy doesn’t translate to global accuracy, and a marketing mix model is a global tool. It’s supposed to reflect how your marketing works across all channels, all marketing activities, all geographies, and all time periods, not just the conditions of a single test.
The structural challenges that make most incrementality tests fall short of true randomized controlled trials are worth understanding in full before treating their results as calibration data. What matters for this article is the implication: when you calibrate your MMM with experiment results, the accuracy of your model becomes a direct function of the experiment quality and its accuracy. Low quality experiments don’t just fail to improve your model. They embed their errors into it structurally, in ways that are harder to detect precisely because the model has been anchored to confirm them.
Problem two: Calibration can’t fix a poorly built MMM
Even if your test data is high quality, calibration has a second (and more fundamental) limitation. It can only adjust what the model produces. It cannot change what the model is.
A poorly constructed MMM—one that doesn’t accurately reflect how marketing works in the real world, that misrepresents how channels interact with one another, or that fails to account for how marketing effects build and decay over time—will produce wrong marketing measurement no matter what data you feed into it. Calibration adjusts the parameter values the model uses. It does not repair the structural assumptions underneath them. If those assumptions are wrong, the model remains wrong. It just produces its wrong answers with more confidence because it has been anchored to real-world data from the experiment results.
This is the more dangerous failure. A model that produces visibly implausible outputs is easy to question. A model that has been calibrated to align with experiments you ran and trust can look entirely reasonable while systematically misattributing revenue across channels and time. You think you’re getting actionable insights. Budget decisions get made on that basis. Channels get cut or scaled based on attribution that has the appearance of rigor without the substance of it.
The result is a model that is locally calibrated but globally inconsistent: it aligns with your test results in the conditions where those tests were run, and drifts in every other context (across different channels, different time periods, and different spend levels). Those are exactly the contexts where marketing mix models are supposed to be most useful.
Test-calibrated MMM vs. “causal MMM”: A distinction without a difference
You may have seen test-calibrated MMM positioned alongside—or contrasted with—something called a “causal MMM.” The framing suggests that traditional MMMs only surface correlations in historical data, while a causal MMM goes further by incorporating experimental data to establish “true” cause and effect. Test calibration, in this framing, is either a step toward causal modeling or a lesser version of it.
It’s a compelling distinction. It’s also one that overstates what any marketing model on the market can deliver.
No MMM establishes true causality in the rigorous sense that word implies. Please read that again. No MMM. Not theirs, not ours.
Marketing systems are too complex, too interconnected, and too influenced by variables that no model fully observes. Incorporating experimental data into a model can, in some cases, improve its grounding in real-world results. It does not and fundamentally cannot transform a statistical model into a proof of cause and effect.
If a model could genuinely establish that your marketing spend caused your revenue outcomes with that level of certainty, the same methodology would be predicting financial markets, and the people selling it to you would be making a lot more money than they are now. The vendors making causal claims are describing something aspirational, not something their models currently do.
We’ll be publishing a dedicated article on this topic shortly and will update this section with a link when it’s live.
What Prescient believes instead
Rather than assuming that incrementality test data either always improves or always degrades model calibration, Prescient’s approach is to find out. The platform runs two parallel model versions: one that incorporates a client’s test data, and one that doesn’t. Both receive accuracy scores. Marketers can compare MMM results from each version and see directly whether the test data is helping or hurting before deciding which model to use going forward.
This is what Prescient’s Validation Layer is built to do. Some clients find their experiment results significantly improve model accuracy: the tests were well-designed, the data is clean, and incorporating it produces a more accurate picture of how their marketing activities drive sales. Others find that their test data introduces bias and reduces reliability. Neither outcome is assumed. The model tells you which one you’re looking at and the model you choose to use is ultimately up to you.
That’s a fundamentally different relationship between an MMM and incrementality data than treating any single experiment as ground truth. The test data is what’s being assessed. The model is doing the assessing. If you want to see how that works in practice, book a demo.
