Last week I closed with a line worth picking up here: a lift test rescales an answer but does not change the math that generated it. A lot of teams run lift tests, and today I want to work through what that distinction means in practice.
Lift tests are a useful tool. They measure the causal impact of an ad campaign, and there are situations where they are exactly the right call. What they do, though, is answer a narrower question than they are often given credit for.
I want to be specific about that distinction, because there is a real difference between calibrating the amplitude of a response and changing how the model interprets the relationships between channels. In practice, the gap between those two things is wider than it looks.
What a lift test does well
A lift test is an experiment with a control group. You expose a population to advertising, compare their behavior to a population that did not see the ad, and the difference in conversion is the incremental effect attributable to the channel.
So, here is what a lift test handles cleanly:
• It confirms whether the channel has an incremental effect. A valuable answer when there is doubt about whether the channel works at all.
• It corrects the magnitude of that channel's contribution for that specific period and context. If the model estimated Facebook drove $500K and the test shows the real number is $800K, you have a scale factor of 1.6x and you can apply it.
• It improves calibration when you feed that result back as a prior. Bayesian models like ours accept lift tests as external evidence.
This issue starts where the lift test ends.
The thermometer analogy
A lift test tells you whether the fever is real. If the channel has an effect, the thermometer reads a temperature. What the thermometer cannot tell you is what illness is there or how to treat it, because it was not built for that. A doctor who only adjusts the painkiller dose based on the fever, without revisiting the underlying diagnosis, can see improvement for a day. Then the problem comes back, because rescaling the dose does not change the treatment hypothesis.
A lift test does exactly the analogous thing with a model: it adjusts the dose but does not change the diagnosis.
Amplitude versus shape of the curve
Here is the distinction that matters.
When a lift test detects that a channel's real return is twice what the model estimated, you can multiply that channel's curve by 2. The amplitude changed, but the shape of the curve, the specific way the model represents how return behaves as spend goes up, is the same as before.
If the curve has the wrong shape, say the model assumes the channel saturates quickly when in reality it scales close to linearly during peak demand, multiplying by 2 gives you a bigger number with the same incorrect geometry. The conclusion about when to stop spending stays the same as it was before the test.
In technical terms, the distinction is between calibrating the response and changing the model's hypothesis class, which is the set of all the shapes the model can possibly interpret in the data. A lift test adjusts the first. The second only changes when the underlying mathematical structure of the model changes.
Identifiability, in plain English
The OMEN paper we published this year formalizes a concept we had been seeing in the data for a while: identifiability. The idea, in operational terms, is that the same data series can have infinitely many equally plausible explanations. The equation is ambiguous. The model ends up picking an answer from its internal assumptions, because the data alone does not have enough information to distinguish between the alternatives.
The clearest example shows up at peak. Meta spend goes up and revenue goes up too. At the same time, organic demand also goes up because it is Black Friday and that was going to happen anyway. The model has infinitely many ways to split that revenue between "what Meta drove" and "what was going to happen regardless." All of them explain the data with the same fidelity. The split that ends up in the report is more a property of how the model is parameterized than a property of the reality it observed.
A lift test in Meta gives a reliable measurement of Meta over the test window. What it does not touch is how the model splits paid demand and organic demand when both move together. That ambiguity lives in the structure of the equation, not in its sample size. More data will not close it.
Three outputs, one set of equations
There is one detail about how an MMM works that is worth making explicit here, because it changes what "calibrating the model" actually means.
Worth remembering: an MMM does not produce one number. It produces three outputs, and all three come from the same set of equations.
• Measurement what happened. Attribution of revenue to each channel in the observed period.
• Forecasting what is going to happen. The expected trajectory of return for each channel going forward.
• Optimization what to do next. The recommendation for how to allocate the next dollar across channels.
All three depend on the same underlying mathematical structure. If that structure represents how channels actually behave, all three outputs are reasonable. If the structure was designed for conditions different from the ones a brand is operating in, all three outputs inherit the same problem at the same time.
A lift test enters this system through one specific door. It measures the incremental effect of a channel in a concrete experimental window. That helps recalibrate one part of the first output, attribution, for that channel and that period. What it does not touch is the other two doors. The way the model projects that channel's return forward still depends on the same saturation curve, the same assumption about how it interacts with other channels, and the same way of separating paid demand from organic demand. The budget recommendation at the end is built from that structure, not from the point estimate the test adjusted.
That is why we say the math matters. Not as an abstract technical argument. It matters because it defines the three outputs the team uses to make budget decisions across different horizons, from the next week all the way to BFCM planning.
When it matters more, and why BFCM is the edge case
Under normal conditions, the three outputs degrade in contained ways. The errors exist but they do not amplify much, and the difference between a well-specified model and an approximate one stays manageable in practice.
At peak the scenario changes. The three outputs come under stress at the same time.
Attribution gets harder because paid spend and organic demand move together with more velocity than usual. Forecasting, meanwhile, loses fidelity: week-to-week efficiency stops being stable, and next week's projection gets built on assumptions that no longer hold. And optimization enters the period recommending cuts on channels that still have room, because the response curves the model assumes saturate before the market actually saturates.
All three degradations happen with the same set of equations. A lift test corrected on one channel does not reach the other two outputs, because those two are still inheriting the same underlying structure.
Across the ten datasets with synthetic ground truth that we analyzed in the OMEN paper, standard models recommended spending up to 81% more than the true optimum at BFCM. Not because someone running the model made a bad call, but because math designed for stable conditions behaves that way when it gets pushed to the edge.
The numbers from the paper, against known ground truth
Measured against synthetic ground truth across ten independent datasets, in conditions where we knew the right answer because we built it into the experimental design:
• Measurement. Standard models tend to over-attribute revenue to paid channels relative to a model built to reflect how marketing actually works in each company. The difference in absolute error against ground truth is on the order of 3x.
• Forecasting. When predicting incremental lift during peak periods, standard models accumulate roughly twice the error of Prescient's model.
• Optimization. Average error against the true optimum lands at 32-45% for standard models and at 5.6% for Prescient's. At BFCM specifically, standard models recommended up to 81% overspend versus the optimum, while the mechanistic model stayed within roughly 1%.
All three are consequences of the same structural problem, measured at three different points of the system.
What to do with this
The operational implication is to understand precisely what a lift test corrects and how far that correction reaches.
A lift test is good for calibrating the magnitude of a channel's effect in a specific window. Useful. What that adjustment does not do is revisit how the model projects that channel forward, how it assumes the channel interacts with the rest of the portfolio, or how it separates paid demand from organic demand when both accelerate at peak.
If you are planning for BFCM right now, it is worth sitting down with your team and looking at the model that sits underneath Measurement, Forecasting, and Optimization. When a lift test result enters the system, what gets updated? Just the amplitude of the measured channel's curve? Or also the way the model projects that curve, the way it connects to other channels, and the separation between what marketing is generating and what was going to come in regardless?
The answer says quite a lot about which of the three outputs is being corrected, and which ones are still resting on the same mathematical structure as before.
The Halo
Exclusive insights, every week.
Subscribe to The Halo for sharper marketing thinking.