Marketing mix models still can't prove ChatGPT visibility drives sales

Advertisers have spent the past year asking whether showing up in ChatGPT, Gemini or Claude answers drives business. None of them has gotten a hard number.

Of the marketers surveyed, 73% have already invested in a tool that tracks their visibility in AI-generated answers. The survey, by AI search visibility platform Scrunch and content marketing agency Scribewise, was first reported by Digiday on Aug. 13. Those tools show whether a brand gets cited, how often, in what tone and against which competitors. They can't show whether that visibility drives incremental sales.

Part of the problem is how people shop. A user can ask ChatGPT for a product recommendation, click no link and buy a few days later on Amazon, from a retailer or in a store. The recommendation may have shaped the decision, but no attribution tool picks it up. Even when an assistant drives a click, the traffic is small and captures only part of its influence.

Brands need to know two things. First, how advertising, PR, creator work, content and technical site structure change their presence in AI answers. Second, whether that presence lifts awareness, consideration or sales.

Generative engine optimization (GEO) tools mostly cover the first question. In theory, MMM can cover the second.

Marketing mix modeling is an econometric method that estimates how much each factor contributes to a business metric such as sales, leads or subscriptions over a given period. It weighs media spend alongside other drivers: promotions, pricing, distribution, seasonality, weather and competitors' activity.

MMM runs on aggregated data. It doesn't need to follow each consumer from exposure to purchase. That makes it well suited to clickless journeys and indirect effects, such as a ChatGPT recommendation followed by a purchase on another platform.

So an MMM could, in principle, take in a time series of a brand's visibility in AI assistants. The model would then test whether changes in that visibility move with sales once other factors are accounted for.

That variable wouldn't necessarily be treated as a standard media channel. Organic presence in AI answers has no certified impressions and no spend to calculate a return on ad spend (ROAS) against. It could instead help break down the baseline more precisely: the share of sales the model doesn't attribute directly to observed media spend.

But MMM needs three things: a variable that moves enough over time, a long enough history and a stable measurement method. Data from AI assistants only partly meets those conditions so far.

French marketing mix modeling firm MediaROI is working with AI visibility platform GetMint on a first method to bring AI assistant visibility into its MMMs. The approach is still in beta and doesn't claim to deliver a GEO ROI right away. Its first goal is to build an exposure variable the model can test.

The method has two parts.

The first is an index of estimated demand in AI assistants for the advertiser's category. No reliable public data currently shows how many queries ChatGPT, Gemini or Claude receive in a given sector.

MediaROI rebuilds that trend from three inputs, co-founder Antoine Szalewski told Open Garden: "the size of the search market, estimated with Google Keyword Planner, seasonality of demand, observed with Google Trends, and public data on the overall growth of queries and usage in LLMs."

The goal isn't actual query volume. It's a relative index that changes daily or weekly. An MMM can work with a variable's movement even when its absolute level is unknown.

The second part is GetMint's visibility score. MediaROI focuses on prompts close to purchase: where to buy a product, which brand to choose or which supplier to go with. The estimated demand index is multiplied by that score to create an exposure variable for the model.

MediaROI started collecting the data this summer with several advertisers. The company estimates it needs about six months of history and hopes to have first results by the end of the year. "The method could also work with scores from platforms other than GetMint if the advertiser already has enough history," Szalewski said.

The approach turns a blind spot into a testable hypothesis. It also starts building the data series the market lacks today. Still, at this exploratory stage, it rests on several assumptions that have yet to be tested.

The first limit is mapping search data onto AI assistants. A Google search and a ChatGPT conversation don't always serve the same need. Search engines have historically run on fairly short queries, while assistants take longer, more contextual questions.

Their role can also vary by category: exploration in travel, comparison in autos, tech support in electronics, recommendations in beauty.

Average LLM growth doesn't reveal how usage is actually changing when people buy car parts, pick a telecom carrier or compare food products. Szalewski acknowledges the limit. His indicator reflects an estimated trend, not an observed volume.

The second unknown is whether AI queries add to search or replace it. The first version of MediaROI's method assumes some assistant queries come on top of traditional search. Marketing effectiveness consultancy Ekimetrics also thinks some of that usage could gradually replace searches people used to run on Google.

The distinction matters. If ChatGPT creates new discovery moments, the LLM variable gives the model new information. If it mostly diverts queries from search, the two variables partly move as two sides of the same behavior. The MMM may then misallocate their contributions.

The method also depends on how reliable the visibility score is. Results vary by vendor, prompt panel, models queried, language, test frequency and how responses are aggregated.

"A GEO KPI isn't comparable from one partner to another," said France Hassenforder, a partner at Ekimetrics. Before anyone measures a business contribution, the score fed into the MMM needs a precise definition and a methodology stable enough to hold.

A brand can score high on a set of generic questions and be missing from the prompts people actually use before buying. A change in the panel can also push the indicator up or down with no change in the brand's real visibility.

LLM answers vary on their own, too. The same question can produce different recommendations depending on timing, the account used, location or model version. Updates to ChatGPT or Gemini can also break the historical series.

"We see our clients very excited about testing, but cooled by the lack of stable KPIs. The topic is still in its infancy," Hassenforder said.

Causality is the biggest problem. A well-known brand is more likely to get recommended by an AI assistant because it has more content, press coverage, social conversation and authority signals. That same brand strength can also explain part of the sales the MMM observes.

A correlation between LLM visibility and revenue doesn't prove the recommendations drove sales. Both can stem from a third factor: brand equity, the value and awareness a brand has built up over time.

Brand campaigns, PR, creator work and content can also lift the GEO score and sales at the same time. If those effects aren't properly separated, the MMM may credit AI visibility with a contribution that really comes from the marketing spend that created it.

The problem can run the other way, too. LLMs draw heavily on content already on the web. Their answers may reflect a brand's past popularity rather than new exposure that changes behavior. The GEO score then becomes a brand health indicator, not necessarily an independent lever.

Ekimetrics considers the topic a priority, but its work remains exploratory. The firm is testing several datasets and looking at both econometric methods and causal approaches that use control groups. "We don't yet have an indicator robust enough to build into MMMs in a standardized way," Hassenforder cautioned.

The first question is which KPI to use: a simple citation, share of voice, position in the answer, sentiment, presence in a purchase recommendation or a price mention. The right one depends on the assistant's role in the journey and on the marketing outcome the advertiser wants to explain.

Hadrien de Nijs of measurement consultancy m13h is even more cautious. Among his clients, traffic directly attributed to AI assistants ranges from 0.1% to 0.5% and often stays below 0.2%. That figure understates their influence because it misses clickless journeys. Still, the directly observable signal is weak next to the other variables in an MMM.

Lack of history is the main blocker. "An MMM is usually built on two or more years of data," de Nijs said. Most advertisers have only a few months of GEO scores, sometimes just a handful of data points. "So the variable shows up abruptly at the end of the series, even though consumers have been using ChatGPT for several years," he added. Building LLMs into MMMs today, he said, is "reckless." The risk is asking the model for a precise contribution from a variable that is recent, unstable and partly reconstructed.

Neither m13h nor Ekimetrics is walking away from the subject. Both advise starting data collection right away. "It's important to start monitoring now, because you won't be able to rebuild the history after the fact," Hassenforder warned.

The first step is to define a prompt panel that reflects real usage and keep it consistent over time. De Nijs breaks it into three dimensions: "salience, or the likelihood that the brand gets cited in a search about its category; favor, meaning how positively or negatively it's presented; and power, its ability to show up when its main competitors are cited too."

Advertisers then need to document measurement conditions precisely: the assistant queried, model version, language, country, any persona, test frequency and changes to the panel. They should keep raw responses, citations and sources so indicators can be recalculated if the methodology changes.

Advertisers can also start studying the small amount of traffic coming from assistants now: conversion rate, basket size, new customers and on-site behavior. Panels and surveys can fill in the picture by identifying consumers who used an LLM without clicking a link.

Finally, tests on control markets or populations will show whether a higher GEO score actually comes with a change in behavior.

Ads in ChatGPT will bring more data on the paid side: impressions, clicks, conversions and spend. They won't solve the measurement of organic visibility. Advertisers will have to separate the effectiveness of a campaign running in an assistant from the influence of a recommendation the model generates.

MMM remains one of the most credible ways to measure that influence without individual tracking. But being able to add an "LLM visibility" column isn't enough to produce a reliable contribution estimate. The immediate job is to stabilize the indicators and build up history. Measuring real business impact comes next.