Ad Delivery Signal Experiment
Key Insights
— Testing what makes an ad algorithm favor one creative over another
One creative had dominated ad delivery for 675 days, leaving new alternatives with too little exposure to evaluate. I proposed and led an experiment to test which visual and narrative elements influenced delivery, designing and producing 21 variants across three game titles.
The Problem
675 Days, One Leading Creative
New Creatives Couldn't Compete
Unverified Assumptions
What I did
Reframed the Research Question
Designed Controlled Variations
Ran the Experiment
Business Outcomes
675-Day Incumbent Replaced
A new creative reached the top position.
+49% Install Conversion
Improvement after introducing the new creative.
Applied Across 3 Titles
Similar delivery patterns were observed in two additional games.
Background
An algorithm that kept choosing the same ad
Google Ads determines how frequently different creatives are shown based on its delivery system and performance signals. For Bubble Pop Origin, one creative had held the highest share of delivery for over two years, despite the team regularly introducing new alternatives.
The Problem
New creatives weren't getting a fair test
New creatives received too few impressions to judge whether they could outperform the existing winner. The team continued producing variations based on the appearance of successful ads, but had little evidence of which elements influenced delivery.
I began to question whether we were optimizing the creative itself without understanding how the platform selected what users would see.
PROJECT GOAL
Make the delivery algorithm the subject of the experiment
Rather than creating another variation of the winning ad, I proposed testing which elements influenced delivery. I independently designed the experiment and produced the variants, then worked with the UA team to run them across three titles.
EXPERIMENT DESIGN
Break the winning creative into testable signals
I separated the incumbent creative into visual elements, such as color, obstacles, and level geometry, and narrative elements, such as gameplay context and the situation presented to the player. Using the incumbent as a reference, I created variations that changed one element while keeping the others as consistent as possible.
Bubble Pop Origin · Example variations
Obstacle: Spider web → another obstacle
Level geometry: Original layout → exaggerated shape
Narrative context: Same gameplay → a spider introduced into the scene
Across three titles, I produced 21 variants and tested them for approximately one month. The main constraint was exposure: variations that differed too much from the incumbent risked receiving too few impressions to evaluate. Each version therefore needed to remain recognizable while isolating a specific change.
FINDING
The results challenged my original hypothesis
I expected stronger visual cues, including more saturated colors and exaggerated shapes, to attract greater delivery. Instead, the variants that gained more exposure tended to change the narrative context while keeping much of the original visual treatment.
This suggested that the team's focus on reproducing the incumbent's appearance may have overlooked other signals associated with delivery. It did not, however, establish how the algorithm interpreted those differences.
OUTCOME
A new creative replaced the 675-day incumbent
On Bubble Pop Origin, a new variant overtook the creative that had led delivery for 675 days. This gave the team a way to move beyond repeatedly copying the same ad and evaluate new creative directions.
Campaign conversion improved by 49%. I also applied the experimental approach to two other titles, where similar patterns in delivery were observed.
REFLECTION
The experiment revealed a pattern, not an explanation
The experiment changed how I approached creative optimization. Instead of treating a successful ad as a whole to imitate, I began looking at which individual elements were associated with its performance.
What remained unclear was how the algorithm interpreted those elements. The stronger response to narrative changes did not necessarily mean it understood narrative context as a person would; it could also have been responding to correlated visual patterns. That distinction became an important question in how I thought about algorithmic systems.
Thanks for coming by.
© 2026 Hailey Moon