One Variable at a Time: A Facebook Ads Testing Recipe

Twelve sailors, six pairs

In May 1747 a naval surgeon named James Lind had a ship full of men dying of scurvy and no idea what would cure them. Nobody did. There were dozens of theories.

So he ran a test. He took twelve sick sailors and divided them into six pairs.

All twelve were kept in the same part of the ship and fed the same basic diet. Then each pair got one addition. Cider for one.

Vinegar for another. Sea water. And for one pair, citrus fruit.

The two men given citrus recovered far better than the rest.

It is often called the first controlled clinical trial. What made it work was not cleverness. He held everything the same and changed one thing.

Why most ad tests teach nothing

Here is the typical test. Last month's ad did poorly. So this month there is a new ad: new picture, new headline, new offer, new audience.

It does better. What did you learn? Nothing you can use.

It might have been the headline. It might have been the offer. It might have been that this month had a holiday in it.

Statisticians have a word for a second factor that changes alongside the one you are studying and muddies the result. They call it confounding.

An ad test with four changes has confounding built in. You get a winner and no reason, so next month you are guessing again.

The cost is larger than one wasted month. Without reasons you never build up knowledge about your own market, which is the real return on testing. See copy the rules, not the ad.

The One-Variable Recipe

The recipe is short. Duplicate a working ad. Change one element.

Run both.

The order matters, because some elements move results far more than others. We work from the top of the ad down.

First, the opening line and headline. On many placements the first line of the text is the only headline a viewer sees. It decides whether anything else is read.

Second, the image or first frame. Same words, new picture. Third, the angle.

A different reason to care.

Fourth, the offer. Last, body copy and format. These matter, and they matter less than what comes before them.

The platform has a tool built for this. Meta's A/B test feature splits an audience so that each person sees only one version, which keeps the comparison clean.

If you are starting from nothing instead of improving a winner, the approach is different, and we cover it in launching ads for a new offer.

Fair Testers: four rules

Some advertisers get steadily better every quarter. They are not more creative. They run fair tests and write down what they find.

We call them Fair Testers.

One difference. The two ads are the same in every respect but one. If you changed the picture and also fixed a typo, you changed two things.

Same audience, same days. Side by side. An ad that ran in the first week of the month and one that ran in the third are being compared across different weather.

Decide the finish line first. Before launch, write down what ends the test: a number of leads per version, or a spend. Then do not look until you get there.

That last rule is the one people break. Check a test every morning and stop when one version is ahead, and you will often crown a winner that was only lucky. The more looks you take, the more false winners you find, which statisticians call the multiple comparisons problem.

Judge on what you pay for. An ad can win on clicks and lose on booked calls. Read the right number, as we explain in reading ad data.

Keep a log. One line per test: what changed, what happened, what you now believe. After twenty lines you know things about your market that no competitor does.

"My budget is too small to test"

You may be spending thirty dollars a day and thinking this is for bigger accounts. Small budgets make the discipline more important, not less.

The honest limit is this. With few results, a small difference between two ads means nothing. If one version gets seven leads and the other gets nine, you have not found a winner.

So test big differences. Two opening lines that take opposite approaches, not two that swap one word. Large effects show up in small samples.

Small ones do not.

Test fewer things. Two versions, not six. Splitting a small budget six ways gives six results too thin to read.

And use the free test first. Imagine the headline alone in a newspaper classified column with a phone number under it. Would anyone ring?

A headline that fails that test will not be rescued by a good picture. Fix it before spending anything. How to write one is in hooks that work.

What to read next, and why

James Lind (encyclopedia entry): The scurvy experiment of 1747 and why it is remembered.

About A/B testing (Meta Business Help Center): How the platform's own test tool divides an audience and reports a result.

Confounding (encyclopedia entry): What goes wrong when two things change at once.

Multiple comparisons problem (encyclopedia entry): Why checking a test repeatedly produces false winners.

So the simple takeaway is this: duplicate your best ad today and change only its first line. Write down when you will judge it, and do not look before then.