How to A/B Test Copy

The Obama campaign staff favoured a video for their signup page. Dan Siroker, the campaign's Director of Analytics, tested six media items with four buttons in December 2007, creating 24 page combinations.

Before we ran the experiment, the campaign staff heavily favored “Sam’s Video”

Siroker's own account of the signup-page test, written three years later.

The favourite didn't win. The best combination paired a family image with a button labelled "Learn More," and it signed up 11.6% of visitors against 8.26% for the original page.

Those two signup rates compare complete pages. They leave the button's separate contribution unanswered. Before your own test, decide exactly which change you're prepared to adopt.

Start with an explanation you could be wrong about

We'll use Fernspool, an invented inventory tool with hypothetical results, for a simpler two-version comparison. The decision is whether to replace its current button with one that names the sample experience.

Suppose shop owners hesitate because they think trying the tool means uploading their own stock list. Both proposed buttons lead to the same sample-inventory experience. Our hypothesis is that naming that destination will lead more assigned visitors to finish the sample workflow.

A preference and a testable decision

A preference dressed as a plan

Test a friendlier button. Success means the new copy wins.

The plan names no reader concern or defined outcome.

A decision we can test

Compare ‘Start free’ with ‘Try with sample inventory.’ We expect the second label to show owners they can try the tool without their stock list. We'll compare seven-day sample-workflow completion among all eligible assigned visitors.

Put the reader concern on the card before launch so a disappointing result can't change the question afterward.

Use customer research to check the concern before launch, and ask owners separately what they understood. Keep the layout and functionality fixed if wording is the decision you want to isolate. If you change the destination too, record that you're testing the package.

Siroker's design deliberately crossed buttons with media. Yours makes a narrower comparison: eligible visitors are randomly assigned to one of two labels during the same period. The Frameworks hub can help with the research behind that plan.

Choose an outcome that survives the click

For Fernspool, the outcome is completion of the sample workflow within seven days of assignment. Count completions as a share of all eligible visitors assigned to each version, including people who never click. Counting only clickers would select people by their response to the copy you're testing.

Agree with analytics on the event that means completion, and test its recording in both versions. Each visitor needs the full seven-day window, including anyone assigned on the final recruitment day.

Check how returning visitors are matched before launch.

One visitor, several visits

An owner might click on Monday, leave, and finish the sample on Thursday from the same browser. Tracking has to join those visits into one record, or that owner shows up as an unfinished click and a separate completion.

These are teaching images made for the lesson, not documentary evidence.

Keep clicks as supporting information. Also agree on error limits that would make you stop the experiment, often called guardrails. Assign someone to follow paid conversion later, since finishing a sample leaves the purchase question open.

Review the call-to-action copy before testing. A broken link or a false promise needs repair as ordinary maintenance.

How much traffic does a copy test need?

Suppose 5% of eligible visitors currently finish, and reaching 6% would justify changing the page. That's one percentage point, or 20% relative. Have your analyst calculate the sample for that target and your actual assignment method.

For this illustration, use a fixed-sample design: you set the visitor count before launch and analyse only after the planned follow-up. The calculation allows for either an increase or a decrease, with a 5% significance setting and 80% power.

Power is the chance of detecting the specified difference under the design's assumptions. With real rates of 5% and 6%, this design has about an 80% chance of catching it. The significance setting caps false alarms at about 5% when there's no real difference.

The calculation assumes independent outcomes and equal assignment chances. Ask whether people from the same shop could affect one another. Revenue needs a different calculation, and repeat visits mustn't count as new people.

Analyst sign-off before recruitment

Outcome

• Confirm visitor-level or shop-level assignment.
• Verify the completion event in both versions.

Sample

• Approve the calculation for that assignment.
• Attach the calculation and its assumptions.

Reporting date

• Agree on a reporting date allowing for follow-up and planned interruptions.

Ask your analyst to approve the design before recruitment starts.

If that calendar is impractical, watch people use the page or interview them about the concern. You can make an editorial choice and record that its effect is unmeasured.

Make the comparison fair before launch

Define eligibility and staff exclusions before assignment begins. Keep returning visitors on the same version where your identity system allows, and note its limits, such as cookie resets. Preview the longer label on a phone to catch wrapping that changes the button's layout.

Configure a check comparing counted visitors with the planned split. Equal assignment chances still allow some variation in totals. A statistically unexpected imbalance, called sample ratio mismatch, needs investigating before you read any effect.

Microsoft's 2020 account of an MSN carousel experiment shows why. A bot filter threw out real users who engaged so heavily with the new version that they looked like bots, making 16 cards seem to do worse than 12. Correcting the filter reversed the result.

Write the adoption rule with your analyst before launch. In this example, the test counts only if the data is valid, follow-up has finished and errors stayed acceptable.

If those checks pass, adopt B only when the estimated gain reaches one percentage point and the planned interval excludes zero.

Resist the urge to stop on a pleasing afternoon

Suppose the dashboard turns green in week two. In a fixed-sample design, stopping at the first favourable significance reading raises the risk of a false alarm. Follow the planned count and let the outcome windows finish.

You can keep monitoring technical problems and the agreed error limits throughout. Sequential methods allow repeated decisions under their own rules, so ask which method your platform uses before you launch.

Tell your analyst about any extra variants or outcomes. A 24-combination design like Siroker's has far more comparisons to plan for than two labels. If you stumble on a mobile-only result nobody planned, record it as a question for a follow-up test.

Keep the decision rule intact

Do

• Put the planned reporting date and analysis method on the test card.
• Record what caused a visitor-count warning and how it was resolved.
• Record when the last visitor's seven days will finish.
• Mark an unplanned mobile-only result as an idea for a follow-up test.

Avoid

• Stop because the dashboard turns green once.
• Replace sample completion with clicks because clicks look better.
• Remove disappointing visitors after seeing their behavior.
• Treat an unfinished test as evidence of no difference.

Read the size and uncertainty together

Take a separate completed example: 10,000 visitors per version, with 500 completions in A and 550 in B. The rates are 5% and 5.5%. The estimated gain is half a percentage point, or 10% relative.

An approximate 95% confidence interval runs from minus 0.12 to plus 1.12 percentage points under the calculation's assumptions. At this scale, that's about 12 fewer to 112 more completions per 10,000 visitors.

A 95% confidence procedure is designed to cover the true difference about 95% of the time across repeated uses. This interval includes a small decline as well as a worthwhile gain, so the result is inconclusive under the adoption rule.

The dashboard may also show a p-value: how unusual a result this extreme or more would be under a model, usually one with no difference. The American Statistical Association's 2016 statement explains that it doesn't measure effect size or the probability that a hypothesis is true.

Keep the estimate and its interval in the decision record, alongside the validity checks. Use direct-response economics to plan the paid-customer follow-up before spending more to attract visitors.

Write the decision onto the test card

Record the reader concern and both versions, then use the editing process to check their promises. Specify who gets assigned and what counts as completion. Mark the sample calculation and stopping rule for analyst review.

On the card, write what you'll do with invalid or inconclusive data, with an owner's name beside each open question.

For the completed 500-versus-550 example, enter the decision:

Decision: retain A. B's estimated gain is 0.5 percentage points; the approximate 95% interval is minus 0.12 to plus 1.12 percentage points.

Paid-conversion follow-up owner: the analyst. Agree on the observation window and reporting date before the next launch.

Fernspool's live page keeps "Start free" because B has not met the adoption rule. The card records an inconclusive result, with paid conversion still to investigate.