How to know your campaign worked: leave some customers out

To know whether a campaign worked, leave some customers out of it and compare. Your email tool will tell you how many people ordered after the send. It cannot tell you how many of them would have ordered anyway, and that second number is the one that decides whether you made money.

Why can't my email tool tell me?

Say your platform reports that the last campaign brought 810 orders. Many of those people would have ordered without your message. Regulars order. The question is how many.

Think of two identical gardens. You water one and not the other. The difference in how they grow is what the water did. In a business the gardens are two groups of customers, picked by chance, and the water is your message.

How do you set it up?

On your next send, pick one customer in ten at random and leave them out. Say 9,000 get the message and 1,000 do not. Four weeks later, 9% of the 9,000 have bought again, and 7% of the 1,000.

The campaign's effect is the gap: 2 percentage points, which is about 180 extra buyers out of the 9,000. The other 630 of those 810 orders were coming back anyway. Even with 1,000 people held back, treat 180 as a rough figure. The true number could be anywhere from a few dozen to a few hundred. All numbers here are an example.

People call the left-out group a holdout or a control group. I prefer the plain description: the people we did not message.

Some details keep the comparison fair:

• Pick by chance, never by hand. If you choose who to leave out, you will pick customers who seem less engaged, and the groups stop being comparable. A spreadsheet with a random number next to each row, sorted, works. A stable rule such as a hash of the customer id works better, because a customer stays in the same group tomorrow.
• Decide the measure first. Write it down before the send: orders in four weeks, revenue per customer, bookings. Do not change it after you see the result.
• Do not peek and adjust. If you change the message or the split halfway because the early numbers look odd, the comparison no longer means anything.
• Keep the left-out group clean. If they get the same offer from another tool or from the shop assistant, the groups are no longer different.

I explain the same method for subscription businesses in what a holdout group is, and the casino version in how to run a control group test for casino retention.

What does leaving people out cost?

The few sales you lose from them. If the campaign works, the left-out customers missed something that would have helped. I will not pretend that costs nothing.

I think it is the cheapest price in retention for knowing what works. The alternative is to keep sending campaigns whose real value you cannot tell, and some of them lose money, especially anything with a discount in it. If a campaign gives 20% off and half of the buyers would have bought anyway, you paid for sales you already had. Without a left-out group you would see only the orders and call it a success.

What if my list is small?

Size matters. With a list of a few hundred, one send tells you almost nothing, because a few customers more or fewer swing the percentages. Two things help.

First, pick a fresh random tenth for each send and add the results up over several sends before you trust the gap. The noise averages out.

Second, run campaigns for longer windows. A reminder sent to 300 people with 30 left out will not settle anything in a week. In a retention pilot I run, a group with fewer than about 30 events is reported as an early signal, never a verdict. Use the same caution: if the left-out group has produced very few purchases, wait.

Making the left-out group smaller does not help a small list. It makes the problem worse. The fix is patience.

Are there customers I should never leave out?

Yes. If you run a clinic, never hold back medical reminders. A patient who misses a check-up because you wanted a clean comparison is a real cost. Test marketing messages only, or send to the left-out group a week later, which still gives you a useful comparison of timing.

The same goes for anything a customer needs to know: a failed payment, a changed booking, a safety notice. Treat those as service messages.

What do I look at when I compare?

Look at outcomes. Opens and clicks measure whether the message was read. They do not measure whether anyone came back. Compare:

• The share who ordered or booked again in the window.
• Revenue per customer in each group.
• For anything with an offer, revenue per customer after subtracting what the offer cost.

If the two groups come out the same, you were paying people to do what they would have done. That is a useful discovery, and it saves you money from then on.

What can you do today?

On your next send, set aside a random tenth of the list before you hit the button. Write the date, the size of each group and the one measure you will compare, on a piece of paper or in a note.

In four weeks, work out the percentage in each group and subtract. That one subtraction is the first honest number you will have about whether your marketing to existing customers earns its keep.