What we got wrong when we automated customer messages

Forty-nine people got two emails from one of our campaigns ten minutes apart, when the second was meant to follow three days later. A bot we run went quiet for almost two days with no alert. These are two of the failures I know first-hand, and they teach one thing: when you automate messages to customers, the mistakes are quiet and some cannot be undone.

I run an automation studio, 2pizza.team, and I write up what broke. Everything below comes from systems we run for a client and inside our own companies. I do not name the client or the people. The numbers are the ones I published in two longer articles, linked below.

What happened when a follow-up fired too early?

The follow-up was supposed to go three days after the first message. It went ten minutes after, to 49 people.

The cause was a misunderstanding about one field. In the sending tool we used, the delay on a step means the wait after that step, not before it. We set the three-day gap on the follow-up step, as anyone would read it. The tool waited three days after the follow-up instead, and the first step had almost no delay.

Nobody caught it in the moment. We found it two days later through a random check, not through an alarm. By then 49 people had received the same pitch twice in minutes, which looks like a bug or a spam script. Both are bad for a first impression. The full story is in automated cold email: the mistakes that only happen once.

What happened with personalization?

Each personal opening looked fine alone. Checked as a batch, 27% of the "personal observations" were the same template, and 20% were compliments.

The cause was an example sentence in the prompt. The model treated it as the form to copy. Reading one email, you would never notice. Reading fifty in a row, the same skeleton kept returning with a different company name dropped in. If you ever write messages with AI, check the whole batch, not one email.

What about the list?

In one client base of 5,764 addresses, 2,562 were valid, 2,494 were catch-all, 560 were invalid and 148 were unknown. We sent only to the valid ones. That is a lot of list to leave unsent, and it was the right call, since bounces damage the sending domain and a young domain is watched closely. On domains about five weeks old we kept the limit at 15 emails a day per campaign.

What broke in our company brain?

A company brain listens to calls, pulls out decisions and tasks, and answers questions later. On real calls it failed in specific, repeatable ways. The ones that stand out:

• The same meeting twice. Three note-taking bots joined one call. Our duplicate check matched speaker names, and one copy had only "Speaker 1" and "Speaker 2", so both went in.
• A proposal logged as a decision. Long transcripts were cut at about 30,000 characters and each piece analysed alone. An early proposal was stored as a decision next to the real one from the end of the call.
• A changed weekday. The transcription service translated the speech and got a day wrong. Nobody noticed until someone asked when a meeting was.
• A pile of unreviewed guesses. About 326 proposals were waiting for approval and the search ranked them next to confirmed facts. We made proposals expire after 7 days, and 135 expired in the first pass.
• A silent bot. The model key hit its spending limit and the bot went quiet for almost two days. There was no alert.
• Checks that never ran. Daily health checks lived on a laptop. The app was closed for a week and they did not run, though the schedule still looked alive.

The long list, with a question to ask any vendor for each item, is in what breaks when you run a company brain on real calls.

What did we put in place?

Each failure became a rule. I would treat these as the minimum for any system that talks to your customers.

• Read back what the tool stored. A preflight check reads the delays, limits, links and empty fields from the sending tool itself, not from what we typed in.
• Send the whole sequence to your own address first. A human sees every step arrive in the real order.
• A watchdog that pauses on its own. Ours runs on a server, checks every 30 minutes, and pauses a campaign on 3 or more bounces, or on a bounce rate above 3% once 30 or more emails have gone.
• Someone other than the writer approves. A failed check is a hard stop.
• A written incident log. We have 26 incidents in ours, and we read it before any new launch.
• A fixed set of real questions for the brain. We use 19, each with phrases the answer must never contain. The strong model passes all 19 at about $4.69 a run, which is cheap enough to repeat after every change.

Where did the channel itself lose?

For one low-priced product, cold email to 521 contacts produced zero interested replies in the first wave. A warm channel, a webinar invitation, produced 53 registrations and a call request within 48 hours. One result is not a law, and I would not build a strategy on a single campaign. But it changed what we recommend: cold email suits higher-priced offers, and for cheap products the warm channel won for us.

What does this mean for your retention messages?

Every automated message to your own customers is one of these. A reminder that fires at the wrong time, a personal line that is obviously a template, a list with dead addresses, a system that stopped and nobody noticed.

Retention messages go to people who already trust you, so a mistake costs more than a stranger's annoyance. That is a reason to be slower at the start, not to avoid automation.

What can you do today?

Before your next campaign goes to customers, send the whole thing to yourself and one colleague. Read every step in the order and at the timing a customer would see. Then open the tool's own settings and check what it stored for delays and limits, because it may not match what you meant.

Write down the one thing that surprised you. That sentence is the first line of your own incident log.