Most bad split tests look fine on the dashboard. Two groups, two numbers, one winner. The damage happened earlier, in how the guys got into those groups and whether they stayed there.
Rule one: nobody chooses
If a chatter decides who gets the discount, she'll pick guys who look ready to buy. Her group will win, because of who's in it. The discount had nothing to do with it.
The group has to come from something that has nothing to do with how he behaves. We use his fan ID. The ID gives the same answer every time, so he's placed once and it can't drift. No human picks, and neither does the AI.
Rule two: he stays in his group for good
This is the rule people skip because it's less convenient. It matters for four reasons.
He remembers. Say a guy sees a PPV at $20 on Monday and a similar one at $12 on Thursday because a test moved him. He's learned that waiting pays, and that lesson doesn't go away when the test ends. Our rule that a fan never sees a price lower than he already paid would break too.
The effect arrives late. A first offer today changes what he does next week. If he's moved groups by then, you don't know which group earned the money.
The chat has to make sense. One guy should get one consistent story. A fan who gets a different kind of answer every other day notices.
You count guys, not messages. This is the big one. If you count per offer, a guy who gets lots of offers counts lots of times. If a whale moves between groups, he's feeding both. The unit is one guy, one group, one line in the results, even if he never bought anything.
The bug that taught us this
We got this wrong ourselves. For a while every test we ran shared one coin. Each guy's group came from the same simple calculation on his ID, so the side he landed on in one test fixed the side he landed on in the others. The guys with one tease gap were the same guys with one first-PPV price and the same guys with one discount size.
It looked like several tests. It was one. Any difference between groups could have come from any of those changes, and there was no way to tell which. We fixed it by giving every test its own coin, so a guy's group in one test says nothing about his group in another.
Another time, a change meant for one group of a social test broke replies for that whole group until we caught it. A test like that measures a bug. Before you read any result, check that both groups really got what you think they got.
Checks before you believe a result
- Are the groups the size you expected? A big gap means something's off in how guys get placed.
- Did anything change halfway? Count from the moment both sides were live. If you change the rules mid-test, start the count again.
- Did something else hit one group harder? A mass message with a locked bundle reaches both groups, and it can blur everything. Note the date.
- Did you count guys who reached the moment? A discount test should only count guys who could have got a discount. Everyone else is noise.
- Did you split by type of guy? We read never-buyers, buyers and whales separately. An answer that's right for one is often wrong for another.
The cost of doing it right
Staying in one group for good means some guys sit on the worse side until the test ends. That's the price of knowing. When you've got your answer, you end the test and everyone gets the winner. It's a far smaller cost than running the worse option on everyone for a year because nobody checked.
Doing it by hand
You don't need software for a small test. Keep a sheet with one row per fan. Flip a coin once when he enters the test, write A or B next to his name and never change it. Your chatters look up the letter before they act. It's boring, and it works, as long as nobody "just this once" gives a B guy the A deal. On a human team, that's the part to watch.