Plenty of tools in this market show you a chart going up. Few tell you how the chart was made. So don't take our word for it either. You've got the fans, the switches and the numbers, and a fair test isn't hard.
Why totals lie
Here's the first 8 days of our own AI chatter on Fanvue, on one AI model:
- 645 paid offers sent
- 64 bought, by 24 buyers
- $1,865 gross, about $29 an average sale
- 3 tips, averaging $45.58
Looks great. Now take out one fan. He made $1,345 of that $1,865, about 72%. Without him the AI sold about $520. Same week, same AI, and two very different stories.
That's why a before-and-after comparison proves almost nothing. One big spender showing up, a new shoot, or a payday falling in the wrong week can beat any tool. Our 8 days are a small sample and we treat them as an early look, nothing more.
What to measure
- Gross per fan, by group. Report it with and without your top spender.
- Buyers. How many different guys paid. A wider base is healthier than one big number.
- Offers bought. Bought divided by sent, and the average sale. A high rate at low prices can make less money than it looks.
- Repeat buyers and renewals. A pushy setup looks good for a week. This is where it shows up.
- What's left. Gross minus what the chatting cost you: fee or commission, base pay, manager time.
- Your own hours. If the money's the same and you got your evenings back, that counts.
Most of this is on the dashboard: daily sales with the AI's part in green, where the money came from, conversion rates and latest purchases. Compare gross with gross and net with net.
How to run a fair test
The idea is simple. Same account, same period, two groups of fans.
- Pick one model. Mixing accounts mixes everything else too.
- Split at random. Flip a coin for each fan, or take every second name. If you give the AI the cold guys and keep the warm ones, you've decided the result already.
- Handle the whales first. Either split your top spenders evenly by hand, or keep them all out of the test and with your humans. Don't let chance decide where your biggest guy lands.
- Use the switches. Set her to answer only the guys you switch on, then switch on the AI group. The other group stays with your team, working as normal.
- Nobody changes groups. We do this in our own tests: half of guys get scripts and half get singles, and every guy stays in one group for good. Once a fan moves, you can't say who earned what.
- Keep everything else equal. Same vault, same price levels. A mass message goes to both groups or neither.
- Run a full month. You want both groups to cross a payday.
- Keep the balance topped up. At -$2 the AI pauses until top-up, and a pause in the middle of a test wrecks it.
Reading the result
Compare gross per fan first, with and without the top spender. Then subtract costs. The AI group's cost is 10% of the gross of what the AI sold, and 0% on subscriptions, renewals and human sales. The human group's cost is whatever you really pay.
Be careful with small groups. If one guy swings the result, you don't have a result yet. Run it longer or use more fans.
A split answer is a normal outcome. Humans may do better on the top spenders and the AI on everyone else. That's useful: the per-fan switch lets you run exactly that.
Keep testing after you've decided
Inside EmpireChat you can keep going on the smaller questions. There's a price split test, scripts against singles, and follow-up style splits. Follow-up lines are ranked by how often fans answer, so the weak ones show themselves.
Start with the $10 of credit a new account gets, which covers the fee on $100 of AI sales. Set up the groups this week, write down the rules before you start, and don't change them halfway.