Every AI guest messaging tool now has a slider. It's usually labelled "confidence". Drag it to 90% and the AI sends any reply it's at least 90% sure about. Anything below waits for a human. Sounds sensible. Sounds scientific. Sounds like someone in a lab coat signed it off.
Here's one vendor describing theirs, in its own words:
Set a confidence threshold, a maximum risk level, the quality signals you require and which intent categories qualify. Anything that misses a check waits for a human.
That's Sparrow Intel's Conversations page, and to be fair to them, they're more careful than most. They keep risk separate from confidence ("a confident refund reply is still high risk"), they block emergencies, refunds, chargebacks and legal threats outright, and they say the default failure mode is "ask a human". Guesty's ReplyAI Autopilot also comes with a confidence threshold. Besty sends when it's confident and drafts when it isn't. This is the industry standard.
Our view: it's also the elephant in the inbox.
What "90% confident" actually means
Let's do the maths nobody puts on the pricing page.
If a confidence score means anything at all, a reply scored 90% is a reply the AI expects to get wrong one time in ten. Not "a bit off". Wrong. The guest is told the pool is open in November. The early check-in is agreed on the day the cleaner has four turnovers. The door code goes out for the flat upstairs.
And the replies most likely to go wrong are exactly the ones you've told it to send: the ones sitting just above your line, at 90 or 91%. The slider doesn't remove the risk. It picks which risk you're comfortable with, then sends it to the guest.
Now add volume. A property typically brings 10 to 20 guest messages a week. Say you run 300 properties and 60% of messages clear your threshold.
| Per week | |
|---|---|
| Guest messages | about 4,500 |
| Sent on autopilot (60%) | about 2,700 |
| Wrong if just 2% of those miss | 54 |
| Wrong if the score is honest at the line (10%) | up to 270 |
Fifty-four guests a week receiving a confident, polite, wrong answer. None of them asked to be part of your statistics.
It's grading its own homework
Here's the bit the slider doesn't mention: the AI is the one deciding how confident it is.
A confidence score isn't a confidence interval, and it isn't a measurement from outside the system. It's the model's own estimate of its own answer. Language models are famously good at sounding sure. "I'm 92% sure" from a model is closer to a mood than a measurement.
So the setup is: the AI writes an answer, the AI marks the answer, and if the AI gives itself an A-minus, the guest gets it. You wouldn't run a turnover that way. "The cleaner is 90% confident the flat is clean" is not a sentence any operations manager wants to hear at 2:55pm.
Why it's the wrong question
Most guest questions don't have a "probably" answer. Either the cot is in the hall cupboard at Harbour House or it isn't. Either your policy allows a 1pm check-in on a turnover day or it doesn't. Either the parking moved round the corner last month or it didn't.
There are really only two situations:
- The answer is known. It's in the listing, the reservation, your written policy, or something your team has confirmed before. And your rules say the AI is allowed to act on this kind of request alone. Send it.
- The answer isn't known, or the rules say a person decides. Then don't guess at 90%. Ask the housekeeper, the manager, the owner. Tell the guest you're checking. Save the answer so the next guest gets it straight away.
That's not a slider. It's a switch. And the switch is set by your rules and your facts, not by how the model feels about its own prose.
None of this means a system built this way is never wrong. Facts go out of date, people make mistakes, rules have gaps. But when it is wrong, you can point at the reason (this fact, this rule) and fix it once. "It was 91% confident" isn't a reason. It's a shrug with a decimal point.
The copilot waiting room
The other industry favourite goes like this: "Start in copilot mode. Your team approves every draft while the AI learns your operation. When we've learnt enough, we'll switch on autopilot."
How long is "enough"? Usually nobody says. Weeks. Months. A quarter, if the onboarding call is on a Friday.
Meanwhile, your PMS is sitting on thousands of replies your team has already written. Every check-in question, every lockout, every "can we bring the dog", answered by the people who know your properties, in your tone. That is the best training data your operation will ever produce, and it's free, and it already exists.
Copilot-as-training asks your team to hand-feed the AI a few dozen approvals a day, one thumbs-up at a time, when the answers are already in the archive. It's like hiring a new team member, refusing to let them read a single past conversation, and making them shadow you for three months instead. Your team isn't getting time back. They're running a finishing school.
To be clear, there's a difference between that and checking the AI's work. A short supervised period, a week of reading what it would have sent and why, is a sensible way to build trust. We recommend it in our enterprise rollout playbook. The point of that week is for you to check the AI. It shouldn't be the AI's only way of learning who you are.
What to ask for instead
Next time a vendor shows you the confidence slider, ask these:
- "What decides whether a reply is sent, apart from the score?" You want to hear: which policy applies, whether that policy lets the AI act alone, and where the answer came from.
- "Show me the source for this reply." Every automatic answer should point at a fact: the listing, the reservation, a confirmed answer from your team. If the source is "the model", it's a guess.
- "What happens when it doesn't know?" The right answer is that it asks the right person, tells the guest, and saves the answer. Not that it drafts something at 74% and waits.
- "What did it learn from our past conversations on day one?" If the answer is "nothing until your team starts approving drafts", that's the copilot waiting room.
- "What's your error rate on replies sent automatically?" Not the automation rate. The error rate, measured on a random sample. Our guide to measuring automation shows how to check it yourself on 200 requests.
And if a tool only works by confidence threshold, at least set it high, keep refunds, complaints and anything involving money or access firmly off the list, and audit what it sends every week.
The short version
- A 90% confidence threshold means some guests, every week, get a wrong answer on purpose.
- The confidence score is the AI's opinion of itself. It's often overconfident.
- Guest questions are mostly known or not known. Send the known ones, ask about the rest, remember the answer.
- You don't need months of copilot to teach an AI your operation. You already wrote the textbook. It's called your inbox.
For how an AI should decide between replying, checking and handing over, see agentic AI for short-term rental operations. For the other ways AI guest messaging goes wrong, see why AI guest messaging goes wrong.
Sources
- Sparrow Intel, Conversations product page (checked November 2026)
- Guesty, ReplyAI Autopilot and AI Task Creation launch, including confidence threshold controls (April 2026): press release
- Besty AI review, STR Specialist: strspecialist.com