Guide · Updated
How to measure whether an AI agent is working
Measure an AI agent on the business outcome it was deployed for, not on how many messages it sent. Agree the target before launch: leads qualified, orders placed, customers reactivated, or tickets resolved without a person. Compare against your own baseline for the same period and segment, ideally with a holdout group the agent does not touch. Then track the supporting signals: time to first reply, follow-ups completed on schedule, the share of conversations handled start to finish, and how often handovers needed a person to repeat questions.
Outcomes, not activity
Messages sent and calls made are easy to inflate and tell you nothing about value. The question is what changed in your numbers: more qualified leads, more orders, more customers back, fewer tickets reaching the team.
The metrics that matter
| Metric | What it tells you |
|---|---|
| Primary outcome vs target | Whether the agent is doing the job it was deployed for |
| Outcome vs holdout group | How much of the result the agent caused |
| Time to first reply | Whether leads and customers are caught while interested |
| Follow-ups on schedule | Whether the cadence actually holds |
| Handled start to finish | How much volume no longer needs a person |
| Handover quality | Whether people pick up without starting over |
Set the target before launch
Agree what success looks like, on which numbers, and over what period before the agent goes live. Targets set afterwards tend to fit whatever happened.
Read the conversations
Numbers show what happened; transcripts show why. A weekly read of a sample of conversations catches the questions the agent handles badly and the reasons customers give, which the dashboard cannot.
Frequently asked questions
- How long before an AI agent shows results?
- It depends on the cycle of the outcome. Response time changes on day one; reorders and conversions need at least one full buying cycle to judge.
- What is a holdout group?
- A comparable set of customers the agent does not contact, so you can see what would have happened without it.
In practice
- 2,540 retailers onboarded without adding a rep · TrueWholesale
- 25,619 dormant leads reopened on one workflow · Labour Law Advisor