How to Benchmark Your Own Support Automation (Instead of Trusting Someone Else's Numbers)

Every vendor publishes a resolution rate. Almost none of them define it the same way, and none of them measured it on your customers, your product, or your help center.

So the honest answer to "what resolution rate should we expect?" is that you cannot know until you measure it yourself. The good news is that running a defensible internal benchmark takes about two weeks and no special tooling.

Here is the method.

Why published benchmarks mislead

Three problems make cross-vendor numbers hard to compare.

Different denominators. Some products count every conversation, including one-line greetings and bounces. Others count only conversations where the customer asked a real question. The same underlying performance can produce very different percentages.

Different definitions of resolved. A conversation with no follow-up within 24 hours is sometimes counted as resolved. That includes the customer who gave up.

Different populations. A company with a mature help center and a simple product will see automation rates that a company with a complex configurable product never will. The variable that moved the number was the content, not the model.

None of this means published numbers are dishonest. It means they answer a question about someone else's operation.

Step 1: Define your metrics before you collect anything

Write these down and freeze them. Changing a definition mid-study is how internal benchmarks lose credibility.

The last one is the metric almost nobody tracks and the one that should carry the most weight in your decision.

Step 2: Build a real test set

Pull a sample of past conversations, ideally 200 or more, from a period that reflects normal operations. Avoid your peak season and avoid the week you launched something.

Then classify them by intent. You do not need a taxonomy. Group them until every bucket holds at least five conversations and you can name each bucket in a few words: order status, returns, sizing, billing, integration setup, and so on.

Record the volume share of each bucket. This is the map that makes the rest of the study meaningful, because a 70% resolution rate spread evenly across your top intents means something very different from 70% concentrated in your two easiest ones.

Step 3: Run the sample, blind

Replay the test set through the agent. Two rules make the result trustworthy:

1. Do not tune between questions. Fixing the knowledge base while testing measures your reflexes, not the product.
2. Have a second person grade the answers. Whoever configured the agent should not be the one deciding whether its answers were good.

Grade every answer on two axes: correct or not, and complete or not. Keep the wrong answers in a separate list with the exact question that produced them.

Step 4: Read the results by intent, not in aggregate

Build a simple table.

Now the useful questions become answerable. Which high-volume intent has the worst resolution rate? That is your first content project. Where do wrong answers cluster? That is your first guardrail. Which intents resolve well but escalate anyway? Those are usually over-aggressive escalation rules, and they are cheap to fix.

Step 5: Separate the content problem from the product problem

Before you conclude that a product underperformed, check whether the answer existed in your content at all. For each failed question, mark one of three causes:

• Content gap. The answer was nowhere. Not the product's fault.
• Retrieval failure. The answer existed and the agent missed it. That is the product.
• Judgment call. The question had no single right answer. That belongs to a human.

In most first benchmarks, content gaps dominate. Teams that fix their top ten gaps before re-running the study often see a bigger improvement than they would have gotten by switching vendors.

Step 6: Re-run on a schedule

A benchmark is only useful as a trend. Keep the test set, add new intents as they appear, and re-run it monthly or quarterly. Track the same numbers every time.

The first run tells you where you stand. The third run tells you whether the work you did between them was worth doing.

What good looks like

Rather than chasing a headline percentage, judge your program on four things:

• Resolution improving in your highest-volume intents, not just overall
• Wrong answers trending to near zero
• Content gaps closing faster than they open
• Escalations arriving with enough context that human handle time drops

Those four move together in a healthy support operation. A resolution rate that climbs while wrong answers climb with it is not progress, it is risk.

The takeaway

Benchmarks you did not run measure conditions you do not have. Two weeks of structured internal measurement will tell you more about your automation than any published figure, and it gives you something a vendor comparison never will: a prioritized list of what to fix next.