Bots for Business

Tracking These KPIs Can Help Your Chatbot Succeed

What KPIs are important for improving chatbot performance? By tracking these KPIs, you can discover what’s working for your users—and what isn’t.

At first, the Scout chatbot was only an idea. Developers wanted to build a chatbot that helped retailers curtail shopping cart abandonment, but their aspirations were no more than, well… aspirations. The idea hadn’t been proven to work.

But when the numbers came in—numbers that showed a 55 percent recovery rate versus a 10 percent recovery rate using traditional tactics—Scout’s creators knew that the bot was a success. Of course, they couldn’t have seen those numbers without monitoring the right KPIs.

Although data on chatbot performance can show you where to direct your development efforts, KPIs are essential for parsing that data efficiently. Instead of taking shots in the dark, you can monitor KPIs to pinpoint areas of success and weakness, perform the requisite adjustments, and continue on the path that’s working. It’s how you make sure your bot is a success.

Why KPI metrics matter for chatbots

The story of Scout underscores an important fact about chatbots. At their core, chatbots are utilitarian tools. No matter how magnetic your bot’s personality, how well it anticipates wild-card user queries, or how fast it operates on the backend, ultimately you need to be sure that it’s successful at whatever you’ve designed it to do.

Success, of course, is relative and depends on your unique business goals, technology objectives, and customer experience benchmarks. A chatbot designed to help bank customers avoid long phone calls has different success metrics from one designed to take a hamburger order. And to effectively gauge each bot’s success, you have to monitor the right KPIs.

For the banking bot, one KPI might be user responses to an exit question. The bot might ask, “Was I able to solve your problem today?” Users’ responses will help you track the performance of the chatbot with respect to its primary purpose. For the fast food bot, measuring the time elapsed between order initiation and payment confirmation might be a KPI. Although this KPI wouldn’t be relevant for banking (there’s no order being placed, after all), it may be the most important one for the fast food bot.

KPIs matter for chatbots for the same reason that they matter for other interactive technologies. If a chatbot isn’t performing adequately, you need to make adjustments. But to make the right adjustments, you’ve got to know why and under what conditions the bot isn’t performing adequately. KPIs can help you get there, but you have to choose the right ones.

Chatbot KPI examples

Although the KPIs you select should always align with your chatbot’s specific performance objectives, there are several KPIs (or classes of KPIs) that are relevant across industries. These highly common and consequential KPIs are a good place to start:

Activation rate

This KPI refers to the percentage of users who decide to engage with (or activate) your bot. Is it important? Definitely. But is activation rate, or AR, the most important KPI to monitor, the end all, be all of your chatbot’s success? Well…

For starters, let’s affirm that you definitely want people to use your bot. Determining the point at which someone has activated the bot, however, is another matter entirely. When Skyscanner’s travel chatbots exceeded one million unique interactions in early 2018, what did that entail for its activation rate? Was every interaction an activation, or did each user have to actually extract valuable information from the bot? Or would activation only occur when users chose to use the bot instead of taking some other action?

Although only Skyscanner can provide the answers, the questions themselves speak volumes about the lack of clarity around activation rate as a KPI. You might say that a user has activated your bot when he or she:

• Opens a chat window.
• Responds to a prompt or introductory call to action (CTA).
• Asks a first question.
• Reads a message from the chatbot.

When monitoring activation rate, just be sure that everyone making decisions around that KPI has a mutual understanding of what it entails for your bot.

Length of session

What’s the ideal length of a chatbot session? Long enough to solve the user’s problem and short enough to prevent them from giving up.

Ideal session length varies by industry. A bot that helps you place a burrito order has shorter session lengths than one that exists for pure entertainment. Regardless, monitoring this KPI can show whether you’re meeting users’ needs.

To optimize session length, Botanalytics, a chatbot analytics provider, recommends onboarding chatbot users by:

• Telling them up front that they’re conversing with a chatbot, not a human.
• Starting with a call to action that encourages users to interact.
• Setting realistic expectations by telling users what your bot is capable of.

Botanalytics explains that, by onboarding users in this way, you’re less likely to disappoint them. They won’t abandon your bot before it’s had a chance to solve their problem.

Conversion rate

Most chatbots have a specific, tangible end goal. That being the case, it almost goes without saying that one of your KPIs should relate to conversion. Of the people who use your chatbot, what percentage do what it’s designed to make them do?

As a chatbot KPI, conversion rate is most informative when you can compare it to conversion or goal completion across other modes of customer communication—or to your other chatbots that are designed to deliver a different kind of experience. At Inbenta, according to company CEO Jordi Torras, analysts compare the rates of online sales and appointments set by human agents to those converted by chatbots. That gives them a sense of how well their chatbot investments are supporting company growth.

Exit question responses or ratings

This is the same KPI mentioned earlier for the hypothetical banking chatbot. As the conversation comes to a close, your chatbot can ask users a simple yes/no question about their experience. For example:

Bot: Did I manage to answer all your questions today?

User: No

Bot: Sorry about that. If you have a moment, please let me know how I can improve.

If the user answered, “Yes,” that would be that. You can also turn your exit question into a rating by requesting that users choose between thumbs up/thumbs down emojis, happy/sad emojis, or a numerical rating between 1 and 10.

Fallback response frequency

If you’ve ever told anyone, “I’m sorry, I didn’t understand you,” you know what a fallback response is. But when your chatbot does it—and then does it again and again—it’s bound to frustrate your users.

The frequency of fallback responses is a KPI that shows how well your bot interacts with your audience. Broadly speaking, the fewer the number of fallback responses, the better your bot’s performance.

Retention rate

The more frequently people come back to use your bot, the greater your retention rate. In terms of significance, retention rates vary from largely irrelevant to absolutely essential.

For example, retention might not tell you much about the performance of a chatbot designed to help someone buy flowers. If the average consumer only buys flowers once every few months, retention might not be high. No big deal. You can gauge performance based on other metrics, like conversion rate and exit question ratings.

But what if you’re Forksy, a chatbot that helps users track their daily caloric intake? To be effective, bots like these require repeated, habitual user engagement. If your chatbot has similar requirements, retention is almost certainly going to appear in your KPI roster. In the absence of high retention, bots like Forksy might not help users achieve their weight-loss goals. If that’s happening, you want to know about it.

Industry-specific KPIs

Some KPIs are useful across all industries. Activation rate is one example—you need to know the percentage of users who decide to converse with your chatbot. But certain KPIs are more industry-relevant than others.

Customer service

Exit responses and user ratings can show whether your bot is actually solving users’ problems. Customer service, after all, is about service. If your bot can’t help somebody access an account statement or request a refund, it’s important to find out why.

Fallback response frequency matters, too. Remember that transferring the user to a live representative is a form of fallback response. If that’s something your bot does, you want to monitor it.

Retail/e-commerce

Conversion and goal completion are the obvious ones here. When the purpose of your bot is to sell more wares, the conversion rate shows whether it’s succeeding or failing in that regard.

Session length comes into play, as well, especially if you notice that your bot is taking a long time to determine what users are asking and if it’s resorting to fallback responses more often than you’d like.

Professional services

Not to be confused with customer service bots, these are chatbots designed to connect people with professional service providers. From HVAC contractors to attorneys to insurance agents, these chatbots are designed to qualify leads, set appointments, and request information. Like retail bots, conversion rate and fallback response rate are important. Just be sure to compare your chatbot conversion rate to the conversion rate for other online lead-generation tactics, like web forms.

Financial/banking

Several KPIs may be useful for financial bots. Exit ratings, length of session (the shorter the better, for customer service), and fallback response rate all matter.

Retention rate can be a big one, as well. Just consider Eno, Capital One’s chatbot that texts customers to confirm purchases, remind them of upcoming bills, and prevent fraud. When your bot is designed to engage users about their finances on an ongoing basis, monitoring retention won’t be optional—it might be your most important KPI.

Health and fitness

Staying fit is a lifelong endeavor. So, too, is maintaining a healthy diet. And keeping our minds focused. And feeling happy.

Given the ongoing, continuous nature of health and fitness, retention rate is a KPI you want to monitor for any chatbot built for this niche. Whether your chatbot reminds people to exercise, greets users each morning with a meal plan, or helps them design and follow a fitness regimen, it’s only successful when it’s regularly used.

Fallback response rate is also important, but retention may be your biggest focus. The greater your retention, the more certain you can be that users are engaged.

KPI reporting

Now that you know which KPIs to monitor, you need an efficient way to capture the data and view it in an easily digestible, readily available format. Luckily, there are several chatbot analytics tools out there that can help you do just that.

Solutions like Dashbot and Botanalytics help you track all of the KPIs just discussed, along with others that may be relevant to your industry. With a chatbot-specific analytics solution, you quickly get a sense of the queries and responses most common among your users. You may want to start adjusting your bot’s dialogue in ways that hadn’t even occurred to you before!

Unless you’re using a solution-specific bot, like a Facebook Messenger chatbot, look for a tool that works with any chatbot solution. Ideally, it will include a machine learning component (so that your chatbot gets smarter over time) and will provide conversation transcripts. If you’re already developing custom funnels based on user intent or other chatbot metrics, an analytics solution might help you do so more efficiently. Look for one that supports funnels and helps you analyze them.

The goal of chatbot KPI reporting, of course, is to discover the best ways to help users. After pulling the numbers, you can determine whether your chatbot is succeeding or whether adjustments are in order.