Chatbot Development, Chatbots
Chatbot KPIs to Track for Success and Improved Performance
Wondering which KPIs for chatbots are important for improving performance? By tracking these KPIs, you can discover what’s working for your users—and what isn’t.
June 1, 2021
At first, the Scout chatbot was only an idea. Developers wanted to build a chatbot that helped retailers curtail shopping cart abandonment, but their aspirations were no more than, well… aspirations. The idea hadn’t been proven to work.
But when the numbers came in—numbers that showed a 55 percent recovery rate versus a 10 percent recovery rate using traditional tactics—Scout’s creators knew that the bot was a success. Of course, they couldn’t have seen those numbers without monitoring the right key performance indicators (KPIs).
Although data on chatbot performance can show you where to direct your development efforts, KPIs to measure chatbot results provide essential information for parsing that data efficiently. Instead of taking shots in the dark, you can monitor KPIs to pinpoint areas of success and weakness, perform the requisite adjustments, and continue on the path that’s working. It’s how you can help make sure your bot is a success.
Why KPI metrics matter for chatbots
The story of Scout underscores an important fact about chatbots. At their core, chatbots are utilitarian tools. No matter how magnetic your bot’s personality, how well it anticipates wild-card user queries, or how fast it operates on the backend, ultimately you need to be sure that it’s successful at whatever you’ve designed it to do.
Success, of course, is relative and depends on your unique business goals, technology objectives, and customer experience benchmarks. A chatbot designed to help bank customers avoid long phone calls has different success metrics from one designed to take a hamburger order. And to effectively gauge each bot’s success, you have to monitor the right chatbot KPIs.
In the banking sphere, one KPI for chatbots might be user responses to an exit question. The bot might ask, “Was I able to solve your problem today?” Users’ responses will help you track the performance of the chatbot concerning its primary purpose. For the fast-food bot, measuring the time elapsed between order initiation and payment confirmation might be a KPI. Although this KPI wouldn’t be relevant for banking (there’s no order being placed, after all), it may be the most important one for the fast-food bot.
Using KPIs to measure chatbots is important for the same reason that it’s important for other interactive technologies. If a chatbot isn’t performing adequately, you need to make adjustments. But to make the right adjustments, you’ve got to know why and under what conditions the bot isn’t performing adequately. Chatbot KPIs can help you get there, but you have to choose the right ones.
Chatbot KPI examples
Although the KPIs you select should always align with your chatbot’s specific performance objectives, there are several KPIs (or classes of KPIs) that are relevant across industries. The following highly common and consequential KPIs are a good place to start.
This KPI refers to the percentage of users who decide to engage with (or activate) your bot. Is it important? Definitely. But is activation rate (AR) the most important KPI to monitor—the end all, be all of your chatbot’s success? Well…
For starters, let’s affirm that you want people to use your bot. Determining the point at which someone has activated the bot, however, is another matter entirely. When Skyscanner’s travel chatbots exceeded one million unique interactions, what did that entail for its activation rate? Was every interaction an activation, or did each user have to extract valuable information from the bot? Or would activation only occur when users chose to use the bot instead of taking some other action?
Although only Skyscanner can provide these answers, the questions themselves speak volumes about the lack of clarity around activation rate as a KPI. You might say that a user has activated your bot when he or she:
- Opens a chat window.
- Responds to a prompt or introductory call to action (CTA).
- Asks the first question.
- Reads a message from the chatbot.
When monitoring activation rates as a KPI for chatbots, be sure that everyone making decisions around that KPI has a mutual understanding of what it entails for your bot.
Length of session
What’s the ideal length of a chatbot session? The ideal length is long enough to solve the user’s problem and short enough to prevent them from giving up.
Ideal session length varies by industry. A bot that helps you place a burrito order has shorter session lengths than one that exists for pure entertainment. Regardless, using the length of the session as one of the KPIs to measure a chatbot's success can show whether you’re meeting users’ needs.
To optimize analytics from session length, Inform Communications, a customer service provider, recommends in a few key ways:
Use session length to identify failed interactions by checking those that fall outside your optimal length to see what went wrong.
Use session length as an alarm—for example, increases in session length outside the optimal window, users interacting with your bot differently, or a potential issue to be resolved.
After you and your organization have found the optimal length of the session for your bot to operate with, variations and anomalies become much clearer, allowing you to use this KPI for chatbots to better understand your customers and their concerns.
Most chatbots have a specific, tangible end goal. That being the case, it almost goes without saying that conversion rate is one of the key KPIs to measure the chatbot’s actual success. Of the people who use your chatbot, what percentage meet the goal of the chatbot (do what the chatbot is designed to make them do)?
As a chatbot KPI, the conversion rate is most informative when you can compare it to conversion or goal completion across other modes of customer communication—or compare it to your other chatbots that are designed to deliver a different kind of experience. At Inbenta, according to company CEO Jordi Torras, analysts compare the rates of online sales and appointments set by human agents to those converted by chatbots. That gives them a sense of how well their chatbot investments are supporting company growth.
Exit question responses or ratings
This is the same KPI mentioned earlier for the hypothetical banking chatbot. As the conversation comes to a close, your chatbot can ask users a simple yes/no question about their experience. For example:
Bot: Did I answer all of your questions today?
Bot: Sorry about that. If you have a moment, please let me know how I can improve.
If the user answered Yes, that would be that. You can also turn your exit question into a rating by requesting that users choose between thumbs up/thumbs down emojis, happy/sad emojis, or a numerical rating between 1 and 10.
Fallback response frequency
If you’ve ever told anyone, “I’m sorry, I didn’t understand you,” you know what a fallback response is. But when your chatbot does it—and then does it again and again—it’s bound to frustrate your users.
Although some developers are working to reduce the banality of current fallback responses, the frequency of these responses is one of the KPIs you can use to measure chatbots on how well they interact with your audience. Broadly speaking, the fewer the number of fallback responses, the better your bot’s performance.
The more frequently people come back to use your bot, the greater your retention rate. In terms of significance, retention rates vary from largely irrelevant to essential.
For example, retention might not tell you much about the performance of a chatbot designed to help someone buy flowers. If the average consumer only buys flowers once every few months, retention might not be high. No big deal. You can gauge performance based on other metrics, like conversion rate and exit question ratings.
But what if you’re Forksy, a chatbot that helps users track their daily caloric intake? To be effective, bots like these require repeated, habitual user engagement. Among the other KPIs for chatbots, the retention rate is particularly important for bots requiring daily engagement. In the absence of high retention, bots like Forksy might not help users achieve their goals. If that’s happening, you want to know about it.
Some KPIs are useful across all industries. Activation rate is one example—you need to know the percentage of users who decide to converse with your chatbot. But certain KPIs are more industry-relevant than others.
Exit responses and user ratings can show whether your bot is solving users’ problems. Customer service, after all, is about service. If your bot can’t help somebody access an account statement or request a refund, it’s important to find out why.
Fallback response frequency matters, too. Remember that transferring the user to a live representative is a form of fallback response. If that’s something your bot does, you want to monitor why and how often it happens.
Conversion and goal completion are the obvious ones here. When the purpose of your bot is to sell more wares, the conversion rate shows whether it’s succeeding or failing in that regard.
Session length comes into play, as well, especially if you notice that your bot is taking a long time to determine what users are asking and if it’s resorting to fallback responses more often than you’d like.
Not to be confused with customer service bots, these are chatbots designed to connect people with professional service providers. From HVAC contractors to attorneys to insurance agents, these chatbots are designed to qualify leads, set appointments, and request information. Like retail bots, conversion rate and fallback response rate are important. Just be sure to compare your chatbot conversion rate to the conversion rate for other online lead-generation tactics, like web forms.
Several KPIs may be useful for financial bots. Exit ratings, length of session (the shorter, the better for customer service), and fallback response rate all matter.
Retention rate can be a big one, as well. Just consider Eno, Capital One’s chatbot that texts customers to confirm purchases, remind them of upcoming bills, and prevent fraud. When your bot is designed to engage users about their finances on an ongoing basis, monitoring retention won’t be optional—it might be your most important KPI.
Health and fitness
Staying fit is a lifelong endeavor. So, too, is maintaining a healthy diet, keeping our minds focused, and feeling happy.
Given the ongoing, continuous nature of health and fitness, the retention rate is a KPI you want to monitor for any chatbot built for this niche. Whether your chatbot reminds people to exercise, greets users each morning with a meal plan, or helps them design and follow a fitness regimen, it’s only successful when it’s regularly used.
Fallback response rate is also important, but retention may be your biggest focus. The greater your retention, the more certain you can be that users are engaged.
Now that you know which KPIs to monitor, you need an efficient way to capture the data and view it in an easily digestible, readily available format. Luckily, there are several chatbot analytics tools that can help you do just that.
Solutions like Dashbot and Botanalytics help you track all the KPIs just discussed, along with others that may be relevant to your industry. With a chatbot-specific analytics solution, you quickly get a sense of the queries and responses most common among your users. You may want to start adjusting your bot’s dialogue in ways that hadn’t even occurred to you before.
Unless you’re using a solution-specific bot, like a Facebook Messenger chatbot, look for a tool that works with any chatbot solution. Ideally, it will include a machine learning component (so that your chatbot gets smarter over time) and will provide conversation transcripts. If you’re already developing custom funnels based on user intent or other chatbot metrics, an analytics solution might help you do so more efficiently. Look for one that supports funnels and helps you analyze them.
The goal of chatbot KPI reporting, of course, is to discover the best ways to help users. After pulling the numbers, you can determine whether your chatbot is succeeding or whether adjustments are in order.