Chatbot Challenge #3 Maintenance and Optimization
Many articles talk about building a chatbot in hours, or days. What they don’t tell you is that building a chatbot is an endless journey, even with a chatbot software or framework. I didn’t believe it when I was interviewing for my current job, which is to build a chatbot for internal employees in a big corporation. I asked my director, with a third-party tool, building a chatbot should only take a few months. What do I do after it? Well, after two years in the job, maintaining that chatbot is still my main project and there are still a lot of things to do.
Here are the common tasks involved in maintaining and optimizing a chatbot and why they are challenging:
1. Promoting the Chatbot
After releasing the chatbot, the first thing to do is to announce it to the users and tell them what it can do. Every time the chatbot has new features, we need to communicate with the users as well.
One dilemma is we want the users to ask questions within the chatbot’s scope. However, we don’t want them to only ask questions that the chatbot can answer, and we don’t want to tell them the exact questions to ask. The reason is we want to know what the users want to know or achieve via the chatbot so that we can add more content and functions to it. We also want to learn our users’ word and sentence choices and use them to train the chatbot.
Another challenge is many users don’t pay attention to the marking stuff, no matter what channel we use to communicate with them. And for some users, even they have found the chatbot, they don’t bother to click on it or try it, they prefer to stick to the old ways.
Top 4 Most Popular Bot Design Articles:
2. Adding More FAQs and Intents
Adopting agile methodology, it is common to launch a chatbot with a few most popular FAQs or intents first, and then gradually adding more content and functions to it.
The more FAQs we have, the harder to keep track of everything. To add a new FAQ, we need to make sure it is not in the knowledge base already. With hundreds of FAQs, it is not that easy to tell.
Also, we need to consider whether adding new FAQs will mess up the search experience of existing FAQs. The more FAQs the bot handles, the longer the searching is. Sometimes several FAQs share the same keyword. When users search that keyword, only some FAQs show as candidates and the most important one is not in it.
Finally, some chatbot platforms, like Dialogflow, only support a limited number of intents (2000) and entities (250). For a pizza shop chatbot, 2000 intents are more than enough, but for an Amazon Web Services Q&A bot, with more than 16000 FAQs right now, it is nearly impossible to use Dialogflow for it. If your chatbot tool has a hard limit on the FAQ or intent amount, it may be challenging to keep them under the limit.
3. Adding More Access Points
It is also common to launch the chatbot at one website or messaging system first, then add more widgets to other channels.
Different systems (e.g. Facebook Messenger, Google Assistant) require different response formats (e.g. different JSON structures).
The widgets at different access points might have different looks (e.g. web vs desktop vs mobile widget), which means different JavaScript and CSS code.
It is also possible that the chatbot has both “search mode” and “chat mode”. If so, the response designed for the “search mode” might be different from the response for the “chat mode”. In “search mode”, the response can be a bit longer and more comprehensive, while in “chat mode” the response needs to be shorter and more specific. Therefore, for different widgets, we might need to define multiple variants of the answer/response for each knowledge article/intent.
If the chatbot requires user information (e.g. user id, name, phone number, email) to get the user-specific responses, different systems might provide different user information. Thus the fulfillment functions or API calls need to be different to accommodate different parameters the target systems can provide.
Finally, If each widget has different stages like production, staging, and dev, the number of widgets to maintain could easily be doubled or tripled.
4. Updating Existing Content
Things change! We need to constantly update the existing content based on new policies, procedures, tools, features, and links, etc. and remove the outdated ones. In addition, based on user interactions, training examples need to be reviewed and revised to optimize user intent recognition.
It is very likely that we need to work with domain experts from different teams to update the content, which means the update might be delayed.
Also, it is possible that the chatbot is not the only knowledge management system. Syncing the existing chatbot content with other knowledge management systems or resources (PDF files, knowledge articles in ticketing systems, etc.) is sometimes more challenging than adding new contents to the chatbot. For example, how to map intents or knowledge articles from one system to another? How to make sure the change in one system will trigger the same change in another system?
5. Monitoring User Interactions
By analyzing the user interactions with the chatbot and the users’ feedback, we can get data about the most or least popular questions, the questions with no answers, the questions with wrong answers, the user intents that are recognized incorrectly, the terms and phrases that need to be added to train the user intents, etc.
However, if the chatbot has thousands of interactions every day, it is nearly impossible to manually review all of them.
6. Measuring the Impact of the Chatbot
To prove that the chatbot worth the effort and the money, it is important to measure the business impact or return on investment of the chatbot. In an ideal world, we can easily compare the volume of tickets, emails, and phone calls of a question before and after we add it to the chatbot and tell the stakeholders how much time and money the chatbot has saved for the company as well as for the end-users.
However, the world is not always ideal. Some companies don’t track the number of tickets, emails, and phone calls. Some companies do have the data, but the grouping can be fuzzy. Also, some questions are seasonal (or temporary) and do not have a steady volume. Therefore one cannot tell whether the chatbot has reduced the inquiry volume without analyzing the user interactions of the chatbot. However, in the chatbot, grouping user interactions by questions or topics are not that straightforward, since user queries can be matched to wrong FAQs and user intents might be recognized incorrectly.
7. Learning New Features and New Chatbot Tools
The scale of the chatbot may change. It might start as a simple FAQ search, and turns into a conversational bot. It might be text-only at first, but later the voice module is added. Also, we want to keep improving the performance of our chatbot. Therefore, we need to keep learning new features of existing chatbot software/frameworks while looking for other software/frameworks that might work better.
New features, tools, and technologies for chatbot come up constantly. It takes a lot of time and effort to keep up with them. Attending demos, workshops, webinars, comparing plans and prices, tool selection takes a lot of time. The continuous learning of a tool also takes a lot of time.
8. Keep an Eye on Competitors’ Chatbot
Chances are, your chatbot is not the only one, either inside or outside your company. Internally, multiple teams may utilize chatbots to achieve various tasks. While externally, your company’s competitors are also doing similar things. It is possible that the scope and priority of different chatbots are different, so the competition is not necessarily direct. However, if your competitors' chatbot has a nice feature that yours don’t have, guess what your users will do and what your boss will say?
Depends on how many competitors you have, it will cost some time to keep track of them to stay ahead of the game.
Final Thoughts
Based on my experience, building a chatbot is much more than spending a couple of hours to put together a quick demo. If not maintained carefully, for example, the information is not updated or there is no proper escalation channel, a chatbot might cause more confusion and dissatisfaction than without a chatbot. Building and maintaining a chatbot also need cross-team collaboration. Depends on what system the chatbot needs to integrate with, what content and functions need to include, how to communicate with the end-users, support from many other teams are needed. Many tasks and challenges discussed in this article could be trivial on a small scale. However, to scale up without adding too much cost, a lot of automation and integration are needed.
I hope this article has provided some useful information for you to calculate the budget and recruit the people you need before initializing the chatbot effort. Thanks for reading!
Don’t forget to give us your 👏 !
