How Regression Testing will improve your bot
Chatbot testing can be really complex and confusing. Therefore, regression testing gains high importance for chatbots in particular. Let us see what exactly regression testing is and how do we implement it for chatbots.
What is regression testing?
Regression testing is re-running functional and non-functional tests to ensure that previously developed and tested software still performs well after a change. In simple words, we keep a record of certain test cases required to test a software. After making any change to the software, we verify if the existing functions and performance are not impacted.
Why is regression testing important?
Regression testing is usually done during the maintenance phase. As a software system ages, the cost of maintaining the software increases. To ensure that the software keeps performing optimally, we need to do regression testing after every modification made. It is used to determine if the modification impacted the existing workflows or created new errors. Improving the regression testing process of a software helps reduce the cost of software development and maintenance considerably.
What does regression testing mean for chatbot?
Chatbots evolve continuously, with addition of a new training phrase, the underlying natural language model gets updated. The updated model might not understand some phrase, which was understood earlier. Hence, it is important to keep track of all test cases and run them after changes are done. Consider a banking chatbot that supports customer support for FAQs, like checking account balance and asking for the statement of credit card. Now the bank wants to add the feature of paying their credit card bill using the chatbot. To add this feature, there will be new intent and entities added to the chatbot. The entire NLP model for the chatbot would be built again, it may so happen that the chatbot might fail for some user inputs which used to work well earlier. Consider the user says “show credit card bill” earlier this would return the bill amount if the model is incorrectly trained this might fail as paying credit card bill intent would have similar words. We would have to run a regression test to ensure there is no overlap in this case.
There are two approaches to implement regression testing for any software, which we can extend to chatbots
Full regression
In this approach, we do a complete product regression test using relevant test cases and test cases modified as per the current product. This is generally done when changes are till the root of the software. In case of chatbot which is simpler (5–10 intents), it is better to run a full regression testing after adding every new training phase or intent as it helps to find the bugs soon, effectively. In the case of complex chatbot (more than 10 intents), it might not be possible to run after every change. The number of test cases or workflow might be too large for complex chatbots. But it should be done periodically preferably twice a week to ensure all the workflows are working correctly.
Top Articles on How Businesses are using Bots:
1. Why Messenger is the Ideal Bot Platform for Now
2. What I learned in making three chatbots for small businesses
Partial regression
In this approach, we test the modified part of a product and the adjacent areas that might have been affected. It is important to single out the application areas that might be affected by recent changes and select relevant test cases from the test suite. Considering the above example of adding pay credit card bill option to the chatbot, we can run the test cases concerning credit card and skip the others related to account statement, net banking queries, etc. This selection technique significantly reduces testing time and effort. It also helps identify and discard obsolete test cases.
Using the above two approaches, we can either test chatbots manually or automatedly.
Manual
Manual regression testing is the basic method for regression testing for every product. This type of testing always precedes automation, in some cases, it could be the only option. For example, it’s impossible to write test scripts for certain responses from chatbot or certain UX changes in the chatbot. Today chatbots support a lot of UI elements like cards, images, etc.which is difficult for a script to test. To ensure bot works as expected, we would manually chat with the chatbot and document these cases, then re-run it for every modification. This is a time-consuming process and constant efforts are made to automate it. There are like a hundred ways to say the same thing and covering all would be tough.
Automation
Automation reduces the efforts and time spent on tedious and repeatable tasks and spares time for testing that requires the human opinion, such as exploratory testing and UX testing. A way to automate chatbot test would be to code test script to run against all your test cases. There are certain products in the market that help you automate the regression testing of your chatbots. One such product is Record and Run tool by Zypnos, it is a very simple-to-use tool. You can record your test cases and run them easily to find out the cases failed.
Chatbots need to be constantly learning. The bot should be able to understand the users and help them complete their task effectively. The only way to do this is to keep testing and making the model better.
I hope the above article helps you make your chatbot better. If you have any queries or require help regarding chatbot testing, please feel free to reach out to me on Twitter or LinkedIn. Will be happy to help.
