Intent Recognition beyond Confidence Scores
“Sorry, I did not understand your question. Can you phrase your request in a different way?”.
It’s the response we all dread when talking to a chatbot and it’s the last thing a conversational designer would want users to see when engaging with their solution.
That’s exactly why so much work is put into the Natural Language Understanding (NLU) components of conversational AI implementations, CAI specialists want their solutions to offer real help and support.
The process of creating a truly responsive solution involves testing and tweaking the performance of their Machine Learned model, enhancing the match requirements on triggers and transitions, as well as other quantitative and qualitative QA processes, so it’s not so easy.
Anybody who has worked with conversational AI, machine learning and NLU knows that getting intent recognition right can be a difficult task. When your intent recognition runs into issues, you may try to resolve it by adding more data to your ML model and this may resolve some issues. However, the experienced conversational AI developer will also know that machine learning is an algorithm and that adding more data does not guarantee resolution. It may help somewhere, but next time you modify the model, the algorithm may send you back to square one on previously fixed issues.
It’s a real problem but one that can be avoided in Teneo as CAI developers have more tools to resolve and control the behavior of the NLU in the long run.
Firstly, developers can look to use TLML syntax to create advanced match requirements that combine class match requirements and language conditions. This can be a very efficient way of dealing with intents that will be triggered by almost identical inputs, or to pick up simple keyword-based inputs.
Secondly, control over intent recognition can be taken one step further with what is called the Trigger Ordering, and that is what we will explore in this article.
What is Trigger Ordering?
In Teneo, intent recognition can happen in two places, Intent Triggers and Transitions.
Intent Triggers are the ones that tell Teneo what kind of inputs should activate a flow. Transitions have a similar role, but within an already active flow where Match Intent transitions determine what path of the flow the dialogue should follow. In this article we will focus on the role of the Intent Triggers in the NLU process.
When an input is passed to Teneo, the Teneo Engine will start looking for a suitable trigger in the solution. The triggers are ordered as illustrated below, and the Engine will start evaluating at the top. When the Teneo Engine finds a trigger where the Match requirements are met, it will stop the input trigger evaluation process and proceed to find the flow that should be activated and the answer that should be returned.
In other words, trigger ordering is how your triggers are ranked, and, together with Match Requirements, it is used by the Teneo Engine to determine the trigger that will be matched. How your triggers are ordered is therefore an essential part of how the input will be processed and what flow will be activated.
How can you configure Trigger Ordering?
The rank of a trigger in the overall trigger ordering can be defined in two places:
- On the trigger itself
- In the Trigger Ordering panel
When you add a trigger to your flow you have the option to define the desired Order Group in the Triggering properties of the trigger
Our example project makes use of seven order groups. The order groups are ways of organizing your triggers into smaller groups of triggers with similar characteristics, and this is the first step in controlling what trigger is matched based on the Trigger Ordering.
The general recommendation is to put the triggers with more precise/demanding Match Requirements in the higher ranked order group, as the Teneo Engine will start by evaluation Group 1 before continuing to Group 2, etc. In other words, the order of the order groups is absolute.
For more advanced options, you will go to the Intent Triggers’ Ordering window which can be found in the top ribbon of Studio Home tab.
Here you can see where in the ordering all Intent Triggers in your solution are placed, you can manage your Order Groups and much more.
One of the features that you can use to fine-tune the matching of your triggers even further can be found inside the order groups, namely relations among triggers. On occasions, you may have multiple triggers in the same order group that will compete for the same user inputs. In those cases, you will want to consider adding a relation between the two triggers to make sure that the more important trigger gets a first shot at the input.
In other words, inside an order group, the ordering among the triggers can be relative when you need it.
Why even worry about Trigger Ordering?
Now we have seen the tools that you have to define the Trigger Ordering in your Teneo solutions, and at this point you might wonder why you should even worry about trigger ordering. Why not simply let the match requirements handle the intent matching process? Good match requirements can by no means be ignored, and they will always be your starting point for a powerful NLU. Nonetheless, here are some reasons why using Trigger Ordering will make your NLU setup even stronger:
Triggers with different levels of precision all mixed up = potential trouble
Theoretically, all your triggers can coexist in the same order group, but the likelihood that this can lead to undesired trigger matching is a real risk.
Imagine you sell coffee like our Longberry Baristas bot. Your customers can ask different questions about coffee, and you have also implemented an Area SafetyNet with a simple Language Object Match requirement in case the user only inputs a keyword or you fail to understand the full intent. If all these triggers are placed in the same order group, what may very well happen is that the Area SafetyNet trigger fires on many more inputs than you intended to.
So, by following the principles of storing triggers in different ordering groups according to the precision of their match requirements, you can make sure that this situation is controlled, and that the Area SafetyNet only picks up on the inputs that you were actually not able to understand in a more specific Intent Trigger.
Avoid overengineering on the NLU part
By ordering your triggers by level of precision, you can save time and efforts on refining your NLU layer. On one hand, you can spend less time on retraining your ML model to try to make it detect one class instead of another, with the implicit risk of creating an unbalanced dataset.
On the other hand, you can avoid complex language conditions that try to cover all potential ways of expressing a certain intent, or, on the contrary, negated language conditions that explicitly look to avoid that certain words or phrases are present in the input.
A suggestion would be to have one order group for advanced triggers at the top and another for the triggers with only a class match requirement lower in the ordering group ranking.
Design the desired UX
Should all users get the same response and go through the same process, or do you adapt according to contextual information that you have about the user, the channel, previous interactions with the bot…? Imagine that you are a utilities company and one of your customers has previously given you a notice that he/she wants to change to the competitor. If your bot knows that, next time this customer talks to the bot you may want to adapt the user experience to try to revert the situation. By having one, more specific trigger that is tested first and kicks in when the contextual information is available, and another trigger for the general approach, you can make your bot a better ambassador for your brand.
Give preference to certain, business critical intents
Another typical usage is to give preference to certain business critical intents. Imagine the case of an airline or a train operator where climate conditions or labor strikes may affect the normal business processes. In those scenarios, the company may want to take the user down a different path in case they ask for a refund or a rebooking directly related to the current business interruption, whereas somebody looking for general information about refunds and rebookings will still be able to get the standard reply.Again, this could be managed by having two triggers where the trigger that also looks for mentions of a business interruption would get higher priority than the regular one.
Fit in the small talk intents without interfering with your business intents
Even if your bot is not going to be chit-chatting away with your users, some level of general dialogue capabilities is required in any conversational AI implementation. Greetings, Acknowledgements, SafetyNet – all of these are part of the Teneo Dialogue Resources and can be plugged into your project. You want to make sure that they do not interfere with your business processes. A user that says “Hello, my name is Andy, I would like to order a large flat white” should be taken to the coffee order process and not to a greeting intent. If you stick to confidence score only to determine what trigger should be matched, small talk flows may interfere with business-critical conversations and mistakenly take the users into the wrong conversation.
Summary
Trigger Ordering is one of the ways the Teneo Developers can control what Intent Trigger is matched, and the triangle Machine Learning - TLML Syntax – Trigger Ordering is a very powerful toolkit that puts the control over the intent matching process in the hands of the developers, as they do not have to depend only on machine learning and confidence scores to determine what flow process is kicked off.
Tells us in the comments, what do you think is the absolute superpower of Trigger Ordering? What kind of situations do you think it solves well?