Handling Disambiguation with Teneo
Virtual Agents (VA) are exposed to new challenges daily, new users with new requests and different conversational styles. Conversational AI Developers keep that in mind during the development process. The goal is to help the VA understand the user intent, as accurately as possible, to be able to provide the right answers and services.
The challenge in understanding user intent lies in unclear user requests. Users may send in requests that are irrelevant, unclear, or ambiguous, which may lead to inaccurate answers from the bot, and consequently result in a frustrating user experience.
In a different article, we present ways to handle off-topic requests, by building flows that activate When Things Get Out of Scope. In another article, we describe the issue of irrelevance and explain how to Handle Irrelevance with Teneo. In this article, we present the topic of ambiguity and discover ways to handle that with the Teneo Platform.
How Can You Prepare your Bot for Ambiguous User Inputs?
Ambiguity in user input implies that the request may have different interpretations, and therefore, be linked to multiple intents. There are cases where the user would send in unclear requests, such as, “ticket”: thinking he/she will be assisted in booking a ticket. However, from the point of view of a VA, in most cases more specific information will be required to be able to narrow the intent down to ticket booking as there can be many other possible intents related to the word ticket, such as ticket canceling, rebooking a ticket, confirming a ticket…etc.
Preparing your bot to disambiguate ambiguous user inputs is essentially making sure that your end user gets directed to the right place in your content. Let’s explore a couple of ways of identifying ambiguity in a user input.
Confidence Score of Classified Input
In Teneo, training data is labeled with intent classes during development. Additionally, for each solution, a confidence score threshold is defined to ensure accurate intent selection. When the user sends in a request to the virtual agent, the machine learned model returns a predicted intent and the corresponding confidence score. In a simple scenario where the confidence score of the intent with the highest confidence score, also known as the top intent, is above the limit of the confidence threshold, the predicted class will be considered in the trigger matching process.
Teneo, however, will sometimes return more than one class annotation for a single input. That happens when the machine learned model was uncertain about which intent was the best match, and this could be an indicator of an ambiguous input. By calculating the delta between the annotated classes, you may identify cases where the difference is so small that those cases could be candidates for a disambiguation where you want to configure the bot to ask more questions before providing an answer, or where you display a selection of options that may be relevant to the topic for the user to select from.
In Teneo Studio, you can use the Advanced Tryout to test potential user input. You will have a view over the classification result and the confidence score for each class. Below is an image of the classifications of a very simple input, “coffee”. We can see that two rather low scoring classes have been returned by the Teneo Classifier. This could indicate a lack of data in the classifier, but also an ambiguous user input, as we see in the example.
You can find more information on how Teneo annotates intents here: The Teneo Predict Input Processor.
Linguistic Conditions to Identify Relevant Keywords
When the user input is either too short to be mapped to a specific machine learned intent, or so unclear that it is only partially understood by the bot, a different approach can be taken to disambiguate the user’s intention with the request.
The image below shows an example of a user request consisting of one word. This type of input is called a keyword request. Keyword requests are important because, unlike irrelevant words, keywords are usually in the scope of the bot knowledge. However, they need to be disambiguated to be able to give a specific response to the user.
As you can see in the classification details in the image, the input got annotated with two classes and both with a low confidence score. With the default confidence threshold in Teneo, this input will fail to match any class-based trigger. However, with a TLML syntax-based trigger, the input can be picked up and led into the desired disambiguation flow instead of falling into the Safetynet as in the first example below.
In the second example, the VA fails to understand the specific intent of the user but thanks to a trigger with a linguistic condition it manages to catch the central word “macchiato” as a synonym of coffee, and this allows the VA to initiate a disambiguation of the intention of the user request.
How Can You Disambiguate the Input in Teneo?
In a manner similar to human conversations, where a person would have to ask follow-up questions to disambiguate unclear phrases , conversational AI should be designed to ask questions if the user request is unclear, or if it has more than one interpretation .
In the previous section, we saw how the virtual assistant can identify ambiguous user input. In this section, we explore the strategies that can be used to disambiguate unclear user intent .
Conversational Design with Linked Flows
As you design the dialogues of the conversational AI, you might find certain topics being closely related or easily mixed-up. One of the methods that can be used is creating a disambiguation flow that is triggered when the user sends in a request that can be linked to multiple intents . This flow would in turn guide the user through possible topics related to the input.
In Teneo, you can prepare such disambiguation flows to Handle Ambiguous Input by linking to other flows that handle specific intents. The image below is an example of using linked flows in Teneo. The disambiguation flow is activated when a user sends in requests with unclear intents, such as “Coffee”. Then the bot response would be a follow-up questions to disambiguate the request: does the user want to order a coffee, or see the selection of coffees available? Depending on the user’s response, the flow branches out to either of the two possible sub-flows: “Order”, or “Selection”.
This design is very helpful when the list of possible options can be predefined as it allows you to create a very controlled and context-close handling of the response to the disambiguation question. Imagine that the user responds “the latter”, instead of “see selection”. By adding TLML Syntax Match requirements on the transitions you can easily prepare your bot to identify these inputs and direct the user to the proper coffee related flow.
Furthermore, it can easily be combined with buttons and other UI options when the VA interacts with the user in a channel with graphical support, which we will get into later in the article.
Open Interaction with the User to Disambiguate
A flow design with linked flow is however not always going to be an option: the list of potential relevant flow links is too long or your VA does not have visual support to help display the options. In those cases, you need to design your VA to be able to handle the disambiguation with an open question.
One important point here is that your bot should take the information that has been shared so far in the conversation into account, in other words the bot should know the conversation context. Teneo offers many great ways to work with context, such as global contexts which can be used as context restrictions in follow-ups. The Teneo Dialogue Resources provide predefined context restrictions such as “Follow up of flow” or “Follow up of Folder” that will help you configure your contextual follow-up handling in combination with a TLML Syntax match on your trigger.
In the above example, if user states “I want to buy one” as a follow up, the bot should be aware that the current topic is “coffee” and trigger the corresponding flow even if no flow link is in place. In the coffee order flow, this could be the implementation:
Optimize the User Experience when Disambiguating
How to achieve the best user experience when doing a disambiguation via your bot depends also on the channel you are using and the functionalities that are available on this channel. Each frontend tends to have some differences, and a voice channel requires also a different approach. Let’s take a look.
Use Quick Messages in Chat Conversations
You can style your bot messages to be more user friendly and to ensure smoother user interaction by providing suggestive options that the user can select with the help of buttons or other clickable elements.
If you are using Teneo Web Chat (TWC) as your frontend, you can check out the Messaging types in Teneo and select from the various messaging styles supported. The image below is an example of how buttons can look like using TWC.
Other channels with graphical support offer similar features, such as the Quick Replies in Facebook Messenger.
Disambiguation in Voice Bots
In terms of user experience, a voice channel needs to be handled differently than our previous example since there is no visual interface in front of the user in which we could display the buttons of the suggested flows. It is a good idea in this case to state clearly to the user what has been understood, and ask an open question for getting the needed information. For example, if the user just tells us on the phone “contract”, our voice bot could confirm and request more info with “I understand you have a contact-related question, can you give me more info on what you would like me to help you with?” Teneo can make use then of the “contract” context for the matching process and give a detailed response after this follow up.
Strategies of disambiguation are used to enable conversational AI solutions to be more accurate with their answers and services. But if you’re still having issues, why not tell us about your conversational AI project? Is it well-prepared to disambiguate user input? What are the most challenging user requests sent to your bot that you found troublesome to interpret? Share your experience, knowledge, and questions on our Teneo Community Forum.