Intent Disambiguation Setup

[chunlin.wang]

In a previous article, (Handling Disambiguation with Teneo), we highlighted how Intent Classifier in Teneo Studio can return more than one class annotation for a single input.

This article describes an approach to deal with such inputs for which the classifier cannot identify a strong top intent. More specifically, for some inputs, the top ranked intent classes have similar confidence scores, suggesting that there are several classes competing for the first position.

For such inputs, it is advisable to provide the user with alternative interpretations, letting the user select the correct intent among two or more options. This will reduce the number of misunderstandings and the users’ response can be used to annotate inputs which in turn can be fed back into the classifier to incrementally improve its quality.

The approach presented also deals with situations where the top intent has confidence score below the class match threshold (default value is 0.45) and the Safetynet has been triggered. The examples below outline the approach in more detail.

Multiple intents with similar confidence scores

In the following example, there are two or more intents with similar confidence scores and the difference between those is below a custom set threshold. Hence, there is no obvious “winner” and instead of providing a response, the user is asked to select between two or more alternatives.

• User: I want to book a flight to see a lovely dog in London.
• Classification: <SEARCHING_FOR_FLIGHTS 0.40, TRAVELLING_WITH_A_PET 0.30>
• Bot: Is your request about:
  • searching for flights
  • traveling with a pet

One class with low confidence score

In this example, the confidence score of the top class is below the default threshold (0.45) and Safetynet response is provided with the low confidence intent presented as an alternative. The reasoning behind this this approach is that it is better to let the user know that the system is unsure about its interpretation than giving the wrong response. In this way, the user is not entering a possible wrong flow, but can enter the suggested flow if it was the correct prediction.

• User: Hi there I want to know the fastest way to go to London?
• Classification: <SEARCHING_FOR_FLIGHTS 0.38>
• Bot: I am sorry I didn’t get that. Is your request about:
  • searching for flights

Implementation overview

The approach is implemented by means of a helper class (DisambiguationHelper) that can be uploaded into your solution under Resources, five global variables, a global listener and a flow with a script condition trigger and a prompt trigger. The global listener creates an instance of DisambiguationHelper which stores the annotated intents (one or multiple depending on the result). DisambiguationHelper comes with two methods:

• hasAlternativeIntents : this method returns true or false depending on if alternative intents should be presented or not when Safetynet was triggered.
• hasSmallDifference : this method returns true only when two or more intents have a very small difference (lower than a custom set threshold set up in DisambiguationHelper) in confidence score and all of them are below the confidence threshold for class match.

The approach is general and can be implemented in any solution that has classes in the Class Manager. The suggestion list is built dynamically based on how many alternatives are available. The manual work needed to implement the approach (except for copying and pasting the available resources described above) is to create a list with the class names as a global variable, together with two number type global variables for thresholds used in the code. Not all intents should necessarily be included here, only those for which disambiguation is motivated. For example, it may not be worthwhile to disambiguate between or suggest non-business relevant intents such as the TDR.

Implementation instructions

• Create a groovy file named DisambiguationHelper.groovy (or any other names you prefer) containing the code in this Code Snippet and upload into your solution under Resource, use /script_lib as Published Location. You can also add the code to the global script Solution loaded if you do not want to create a resouce file.

• Add the following line of code to the global script Pre-matching:
  disambiguationHelper = new DisambiguationHelper(binding)

• Add the following line of code to the global script On top:
  sFlowName = _.getThisFlow().getName()

• Add the following line to the global script On drop:
  disambiguationHelper.prevFlowName = sFlowName

• Add the following Global variables, which you need to change the content according to the classes you want to disambiguate in your own solution.

  1. intentsForDisambiguation: a list containing the class names of all the intents to be included in Intent disambiguation. For example:
     
  2. diffThreshold: a number indicates the threshold that the difference between the confidence score of an intent and the confidence score of the top intent should be lower than in order to be considered as intents having small difference.
  3. minThreshold: a number indicates the threshold that the confidence score of an intent should surpass in order to be considered a possible choice.
  4. sFlowName: a variable with empty string as initial value. It temporally saves the flow name of the last-triggered flow, used in the Global scripts On-top and On-drop.
  5. disambiguationHelper: a variable used in the Global listener to create an object of the DisambiguationHelper class. It should have null as initial value.

• Create a disambiguation flow with the following structure by implementing the following steps:
  

  

  1. Add a new flow. It should contain an Intent trigger and an output node by default.

  2. Choose the Intent trigger, add a Script match requirement with the following condition: disambiguationHelper.hasSmallDifference()
     Note that the Script match trigger need to be at top in the trigger ordering (or at least higher than the triggers using the intents included in the list intentsForDisambiguation).

  3. Add a Prompt trigger with the following expression: disambiguationHelper.hasAlternativeIntents()

  4. Add a script node and name it Build intent suggestions (or any other name indicates its functionality), then set it as the starting node by clicking on the Set Start Node button from the ribbon bar.

  5. If you are using Teneo Web Chat, add a Flow variable named JSON with empty string as initial value, and the following code in the script node “Build intent suggestions”:

     // Create clickable list
     def clickableList = [:]
     clickableList.type = "clickablelist"
     clickableList.list_items = []
     for (intent in disambiguationHelper.intents){
         def sIntentName = intent.getName().replace(".INTENT","").replace("_"," ").toLowerCase()
         def item = [:]
         item.title = sIntentName.capitalize()
         item.postback = sIntentName.capitalize()
         item.parameters = ["intent_disambiguated":intent.getName().replace(".INTENT","")]
         clickableList.list_items << item
     }
     
     JSON = new groovy.json.JsonOutput().toJson(clickableList)
     

     The code above will create several buttons corresponding to the intents which are waiting for disambiguation. If you are using other frontends, you can also create something similar to realize this functionality.

  6. Connect the script node to the output node and add an answer text to encourage the user choose one of the intent from the list, for example Is your request about:. If you are using Teneo Web Chat, add an output parameter named teneowebclient (obligatory) with value ${JSON} as well. See the screenshot below as an example:

     

• Add a Script match trigger in every flow that is included in the intent disambiguation options. If you use the code in the step above, please add the following code as the Evaluation Script (replace INTENT_NAME with real intent name in your Class Manager): engineEnvironment.getParameter("intent_disambiguated")&&engineEnvironment.getParameter("intent_disambiguated")=="INTENT_NAME"
  This trigger should be at top in the trigger ordering. See the screenshot below as an example:

  

Threshold setup

It is very important setting up an adequate threshold in this approach. Here is advice for the threshold setup:

• If you are using the classifier provided by Teneo Studio, the reasonable number for diffThreshold and minThreshold is usually between 0.1 and 0.2. Please be aware that not all classes will be included in the annotation. Check this page about how intents are annotated during Input Processing.
• If you are using Conversational Language Understanding (CLU) of Microsoft Language Studio, you will probably have in the average higher confidence scores than with a setup based on the Teneo Classifier. A reasonable minThreshold in this case might be greater than 0.8, and the diffThreshold could be around 0.03.
• After you set up the initial value of diffThreshold and minThreshold, please always remember to adjust them by analyzing more testing data or real user input. If it happens that the intent disambiguation flow has been wrongly triggered frequently, please consider changing these two thresholds as soon as possible.

Intent disambiguation examples in Teneo Web Chat frontend

The following screenshot shows an example of multi-intents with similar confidence score in Teneo Web Chat. The classifier still needs to learn from more data and is therefore not sure if the user is asking about selecting the seat for a booked flight or searching for a flight, so both options has been provided. Once the user clicks on one of them, the corresponding flow will be triggered:

The following screenshot shows an example of providing an alternative intent when the SafetyNet has been triggered. The classifier is lacking data and is therefore not sure if the user is really asking for booking a flight, so after the fallback answer is provided, the booking flight intent is also provided as a possible alternative. Once user clicks the button, the corresponding flow will be triggered:

Conclusion

This article gives an introduction on how to implement intent disambiguation according to the classification results from Teneo Classifier. This approach can easily be adapted to other classifiers by adjusting the diffThreshold and the minThreshold. If you are using intent disambiguation in your solution, we encourage you to share your use case in the forum to help other Teneo developers. We hope you found this article useful, and feel free to ask here any questions you might have on this topic.