After you launched your own project developed on Teneo Studio, it is very important to analyze the real dialogue sessions to check out the performance of you chatbot using Teneo Query Language (TQL), which is a proprietary query language designed for retrieving information from the Teneo Engine log data.
Although TQL query is only applied to log data, we suggest you start thinking about it in the development phase. In this article, we will provide you with 10 most useful TQL queries covering those you must know about your project, including how many conversations per month, what are the user inputs that the chatbot has failed to understand, etc.
Number of sessions/transactions per month
The monthly traffic is the most basic data you must know about your chatbot. It helps you understand how many dialogues your chatbot have made. If you find the number is lower than expected, or keep decreasing every month after launched, you may need to encourage the potential users to use your chatbot and investigate if your chatbot provides bad user experience to make users stop using it.
Query 1: Counting the number of sessions in last month
ca s.id: s.beginTime == in {"now-1M/M"}
You can use a very similar query for counting the number of transactions by replacing s.id with t.id:
ca t.id: s.beginTime == in {"now-1M/M"}
In many cases, the user may just open the chatbot and then close it without saying anything. These sessions without user input may not generate any value to your business. Thus, it is also very important know how many sessions have real user inputs. You can do this by the following query:
Query 2: Counting the number of sessions with user input in last month
ca s.id: t.e.userInput != "", s.beginTime == in {"now-1M/M"}
This query counts the number of sessions where there is at least one transaction containing a non-empty user input. The same as Query 1, you can replace s.id with t.id to count the number of non-empty transactions:
ca t.id: t.e.userInput != "", s.beginTime == in {"now-1M/M"}
If your chatbot has huge amount of traffic every month which can slow down the queries, you can count the sessions and transactions weekly by changing {“now-1M/M”} to {“now-1w/w”}. Read this article: Periodic usage counting queries for more details about these queries. The time restriction is usually added to all kind of queries, so in this article we will not add it in all the queries below. If you need to copy these queries, remember to add an adequate time restriction according to the traffic of your chatbot.
Querying business-relevant dialogues
Most of the projects developed on Teneo Studio contain small-talk flows from Teneo Dialogue Resources and may receive many chit-chat dialogues which have nothing to do with the business. In this case, to know how much value your chatbot has created, it is very important to know how many dialogues are related to your business within all dialogues in the Log Data.
In order to query business-relevant dialogues easily, we suggest you create one unique folder called Project (or any other name you prefer to differentiate it from small-talk and test flows) in the root directory and put all flows which are related to your project in it.
Now you need the following query to count the dialogues in which any of the flow in the Project folder has been triggered at least once.
Query 3: Counting the number of business-related sessions:
ca s.id: t.e.folder ~= "Project.*"
if you have already created several project related folders in the root directory, you can either move them to one unique folder, or you can change the constraint to t.e.folder ~= "(ProjectFolder1|ProjectFolder2|ProjectFolder3).*" (you can add more folders if needed). Click here to learn more about constraints in TQL.
Besides the counting, the business-related inputs, and the flows that they have triggered are also frequented queried by Teneo developers in order to know the if the business-related flows are working as expected. The following query shows you how to do this:
Query 4: Listing all business-related inputs with the flow they triggered:
lu t.e1.userInput, t.e2.fname:
t.e2.pathType=="flow-trigger",
t.e2.folder ~= "Project.*"
The result of this query will be a table containing the user input and the flow names it triggered:
Please remember that the property userInput and fname (representing flow name) are two properties that belong to different events, so you need to use e1 and e2 to differentiate them; while the properties fname, pathType and folder belong to the same event, so they share the same notation e2. You can click here to learn more about query constructs.
Analyze Safetynet logs
Safetynet is a flow that all projects should have to cover the user input that the chatbot cannot understand. Analyzing inputs that have dropped to Safetynet is very important for developers to know the efficiency of your chatbot and to find out new knowledge areas that are worth to be added to the chatbot.
Let’s first started with the basic query for counting the Safetynet input.
Query 5: Counting the number of SafetyNet input:
cu t.id: t.e.pathType == "flow-trigger", t.e.fname ~= "(?i).*safety\s?net.*"
Please note that this query assumes that your Safetynet flow is named safetynet or safety net (case insensitive). If this is not the case (although we recommend that you keep the flow name as Safetynet even if your solution is not in English), you may need to change the constraint t.e.fname ~= "(?i).*safety\s?net.*" to f.e.fname == "SAFETYNET_FLOW_NAME" (replace SAFETYNET_FLOW_NAME with the name you use for Safetynet flow) in the query.
Now you may also need to know what user inputs have triggered the Safetynet flow. The following query will extract such inputs.
Query 6: Listing all user inputs that have triggered Safetynet:
lu t.e1.userInput: t.e2.pathType == "flow-trigger", t.e2.fname ~= ".*[s|S]afety\s?[n|N]et.*"
Another thing you may find very useful is to know the context where the Safetynet has been triggered by querying the flow that came before it. This helps you to know under which situation you often receive inputs that are not covered by the current knowledge of the bot.
Query 7: Counting each flow that precedes the Safetynet:
d t1.e.fname:
t1.e.pathType=="raise-flow",
t1.e-{pathType=="raise-flow"}>t2.e,
t2.e.fname ~= ".*[s|S]afetynet.*"
order by count desc
The query above will generate the following table, in which you can find which is the most frequent flows that lead to Safetynet. According to the result, you can decide which flow needs to be improved (for example the output text of this flow contains information that is usually interpreted incorrectly) in order to avoid triggering Safetynet in the future.
Querying most and less frequent knowledge areas
When your chatbot has been in service for several weeks, it happens a lot that some of the knowledge areas are very popular among the users while some of them are hardly triggered. Knowing such information can help you distribute your time and effort focusing on those flows which are very often triggered while doing maintenance.
The following query will show you the frequency of each flow. Usually, we are less interested in this type of analysis in greeting, small-talk and Safetynet flows, so the constraint for Querying business-relevant dialogues is added here as well.
Query 8: Counting each flow which is business-relevant
d t.e.fname:
t.e.pathType=="flow-trigger",
t.e.folder ~= "Project.*"
order by count desc
You can also replace the desc with asc in the query above to let the less frequent flows show first. The following table shows an example of the query result in descending order:
Querying user feedback
User feedback question or user survey is essential for Teneo developers to have a view of the quality of the chatbot. They are usually appended to the response text after a business-related flow has been finished, and there are many ways to append the feedback questions according to different kinds of frontends. Here is an example of feedback question implemented in the Teneo Web Chat frontend.
User’s choice here will be captured by a specific flow in which provides response for positive and negative feedback. In order to be easily queried by TQL, a common way is to add Metadata (created in Solution > Globals > Metadata) to the output node like this:
If user provides negative feedback, developers usually ask the user to choose a reason of unsatisfaction and store it in another Metadata.
You can also get the similar effect by using Augmenters, which requires knowledges in groovy. However, we suggest you use Metadata here as it is a simpler and more intuitive approach.
Now let me show you how to query user feedbacks using the Metadata. The following query shows the distribution of positive and negative feedback:
Query 9: Showing distribution of positive and negative feedback
d t.e.md:FEEDBACK_RATING :
t.e.md:FEEDBACK_RATING ~= ".+"
order by count desc
This query will give you the following table, showing the counts of positive and negative feedback:
And you need the following query to show the reasons of negative feedback:
Query 10: Counting each reason of negative feedback
d t.e.md: FEEDBACK_REASON :
t.e.md:FEEDBACK_REASON ~= ".+"
order by count desc
This query will show you the counts of all negative feedback reasons provided by the user:
Conclusion
In this article we provide you with 10 must-have TQL queries for any project developed in Teneo Studio, covering different query commands (counting, listing and distribution) and commonly used query constructs (session id, user input, flow name, etc.). We encourage you to share more queries that can be considered must-haves for all kinds of projects. We hope you found this article useful, and feel free to ask here any questions you might have on this topic.