Conversations with Persistent Memory
Introduction
A published Teneo solution is made available as a Java web application. When a user talks to a virtual assistant (VA), it interprets each user input in the context of previous inputs. In this sense a VA is a stateful application whose conversation state is maintained via the session mechanism of the web container the Teneo engine (the VA backend) is deployed in. By default, the session times out after 10 minutes of user inactivity. Increasing it is not advisable as it can lead to a significantly increased number of concurrent sessions and more costs.
There are scenarios, however, where a VA should “remember” the dialog state much longer, so the user can go back to that same dialog some hours, days, weeks or even months later. Increasing session timeout is not an option in this case. Moreover, depending on the requirements of the hosting setup, Teneo engine can be shut down and started up again in some situations, and the dialog state is not kept in these cases either. Thus, application restarts lead to interrupted dialogs, just like waiting beyond the session timeout does.
To summarize it: when a session expires or the application is restarted due to specific hosting requirements, the dialog state is lost. The data collected during the dialog is only stored in the dialog logs for analysis and statistics tools.
So the question is how to maintain the dialog state beyond the session timeout and across application restarts. A short answer to this question is, even thought the dialog state itself cannot be maintained, we can store relevant information outside of the session and re-use it for the next interaction with the user. To achieve it, this data should be stored somewhere outside Teneo engine sessions.
Teneo Engine setup
In the commonly used production setup the VA backend consists of several - usually two or three - independent Teneo engine instances placed behind a load balancer. A client software (a frontend or a customer-specific middleware) talks to the load balancer, which then decides which Teneo engine instance the request should go to. Once a session is started with one particular Teneo engine instance, it also continues with it (session stickiness). Which Teneo instance the load balancer chooses to start a session with should be seen as random. In this setup the different Teneo engine instances are independent of each other, can be hosted on different physical or virtual machines and don’t share any data between themselves.
Storing dialog data externally
If we want to persist dialog data across session timeouts and application restarts, we should store it externally. It can be done either directly in a database or other storage service accessible from all the Teneo engine instances or indirectly via the middleware or the frontend. In either case the data should be linked to a specific user’s dialog. Implementations of both approaches and of the data-user link are highly project-specific, and Teneo is flexible in this respect. It is up to the conversational AI developer to implement the mentioned data access by using in-solution Groovy scripts which have the full power of the Java SE API, including adding any third-party Java libraries if needed. Depending on the approach you choose, you might need the following:
- a database or some other storage service shared by all the Teneo instances,
- adjustments to the frontend and/or the middleware to keep relevant user-related data across different sessions.
Re-joining an interrupted dialog
Whether a dialog has been interrupted by a session timeout or through an application restart, resuming the dialog means starting a new session from scratch. Even though there is no universal way to restore any dialog to the state the dialog was in when it was interrupted, it is still possible to add to the newly-started dialog the stored data by submitting it in additional request parameters or by fetching it from a database or from some other service the data was stored in. How this data is incorporated into a new session will largely depend on the dialog flow structure and on the project requirements.
Let’s consider an example of a flow to book flight tickets for a trip. The user is first asked to explain the full travel plan. Then the VA asks the user to provide a date for each flight, collects this information and books the flight tickets in the end of the dialog. Even though this example is simplified (details unrelated to persisting the dialog information are left out), it illustrates how key dialog data can be persisted across session timeouts and application restarts. If the dialog session is interrupted, the user can still get back at a later point to the logical step where s/he left off.
In what follows, the dialog elements in the picture and the explanations of these elements are marked with the same numbers in parenthesis, for example (1), (2), etc, so it is easy to see which description corresponds to which element.
Steps
(1) A user says s/he wants to book a trip.
(2) VA verifies if there exists any persisted booking data for this user from previous sessions. It can be done in different ways:
If the data persistence is handled by the frontend or middleware, VA can receive it in request parameters, for instance
airports = TravelDataParser.parse(engineEnvironment.getParameter('airports'));
flights = TravelDataParser.parse(engineEnvironment.getParameter('flights'));
where TravelDataParser.parse() stands for your project-specific travel data parsing.
If the data is persisted in an external storage and is not passed in via the client software, the VA should be able to identify the user. It can be done via user ID, for example. This ID is then used to get the data from your external storage, for instance
sUserId = engineEnvironment.getParameter('userId');
airports = ExternalStorageAccess.getAirportsByUserId(sUserId);
flights = ExternalStorageAccess.getFlightsByUserId(sUserId);
Both the procedure to read the user ID and the implementation of the ExternalStorageAccess class are project-specific. You will need a database or some other data storage and you will have to implement access to it via static methods of this class.
Assume in both cases airports is a List of airport codes and flights is a List of planned flight dates. There are two possible alternatives to continue the dialog: (3) and (4).
(3) No persisted booking data is found. It corresponds to the condition flights.size()==0. In this case, VA proceeds to ask the user about his/her travel plan in (9).
(4) A previous booking attempt was interrupted by session timeout or application restart. It corresponds to the condition flights.size()>0. VA proceeds in this case to (5).
(5) VA show the bookings from the previous session to the user and asks him/her if s/he wants to continue with them or drop them and start everything from scratch. If s/he wants to start from scratch, (6), VA proceeds to (7) to delete the stored data. If s/he wants to continue, (8), VA proceeds to (13).
(7) The stored data in cleaned up. It can be done in different ways depending on the project-specific implementation.
If the data persistence is handled by the frontend or middleware, Teneo engine can return an output parameter to the client software (middleware or frontend) indicating the corresponding data should be deleted. If the data is persisted in an external storage and not passed via the client software, it can be deleted similar to how it is read in (2), for instance:
airports = ExternalStorageAccess.clearAirportsByUserId(sUserId);
flights = ExternalStorageAccess.clearFlightsByUserId(sUserId);
After that, VA goes on to ask the user about his/her travel plan (9).
(9) VA asks the user about his/her planned travel itinerary. Let’s assume for simplicity the user is supposed to indicate just the list of the airports. Once VA received the user input, it goes on to analyze it in (10).
(10) VA analyzes the list of the airports mentioned by the user, assigns it to the airports variable and stores it persistently. The procedure to store it is project-specific.
If the data persistence is handled by the frontend or middleware, VA can return it in an output parameter to the client software (middleware or frontend). If the data is persisted in an external storage and not passed via the client software, it can be stored similar to how it is read in (2), for example:
ExternalStorageAccess.storeAirportsByUserId(sUserId, airports);
If less than two airports are indicated, (12), the conversation is dropped by proceeding first to (14) to tell the user too few airports have been entered and then to (20) to clean up the stored data. If two or more airports have been mentioned, (11), VA goes on to ask the user for the desired date for each travel segment one by one looping (13), (15) and (16). So, if the user said s/he wanted to travel from New York (JFK) to Madrid (MAD), then to Stockholm (ARN) and then back to New York (JFK), VA first asks the user to indicate the date for the flight from JFK to MAD, then the date for the flight from MAD to ARN etc till the dates for all the parts are collected. When the user indicates a date, it is stored.
(13) The user is asked to enter the date for the current flight.
(15) The date is detected and stored in the flights variable. If the data persistence is handled by the frontend or middleware, VA can return it in an output parameter to the client software (middleware or frontend). If the data is persisted in an external storage and not passed via the client software, it can be stored similar to how it is read in (2), for example:
ExternalStorageAccess.storeFlightsByUserId(sUserId, flights);
After that, VA checks if all the flights have been provided with their date, which would correspond to the condition
flights.size() == airports.size() - 1
If it is true (17), VA tells the user it will book these flights (18), books them (19) and proceeds to (20) to delete the stored data and to quit the flow. If it is false, i.e. there are still flights without date, step (13) is repeated.
If during any of the iterations of the loop (13-15-16) the session is lost because of timeout, application restart or any other cause, the (partial) booking data will still be preserved so this user will be able to just fill the missing data in a new session, without having to explain the travel itinerary and enter the already indicated travel dates again.
Please note that depending on the user experience you want to achieve, Skip Conditions and Prompt Triggers might also be interesting options for the design of the flows.
Conclusion
We hope this example has been useful for illustrating the persistent memory approach.
Do you have any questions? Have you designed already such an approach in your project? Let us know in the comments.