In one of our previous articles Going Multilingual: Bot Expansion Strategies, we explained how you can use machine translation to build a multilingual chatbot with only one Master solution.
Now, we´re going to explain how you can integrate a machine learning translation service to further improve your solution and help to cover more languages if your chatbot is required to scale across different regions.
Get your Machine Translation API
Teneo Studio itself does not contain any Machine Translation service. You need to purchase it and set up the API in your Teneo solution. The mainstream Machine Translation services, such as Google Cloud Translate or Microsoft Azure Translator, usually require knowledge of their Cloud Platform. If you are not familiar with these Cloud platforms and you only need the translation service, Rapid API is a good choice. In this article, we are going to show you how to set up a Machine Translation service in your solution via Rapid API.
First of all, you need to create your account in Rapid API, and choose one Machine Translation API in the API hub which best meets your needs according to the supported languages, pricing plan and maximum calls per month. In this article we are using the Microsoft Translator Text API. This API includes both language detection and translation, which lets your chatbot speak all the languages that Microsoft Translator supports with only one solution built in Teneo Studio. Open the Pricing page and subscribe to one of the plans according to the expected usage of your chatbot.
After subscribing to this API, you need to go to the Endpoints page to find the essential information you need, such as request URL, host, key, and the structure of the request body. On the right-hand side you can find the Code Snippets. In java you have many different packages containing methods related to API calls. In this article we are going to use the code in Java > java.net.http, which is very easy to be adapted to the groovy code you will need in Teneo Studio. If you are more familiar with other packages, feel free to choose the one you are most comfortable with.
Set up Machine Translation API in Teneo Studio
Now let us set up the Machine Translation API in the solution in Teneo Studio. First, we need to create the following global variables:
-
sSolutionLanguage: The language of your solution. Use ISO 639-1 language codes, in this case, en.
-
sUserLanguage: The language of the expected user input, which is detected automatically so you can put an empty string as initial value.
-
translationApiHost: The API host, which can be found in the header parameters list or code snippets in the Endpoints page in Rapid API
-
translationApiKey: The API key, which can be found in the header parameters list or code snippets as well
Second, define the method of translation in the Global > Scripts > Solution loaded.
import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import groovy.json.JsonBuilder
import groovy.json.JsonSlurper
class TranslationApi {
public static String translate(String apiHost, String apiKey, String input, String source, String target, Map extraParams = [:]) {
def apiUrl = "https://" + apiHost + "/translate?api-version=3.0&from=" + source + "&to=" + target + "&profanityAction=" + (extraParams.get("profanityAction")?extraParams.get("profanityAction"):"noAction") + "&textType=" + (extraParams.get("textType")?extraParams.get("textType"):"plain")
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(apiUrl))
.header("content-type", "application/json")
.header("X-RapidAPI-Host", apiHost)
.header("X-RapidAPI-Key", apiKey)
.method("POST", HttpRequest.BodyPublishers.ofString(new JsonBuilder([["text":input]]).toString()))
.build()
HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString())
def parsedResponse = new JsonSlurper().parseText(response.body())
return parsedResponse.translations.text[0][0]
}
public static String detect(String apiHost, String apiKey, String input) {
def apiUrl = "https://" + apiHost + "/Detect?api-version=3.0"
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(apiUrl))
.header("content-type", "application/json")
.header("X-RapidAPI-Host", apiHost)
.header("X-RapidAPI-Key", apiKey)
.method("POST", HttpRequest.BodyPublishers.ofString(new JsonBuilder([["text":input]]).toString()))
.build()
HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString())
def parsedResponse = new JsonSlurper().parseText(response.body())
return parsedResponse.language[0]
}
}
For translating the user input to the language of the solution, add the following line in the Global > Scripts > Pre-processing:
sUserLanguage = TranslationApi.detect(translationApiHost,translationApiKey,_.userInputText)
if(!sUserLanguage.equals(sSolutionLanguage)) _.userInputText = TranslationApi.translate(translationApiHost,translationApiKey,_.userInputText,sUserLanguage,sSolutionLanguage)
The code above first detects the language the user is speaking and store it in the variable sUserLanguage, then translate it to your solution’s language. For translating the output text to the user’s language, add the following line in the Global > Scripts > Post-processing:
if(!sUserLanguage.equals(sSolutionLanguage)) _.outputText = TranslationApi.translate(translationApiHost,translationApiKey,_.outputText,sSolutionLanguage,sUserLanguage)
Now the machine translation API is successfully added to the solution. The following screen shot of the Tryout panel shows that the chat bot understands and responses in user’s language, while the solution is in English.
Let us check the processing of this dialogue from the Advanced Tryout. You can see that on the input side, the translator first detects the language of the user input as Spanish, then translates it to English. While on the output side the original output is in English, and the translator translates it back to the user’s language Spanish.
Profanity filter
Microsoft Translator API provides you with profanity filter which can help you delete or mark with asterisks the profanity words in the user input. You can add an extra argument ["profanityAction":"Marked"] or ["profanityAction":"Deleted"] to activate this functionality in the code for translating user input in the Pre-processing script, for example:
if(!sUserLanguage.equals(sSolutionLanguage)) _.userInputText = TranslationApi.translate(translationApiHost,translationApiKey,_.userInputText,sUserLanguage,sSolutionLanguage,["profanityAction":"Marked"])
Let’s try an input with profanity: Hola gilipollas, then check the result:
You can see that the profanity word gilipollas is replaced by asterisks during translation of the input. If you add ["profanityAction":"Deleted"] as the extra argument, the profanity word will be removed like this:
HTML translation
HTML translation is another important functionality Microsoft Translator API provides. If your solution contains outputs with HTML elements, adding ["textType":"html"] in the code for translating output in the Post-processing script can make sure that all the html tags will be kept during the translation, for example:
if(!sUserLanguage.equals(sSolutionLanguage)) _.outputText = TranslationApi.translate(translationApiHost,translationApiKey,_.outputText,sSolutionLanguage,sUserLanguage,["textType":"html"])
Let’s check the result:
You can see that all the html tags remain untouched, while the texts in the title and body have been translated form English to Spanish.
Google Translate API setup
Besides Microsoft Azure Translator, you can also use the Google Translate API in your solution with very similar setup. Just change the value of the global variable translationApiHost to google-translate1.p.rapidapi.com, and slightly modify the class TranslationApi in Global > Scripts > Solution loaded:
class TranslationApi {
public static String translate(String apiHost, String apiKey, String input, String source, String target) {
def apiUrl = "https://" + apiHost + "/language/translate/v2"
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(apiUrl))
.header("content-type", "application/x-www-form-urlencoded")
.header("Accept-Encoding", "application/gzip")
.header("X-RapidAPI-Host", apiHost)
.header("X-RapidAPI-Key", apiKey)
.method("POST", HttpRequest.BodyPublishers.ofString("q=" + URLEncoder.encode(input, "UTF-8") + "&target=" + target + "&source=" + source))
.build()
HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString())
def parsedResponse = new JsonSlurper().parseText(response.body())
return parsedResponse.data.translations.translatedText[0]
}
public static String detect(String apiHost, String apiKey, String input) {
def apiUrl = "https://" + apiHost + "/language/translate/v2/detect"
HttpRequest request = HttpRequest.newBuilder()
.uri(URI.create(apiUrl))
.header("content-type", "application/x-www-form-urlencoded")
.header("Accept-Encoding", "application/gzip")
.header("X-RapidAPI-Host", apiHost)
.header("X-RapidAPI-Key", apiKey)
.method("POST", HttpRequest.BodyPublishers.ofString("q=" + URLEncoder.encode(input, "UTF-8")))
.build()
HttpResponse<String> response = HttpClient.newHttpClient().send(request, HttpResponse.BodyHandlers.ofString())
def parsedResponse = new JsonSlurper().parseText(response.body())
return parsedResponse.data.detections.language[0][0]
}
}
Please note that the Google Translate API on Rapid API does not support profanity filter and HTML translation.
Conclusion
This article gives an introduction on how to implement Microsoft Azure translator and Google Translate via Rapid API to make your solution multilingual. If you are using other machine translation APIs, we encourage you to share your use case in the forum to help other Teneo developers. We hope you found this article useful, and feel free to ask here any questions you might have on this topic.