Skip to content

Getting started with AI speech translations in live streams

This tutorial explains how to configure and manage a live stream with multilingual AI generated speech translations (synthetic voices).

These can be combined with AI-generated closed captions, as well as Remote Simultaneous Interpretation and human real-time subtitling. Although these workflows are largely similar, we recommend reading the respective tutorials.

The speech translations are based on an AI speech-to-text conversion. By default, they are generated automatically, without human intervention. To increase accuracy, you can have the speech-to-text source edited in real-time by a human corrector (see the speech-to-text correction manual).

This tutorial applies to enterprise and webinar plans. Skip step 1 for webinars, since an event is automatically created when you create a webinar.

1 Creating an event

Select the Live > Events menu. On the events page, press the Create Multilingual Event button in the sidebar. In the popup dialog, enter the following info and press Create:

  • an event name (for your own information only)
  • select one of the broadcast protocols with support for AI.
  • select the origin of your broadcast
  • select the Default Language. This is the language being spoken in the live stream. Currently, this must be a real language: 'Original' is not (yet) supported for speech translations.
  • select the Latency. Choose 'Default Latency' for (slightly) more AI accuracy and more time for human correctors (if you use them), or 'Low Latency' for a shorter delay.
  • Optionally, select the Streaming Resolutions. Clevercast does cloud transcoding for adaptive bitrate streaming. Make sure the highest resolution doesn’t exceed the resolution in which you are broadcasting.

2 Configuring the event

When you press the Create button, Clevercast creates the event and redirects you to its detail page. Among other things, the event page allows you to do the following:

  • Copy the event’s embed code from the Management tab to your website or 3rd party platform.
  • On the Appearance tab, you can upload a poster image and show a countdown timer in the embedded player (among others)

To add AI Interpreter languages to the event, go to the Audio Languages tab and select the speech translations that you want to make available to your viewers.

For each AI Interpreter language, a closed caption language is automatically created on the ‘Caption languages’ tab (if not already present). Clevercast uses the text of the closed captions to generate the synthetic voices. This means that the tools to improve the quality of the closed captions, will also improve the quality of the speech translations.

Notes:

  • When using a human corrector to edit the source text of AI interpreter languages, make sure that 'Allow captions to be corrected' on the Caption Languages tab is checked (= default). This will add a slightly higher latency to the live stream.
  • Consider adding an AI Vocabulary containing terms (names, acronyms, industry jargon, technical phrases ...) that may appear in the live stream, so they will be correctly captured and translated.

3 Test broadcast

Configure your encoder by using the settings on the Broadcast tab and our broadcast guidelines.

When you are ready to test, go to the Management tab and set the event status to Preview by pressing the Set to preview button. This is required for your broadcast to be processed, and for correctors to connect to their rooms.

Start your broadcast.

You can use the Preview player to hear the speech translations. The live stream in the preview player has a delay of about 2 minutes. As soon as you start broadcasting, the preview player will indicate the number of seconds before the live stream is available.

After the event is set to Preview, correctors are already able to connect to the Correction room. After about one and a half minute, they can see the live stream and the first sentences, and are able to correct them. If you are the event manager, you can use the Realtime Management page to communicate with correctors through text chat. Unfortunately, it is not yet possible to hear the speech translations in the realtime player on this page.

4 Starting the live stream

When the event is about to start, go the Management tab and press the Start event button. We recommend to start the event at least four minutes before the live action begins. This allows the player to start buffering and ensures that nobody misses the start.

When you start the event, Clevercast also starts recording the live stream. So, starting the event in time also ensures that the cloud recording is complete.

The embedded player automatically detects the status change and makes the live stream visible to your participants. You can also see the live stream in the Public Player on the Management tab. If you want to display a message in the embedded player, use the Service Message on the Management tab.

5 Stopping the live stream

After the live action has ended, wait at least 4 minutes before setting the event status to Ended, by pressing the Change Event Status button and selecting End event. The embedded player automatically detects this and shows a poster image or message to your viewers (see the Appearance tab) instead of the live stream.

Clevercast automatically completes the cloud recording and converts it to an MP4 file. You can download it via the Events > Recording menu. If your plan includes VoD, you can publish the recording to Video on-Demand (this also allows you to turn the multilingual recording into single-language MP4's).

Watch the video tutorial