Skip to content

Auto-generated captions

HeapStream can generate captions for on-demand videos automatically, giving viewers subtitles and you a transcript, and can translate those captions into other languages in the same request.

Overview

Captioning uses OpenAI's Whisper model. It works best with clear audio and can struggle with heavy non-speech sound or long silences, so test it on your typical content before relying on it.

Enabling auto-captioning

  • Enable auto-captioning for all videos in the Project Settings page.
  • Enable auto-captioning when uploading a video in the Video Upload page.
  • Generate captions for an existing video in the Video Edit -> Text Tracks page.
  • Enable auto-captioning for all videos of a project in the API ⧉ "Update project settings" endpoint.
  • Set the auto_tt array when uploading ⧉ or fetching ⧉ a video.
  • Generate captions for an existing video through the endpoint "Generate auto captions" in the API ⧉

Captions in other languages

Ask for more than one language and you get one caption track per language: the speech is transcribed once, in the audio track's language (the one you set on the track, or the one Whisper detects for the main track), and every other language you asked for is a machine translation of that transcript. You do not need to include the spoken language in the list; a video in English with de and fr requested gets a German and a French track.

  • Dashboard: the language picker on Project Settings, Video Upload and the Video Edit -> Captions dialog takes several languages.
  • API: pass one entry per language in auto_tt (upload, fetch, project settings) or generated_subtitles ("Generate auto captions").

Translation follows the transcript, so a mistake in the transcription shows up in every translated track, and editing one track by hand does not regenerate the others. The translated tracks are counted on the Analytics page under AI translations. The target languages are the ones in the table below.

FAQ

List of supported languages

Below are supported VOD auto-captioning languages and codes (note that beta languages may have lower accuracy):

Language Language Code Status
English en Stable
Spanish es Stable
Italian it Stable
Portuguese pt Stable
German de Stable
French fr Stable
Korean ko Stable
Dutch nl Stable
Thai th Stable
Russian ru Stable
Polish pl Stable
Japanese ja Stable
Swedish sv Stable
Turkish tr Stable
Catalan ca Stable
Indonesian id Stable
Ukrainian uk Stable
Malay ms Stable
Mandarin zh Stable
Finnish fi Stable
Norwegian no Stable
Romanian ro Stable

What is the cost of auto-captioning for VOD?

It's free, translations included.

Time needed to generate captions?

About 0.2x the duration of the speech for English, and about 0.5x for other languages.

My audio speech is in multiple languages

Auto-captioning assumes one language per audio track and gives poor results on mixed-language speech, so we don't recommend it for such content.

My video has multiple audio tracks (e.g. a dub or a commentary)

Captions are generated per audio track. By default the video's main audio track is captioned. To caption a specific alternate audio track, pass its audio_track_id to the "Generate auto captions" endpoint. Each audio track gets its own captions, even when two tracks share the same language (for example a main track and a separate commentary track). Deleting an audio track also removes the captions that were generated from it.

The speech is in a language you don't support

Contact us if you need a language that isn't listed.

Auto-generated captions may have mistakes

Automatic speech recognition still makes mistakes. To fix them, edit the generated track:

  1. Download the VTT file through the API or GUI
  2. Correct mistakes in a text editor
  3. Delete the auto-generated track
  4. Create a new track with the edited VTT file