Karaoke captions that don't sound AI: Whisper + forced-align explained

A viewer wrote to me in April 2026, under a Cocorico episode: "Your captions are clean, unlike 95% of AI channels. How do you do it?". The question surprised me, because I'd never considered captions a quality signal for AI content — and because during the first months of production, mine were exactly as mediocre as everyone else's.

The realization came reading two versions of the same episode back to back. Version 1: captions generated directly by Whisper, no intervention. Version 2: captions aligned to the original script, word by word, with timing derived from the audio. The difference between the two wasn't in the transcribed content, which was almost identical. It was in the visual fidelity to the script. Version 1 broke sentences at the wrong places, swallowed words, fumbled proper nouns. Version 2 displayed exactly what the voice was saying, at the moment it said it.

What follows is the forced-align technique I've used since to produce karaoke captions that never betray the synthetic nature of the voice. With the two parameters that actually matter, and the silent trap to anticipate.

The problem with raw Whisper output

Whisper, OpenAI's open-source transcription model, is remarkable on one count: it transcribes nearly every human language at a respectable error rate. The catch is what it does in addition to transcribing. To produce a readable result, Whisper automatically applies several heuristics that alter the text in subtle but cumulative ways:

• Phantom word insertion. Whisper hallucinates "uh", "hmm", and linking words ("so", "then") that don't exist in the audio but the model judges linguistically plausible.
• Weak word elision. Conversely, Whisper drops some articles and particles it considers negligible. "I'm going to the house" becomes "going to house" in a rushed transcription.
• Spelling reformulation. Whisper aligns written numbers on standard orthographic convention, sometimes disagreeing with what the voice actually pronounced.
• Misplaced semantic pause. Whisper inserts commas and periods following written grammar, not the real oral cadence.

For a passive caption (watched on mute, say), these alterations are tolerable. For a karaoke caption synced to an audible voice, they produce a permanent gap between what's said and what's written. That gap is what viewers feel without always being able to name it.

The forced-align idea

Forced-align flips the relationship. Instead of asking Whisper "transcribe this audio", you ask it "align this audio to this text I'm giving you". The reference text is the original script — the one you wrote and that was passed to voice synthesis.

Concretely, Whisper produces its transcribed tokens with timing. A matching algorithm then walks through each audio token and looks for the best correspondent in the script. When a match is found, the audio token's timing is attributed to the script token. At the end, you have a word-level timing file where the displayed text is exactly the script, and the chronology comes from the real audio.

The operational benefit is total. Whisper's inserted *"uh"*s are ignored (they don't exist in the script). Elided words are recovered (they exist in the script). Displayed orthography is guaranteed identical to the author's intent. And timing stays faithful to what the voice pronounces.

The implementation, in broad strokes

The pipeline fits in three steps. In Python with openai-whisper and a bit of custom matching:

import whisper
from rapidfuzz import process, fuzz

# 1. Whisper transcribes with word-level timing
model = whisper.load_model("medium")  # 'medium' is enough for EN/FR
result = model.transcribe(
    "vo.wav",
    language="fr",
    word_timestamps=True,
)

# Each "word" contains {word, start, end, probability}
whisper_words = [w for seg in result["segments"] for w in seg["words"]]

# 2. Load the reference script and tokenize on whitespace
script_text = open("script.txt").read()
script_tokens = script_text.split()

# 3. Forced-align: match each script token to the closest Whisper token
LOOKAHEAD = 6
MIN_WORD_GAP = 0.08

aligned = []
cursor = 0  # position in whisper_words

for script_token in script_tokens:
    # Look in the next LOOKAHEAD Whisper tokens for the best match
    window = whisper_words[cursor:cursor + LOOKAHEAD]
    if not window:
        break
    best_match = process.extractOne(
        script_token,
        [w["word"].strip() for w in window],
        scorer=fuzz.ratio,
    )
    if best_match and best_match[1] >= 70:  # 70% similarity threshold
        match_idx = best_match[2]
        matched_word = window[match_idx]
        aligned.append({
            "text": script_token,
            "start": matched_word["start"],
            "end": matched_word["end"],
        })
        cursor += match_idx + 1
    else:
        # No match — slide forward and record a placeholder
        # to fill via uniform distribution later
        aligned.append({"text": script_token, "start": None, "end": None})

# Post-processing: fill the None gaps via linear interpolation between
# neighbors (omitted here for readability — in the Cocorico repo)

The idea is simple: for each script token, look in the next six Whisper-transcribed tokens for the one that resembles it most. If similarity is above 70%, that's a match — grab the timing. Otherwise, slide forward, and fill the gaps by interpolation at the end.

The two constants at the top (LOOKAHEAD = 6 and MIN_WORD_GAP = 0.08) deserve a detailed explanation.

LOOKAHEAD: how many tokens to scout before giving up

LOOKAHEAD sets the window in which we look for a match for each script token. Its value arbitrates a trade-off:

• Too low (1-2): if Whisper inserts two consecutive *"uh"*s, we confuse them with the next script word, miss the match, and the rest of the episode drifts.
• Too high (15+): if a script word doesn't exist in the audio (rare but possible), we search very far and may grab an incorrect match several sentences later, creating a violent visual jump in the captions.

The value 6 that I use is empirical. It absorbs typical Whisper artifacts (hesitation inserts, doublets) while staying short enough to avoid distant false positives. Across the dozen channels I've instrumented, this value produces a match rate around 96-98%, meaning 2-4% of tokens get re-synced via interpolation — invisible to the eye.

MIN_WORD_GAP: preventing overlaps

MIN_WORD_GAP imposes a minimum delay between the end of a word and the start of the next. The value 0.08 second (80 milliseconds) corresponds roughly to the minimum interval a human perceives between two distinct words.

Without that constraint, ElevenLabs (or any fast TTS) can produce two consecutive words with the end of one and the start of the next nearly overlapping. Whisper transcribes each at its real audio timing, and the alignment faithfully reproduces the overlap. Result: karaoke captions show two words simultaneously for 20 milliseconds, producing an unpleasant visual flash.

By forcing a minimum gap of 80 milliseconds, you guarantee every word has its clean visual slot. The cost is negligible (the second word's start is nudged into the future), the visual gain is immediate.

The constraint that can break everything: token alignment

Here's the silent trap. Forced-align operates at the token level, whitespace-separated. The displayed text and the spoken text must carry the same number of tokens. Otherwise the algorithm walks through a script with more or fewer tokens than the audio, and matching fails on cumulative drift.

Three typical cases that break alignment:

Case 1 — spelled vs digital number. You write "3000" in the displayed script but want the voice to pronounce "three-thousand". One token in both cases — OK. However, if you write "3 000" with a space, that's two tokens, and the voice has to pronounce two tokens too ("three thousand", no hyphen).

Case 2 — acronym vs expansion. You write "AZERTY" (1 token) but want the voice to pronounce "A Zed E R T Y" (6 tokens). You need to pick: either align on 1 token (write "azerty" lowercase pronounced as a word, not an acronym), or align on 6 tokens (write "A Zed E R T Y" in the displayed script too).

Case 3 — attached punctuation. "Hello," written as a comma stuck to the word, vs "Hello ," with a space. The attached comma is in the same token; the detached comma is a separate token. Whisper doesn't transcribe punctuation separately, so alignment will break on the detached comma.

My Python orchestrator validates this constraint before emitting the config file. For each segment, it counts the tokens in the displayed text and in the spoken tts. If they differ, it emits a warning and distributes timing uniformly across the offending segment — not ideal, but better than a crash. The practical rule I carved into the editorial doc: as many spoken tokens as displayed tokens, or you'll know on the first re-read why a caption skips.

The visual render: the karaoke proper

Once the alignment file is produced, the karaoke render is a CSS templating question. Here's the formula I use in HyperFrames:

<div class="caption-line">
  <span class="word" data-start="0.12" data-end="0.34">I</span>
  <span class="word" data-start="0.42" data-end="0.78">learned</span>
  <span class="word" data-start="0.86" data-end="1.10">something</span>
  <!-- ... -->
</div>

Each word carries its two timings as attributes. A GSAP timeline iterates over every .word and applies two states: before data-start, the word is gray; between data-start and data-end, it switches to the signature color (e.g. yellow #FFD60A on WhyFactory); after data-end, it stays on the signature color until the end of the segment.

The visual effect is the progressive light-up associated with karaoke. Synchronization is word-precise because timing comes from real audio, not estimation. And text never drifts from the script, because the script drives the display.

The mistake everyone makes at first

A mistake I've seen repeated by several creators starting out: using Whisper's result["text"] directly for captions, assuming Whisper transcribes correctly enough to avoid the complexity of forced-align.

It's understandable and it's wrong. The problem isn't Whisper's precision on an isolated word; it's the accumulation of micro-alterations across a full script. A 60-second short typically contains between 100 and 150 words. If Whisper alters 3% (insertions + elisions + reformulations), you have 4-5 visible misalignments per episode. The viewer won't be able to name them all, but they'll catch enough that the channel feels "AI poorly done" rather than "AI well done".

Forced-align fixes that problem in a technical step of 30 lines of Python. The investment / gain ratio is, to my knowledge, the best across the entire AI video production chain.

If you're starting out

Three resources worth knowing if you want to build this pipeline yourself:

• openai-whisper on PyPI. The medium model is the sweet spot for most languages (quality nearly equivalent to large at half the cost).
• rapidfuzz or python-Levenshtein for string matching. C-backed implementations, negligible CPU even on thousands of tokens.
• HyperFrames for HTML/CSS animated rendering via GSAP. The karaoke caption templating takes about three hundred lines of HTML/CSS, debuggable like a web page.

The only serious upfront investment is understanding forced-align and writing the matcher (the thirty Python lines above, plus the interpolation post-processing). Once done, it runs in a loop over all your episodes with no intervention.

---

The publication layer of this pipeline — pushing renders to YouTube, TikTok, Instagram, Facebook, Threads and LinkedIn from Claude via MCP — is what Shortflow does. If you've calibrated your captioning and you're missing the publish tooling, creating an account opens a seven-day free trial.