Manim Director

Background

I take a lot of math notes. Turning notes into something teachable — video with visuals that actually compute things on screen — normally costs hours per minute of output. I wanted four things:

• One file is the lesson. lesson.md holds the narration prose and the animation blocks. Video, notes, slides, and quiz all compile from it. No project format, no binary state.
• Agents as authors. A drafting agent writes the script and the scenes. A chat agent revises them. I review diffs, not blank pages.
• Local voice. Narration synthesized on-device, no API cost per build, and eventually my own cloned voice.
• A real review surface. Timeline, program monitor, per-sentence captions, frame-level QC. Watching the output is the QA loop.

Architecture

Three parts, and a rule that keeps them honest.

• The pipeline (studio, Python): parses lesson.md, renders scenes with Manim, synthesizes narration, assembles with ffmpeg.
• The app (Tauri v2 + Svelte 5): a VSCode-style shell around an NLE — curriculum tree, script editor, program monitor, timeline, inspector, chat.
• The hub (Express + Socket.io + SQLite): REST over the pipeline plus an MCP server (17 tools) so any Claude Code session can operate the studio.

The rule: files are the API. Python is the only parser of lesson.md. The app never interprets lesson content itself; it calls studio state and renders the JSON. Agents edit files, and a filesystem watcher refreshes the app. That one constraint is why agents, the CLI, and the UI never disagree about what a lesson is.

lesson.md → parse → render (manim) ∥ narrate (tts) → assemble (ffmpeg) → lesson.mp4

The pipeline

Parse

A lesson is ## sections with {#id} anchors. Prose is the narration script, verbatim. Visuals are fenced blocks naming a scene and a template:

## The dead load sets the tension {#s03}

Time to put numbers on the bridge itself. Under its own uniform
weight, the cable settles into a parabola:
$y(x) = \frac{4f}{L^2}\,x\,(L - x)$ ...

```animation scene=s03_deadload template=equation_morph duration=50s
{ "beat_offset": 1, "steps": ["y(x) = ...", "y'' = ...", "H_w = ..."] }
```

Ten parameterized Manim templates (vector plots, grid shears, equation morphs) cover most scenes; template=freeform is hand-written Manim for everything else; template=placeholder is a slot carrying only a prompt, for script-first drafting.

Render

Every scene is content-addressed. The hash covers the generated Manim source, the template class source, referenced asset bytes, and the narration beats baked into the scene. Unchanged scenes never re-render:

h = scene_hash(source, template_cls,
               asset_fingerprint(lesson_dir, block.params))
if manifest.is_current(block.scene, h) and entry["quality"] == quality:
    click.echo(f"  {block.scene}: current ({h}), skipping")

Render quality is sticky per lesson (majority of the manifest). That rule exists because an agent once rendered two scenes at 480p into my 1080p lesson and every build died at an assemble gate for an hour.

Assemble

ffmpeg pads each scene to its section's audio (hold last frame, pad silence), loudnorms, and concatenates. Two hard gates from production bugs: sections with mismatched resolutions refuse to assemble, and the whole build takes an flock on the lesson —

@contextmanager
def lesson_lock(lesson_dir):
    f = open(lesson_dir / "renders" / ".studio.lock", "w")
    try:
        fcntl.flock(f, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except OSError:
        raise LessonBusy("another studio process is already building this lesson")

— because the night the app, a chat agent, and a CLI build all assembled the same lesson concurrently, they interleaved writes into the same intermediates and produced mp4s with no moov atom. One writer per lesson; the second gets a clean error.

The agents

The methodology, condensed: give agents a file contract, give them tools instead of prose instructions, and measure them like you'd measure any system.

The queue contract

Every lesson folder has a requests.jsonl. Humans (via the app) append requests; agents work them:

{"ts":"2026-07-11T16:20:00Z","type":"draft-lesson",
 "note":"Draft the full lesson \"The RC circuit: deriving the time
 constant\". Brief: ..."}

The app spawns a drafter as a plain claude -p process with a fenced toolset:

Command::new("claude").args([
    "-p", prompt, "--output-format", "stream-json", "--verbose",
    "--permission-mode", "acceptEdits",
    "--allowedTools", "Edit", "Write", "Read", "Glob", "Grep",
    "Bash(uv run studio *)", "Bash(studio *)",
    "--max-turns", "40",
]).current_dir(&lesson_dir)

stream-json means every agent step is a line the app can parse: the UI shows a live point-to-point trail (tool, target, thought), a ticking clock, and a "quiet Ns" indicator when the model thinks. Progress persists to .agent.json in the lesson folder, so reopening a lesson shows what its last run did.

Chat, two scopes

The chat rail is claude -p --resume <session> per turn, one continuous session per lesson, cwd = the lesson folder. A second scope — Studio — points the same machinery at the workspace root with a different system prompt, so "make me a lesson on Laplace transforms from my notes at ~/..." scaffolds, drafts, and files a new lesson into the curriculum. Messages sent while a turn runs queue up and fire automatically; attachments are just file paths the agent reads.

MCP and the delta contract

The hub exposes 17 MCP tools so external Claude sessions can operate the studio. Script editing follows a read-once + patch contract: get_script_model returns the parsed lesson with line spans and a rev (sha256 of the file, truncated); every mutation returns a structured delta against that rev. Agents stop round-tripping whole files, and concurrent edits fail loudly instead of silently clobbering.

Director discipline

Output quality came from a written authoring bar in each lesson's CLAUDE.md, distilled from my actual rejections. The load-bearing rules:

• Prose is a speech script, read verbatim by TTS. Written-essay register gets rejected.
• Animate the mechanism, not a caption. A frame should still teach with the audio muted.
• You are the director, not a slide generator. Related equations accumulate in one scene; a derivation is one morph, never a scene per line. A lone equation gets a scene only when it's being introduced, and it renders beside the visual it comments on.
• Shot planning is a required step: one line per section — opening frame, what accumulates, payoff frame.

Measuring agents

Rules aren't enough; I audited the transcripts. Across four drafts, every agent burned 5–8 tool calls re-discovering the animation template registry by reading pipeline source, plus 3–5 blocked shell commands. Root cause was mine: the docs told agents to run a python -c one-liner that the permission gate blocks. The fix was a tool, not more prose: studio templates prints every schema in one call.

Then I re-ran a controlled draft (a derivation-heavy brief designed to tempt the old failure modes):

Model: Claude Fable 5 via Claude Code (one baseline draft ran Claude Opus 4.8). Dollar figures are Anthropic API list prices as reported by Claude Code's per-run cost accounting — notional when the agent runs on a subscription login.

The retest also exposed a rule loophole (agents reached for the math-only morph template, which structurally can't dock a visual), which became another guide fix. Audit, fix the tool, retest: the same loop as any system.

Voice

Math-to-speech

Prose keeps $...$ TeX for display; the voice must never say "dollar q dollar". Math spans convert to words at synthesis time only:

def _speak_tex(tex):
    s = tex
    while _FRAC.search(s):                      # innermost-first
        s = _FRAC.sub(lambda m: f" {m.group(1)} over {m.group(2)} ", s)
    s = s.replace("''", " double prime ")
    s = re.sub(r"\^\s*\{?\s*2\s*\}?", " squared ", s)
    ...
# "$H_w = \frac{q L^2}{8 f}$"  →  "H w equals q L squared over 8 f"

Display text keeps the TeX (captions render KaTeX); only the audio changes. The narration cache hashes the speakable form, so exactly the math-bearing sections re-synthesize.

Paragraph prosody

The voice sounded robotic on long sentences. The cause was in my own pipeline: I synthesized per sentence to get exact caption timings, so the model restarted its intonation contour every sentence, and silence-trimmed WAVs were butt-joined with zero gap. The fix inverts it: synthesize each prose chunk in one TTS pass, then recover per-sentence beats by aligning faster-whisper word timestamps against the known text with a Needleman–Wunsch pass over normalized words. Transcript drift ("2.47" vs "two point four seven") survives because the surrounding words anchor the alignment; a proportional split is the deterministic fallback, which also keeps tests hermetic.

Engines and cloning

Kokoro-82M is the default engine: faster than real-time, 54 voices, phonemized by misaki (dictionary-first G2P) after measuring that raw espeak reads "2.47" as "two, forty-seven". Chatterbox (0.5B, MIT, via mlx-audio) is the quality engine at roughly real-time; it zero-shot clones a reference WAV, so any Kokoro voice — or 15 seconds of me reading a script in Settings — becomes the narrator. Engine choice was settled by a blind A/B: same sections, shuffled a/b pairs, answer key sealed until after picking. The pair below is one of the test sections:

The app

The shell is VSCode-max: activity bar, resizable rails, a script editor where prose renders like a document (KaTeX inline) but every block is directly editable, a 16:9 program monitor with hover media controls, and an NLE timeline — filmstrip clips, trim handles, drag-to-reorder, waveforms, click-anywhere scrubbing with a wall-clock-interpolated playhead.

Three WebKit bugs cost the most hours:

• The srcless <track>. An accessibility pass added <track kind="captions"> with no src. WebKit gates media readiness on text-track loading; a srcless track never finishes, so readyState latched at HAVE_CURRENT_DATA forever — full buffer, working seeks, frozen clock. Convicted by a headless WKWebView harness bisecting element configurations. One line removed, playback fixed.
• The media connection pool. Destroying <video> elements mid-stream leaks their loader connections; about six leaks exhaust WebKit's per-host pool and every later load starves with zero network requests. Fix: drain on unmount and never recreate the element via {#key}.
• The mid-flush zombie. In Svelte 5, an exception inside any template expression aborts the remaining update flush. A helper dereferencing undefined on scene-less lessons meant half the UI repainted and half kept the previous lesson's DOM — which looked exactly like a state-management bug and wasn't.

The debugging pattern that worked all three times: build a harness that renders the real page in the same engine, and bisect with measurements instead of theories.

Related work

TheoremExplainAgent (TIGER-AI-Lab, ACL 2025 oral) is the closest published system: a planner agent writes the story and narration, a coding agent writes Manim, and the output is a 5-minute-plus theorem video with audio. They also built TheoremExplainBench — 240 theorems across CS, chemistry, math, and physics at three difficulty levels, scored on five automated dimensions (accuracy, depth, logical flow, visual relevance, element layout). Best reported agent: o3-mini at a 93.8% render-success rate, 0.77 overall. Manimator (2025) maps research papers to Manim explanations.

The difference: those systems generate a video and score it with automated metrics; there is no editing step. This project is mostly the editing step — an NLE for review, per-frame QC, chat revisions, local narration, incremental rebuilds. Their evaluation is stronger than mine. A planned experiment: run TheoremExplainBench's prompts through this drafter and score on their five dimensions.

Results

• 12 lessons produced in 4 days across linear algebra, calculus, transformers, and structural mechanics. The pipeline compiles each lesson file into video, notes, slides, and a quiz.
• Longest lesson: 13 sections, ~12:30 runtime, drafted, revised through chat, rebuilt end-to-end.
• Drafting agent after the audit loop: 20 tool calls, $4.45 (Claude Fable 5, Anthropic API list prices), 9 minutes for a 4-section lesson with 3 worked derivation morphs.
• Pipeline test suite: 161 fast tests plus render/tts/export tiers; the hub has 16 smoke tests over REST + MCP.
• Narration, rendering, and assembly run locally; the drafting agents are the one cloud dependency. A rebuild of a 12-minute lesson with cached scenes takes about a minute.
• The studio is headed for an open-source release.

Takeaways

• Files as the API scales to agents. One parser, everything else renders its JSON. Agents, CLI, and UI stayed consistent through four days of continuous change because there was nothing else to desynchronize.
• Fix the tool, not the prompt. Every agent failure that mattered — wrong render quality, blocked commands, registry source-diving — was cured by changing what the tools do by default, not by adding instructions.
• Measure agents like systems. Transcript audits found a 20% waste pattern invisible from watching any single run. Baseline, fix, controlled retest.
• The webview is part of your stack. Three of the worst bugs were WebKit media and reactivity semantics. A headless harness that drives the real page found all three root causes.

Next steps

• Sharing. Self-contained quiz and notes pages under /q/ on this site, plus a configurable publish webhook so the open-source release can target any host.
• Word-level editing. The script is the transcript — remove-words and remove-silence map cleanly onto the beat contract.
• Creative direction. The authoring bar now demands cold-open hooks, role-colored equations, and concrete objects before abstract axes; equation morphs gained a per-role colors capability. Next: prove it against the bar set by channels like Visually Explained — draft a convex-optimization lesson and compare — and add 3D scenes to the template set.
• Local models for the agent work. Train or fine-tune open-source replacements for what a frontier model does in the studio today. The full task list: drafting a lesson script from a brief or notes; writing freeform Manim scenes; replacing placeholder blocks; revising blocks from change requests; lesson chat edits; studio-level lesson creation and curriculum filing; quiz writing; and QC frame judging. QC already has a local baseline (a small VLM on MLX judging rendered frames); judging and revision look tractable first — drafting is the hard one.