Skip to content

#652: measure live display/monitor sync and reduce WASAPI latency - #655

Merged
iamfatness merged 2 commits into
mainfrom
codex/652-live-av-sync
Sep 26, 2026
Merged

iamfatness merged 2 commits into
mainfrom
codex/652-live-av-sync

Conversation

@iamfatness

Copy link
Copy Markdown
Owner

#652 live operator A/V sync

The same-run clap harness now compares Program shared-texture publish, GoXLR Game WASAPI loopback, and the recorded Program. A matched 24 s A/B at Program depth 2 found ~240 ms of live monitor lag on both accepted beta db9e703 and main d23dc9d5, while the recording was within ~20 ms. This points to the monitor path in the synthetic run; it does not establish why the later human report felt worse.

The monitor adapter now requests the endpoint's low shared period through IAudioClient3 and holds a 20 ms ring target. On GoXLR the actual endpoint buffer fell to 22 ms. The final 24 s run measured video minus monitor audio at -46.2 ms (12.2 ms spread), video minus recorded audio at -33.4 ms (13.5 ms spread), with zero monitor underruns and zero lost samples. The harness fails if any of the three paths is missing or if live skew exceeds 50 ms.

No Program delay, per-guest Zoom offset, ingest change, or SRT/NDI work. docs/BACKLOG.md lists #652 as unranked, awaiting real-human acceptance. Texture publish is measured rather than physical display scanout. Long-show drift remains to be validated because GoXLR's low-period stream does not expose IAudioClockAdjustment.

Detailed commands, SHAs, and artifact paths are in the first PR comment. Relates to #652.

@iamfatness

Copy link
Copy Markdown
Owner Author

#652 measurement evidence

Repository/builds. Accepted beta db9e7033851d784b892b67ac8cb74258cecd73e8; measurement-only cherry-pick 4bee5e10 in an isolated beta worktree. Current-main A/B base d23dc9d56b118e07171e4d13a270421450fd8965 (newer than the reported 550f908f); measurement commit 3798969f. Candidate fix commit 1b2d2e97. Both sides were built as Release on this Windows/GoXLR machine. No running operator app was replaced.

Commands. In each isolated worktree, build with cmake --build native/build-dev --config Release --target corevideo-native corevideo-native-tests corevideo-zoom-engine-fake corevideo-loopback-rec. Run node scripts/validate-av-clap.mjs --seconds 24 --keep-artifact --live-paths --monitor-device "Game (TC-HELICON GoXLR)" --program-buffer 2 from that worktree. The harness sets COREVIDEO_PROGRAM_BUFFER_FRAMES=2 before launching native core; this is not a live setting. The same run samples Program shared-texture publish QPC, WASAPI endpoint loopback packet QPC, and the recorded Program content. The displayed path is texture publish, not physical scanout. A missing path or >50 ms live skew fails the run.

Release run Recorded Program video−audio Program publish−monitor audio Monitor underruns / audio lost samples
Beta db9e703 + probe −19.2 ms median, 17.3 ms spread, −1.15 frames (8 pairs) −242.1 ms median, 36.5 ms spread, −14.52 frames 0 / 0
Current main d23dc9d5 + probe −17.3 ms, 16.2 ms, −1.04 frames (8 pairs) −239.3 ms, 17.6 ms, −14.36 frames 0 / 0
Candidate 1b2d2e97 −33.4 ms, 13.5 ms, −2.01 frames (8 pairs) −46.2 ms, 12.2 ms, −2.77 frames (9 pairs) 0 / 0

Negative sign means audio lags video. One 60 fps frame is 16.67 ms. Beta/current A/B differ by just 2.8 ms on the synthetic monitor clap, so these runs do not reproduce the subjective regression between builds. They do isolate a large monitor delay that a passing Program.mp4 conceals. The candidate cuts that live delay by ~193 ms while keeping the recorded path inside 50 ms.

Artifacts. Each evidence JSON includes all clap times, pair deltas, counter snapshots, selected monitor endpoint, loopback capture log, and its recordingArtifact path; adjacent files include loopback WAV and packet CSV.

  • Beta: C:\Users\walla\OneDrive\Documents\ChatGPT\CoreVideo Pro\av-sync-baseline-652\native\build-dev\Recordings\CoreVideoPro\validate-av-clap\live-paths-1790427794907\evidence.json
  • Current main: C:\Users\walla\OneDrive\Documents\ChatGPT\CoreVideo Pro\av-sync-652\native\build-dev\Recordings\CoreVideoPro\validate-av-clap\live-paths-1790427841699\evidence.json
  • Candidate: C:\Users\walla\OneDrive\Documents\ChatGPT\CoreVideo Pro\av-sync-652\native\build-dev\Recordings\CoreVideoPro\validate-av-clap\live-paths-1790428635389\evidence.json; recorded Program.mp4 at ...\validate-av-clap\av-clap-20260926-091723\Program.mp4.

Buffer experiments. GoXLR shared legacy mode negotiated an actual 44.5 ms endpoint buffer even when asked for 0 ms; 20 ms ring gave −53.2 ms live skew and 0/0 counters. A 10 ms ring gave −53.8 ms and 5 underruns, so it was reverted. GoXLR IAudioClient3 reports a 480-frame (10 ms) supported shared period. The candidate uses that period plus a 20 ms ring; actual endpoint buffer is 22.0 ms. Runtime log: mode=shared-low-period, selected=480 frames, actual=22.0ms, ring-target=20.0ms.

Checks. Release targets rebuilt. native/build-dev/corevideo-native-tests.exe --gtest_filter=AvSyncClapProbe.* and --gtest_filter=MonitorLatencyPolicy.* pass. node --check scripts/validate-av-clap.mjs and git diff --check pass. The hardware harness is a hard gate, not a self-skip. The 24 s candidate run passed. GoXLR low-period mode did not expose IAudioClockAdjustment (0x80070057); long-show drift and real-human lip-sync still need validation. The existing operator app remains on its earlier running binary until restarted.

@iamfatness
iamfatness merged commit 7d0d363 into main Sep 26, 2026
20 checks passed
@iamfatness
iamfatness deleted the codex/652-live-av-sync branch October 1, 2026 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant