Add video-to-texture example driven by requestVideoFrameCallback - #361
Open
matiasngf wants to merge 3 commits into
Open
Add video-to-texture example driven by requestVideoFrameCallback#361matiasngf wants to merge 3 commits into
matiasngf wants to merge 3 commits into
Conversation
Adds a gallery example that maps a playing video onto a rotating cube, using requestVideoFrameCallback so the texture upload happens once per decoded frame instead of once per display refresh. Co-authored-by: v0 <it+v0agent@vercel.com>
Contributor
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The cube is a screen, so the spin has to leave a face turned towards the viewer long enough to watch what is playing on it. At 0.45 rad/s a face swept 268 degrees over the length of the clip; at 0.15 rad/s it sweeps 89, holding a readable angle for about as long as the clip runs. The previous excerpt was a zoom-in on the rabbit's face, which read as an almost-static gradient once cropped to a square cube face. Replaced it with the flying-squirrel glide (398.95s-409.35s, both shot boundaries), which has a large subject that stays inside the centre crop, saturated colour, and continuous motion, so a stalled upload is visible rather than plausible. Co-authored-by: v0 <it+v0agent@vercel.com>
Follow the shape of `gradient` and `clipping`: a canvas-only `index.tsx`, a short `renderer.ts` that wires init/surface/frameLoop, and a `scene.ts` that owns the GPU objects. 732 lines down to 463. - Drop the `useExampleErrorReporter` import. A failed video load now leaves the test pattern on the cube, so the example needs no error channel. - Remove the HTML overlay. The upload/frame counters, `presentedFrames`, `mediaTime` and the `precise` flag only existed to feed it, which is most of what made `video-source.ts` 223 lines. - Fold the test pattern into `scene.ts` and write it at the aligned stride, removing a file and the row-padding helper. Bytes are unchanged, so both thumbnails still match their baselines pixel-for-pixel. - Replace the token bookkeeping with `video.consume()`, so the renderer reads as "upload if there are new bytes, then draw". Also fixes a bug the restructure exposed: the no-rVFC fallback tested `currentTime` for change, but that is a continuous clock that advances on every rAF tick, so the fallback uploaded once per tick and saved nothing. Measured 360 uploads / 360 ticks. It now quantises the clock by the clip's frame rate: 180 / 360, matching the rVFC path's 181 / 360. Verified both paths in-browser at 30 uploads/sec, with rVFC deleted to force the fallback. Co-authored-by: v0 <it+v0agent@vercel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
video-to-textureexample: a looping clip mapped onto a slowly rotating cube, usingrequestVideoFrameCallbackto drive texture uploads. The two clocks are deliberately separate — rVFC fires once per decoded frame and gatescopyExternalImageToTexture, whileframeLoopruns at the display's refresh rate so the rotation stays smooth. Measured in-browser at 181 uploads against 360 rAF ticks, i.e. roughly half the uploads a naive rAF loop would issue.What changed
gradientandclipping: canvas-onlyindex.tsx, a shortrenderer.tswiring init/surface/frameLoop, and ascene.tsowning the GPU objects. Nothing is imported fromlib/beyond the type-onlyexample-meta/example-renderercontracts every example uses.ffmpeginvocation is recorded inrenderer.ts. The committed binary is a judgement call worth a reviewer's opinion.video/video-inputto the controlled vocabulary JSONs and regenerates the examples-API artifact tree.Worth knowing
A fallback bug that was invisible until the branch was exercised. The no-rVFC path (Firefox) tested
video.currentTimefor change — but that is a continuous clock which advances on every rAF tick, so the fallback uploaded once per tick and saved nothing at all: 360 uploads / 360 ticks. Chrome always takes the rVFC path, so no amount of normal testing would have caught it.totalVideoFramesturned out to be no good either — it is a batched decode counter, changing on only 16 of 240 ticks. The fallback now quantises the clock by the clip's frame rate, giving 180 / 360, matching rVFC's 181 / 360. Both paths verified in-browser at ~30 uploads/sec, with rVFC deleted from the prototype to force the fallback.Two clock-coupling traps, made structural.
thumb.timeis in seconds and gets multiplied by the spin rate, so changing the rotation speed would silently rotate the thumbnail too; the thumbnail is now expressed as a fixed pose divided bySPIN_RATE. Separately, the test pattern's orientation marker was originally a brighter top band, which clipped against already-saturated colour bars and rendered invisible — so it silently proved nothing about vertical orientation. It is now a dark band, visible along the top of every face.Nobody has watched the cube spin on real hardware. This sandbox has only software WebGPU with no canvas swapchain; I confirmed that is environmental rather than my code by loading the pre-existing
clippingexample and getting an identicalInstance dropped in popErrorScopefailure. So the render path is verified through the repo's own headless Node thumbnail gate (48/48 matched, no baseline rewritten) and the video/rVFC path separately in a real browser. A reviewer on real hardware should still eyeball the live cube — in particular whether the spin rate feels right in motion, which is a taste call I can't make from stills.How to test
pnpm thumbs:checkshould report 48/48 matched withvideo-to-texturematching its baseline pixel-for-pixel. Then open the example and confirm the cube turns smoothly while the clip plays upright on every face.