Innovations in Real-Time Rendering

I work in VFX and only recently understood the full concept of Gaussian Splats. I had seen the demos and nodded along when people referred to it as "the next photogrammetry." However, if you had asked me to explain what they are, their place in a real pipeline, or their capabilities, I would have struggled. So, I took the time to understand it, and here’s the simplified version I wish I had received earlier A Gaussian Splat consists of millions of tiny semi-transparent ellipsoids in 3D space. You input photos, train a model, and receive a scene in return. Phones serve as capture devices, and it can render at over 100 FPS in real time. Key tools like Nuke 17 now include native support, Houdini 21 offers a tech preview, V-Ray 7 can ray-trace splats, and OpenUSD 26.03 has introduced a first-class schema. Notably, Framestore utilized 4D Gaussian splatting for approximately 40 final-pixel shots in Superman last year. What it can achieve: - Rapid and cost-effective capture of real environments - Rendering at game-engine speeds - Integration into compositing without the need to recreate the world in CG What it cannot do: - Relight scenes (yet) - Provide a clean mesh (yet) - Render with AOVs, as the lighting is baked into the data. This is the trade-off. Thus, it does not replace photogrammetry or CG environments; rather, it serves as a new tool for scenarios requiring photoreal capture and real-time playback, with the understanding that relighting flexibility is sacrificed. For fellow VFX artists who have been quietly nodding along: you are not behind. The foundational paper is from 2023, and most production tools have been released in the past six months. Now is the time to learn it before it becomes a part of your next project. What new technology have you been struggling with?

Two years ago I already said AI is the future of rendering but I'm honestly surprised how far we've come since then. AI now lets you reimagine your 3D layouts with simple prompts and reference images. It doesn't just add textures, lighting, and effects like depth of field, it will also generate smoke simulations, water splashes, and explosive debris based on the movement in your scene. You can go from a rough layout to final render in minutes. You can change the style by just swapping out the reference frame. You can even feed in multiple reference images that get merged together — giving you full control over the final aesthetic. All of this runs on your own computer. Free, open-source tools. No subscription, no cloud, no waiting list. So how does it work? The workflow is built around a model merge by Inner-Reflections that combines two video models: SkyReels V3 R2V and Wan VACE. SkyReels understands reference images really well but can't be guided precisely by ControlNets. Wan VACE accepts ControlNet guidance but its reference understanding isn't good enough for longer scenes. The merge gives you both. It's like the model now speaks two different languages. You export depth maps and outline passes from Blender, generate a style reference with Z-Image Turbo, and render it all through ComfyUI. Is this replacing traditional rendering? No. It is not perfect yet, but for prototyping and indie productions, this is genuinely useful. You can explore ten visual directions in the time it takes to set up one traditional render. For previs especially — being able to show directors what a shot will actually feel like before committing render farm time. Open source is not far behind. No proprietary tool I've seen combines ControlNet-guided geometry with reference-based style transfer at this level of consistency. The community is building faster than any single company can ship. I made a full tutorial with free downloadable workflows — link in the comments.

Sharp Monocular View Synthesis in Less Than a Second https://lnkd.in/djBTUjUE Real-time photorealistic view synthesis from a single image. Given a single photograph, regresses the parameters of a 3D Gaussian representation of the depicted scene. Synthesis in less than a second on a standard GPU via a single feedforward pass through a neural network. The synthesized representation is then rendered in real time, yielding high-resolution photorealistic images for nearby views. The representation is metric, with absolute scale, supporting metric camera movements. Robust zero-shot generalization. SOTA on multiple datasets while lowering the synthesis time by three orders of magnitude. Code and weights (try it on your images!) at https://lnkd.in/dMjfhnP4 . Project page with videos: https://lnkd.in/dGbuDaht with Lars Mescheder, Wei Dong, Shiwei Li, Xuyang Bai, Marcel Santana, Peiyun Hu, Bruno Lecouat, Mingmin Zhen, Amaël Delaunoy, Tian Fang, Yanghai Tsin, Stephan Richter

Neural Directional Encoding for Efficient and Accurate View-Dependent Appearance Modeling Novel-view synthesis of specular objects like shiny metals or glossy paints remains a significant challenge. Not only the glossy appearance but also global illumination effects, including reflections of other objects in the environment, are critical components to faithfully reproduce a scene. In this paper, we present Neural Directional Encoding (NDE), a view-dependent appearance encoding of neural radiance fields (NeRF) for rendering specular objects. NDE transfers the concept of feature-grid-based spatial encoding to the angular domain, significantly improving the ability to model high-frequency angular signals. In contrast to previous methods that use encoding functions with only angular input, we additionally cone-trace spatial features to obtain a spatially varying directional encoding, which addresses the challenging interreflection effects. Extensive experiments on both synthetic and real datasets show that a NeRF model with NDE (1) outperforms the state of the art on view synthesis of specular objects, and (2) works with small networks to allow fast (real-time) inference.

Type a single prompt. Walk through the world you just created. In real time. That's Google's Genie 3. Not pre-rendered. Not a video. A fully interactive 3D environment generated at 24fps that remembers where you've been for minutes, not seconds. What makes this technically significant: → Visual memory up to 1 minute. Leave a location, return, it's exactly as you left it. → 720p, real-time navigation. No latency. → "Promptable world events" let you alter weather, objects, or the environment mid-session. This is a step toward unlimited training environments for robotics, simulation testing without expensive 3D modeling, and design prototyping that doesn't require building anything first. The limitation? Still capped at a few minutes of consistency. That window will expand. What catches my attention: this moves world simulation from "consume a video" to "navigate a space." Fundamentally different. Learn more here → https://lnkd.in/eacn9vX5 #AI #WorldModels #GoogleDeepMind #GenerativeAI #Robotics #Genie3 #FutureOfDesign #ProductDesign

This isn't an Unreal Engine walkthrough video, it’s the entire Quixel's Derelict Corridor running live in a web browser, without pixel streaming. Explore the full environment here: https://lnkd.in/dgQ2SHeH A few months ago, we showcased a portion of this scene running in a web browser. Squeezing a single slice of this corridor into a browser was a massive challenge then. Today, we are unleashing the entire facility. Typically, an asset-dense environment of this scale and fidelity requires at least a dedicated NVIDIA RTX A6000 in the cloud just to stream a single instance. Full Derelict Corridor and its visual fidelity has been preserved, all while maintaining a rock-solid framerate on low-end devices and smartphones. Getting here required a hardcore engineering sprint by the Moshpit team. We bypassed the traditional limitations of WebGL by building a 100% GPU-driven pipeline for UnrealTwin: ‣ WebGPU: We’ve moved beyond the limits of WebGL. UnrealTwin now has direct access to the user's GPU hardware via WebGPU, executing blazing-fast Splat sorting using custom compute shaders. ‣ A Custom LOD Pipeline: Standard Gaussian Decimation destroys immersion, it introduces aggressive popping and turns distant geometry into voxelated mush. To preserve Quixel's visual fidelity, we engineered a custom Splat LOD generation pipeline from the ground up. The result is flawless distant rendering without tanking mobile performance. #UnrealTwin is proof that the open web and non-gaming rigs are finally ready for real-time 3D. Have a walk-through and let me know how it runs on your hardware! #UnrealEngine #UE5 #Quixel #GaussianSplatting #3DGS #WebGPU #PlayCanvas #DigitalTwins #Realtime3D #Moshpit #UnrealTwin #TechArt #GameDev

🚀2D Gaussian Splatting: Real-Time, Geometry-Aware Radiance Field Reconstruction In this week’s deep dive, we unpack how 2D Gaussian Splatting (2DGS) redefines the future of real-time neural rendering and reconstruction. By collapsing volumetric 3D Gaussians into surface-aligned 2D disks, 2DGS achieves unprecedented geometric accuracy and multi-view consistency - outperforming both 3D Gaussian Splatting (3DGS) and NeRF in quality, stability, and speed. The post explores how 2DGS bridges the gap between explicit surface reconstruction and radiance field rendering, setting a new standard for geometry-preserving real-time synthesis. 🔍 What’s Covered? ✔️ Understanding the Transition from Volumetric 3DGS to Surface-Based 2DGS ✔️ Perspective-Correct Ray–Plane Intersection for Accurate Rendering ✔️ Depth & Normal Regularisation for Sharp, Noise-Free Surfaces ✔️ Real-Time Rendering with Physically Consistent Geometry ✔️ Qualitative Comparisons: 2DGS vs 3DGS vs NeRF This blog post covers everything - from 2DGS’s architectural innovations to implementation part - showing how it delivers NeRF-level realism with Signed Distance Function(SDF)-level geometry, all at real-time rendering speeds. Perfect for researchers, engineers, and enthusiasts exploring next-generation scene reconstruction and neural rendering frameworks. 🔗Read More: https://lnkd.in/gJgWhXmz #2DGaussianSplatting #3DGaussianSplatting #Photogrammetry #NeuralRendering #3DReconstruction #RadianceFields #3DGS #NeRF #ComputerVision #DeepLearning #SIGGRAPH2024 #RealTimeRendering #GeometryAwareAI #AIResearch #CVML #pointclouds

From Billion-Cell Solvers to Real-Time AI Surrogates. The New Architecture of CFD Engineering teams building the next generation of aircraft, vehicles, and data centers need the ability to make rapid, iterative design decisions. Waiting days for batch-processed fluid dynamics solvers completely breaks the development loop. To solve this, we are releasing the NVIDIA Omniverse Blueprint for Interactive Fluid Simulation. Check out the live blueprint: https://lnkd.in/gWZxX_ee For the CAE practitioners and enterprise architects scaling these workloads, here is the exact reference architecture to transition your CFD pipelines into real-time digital twins: 🟢 1. The Compute Engine (Blackwell & CUDA-X): We are accelerating traditional solvers by orders of magnitude. The proof is in the hardware: Cadence recently ran a 10-billion-cell large-eddy simulation (LES) of a complete aircraft on a single NVIDIA GB200-NVL72 system. It did the work of nearly 300,000 CPU cores at a 7x lower cost. 🟢 2. The AI Surrogate (PhysicsNeMo): To achieve real-time interactivity, developers are using the open-source PhysicsNeMo framework to embed governing equations (like Navier-Stokes) directly into machine learning models. Using tools like the DoMINO NIM microservice, these AI surrogates predict massive flow fields instantly. 🟢 3. The Digital Twin (OpenUSD & NVIDIA Omniverse): The unified pipeline—CAD → meshing → CFD solve → AI surrogate—is piped natively into Omniverse using OpenUSD. This gives engineers fully interactive, physically based RTX rendering of the fluid dynamics directly in their applications. You get the real-time design exploration of an AI surrogate, backed by the gold-standard accuracy of a high-fidelity solver. Incredible to see ecosystem leaders like Cadence, Siemens, Ansys, and Dassault Systèmes bringing these integrated capabilities to their customers. The interactive blueprint and reference architecture are live today. 🔗 Dive into the technical implementation here: https://lnkd.in/gc4qyj7s What is the biggest compute or data bottleneck your team faces when scaling multi-physics simulations? Let's discuss in the comments. 👇 #NVIDIA #Omniverse #CFD #DigitalTwins #OpenUSD #Blackwell #PhysicsNeMo #CAE #Engineering #DevRel

CGI elevated how anime looked. But Unreal Engine 5 is changing how anime is made. Traditional CGI still depends on long render times and static workflows. Every frame takes hours. Every correction takes days. Unreal Engine flips that. Real-time rendering lets directors see lighting, motion and camera instantly. No waiting for test renders. No guessing how it will look in post. For artists, faster feedback and more freedom. For studios, reusable assets and cleaner pipelines. For fans, worlds that feel more alive built on the same tools as AAA games. With Lumen and Nanite, lighting and materials can breathe like cinema without losing anime soul. With virtual cameras, animators can move inside the scene as if in live action. CGI made anime more beautiful. Unreal makes it smarter. It’s not just a new tool. It’s a new way to think, to direct, to tell stories in real time. The question is how soon before every studio makes the switch? #KhoaTrinh #AnimationStudio #Anime #UnrealEngine