How to Optimize 3d Scene Reconstruction

Streaming 3D reconstruction is fundamentally a memory problem. How do you map a massive, multi-room environment without blowing up your compute budget as the sequence gets longer? Lingbo-Map just introduced a highly elegant architectural solution to this exact bottleneck: Geometric Context Attention (GCA). Instead of brute-forcing the entire scene history into memory, GCA splits the streaming state into three lightweight buckets: an anchor for global coordinate grounding, a local reference window for dense geometry, and a compressed trajectory memory. By squashing the full sequence history into compact per-frame tokens, the memory and compute requirements remain nearly constant. Running through a DINO backbone, the pipeline actively predicts camera poses and depth maps at ~20 FPS—even on continuous 10,000+ frame sequences. This is how you scale real-time spatial computing and large-scale digital twins without needing infinite VRAM. Models: https://lnkd.in/dxY7D4Ar Project page: https://lnkd.in/dKRUEQaq Code: https://lnkd.in/dXQSJB7u Paper: https://lnkd.in/diPQk3Ki #SpatialComputing #3DReconstruction #ComputerVision #MachineLearning #SLAM #DevRel

📢SAM 3D: Single-Image 3D Reconstruction with Foundation-Model Reliability In this week’s deep dive, we break down SAM 3D, Meta’s groundbreaking framework that redefines what’s possible in single-image 3D reconstruction. Unlike earlier pipelines that struggle with occlusions, clutter, and ambiguous textures, SAM 3D produces high-quality 3D shape, texture, and layout directly from a single natural image - and does so with the stability and generalization of a true foundation model. SAM 3D combines a two-stage 3D generative architecture, a massive model-in-the-loop data engine, and a multi-stage synthetic-to-real training curriculum to achieve unprecedented reconstruction fidelity. From indoor scenes to outdoor environments, from tiny objects to full building façades, SAM 3D consistently outperforms traditional 3D methods and even modern diffusion-based models in accuracy, detail, and robustness. Whether you're reconstructing a chair in your living room or digitizing complex real-world scenes, SAM 3D delivers artist-level 3D assets with remarkable consistency - unlocking new possibilities across robotics, AR/VR, gaming, film, simulation, and digital twins. What’s Covered? ✅How SAM 3D Achieves Reliable Single-Image 3D Reconstruction ✅The Geometry Model: Coarse Shape & Layout Prediction ✅The Texture & Refinement Model Explained ✅Synthetic → Semi-Synthetic → Real-World: The Multi-Stage Training Pipeline ✅Model-in-the-Loop Data Engine & Human Preference Alignment (DPO) ✅ How SAM 3D Keeps Getting Better This blog post deconstructs every technical component of SAM 3D - from its architecture and training philosophy to its datasets, refinement modules, and real-world performance. Written to be both technically rigorous and beginner-friendly, the blog post helps researchers, engineers, and creators understand not just how SAM 3D works, but why it works, and what makes it arguably one of the most significant advancements in modern 3D perception. 🔗 Read More: https://lnkd.in/gU8wReJc #SAM3D #MetaAI #ComputerVision #3DReconstruction #FoundationModels #GenerativeAI #3DVision #Robotics #ARVR #GraphicsResearch #AIResearch #SingleImage3D

#2xplat solves: Why your image-to-3D pipeline breaks under scale. What if better 3D reconstruction isn’t about bigger models… but splitting vision into two specialists? That's what 2xplat does. 2xplat: Two Experts Are Better Than One Generalist Most pipelines that convert images into #GaussianSplats (#gs3d) aka #radiancefields, try to do everything at once: geometry, camera pose, and appearance. That’s the bottleneck. If you've ever used high-resolution images with professional tools to make GS3D, you better have tons of RAM, high-end GPUs with lots of VRAM, and high-speed SSDs. Oh, and be prepared to fix the outputs, perhaps multiple times. This paper takes a different approach: Two experts. One job each. • A geometry expert estimates camera pose and structure • An appearance expert generates high-fidelity Gaussian splats Both are trained end-to-end, but they don’t compete for the same representation. Why this matters now: • Monolithic models blur geometry and appearance • Pose errors cascade into visual artifacts • Scaling alone doesn’t fix reconstruction quality What’s different: • Explicit separation of geometry vs appearance • Pose-conditioned generation, not implicit guessing • Robustness to noisy or imperfect camera estimates What this unlocks: • Cleaner edges, fewer artifacts in splats • Faster, more stable radiance field generation • Practical pipelines without heavy optimization loops The shift: From “learn everything at once” → to “specialize, then coordinate” For image-to-3D pipelines, that changes everything. Project page: https://lnkd.in/eMwxGFgw Whitepaper: https://lnkd.in/evm2_dZp Code: https://lnkd.in/gDt-Naj2 My #YouTube Playlist of #LinkedIn content (searchable!): https://lnkd.in/eVQBGUTS

Snap-it, Tap-it, Splat-it Tactile-Informed 3D Gaussian Splatting for Reconstructing Challenging Surfaces Touch and vision go hand in hand, mutually enhancing our ability to understand the world. From a research perspective, the problem of mixing touch and vision is underexplored and presents interesting challenges. To this end, we propose Tactile-Informed 3DGS, a novel approach that incorporates touch data (local depth maps) with multi-view vision data to achieve surface reconstruction and novel view synthesis. Our method optimises 3D Gaussian primitives to accurately model the object's geometry at points of contact. By creating a framework that decreases the transmittance at touch locations, we achieve a refined surface reconstruction, ensuring a uniformly smooth depth map. Touch is particularly useful when considering non-Lambertian objects (e.g. shiny or reflective surfaces) since contemporary methods tend to fail to reconstruct with fidelity specular highlights. By combining vision and tactile sensing, we achieve more accurate geometry reconstructions with fewer images than prior methods. We conduct evaluation on objects with glossy and reflective surfaces and demonstrate the effectiveness of our approach, offering significant improvements in reconstruction quality.

4D3R: Motion-Aware Neural Reconstruction and Rendering of Dynamic Scenes from Monocular Videos https://lnkd.in/e58P_3qm Novel view synthesis from monocular videos of dynamic scenes with unknown camera poses remains a fundamental challenge in computer vision and graphics. While recent advances in 3D representations such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) have shown promising results for static scenes, they struggle with dynamic content and typically rely on pre-computed camera poses. We present 4D3R, a pose-free dynamic neural rendering framework that decouples static and dynamic components through a two-stage approach. Our method first leverages 3D foundational models for initial pose and geometry estimation, followed by motion-aware refinement. 4D3R introduces two key technical innovations: (1) a motion-aware bundle adjustment (MA-BA) module that combines transformer-based learned priors with SAM2 for robust dynamic object segmentation, enabling more accurate camera pose refinement; and (2) an efficient Motion-Aware Gaussian Splatting (MA-GS) representation that uses control points with a deformation field MLP and linear blend skinning to model dynamic motion, significantly reducing computational cost while maintaining high-quality reconstruction. Extensive experiments on real-world dynamic datasets demonstrate that our approach achieves up to 1.8dB PSNR improvement over state-of-the-art methods, particularly in challenging scenarios with large dynamic objects, while reducing computational requirements by 5x compared to previous dynamic scene representations. --- Newsletter https://lnkd.in/emCkRuA More story https://lnkd.in/eMFcEekQ LinkedIn https://lnkd.in/ehrfPYQ6 #AINewsClips #AI #ML #ArtificialIntelligence #MachineLearning #ComputerVision

3DGS looks amazing, but it requires millions of tiny blobs to look that good! There has to be a better way... and it turns out, surfels plus some neural magic are coming to the rescue! Check out Nexels, a new representation that decouples how a scene looks from how it's shaped. Instead of using millions of blobs to capture a simple flat wall with a complex texture, it uses a sparse set of quads and a global neural field to handle the heavy lifting. TLDR: Insane Efficiency: It hits the same quality as Gaussian Splatting but uses up to 31x fewer primitives and a fraction of the memory. Speed: It renders at a smooth 50+ FPS, making it twice as fast as previous textured methods. Details: No more "blurry blobs", it uses a technique create sharp edges and flat surfaces that look like the real deal. This is a step toward getting high-fidelity 3D scenes to run on leaner hardware without over optimizing and losing the "wow" factor. Check out the project page for the paper and code: https://lnkd.in/grxTR96t #3DGS #ComputerVision #3D

How can a single feed-forward model reconstruct a high-quality 3D scene from unposed, uncalibrated images? ETH Zürich, ETH AI Center and Microsoft present "YoNoSplat: You Only Need One Model for Feedforward 3D Gaussian Splatting". It introduces a unified 3D scene reconstruction approach that directly predicts 3D Gaussian splats and camera poses from arbitrary image collections with or without known pose or intrinsics. The model uses a novel mixing training strategy to stabilize the joint learning of geometry and camera parameters and resolves scale ambiguity through pairwise camera-distance normalization. YoNoSplat demonstrates exceptional efficiency, reconstructing a scene from 100 views in ∼2.7 s on a GH200 GPU while achieving state-of-the-art performance in both pose-dependent and pose-free settings. Checkout the links in the comments for more info on the project and the team behind it! #computervision #machinelearning #3dreconstruction #gaussiansplatting #deeplearning #research #novelviewsynthesis #ai