Dmitry Nedov’s Post

AI is rapidly evolving beyond text and 2D images — it’s now beginning to understand the 3D world. A recent breakthrough is SpatialLM, an open-source 3D Large Language Model developed by researchers at Manycore and accepted at NeurIPS 2025. Unlike traditional vision-language models that analyze flat images, SpatialLM can understand real-world physical environments using 3D point clouds collected from LiDAR, RGBD cameras, and even standard videos. What makes this exciting is its ability to convert raw spatial data into structured semantic understanding. It can identify: • Walls, doors, and windows • Furniture and surrounding objects • Precise 3D object locations and orientations Instead of generating only conversational text, SpatialLM produces structured Python-style scene representations that machines can directly reason about. This unlocks huge possibilities for: • Robotics — helping autonomous agents navigate and interact with environments more intelligently. • AR/VR — enabling seamless mixed-reality experiences through instant spatial mapping. • Accessibility — supporting advanced navigation systems for visually impaired individuals. The shift from recognizing 2D pixels to understanding 3D spatial relationships represents a major milestone in AI evolution. Spatial reasoning models could become a foundational technology for the future of robotics, spatial computing, and embodied AI. Read more about SpatialLM here: https://lnkd.in/dmYF5hVp What industry do you think will benefit the most from 3D AI models? #AI #ArtificialIntelligence #MachineLearning #SpatialLM #ComputerVision #Robotics #ARVR #SpatialComputing #NeurIPS2025

The Anatomy of a Ghost Driver: How Robo-taxis Think in Real-Time 🚖🤖 Ever wondered how a vehicle with no one behind the wheel navigates a chaotic intersection? It isn’t just "code"—it’s a high-concurrency, multi-modal engineering marvel. Under the hood, a robotaxi is a symphony of perception, localization, and high-speed actuation. 1. The 360° Digital Bubble (Perception) To drive safely, the car creates a real-time 3D reconstruction of its surroundings. It doesn't just "see"; it measures. * LiDAR: Sends out millions of laser pulses per second to create a high-precision Point Cloud, measuring the exact distance to every object. * Computer Vision: High-res cameras perform semantic segmentation, identifying whether a pixel belongs to a "pedestrian," "red light," or "pothole." * Radar.Provides critical velocity data, detecting how fast the car in front is braking, even in heavy rain or fog. 2. The Map is Not the Territory (Localization) While we use GPS, robotaxis use **HD Maps** with centimeter-level precision. The car compares its live sensor data against a pre-mapped "Satellite View" and street-level geometry. If the LiDAR sees a curb that is 2cm off from the map, the car knows exactly where it is positioned in the lane. 3. Object Detection & Intent Prediction It’s not enough to see a cyclist; the car must predict if that cyclist is about to veer left. Neural networks process the "bounding boxes" of objects and run probabilistic models to forecast movement trajectories for every actor in the scene. 4. From Data to Dirt: Actuation & Control This is where the "Motor Reflection" happens. The "Brain" (the stack) sends digital signals to the **Drive-by-Wire** system: * Acceleration/Braking: Digital commands are converted into precise torque requests for the electric motors or hydraulic pressure for the brakes. * Steering: Electronic actuators turn the steering rack based on calculated "Path Planning" curves. * Feedback Loops: The system uses PID controllers or Model Predictive Control (MPC) to constantly adjust. If the car feels a skid, it compensates in milliseconds—faster than a human "reflex." 5. The Magic of Real-Time Concurrency ⚡ How does it handle this every single second? The architecture is typically asynchronous yet deterministic. * Sensor Fusion: Instead of waiting for all sensors (synchronous), the system uses a high-speed "Data Bus" (like DDS or specialized ROS2 implementations) to fuse data as it arrives. * Edge Computing: Processing happens on-vehicle. High-performance GPUs and FPGAs handle the heavy lifting to keep latency below **100ms**. The result? A vehicle that "grasps" its surroundings hundreds of times faster than a human, making decisions while we would still be processing the "image" of a stop sign. The future isn't just "self-driving" we will soon see AI creating a 3D map of our house through all the cameras and actually work like a maid with distributed systems and real-time robotics. #AI

Most computer vision systems today still operate in a fundamentally flat world. 2D vision can detect, classify, segment, and track objects remarkably well — but it lacks one critical capability: Spatial understanding. That limitation becomes obvious in robotics, industrial automation, autonomous systems, AR/VR, and intelligent manufacturing environments where depth, geometry, and real-world positioning matter more than pixel appearance alone. 2D Vision excels at: • Object detection • OCR and document analysis • Surface-level defect inspection • Human pose estimation • Retail analytics and surveillance • Image-based classification pipelines But 2D systems struggle when: • Occlusion increases • Perspective changes dramatically • Distance estimation is required • Objects overlap in cluttered scenes • Real-world dimensions must be measured • Navigation or manipulation is involved This is where 3D vision changes the architecture entirely. Instead of understanding only pixels, 3D systems reconstruct spatial structure using: • Stereo vision • LiDAR fusion • Depth cameras • Multi-view geometry • SLAM pipelines • Point cloud processing The result is a perception stack capable of: • Depth-aware object localization • Volumetric scene understanding • Collision-aware robotics • Precise industrial measurement • Autonomous navigation • Spatial mapping in dynamic environments What’s interesting is that modern AI systems are increasingly combining both approaches rather than replacing one with the other. The future is not: “2D vs 3D” It’s: “Semantic understanding + spatial intelligence.” That convergence is becoming the foundation of next-generation perception systems across manufacturing, healthcare imaging, robotics, logistics, and autonomous mobility. Computer vision is no longer just about recognizing what is visible. It’s about understanding the physical world itself. #ComputerVision #ArtificialIntelligence #3DVision #2DVision #DeepLearning #MachineLearning #Robotics #AutonomousSystems #IndustrialAI #SpatialComputing #AIInnovation #ComputerVisionAI

Arxiv: https://lnkd.in/ew7X_5Gb Project: [Link not provided] 🔁 At a Glance 💡 Goal: Enable accurate, real-time robot localization by matching hierarchical scene graphs with prior architectural maps. ⚙️ Approach: - Graph Augmentation: Adds intra- and inter-level edges to encode spatial relations. - Shared MLP Encoder: Produces type-aware node embeddings for heterogeneous nodes. - Differentiable Matching: Uses affinity matrices and Sinkhorn algorithm for soft assignment. 📈 Impact (Key Results) 🧪 Accuracy and Speed: - Outperforms combinatorial baseline in F1 score on real LiDAR data. - Runs over 80× faster than previous methods. - Achieves high generalization zero-shot to real environments. 🔄 Scalability: - Handles partial/noisy observations. - Maintains real-time performance in complex environments. 🤖 Robustness: - Tolerates sensor noise and domain shift. 🔬 Experiments 🧪 Benchmarks: Synthetic MSD dataset, real LiDAR RE environment. 🎯 Tasks: Scene graph matching for localization. 🦾 Setup: Nvidia GPU, LiDAR sensor. 📐 Inputs: Architectural plans, LiDAR scans. 🛠 How to Implement 1️⃣ Construct hierarchical A-graphs and S-graphs. 2️⃣ Enrich graphs with relation edges. 3️⃣ Encode nodes via shared GATv2-based encoder. 4️⃣ Compute affinity matrix and normalize. 5️⃣ Apply Sinkhorn and Hungarian algorithms for matching. 📦 Deployment Benefits ✅ Fast, scalable real-time localization. ✅ Zero-shot generalization from synthetic to real data. ✅ Robust against observation noise. ✅ Compatible with existing BIM models. 📣 Takeaway This approach bridges semantic scene understanding with architectural prior maps, enabling more reliable and efficient indoor robot localization. Why it matters: It unlocks scalable, real-time semantic SLAM integrated with high-level structural priors. Follow me to know more about AI, ML and Robotics!

GenAssets: Generating in-the-wild 3D Assets in Latent Space Arxiv: https://lnkd.in/eSDfeRXt 🔁 At a Glance 💡 Goal: Develop a scalable method to generate high-quality 3D assets from in-the-wild sensor data for traffic simulation. ⚙️ Approach: - Reconstruct-then-generate: Build complete 3D assets via neural rendering and latent diffusion. - Occlusion-aware neural rendering: Handle partial observations and occlusions. - Latent space learning: Compress scene and actor info for scalable generation. - Diffusion model training: Enable diverse, controllable asset creation. 📈 Impact (Key Results) 🧪 Reconstruction: - Outperforms state-of-the-art on sparse, novel, and 360° views. - More accurate and complete in diverse scenarios. 🔄 Generation: - Better visual quality with lower FID/KID scores. - Supports conditioning on class, time, and sparse sensor input. 🤖 Applications: - Single-image 3D reconstruction. - Data augmentation improves detection accuracy. 🔬 Experiments 🧪 Benchmarks: PandaSet dataset. 🎯 Tasks: 3D reconstruction, novel view synthesis, conditional generation. 🦾 Setup: 6 cameras, 360° LiDAR, real-world scenes. 📐 Inputs: Camera images, LiDAR point clouds, actor masks. 🛠 How to Implement 1️⃣ Collect in-the-wild LiDAR & camera data. 2️⃣ Train neural scene representation with occlusion awareness. 3️⃣ Encode actors into latent space. 4️⃣ Train diffusion model on latent codes. 5️⃣ Generate and render assets conditioned on various inputs. 📦 Deployment Benefits ✅ High-quality and diverse 3D assets. ✅ Robust to partial occlusions & sparse views. ✅ Supports conditional and single-image generation. ✅ Enhances simulation realism and scalability. 📣 Takeaway This method pushes the boundary of real-world 3D asset generation from sensor data. Enabling safer, more realistic autonomous vehicle simulations. Its scalability and controllability open new avenues in AI-driven simulation content. Follow me to know more about AI, ML and Robotics!

[HCSG]: Human-Centric Semantic-Geometric Reasoning for Safe Navigation Arxiv: https://lnkd.in/e3HTzNpK Project: https://lnkd.in/eD4vHEEn 🔁 At a Glance 💡 Goal: Enable robots to understand and anticipate human actions for socially safe navigation. ⚙️ Approach: - Combines geometric forecasting of human pose and trajectory with semantic understanding via vision-language models. - Fuses human cues into topological maps for instruction-driven planning. - Uses social distance loss to promote respectful human-robot interaction. 📈 Impact 🧪 Key Results: - 14.3% success rate increase in new environments. - 34.5% collision rate reduction. - Improved safety and goal achievement over baselines. 🤖 Real-world: - Robots effectively interpret human activities and proactively avoid collisions in physical tests. 🔬 Experiments - Benchmarks: HA-VLNCE indoor navigation with humans. - Tasks: Goal-reaching with social awareness. - Hardware: NVIDIA GPUs, Habitat, real robot. - Inputs: RGB-D images, instructions. 🛠 How to Implement 1. Build panoramic topological map from scene data. 2. Detect humans and gather short-term observations. 3. Process human pose, trajectory, and language-based actions. 4. Fuse features into map for instruction-guided planning. 5. Enforce social distances in navigation. 📦 Deployment Benefits ✅ Safer robot-human interaction. ✅ Better instruction grounding. ✅ anticipates movement for proactive avoidance. ✅ Generalizes to real settings. 📣 Takeaway Moving beyond obstacle avoidance, understanding human intent and future motion leads to socially aware, safer robots. Explicit semantic and geometric reasoning is crucial for natural human-robot coexistence. Follow me to know more about AI, ML and Robotics!

Depth estimation is one of the most important perception capabilities in modern computer vision systems. But the industry is still divided between two fundamentally different approaches: • Monocular depth estimation vs • Stereo depth estimation At a high level, both attempt to reconstruct 3D spatial understanding from visual input — but the engineering tradeoffs are entirely different. Stereo depth estimation relies on geometric correspondence between two synchronized cameras. By analyzing pixel displacement between left and right views, the system can infer scene depth with strong geometric consistency. This makes stereo systems highly effective for: • Robotics • Autonomous navigation • Industrial automation • Warehouse systems • AR/VR spatial tracking The advantage is physical depth grounding. The challenge is operational complexity. Stereo pipelines require: • Precise camera calibration • Baseline optimization • Synchronization stability • Occlusion handling • Dense correspondence matching • High-quality rectification pipelines Performance can degrade rapidly under: • Low-texture environments • Reflective surfaces • Repetitive patterns • Poor lighting conditions Monocular depth estimation approaches the problem differently. Instead of triangulation, the model learns spatial priors directly from massive datasets using deep neural architectures. The system predicts relative scene geometry from a single image by learning: • Object scale relationships • Perspective cues • Semantic context • Surface continuity • Scene structure priors Modern monocular systems increasingly leverage: • Vision transformers • Self-supervised learning • Multi-scale feature fusion • Temporal consistency modeling • Foundation-model-based scene understanding The operational advantage is significant. Monocular systems require: • Only one camera • Lower hardware cost • Simpler deployment • Easier edge integration • Reduced calibration overhead But they introduce a different challenge: Depth becomes probabilistic rather than physically constrained. This creates limitations in: • Absolute metric accuracy • Long-range precision • Edge-case generalization • Safety-critical autonomy In practice, the decision is rarely about “which is better.” It is about selecting the correct perception architecture for the operational environment. Stereo depth provides stronger geometric reliability. Monocular depth provides scalability and deployment flexibility. The future of perception systems will likely combine both: geometry-driven sensing + learned spatial reasoning. Because true machine perception is not just about seeing objects. It is about understanding space. #ComputerVision #ArtificialIntelligence #DepthEstimation #MonocularDepth #StereoVision #MachineLearning #DeepLearning #AutonomousSystems #Robotics #3DVision #EdgeAI #AIInnovation

Most Physical AI failures are not caused by bad AI models. They are caused by bad infrastructure timing. Right now, much of the industry still believes: “Add a GPU into a standard IPC, and you have an Edge AI platform.” That may work in a demo. But in real deployments, we continue seeing the same failures: ➡️ Thermal throttling ➡️ Random power resets ➡️ Unstable control loops ➡️ Multi-sensor timing drift The last one is the most underestimated. Your cameras, LiDAR, IMU, and encoders may all function correctly individually, but if they are not synchronized at the hardware level, your system is no longer perceiving the same physical reality at the same moment. Once timing uncertainty enters the control loop, reliability disappears. Physical AI is no longer just a compute problem. It is becoming a deterministic infrastructure problem. The industry is spending enormous effort optimizing AI models, while still underestimating synchronization architecture, hardware timing, and deterministic sensor infrastructure. That gap becomes very visible the moment systems leave the lab and enter real environments. We recently compiled a short internal deployment checklist covering: ➡️ Thermal resilience ➡️ Power stability ➡️ Hardware synchronization ➡️ Deterministic timing benchmarks If you are building robotics, autonomous systems, or industrial AI infrastructure, comment “HARDWARE” and I’ll send it over. #PhysicalAI #DeterministicSystems #RoboticsInfrastructure #SensorFusion #EdgeAI #FPGA

[Paper Title]: STAR-Filter: Efficient Convex Free-Space Approximation via Starshaped Set Filtering in Noisy Environments Arxiv: https://lnkd.in/ewW9T8fh 🔁 At a Glance 💡 Goal: Accelerate convex free-space approximation for robot planning in noisy, cluttered environments. ⚙️ Approach: - Starshaped set filtering: identifies key obstacle points for efficient region generation. - Sphere flipping map: constructs starshaped sets from point clouds. - Iterative polytope inflation: optimizes by reducing redundant obstacle constraints. - Boundary augmentation: improves approximation near map edges. 📈 Impact (Key Results) 🧪 Computation Time: - Significantly faster than existing methods like IRIS and Galaxy. - Robust to sensor noise and large-scale obstacle data. 🔄 Polytope Quality: - Achieves similar volumes to benchmark methods with reduced computation. - Maintains collision avoidance and seed inclusion. 🤖 Application: - Demonstrated in quadrotor planning tasks with real-time updates. - Effective in noisy, sparse sensor data environments. 🔬 Experiments 🧪 Benchmarks: Maze maps, real-world LiDAR and event camera datasets. 🎯 Tasks: Free-space polytope generation & safe flight corridor planning. 🦾 Setup: Real-time on onboard quadrotor hardware with raw point cloud data. 📐 Inputs: LiDAR, stereo event camera point clouds. 🛠 How to Implement 1️⃣ Use the point cloud data to construct starshaped sets via sphere flipping. 2️⃣ Filter obstacle points to find extreme boundary points. 3️⃣ Generate an inscribed polytope using supporting hyperplanes. 4️⃣ Iteratively refine the polytope with MVIE and normal updates. 5️⃣ Incorporate boundary points for robust environment approximation. 📦 Deployment Benefits ✅ Fast computation suitable for real-time environments. ✅ Robustness against sensor noise and large obstacle sets. ✅ Reduced conservative approximation, maximizing free space. ✅ Simple integration with existing motion planning pipelines. 📣 Takeaway This framework dramatically speeds up free-space region estimation using starshaped filtering. Reduces unnecessary obstacle computing, enabling agile, noise-robust robot navigation. Its lightweight design suits real-time onboard systems, pushing autonomous exploration forward. Follow me to know more about AI, ML and Robotics!

🚀 3D SLAM is not just about building maps. It is about making machines understand and navigate the real world with precision. Many deployments fail not because of algorithms alone, but because real environments introduce challenges such as: 📍 Localization drift 🚶 Dynamic moving objects 📡 Multi-sensor synchronization 🌙 Lighting variations 🏢 Feature-poor environments ⚡ Real-time edge constraints The real engineering challenge is converting research into reliable production systems. At Visual Grab, we work on solving these challenges through Computer Vision and AI by combining: ✅ Camera + LiDAR + IMU fusion ✅ Dynamic object filtering ✅ Semantic scene understanding ✅ Pose graph optimization ✅ Real-time edge inference ✅ 3D scene reconstruction ✅ AI-driven perception pipelines The goal is not simply SLAM. The goal is robust spatial intelligence that enables: 🏭 Industrial Automation 🚗 Autonomous Mobility 🏙 Smart Cities 🤖 Robotics 📦 Intelligent Warehousing Moving from “Where am I?” ➝ “What am I seeing?” ➝ “What action should I take?” is where the future of perception systems is heading. Follow Computer Vision AI Visual Grab to get more updates on computer vision. #ComputerVision #SLAM #3DVision #ArtificialIntelligence #Robotics #EdgeAI #SmartCities #IndustrialAutomation #SpatialAI #VisualGrab #visualgrab