Engineering Simulation Tools Overview

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

Building AI for the real world is a very different problem than building AI for text. I sat down with Steve Xie, Ph.D., Founder & CEO of Lightwheel on The Ravit Show, to break down what it actually takes to train systems that operate in the physical world. Steve’s journey from Peking University to Columbia University, and then into leadership roles at Cruise, NVIDIA, and NIO, gives him a unique lens into where today’s AI systems struggle when they leave controlled environments and face the real world. One of the biggest takeaways from this conversation is that the core bottleneck in AI is no longer models, it is data. While large language models benefited from massive, passive data sources, robotics has no equivalent. There is no scalable way to collect real-world interaction data, no reliable evaluation layer, and very little infrastructure to continuously improve systems once deployed. This is where simulation becomes critical. In autonomous driving, simulation is helpful. In robotics, it is foundational. You cannot run thousands of parallel experiments in the real world, and you cannot reset physical environments at will. Simulation is what makes learning, testing, and iteration possible at scale. But not everything that looks like simulation actually works. As Steve explains, true simulation needs to be physically accurate, reproducible, and capable of generating actionable feedback. Without that, it cannot train real systems. What makes Lightwheel interesting is their approach to solving this problem. Instead of starting with data collection, they start with evaluation. They identify where models fail, generate targeted data to fix those failures, and create a continuous feedback loop. It is a shift from a passive data pipeline to an active data engine built for physical AI. They are already working with teams like DeepMind, ByteDance, and Alibaba, building infrastructure that sits beneath both robotics companies and AI labs. The bigger idea is simple. You cannot scrape your way to physical intelligence. You have to generate, test, and refine data in closed loops. #data #ai #robot #nvidiagtc #lightwheel #api #training #behaviour #theravitshow

You cannot train physical AI on reality alone. There is not enough of it. Jensen Huang explains why NVIDIA built Alpamayo, a robotics model that learns from synthetic data grounded in physics. The problem is fundamental. Teaching physical AI like autonomous vehicles or robotics requires vast amounts of diverse interaction data. Videos exist. Lots of videos. But hardly enough to capture the diversity and type of interactions needed. So NVIDIA transformed compute into data. Using synthetic data generation grounded and conditioned by laws of physics, they can selectively generate training scenarios reality cannot provide. The example Huang shows is remarkable. A basic traffic simulator output gets fed into Cosmos AI world model. What emerges is physically based, physically plausible surround video that AI can learn from. This solves a constraint that limited physical AI development. You cannot train autonomous systems on every possible scenario by recording reality. There are not enough cameras, time, or situations. But you can simulate physics accurately enough that AI trained on synthetic data generalizes to real environments. Why this matters beyond autonomous vehicles. Any AI learning physical interactions faces the same data scarcity problem. Manufacturing robots, warehouse automation, infrastructure inspection, medical robotics. All require training on scenarios that are rare, dangerous, or impossible to capture at scale. Synthetic data generation grounded in physics laws becomes essential infrastructure for physical AI deployment. The organizations building AI for physical systems will either master synthetic data generation or remain limited by whatever reality they can record. Watch the full presentation to hear Huang explain how Alpamayo generates training data for autonomous vehicles that think like humans. What physical AI application needs synthetic data because reality cannot provide enough examples?

A humanoid robot costs $90K to break once. AI lets you break thousands... and learn from every fall. My background is mechanical engineering, robotics, and integration & test. But this field is moving so fast with AI that reading articles wasn't cutting it anymore. I felt out of the loop, so... I recently upgraded my personal setup to support AI training workloads and ran my first experiment: Teaching a bipedal (two-legged) humanoid robot to navigate a custom parkour course using reinforcement learning in NVIDIA Isaac Lab 5.1. But before I share what I learned, let me explain what's actually happening under the hood. A GPU-accelerated AI agent runs thousands of virtual robots in parallel. Each one learns from its own falls and successes simultaneously. The AI develops a "control policy," which is the brain that tells a robot how to move through the physical world. Why does this matter? Because what once required million-dollar labs and months of physical testing can now run on a single AI-capable GPU in hours. Robotics R&D is becoming software-first. Here's what that looked like for this experiment: 76 minutes of CUDA-accelerated training time. 393 million training steps. 4,096 robots learning in parallel on my RTX 5080. So what did I learn so far? Three things stood out to me: 》The setup before you can hit "Run" is a challenge. It took me seven hours to troubleshoot versioning, packages, and dependencies before I could run anything. I forced myself to do it manually because I wanted to understand what's under the hood. YouTube tutorials hit their limit quickly, but thankfully the NVIDIA developer forums saved me. 》The cost case is undeniable. A Unitree H1 costs around $90K. I *virtually* crashed thousands of them. My damage bill? $0. Simulation lets you fail-forward at scale. This gets you to a solid starting point for physical testing, but... 》The Sim-to-Real gap is real. This policy works well in simulation, but I couldn't get a feel for stress points, sensor behavior, or true stability. Failure is not predictable and happens at the edges. The next step would be to transfer this policy to a physical robot, gather real-world data, and continuously aligning the simulation to close that gap. The key thing here is: Testing real hardware is expensive. Simulation in software is cheap. How can you leverage both, intelligently? The benefit isn't limited to cost savings. This workflow also compresses developmental cycles and allows you to field systems faster. Do you think virtual simulation is a game-changer that is here to stay, or a fad? How would you build confidence in a robotic control policy that is trained in a virtual world? #robotics #ai #nvidia #omniverse #isaaclab ~~~~~~~~ Citations: NVIDIA IsaacLab -> https://lnkd.in/ekVMDnDc RSL-RL -> https://lnkd.in/eJye3XTW Unitree H1-> unitree.com/h1/ Note: this is an educational personal project. Opinions are my own, no affiliation or endorsement.

Closing the sim-to-real gap in humanoid robotics requires massive simulation throughput and high-fidelity physics validation. WPP recently detailed their engineering pipeline, showing how they reduced reinforcement learning cycle times for complex humanoid locomotion from 24 hours down to less than 60 minutes. The hardware architecture relies on Google Cloud’s new G4 VMs (powered by NVIDIA RTX PRO 6000 Blackwell GPUs) running NVIDIA Isaac Sim, integrated closely with DeepMind’s MuJoCo physics engine. The mechanics: The team mapped raw human mocap data (over 200 degrees of freedom) down to a constrained 29-DOF OpenUSD digital twin. By leveraging a P2P GPU topology to bypass central processing bottlenecks, the infrastructure executed over 3 billion simulations in under an hour. The virtual environment continuously introduced physical micro-variances—simulated pushes, shifting floor friction, and momentum changes—to train the model against the chaos of the real world. The resulting reinforcement learning model was condensed into a highly efficient ONNX policy and deployed directly to the physical robot. This edge policy processes live IMU and joint telemetry to output immediate, stabilized motor commands. Reaching this scale of simulation volume is the precise engineering mechanism that allows control policies to handle unstructured physical deployment. To support the research, Unitree has open-sourced the underlying RL code on GitHub. Blog post : https://lnkd.in/g4-gWzTP #Robotics #PhysicalAI #ReinforcementLearning #MuJoCo #GoogleCloud #IsaacSim #Engineering

Are we looking at software engineering the wrong way? What if it’s less about writing code and more about making better decisions? Learn a revolutionary approach to understanding complex software systems in my conversation with Tudor Girba, the CEO of feenk. We explore 𝙈𝙤𝙡𝙙𝙖𝙗𝙡𝙚 𝘿𝙚𝙫𝙚𝙡𝙤𝙥𝙢𝙚𝙣𝙩, a groundbreaking concept that challenges traditional views of software engineering. Learn why treating development as a decision-making process, supported by custom tools, is crucial for tackling today’s software challenges, especially when dealing with legacy systems. Key topics discussed: ⤷ 𝗦𝗼𝗳𝘁𝘄𝗮𝗿𝗲 𝗘𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗮𝘀 𝗗𝗲𝗰𝗶𝘀𝗶𝗼𝗻-𝗠𝗮𝗸𝗶𝗻𝗴: Why software development is fundamentally about making informed decisions rather than just constructing systems. ⤷ 𝗧𝗵𝗲 𝗜𝗻𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝗰𝘆 𝗼𝗳 𝗥𝗲𝗮𝗱𝗶𝗻𝗴 𝗖𝗼𝗱𝗲: Developers spend over 50% of their time reading code, yet this activity remains unoptimized. ⤷ 𝗠𝗼𝗹𝗱𝗮𝗯𝗹𝗲 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁: Learn how creating custom tools tailored to specific problems can revolutionize your workflow and decision-making process. ⤷ 𝗟𝗲𝗴𝗮𝗰𝘆 𝗦𝘆𝘀𝘁𝗲𝗺𝘀 𝗮𝘀 𝗢𝗽𝗽𝗼𝗿𝘁𝘂𝗻𝗶𝘁𝗶𝗲𝘀: Reframe legacy systems as value-creation opportunities instead of burdens. ⤷ 𝗚𝗹𝗮𝗺𝗼𝗿𝗼𝘂𝘀 𝗧𝗼𝗼𝗹𝗸𝗶𝘁: Discover the innovative development environment enabling thousands of micro-tools for better system understanding. ⤷ 𝗧𝗵𝗲 𝗙𝘂𝘁𝘂𝗿𝗲 𝗼𝗳 𝗗𝗲𝘃𝗲𝗹𝗼𝗽𝗺𝗲𝗻𝘁 𝗘𝗻𝘃𝗶𝗿𝗼𝗻𝗺𝗲𝗻𝘁𝘀: Explore how AI, moldable development, and tools like Glamorous Toolkit can coexist to solve diverse class of problems. This conversation will completely transform how you think about software development!

𝗦𝗼 𝗵𝗮𝘀 𝘆𝗼𝘂𝗿 𝟯𝗗 𝗶𝗻𝘃𝗲𝘀𝘁𝗺𝗲𝗻𝘁 𝗱𝗿𝗶𝗲𝗱 𝘂𝗽 𝗶𝗻 𝗳𝗮𝘃𝗼𝘂𝗿 𝗼𝗳 𝘀𝗵𝗶𝗻𝘆 𝗻𝗲𝘄 𝗔𝗜 𝘁𝗼𝗼𝗹𝘀? There's definitely a shift happening : teams chasing AI outputs while 3D work gets put on hold. That move usually comes from a misunderstanding. 𝗔𝗜 𝗮𝗻𝗱 𝟯𝗗 𝗮𝗿𝗲 𝗻𝗼𝘁 𝘀𝘂𝗯𝘀𝘁𝗶𝘁𝘂𝘁𝗲𝘀!!! They drive different outcomes. 📈AI is for fast variation. It’s useful when you need to explore directions quickly, generate multiple visual routes, and support marketing and creative exploration. 📈 3D is for product decisions that hold up. It supports fit intent, construction logic, consistent asset creation, and development decisions that can travel across teams without being reinvented every week. When 3D is replaced with AI, the business often gets a higher volume of visuals, but loses the thing that reduces rework: a reliable source of product truth. There’s also a second effect that decision makers should take seriously: more output can slow decisions. When it becomes effortless to request another version, review cycles expand. The work shifts from progressing the product to debating options. So the question isn’t which tool is “better.” The question is: what decision are we trying to improve? If the decision needs speed and breadth of visual routes, AI helps. If the decision needs accuracy, continuity, and fewer late-stage changes, 3D helps. If you want both, you need boundaries, so exploration has an end point and commitment has a clear owner. Question for you Fashion business, Design and PD leads: Have your newer AI tools improved decision speed in your organisation, or increased the number of options you’re debating? Comment 👇🏾👇🏾👇🏾 #DesignLeadership #RetailStrategy #DecisionMaking

🔋 Calculation of Total Remaining Energy in a Cell – The Crucial Role of Battery Management Systems (BMS) 🔋 In advanced battery-powered systems—whether in electric vehicles, grid-scale storage, or renewable integration—the accurate estimation of remaining energy is not merely informative, but mission-critical. This estimation hinges on the intelligent functioning of the Battery Management System (BMS), which serves as the analytical core of every modern battery pack. At the center of this functionality lies the State of Charge (SOC)—a dimensionless quantity (typically expressed as a percentage) representing the ratio of the current charge to the cell’s rated capacity. ⚙️ BMS Techniques for SOC & Energy Estimation: 🔸 Coulomb Counting – Integrates charge inflow/outflow over time. Accurate under stable conditions but accumulates drift over long durations. 🔸 Open Circuit Voltage (OCV) Mapping – Relates terminal voltage to SOC at rest. Highly accurate but impractical under dynamic loads. 🔸 Equivalent Circuit Modeling (ECM) – Represents battery dynamics using RC networks, enabling real-time estimation under varying operating conditions. 🔸 State Estimation Algorithms – Techniques like the Extended Kalman Filter (EKF) or Unscented Kalman Filter (UKF) provide robust SOC and energy estimation even under noise and system uncertainties. 🔸 Data-driven & ML Approaches – Growing rapidly in popularity, these methods exploit real-world data for adaptive estimation and degradation tracking. 🌐 Practical Relevance: The ability of a BMS to precisely estimate remaining energy directly impacts: 🔸 Range prediction in EVs, 🔸 Optimal power dispatch in BESS applications, 🔸 Thermal and safety management, 🔸 Cycle life optimization, 🔸 Grid reliability when integrated with renewable energy systems. As we transition toward a cleaner and smarter energy future, robust and intelligent BMS design will remain pivotal to the safety, performance, and longevity of energy storage technologies. 🔍 Whether you're in research, product development, or system planning—understanding the interplay between electrochemical theory and algorithmic estimation is key to innovating in the battery domain. #BatteryManagementSystem #StateOfCharge #RemainingEnergy #EnergyStorageSystems #BESS #EVTechnology #KalmanFilter #SOCEstimation #BatteryModelling #SmartGrids #PowerSystemsEngineering #CleanTech #SustainableEnergy #EnergyAnalytics #ElectricalEngineering

Publicly Accessible Energy Storage Systems (ESS) Simulation Price-taker models are suitable for small-scale ESS as their capacity does not influence market prices or system dispatch. This post highlights DOE price-taker valuation tools. 🟦 1) QuESt  QuESt is a free, open-source Python application suite for energy storage simulation and analysis, developed at Sandia National Laboratories. It includes three interconnected applications:  1- QuESt Data Manager,  2-QuESt Valuation, and  3-QuESt BTM, Eligible technologies include BESS (Li-ion, advanced lead-acid, vanadium redox), flywheels, and PV, using a shared model for different BESS and flywheel types based on their parameters. 🟦 2) Renewable Energy Integration and Optimization (REoptTM)  The REopt™ platform, developed by the National Renewable Energy Laboratory (NREL), optimizes energy systems for various applications, recommending the best mix of renewable energy, conventional generation, and energy storage to achieve cost savings, resilience, and performance goals. Eligible technologies include: PV, wind, CHP, electric and thermal energy storage, absorption chillers, and existing heating and cooling systems. 🟦 3) Distributed Energy Resources Customer Adoption Model (DER-CAM)  DER-CAM is a decision support tool from Lawrence Berkeley National Laboratory (LBNL) designed to optimize DER investments for buildings and multienergy microgrids. Eligible technologies include conventional generators, CHP units, wind and solar PV, solar thermal, batteries, electric vehicles, thermal storage, heat pumps, and central heating and cooling systems. 🟦 4) System Advisor Model (SAM) SAM is a techno-economic computer model that evaluates the performance and financial viability of renewable energy projects. It includes performance models for various systems such as PV (with optional battery storage), concentrating solar power, solar water heating, wind, geothermal, and biomass, and a generic model for comparison with conventional systems. Eligible technology types focus on electrochemical ESS, supporting lead-acid, Li-ion, vanadium redox flow, and all iron flow batteries. Users can also model custom battery types by specifying their voltage, current, and capacity. SAM offers detailed modelling of battery cells, power converters, and factors like degradation, voltage variation, and thermal properties. 🟦 5) Energy Storage Evaluation Tool (ESETTM) ESETTM is a suite of modules developed at PNNL that allows utilities, regulators, and researchers to model and evaluate various ESSs. ESETTM features a modular design for ease of use and currently includes five modules for different ESS types, such as BESSs, pumped-storage hydropower, hydrogen energy storage, storage-enabled microgrids, and virtual batteries. Some applications also include distributed generators and photovoltaics (PV). Source: see post image. Link to the modellers: in the comment section This post is for educational purposes only.

Electrochemical Impedance Spectroscopy (EIS) is a crucial non-destructive technique used to characterize complex electrochemical processes within batteries, vital for parametrizing advanced models. The provided image illustrates typical EIS data through two plots: the Nyquist plot (top) displays the real versus negative imaginary impedance, while the Bode plot (bottom) shows impedance magnitude against frequency. These plots reveal distinct regions corresponding to specific battery phenomena. The high-frequency intercept on the Nyquist plot signifies Series Resistance. Moving clockwise, a semicircle emerges, representing Charge Transfer Kinetics at electrode-electrolyte interfaces. At lower frequencies, further features indicate Electrolyte Diffusion, and a characteristic low-frequency tail corresponds to Particle Diffusion within the active materials. Our research highlights how EIS data is inherently valuable for parametrizing physics-based battery models. We studied various models like the single-particle model (SPM), Doyle-Fuller-Newman (DFN), and, critically, the Single-Particle Model with electrolyte (SPMe). The SPMe emerged as a parsimonious choice, effectively capturing the typical impedance features observed in measured lithium-ion cell data, as depicted. This work involves a grouped-parameter SPMe, analyzing impedance features related to each parameter. We successfully parametrized the SPMe for an LG M50LT cell using measured EIS data across different states-of-charge, validating it on a drive cycle with errors below 15 mV. While challenging, this framework allows for easy model extension to achieve better fits, offering a robust approach for battery characterization and simulation. #EIS #Electrochemistry #BatteryResearch #LithiumIonBatteries #PhysicsBasedModels #BatteryModeling #ImpedanceSpectroscopy #EnergyStorage #ScientificPublication #OpenScience #PyBaMM #PyBOP #OxfordUniversity #FaradayInstitution