Robotics Engineering for Advanced Human-Robot Interaction

Happy to share our latest paper, "Enabling Novel Mission Operations and Interactions with ROSA: The Robot Operating System Agent". This work was led by Rob R. in collaboration with Marcel Kaufmann, Jonathan Becktor, Sangwoo Moon, Kalind Carpenter, Kai Pak, Amanda Towler, Rohan Thakker and myself. Please find the #OpenSource code, paper, and video demonstration linked below. Operating autonomous robots in the field is often challenging, especially at scale and without the proper support of Subject Matter Experts (SMEs). Traditionally, robotic operations require a team of specialists to monitor diagnostics and troubleshoot specific modules. This dependency can become a bottleneck when an SME is unavailable, making it difficult for operators to not only understand the system's functional state but to leverage its full capability set. The challenge grows when scaling to 1-to-N operator-to-robot interactions, particularly with a heterogeneous robot fleet (e.g., walking, roving, flying robots). To address this, we present the ROSA framework, which can leverage state-of-the-art Vision Language Models (VLMs), both on-device and online, to present the autonomy framework's capabilities to operators in an intuitive and accessible way. By enabling a natural language interface, ROSA helps bridge the gap for operators who are not roboticists, such as geologists or first responders, to effectively interact with robots in real-world missions. In our video, we demonstrate ROSA using the NeBula Autonomy framework developed at NASA Jet Propulsion Laboratory to operate in JPL's #MarsYard. Our paper also showcases ROSA's integration with JPL's EELS (Exobiology Extant Life Surveyor) robot and the NVIDIA Carter robot in the IsaacSim environment (stay tuned for ROSA IssacSim extension updates!). These examples highlight ROSA's ability to facilitate interactions across diverse robotic platforms and autonomy frameworks. Paper: https://lnkd.in/g4PRjF4V Github: https://lnkd.in/gwWXmmjR Video: https://lnkd.in/gxKcum27 #Robotics #Autonomy #AI #ROS #FieldRobotics #RobotOperations #NaturalLanguageProcessing #LLM #VLM

Researchers at Osaka University have developed a new technology enabling androids to express mood states, such as "excited" or "sleepy," through dynamic facial movements modeled as overlapping, decaying waves. Traditional androids have relied on pre-programmed scenarios to convey emotions, often resulting in unnatural, disconnected expressions that can make human interactions uncomfortable. The new method, led by Hisashi Ishihara, uses "waveform-based" synthesis to create real-time facial expressions. Individual waveforms represent gestures like blinking, yawning, and breathing, which are propagated across the android’s face and overlaid to produce complex, fluid movements. This eliminates the need for pre-configured action scenarios and minimizes abrupt movement transitions. Additionally, waveform modulation allows the robot’s internal state to influence facial expressions instantly, reflecting changes in mood and enhancing emotional communication. Senior researcher Koichi Osuka emphasizes that this innovation can help robots interact with humans in a more expressive and natural manner. Ishihara envisions future androids with deeply integrated emotional cues, making them feel more lifelike and capable of meaningful connections. This advancement has the potential to greatly improve communication robots in various settings. Read more: https://lnkd.in/eRwTGGne

How can we enable robots to fluently collaborate with humans on physically demanding tasks? In our #HRI2025 paper, we focus on the task of human-robot collaborative transport, where a human and a robot work together to move an object to a goal pose. In the absence of explicit or a priori coordination, critical decisions such as navigating obstacles or determining object orientations become especially challenging. Our key insight is that a human and a robot can coordinate fluently by leveraging the transported object as a communicative medium. By encoding subtle, communicative signals into actions that affect the state of the transported object, the robot could effectively convey its intended strategy and role. To this end, we designed an inference mechanism that probabilistically maps observations of joint actions executed by the human and the robot to a set of joint strategies of workspace traversal, drawing from topological invariance. Integrated into a model predictive controller (IC-MPC), this mechanism enables a robot to estimate the uncertainty of its human partner over a traversal strategy, and take proactive corrective actions balancing uncertainty minimization and task efficiency. We deployed IC-MPC on a mobile manipulator (Hello Robot Stretch) and evaluated it in a within-subjects lab study (N = 24). IC-MPC enables greater team performance and empowers the robot to be perceived as a significantly more fluent and competent partner compared to baselines lacking a communicative mechanism. My fantastic PhD student, Elvin Yang, will present this work in the 1A: Human-Robot Collaboration Session on Tuesday, and in the X-HRI workshop today! paper: https://lnkd.in/gidfgq4W code: https://lnkd.in/gR8gAEud video: https://lnkd.in/gzkktKzf #robotics #humanrobotinteraction #artificialintelligence University of Michigan Robotics Department

Special thanks to Prof. Ahmed H. Qureshi for the personalized tour of his CORAL Lab at Purdue University. CORAL’s research spans machine learning, robot planning and control, and my personal favorite area, safe human–robot collaboration. Together, these threads tackle one of the hardest problems in robotics today: how autonomous systems can learn, plan, and act effectively while operating alongside people in real, unstructured environments. What stood out to me is how the lab integrates learning and decision-making with explicit attention to safety, interaction, and shared spaces. CORAL explores how robots can reason about uncertainty, model human behavior, and adapt their plans in ways that support collaboration rather than conflict. This includes work on risk-aware planning, learning-based control, and interaction-aware decision-making that directly addresses how robots should behave around and with humans. Several of my favorite papers from the lab dive deeply into these themes, including work on safe and interactive planning, human-aware risk representations, and learning frameworks that support trustworthy collaboration between humans and robots. I’ve shared links to a few of these papers below for anyone who wants to explore further. As robots increasingly leave controlled environments and enter factories, hospitals, warehouses, and public spaces, this kind of research becomes foundational. Autonomy that ignores the human context will struggle to scale. Autonomy that understands and respects it has the potential to truly transform how we work and live. Many thanks again to Ahmed and the CORAL team for the warm welcome and the great conversations (remember: replace the banana with a beer bottle for social impact! 😉 ). It was energizing to see research that so clearly connects theory, algorithms, and real-world human impact. Purdue Computer Science ---- For those interested in going deeper, here are a few of my favorite papers from the CORAL Lab that really capture the breadth and impact of their work: 🔹 Safe and interactive planning for human–robot collaboration https://lnkd.in/eMMqED3r https://lnkd.in/eQJtSinY 🔹 Risk-aware representations and decision-making around humans https://lnkd.in/ep4dRPHM 🔹 Learning and control frameworks that enable safe, trustworthy interaction https://lnkd.in/eVjcQBpa https://lnkd.in/eSH-_CrV These papers do a great job of connecting learning, planning, and control with the realities of shared human–robot environments. Highly recommend a read if you’re working in robotics, autonomy, or human–robot interaction.

Saturday Robotics Reading Club 08 with ryan punamiya from NVIDIA GEAR, cohosted with Junfan Zhu The whole field of robotics is now trying to answer one question: how do we turn human experience into robot action? 1. Is human data actually the “internet” for robotics? Not yet. When people say “just scale human video,” they are skipping the hardest part: translation across embodiment. The important shift is not “use more human data.” The shift is to treat human behavior as a first-class training signal, not as a side channel for better visual representations or auxiliary pretraining. Early pipelines that used point tracks, value functions, or representation learning were useful, but still treated human data as indirect supervision rather than the thing the policy actually learns from. 2. Why doesn’t naïve human-to-robot transfer work? Human-to-robot transfer breaks because humans and robots differ across kinematics, dynamics, sensing, and visual embodiment. Humans have compliant hands, rich tactile feedback, fluid motion, and implicit body priors. Robots have rigid joints, limited sensing, different cameras, and different control constraints. That is why learning from watching humans is not enough. A model can learn two separate distributions, one for human motion and one for robot control. 3. If representation learning wasn't enough, what's better? For years, the default bet was: learn better visual representations, then fine-tune a policy. But the representation has to be action-aware and embodiment-aware. EgoBridge is an interesting work because instead of naïve alignment, it uses action-aware latent alignment through transport. 4. Do we know what modalities matter the most? Egocentric video is the cheapest proxy for human experience, but ego videos plus pose only capture a thin slice. We still do not know which modalities matter most, what the ROI of each sensor is, or how to unify data from gloves, stereo cameras, tactile insoles, full-body mocap, and robot sensors into one foundation model. It's tempting to just say add more sensors, but more sensors also create more cost, standardization problems, and behavior shift. The long-term answer is probably a data pyramid: broad cheap video at the base, richer calibrated multimodal data above it, and high-precision instrumented data at the top. 5. So, what is the data scaling law that we are still missing? EgoScale suggests that after some point, adding more demonstrators may matter less than adding more scenes, objects, and task settings. One hour of mopping data and one hour of soldering data are VERY different. Some tasks have low variation; others exhibit combinatorial explosion across object geometry, grasp strategy, contact dynamics, tool usage, and scene setup. So “how many hours of data do we need?” is the wrong question. The way to scale data in robotics is not about hours, but about task-dependent entropy.

Presenting FEELTHEFORCE (FTF): a robot learning system that models human tactile behavior to learn force-sensitive manipulation. Using a tactile glove to measure contact forces and a vision-based model to estimate hand pose, they train a closed-loop policy that continuously predicts the forces needed for manipulation. This policy is re-targeted to a Franka Panda robot with tactile gripper sensors using shared visual and action representa- tions. At execution, a PD controller modulates gripper closure to track predicted forces -enabling precise, force-aware control. This approach grounds robust low- level force control in scalable human supervision, achieving a 77% success rate across 5 force-sensitive manipulation tasks. #research: https://lnkd.in/dXxX7Enw #github: https://lnkd.in/dQVuYTDJ #authors: Ademi Adeniji, Zhuoran (Jolia) Chen, Vincent Liu, Venkatesh Pattabiraman, Raunaq Bhirangi, Pieter Abbeel, Lerrel Pinto, Siddhant Haldar New York University, University of California, Berkeley, NYU Shanghai Controlling fine-grained forces during manipulation remains a core challenge in robotics. While robot policies learned from robot-collected data or simulation show promise, they struggle to generalize across the diverse range of real-world interactions. Learning directly from humans offers a scalable solution, enabling demonstrators to perform skills in their natural embodiment and in everyday environments. However, visual demonstrations alone lack the information needed to infer precise contact forces.

The dark matter of robotics is “physical commonsense.” It’s everywhere, yet hard to pin down. From gently nudging an object to make space for fingers to grasp, to placing down a slipping object to get a better grip—these tiny corrections, recoveries, and “obvious” actions are subtle and automatic. We rarely notice them, but together they account for much of our extraordinary human ability to manipulate the physical world. This is the intelligence behind dexterity. And it can be learned on robots with data—but only if it’s the right data. Much of robot data today comes from remote control teleoperation, which often breaks the human sensorimotor loop: latency, limited tactile feedback, and unnatural interfaces push operators away from fast, reactive control (System 1 thinking) and towards slow, deliberate planning (System 2 thinking) e.g. “put one finger here… then another finger there…” The resulting trajectories are stiff, stilted, and slow. The exception is data collection so seamless that it preserves natural human behavior — as though the mind of the operator can act directly through instincts refined over millions of years. At Generalist, our foundation models like GEN-0 are trained on data from lightweight handheld, ergonomic devices that let people manipulate objects almost as they would with their own hands. These devices feel balanced, and the force feedback is there — after a few minutes of doing a task, operators stop “thinking” and start reacting. The results look different. People knit, peel potatoes, paint miniatures. Not only does it expand what tasks are possible to get robot data on—the data itself captures reflexes, micro-corrections, and real-time recovery. Our models trained on this data produce robot behaviors that people consistently describe as “human-like.” This is no accident. And it is scaling. Robots that ship with physical commonsense will be better at just about everything. I wrote about why it’s been so hard for machines to acquire physical commonsense, and why large-scale, real-world physical interaction data may finally change that. Read the full article here 👉 https://lnkd.in/gCjHP-qQ

'A roadmap for AI in robotics' - our latest article (https://rdcu.be/euQNq) published in Nature Machine Intelligence, offers an assessment of what artificial intelligence (AI) has achieved for robotics since the 1990s and proposes a research roadmap with challenges and promises. Led by Aude G. Billard, current president of IEEE Robotics and Automation Society, this perspective article discusses the growing excitement around leveraging AI to tackle some of the outstanding barriers to the full deployment of robots in daily lives. It is argued that action and sensing in the physical world pose greater and different challenges for AI than analysing data in isolation and therefore it is important to reflect on which AI approaches are most likely to be successfully applied to robots. Questions to address, among others, are how AI models can be adapted to specific robot designs, tasks and environments. It is argued that for robots to collaborate effectively with humans, they must predict human behaviour without relying on bias-based profiling. Explainability and transparency in AI-driven robot control are essential for building trust, preventing misuse and attributing responsibility in accidents. Finally, the article close with describing the primary long-term challenges, namely, designing robots capable of lifelong learning, and guaranteeing safe deployment and usage, as well as sustainable development. Happy to be co-author of this great piece led by Aude G. Billard, with contributions from Alin Albu-Schaeffer, Michael Beetz, Wolfram Burgard, Peter Corke, Matei Ciocarlie, Danica Kragic, Ken Goldberg, Yukie NAGAI, and Davide Scaramuzza Nature Portfolio IEEE #robotics #robots #ai #artificial #intelligence #sensors #sensation #ann #roadmap #generativeai #learning #perception #edgecomputing #nearsensor #sustainability

German Humanoid Robot Brings Physical AI to the Factory Floor Introduction A new generation of industrial automation is taking shape as a German startup unveils a humanoid robot designed to work safely and productively alongside humans. The system combines advanced Physical AI, human-like dexterity, and factory-ready endurance to address real-world manufacturing demands. The Breakthrough Agile Robots introduced Agile One, a humanoid robot purpose-built for industrial environments. The robot features 71 degrees of freedom, including 21 in each hand, enabling human-level manipulation. Onboard AI supports audio-based interaction, spatial awareness, and real-time motion and sound tracking. Designed for extended shifts, Agile One offers up to eight hours of battery life. Factory-Grade Capabilities Agile One stands 174 cm tall, weighs 69 kg, and can carry payloads up to 20 kg. It reaches speeds of up to 2.0 m/s, allowing efficient movement across factory floors. Tactile fingertips and force-torque sensing at every joint enable both delicate and forceful tasks. Intended use cases include material transport, machine tending, tool handling, and precision assembly. AI Architecture and Training The robot is powered by a layered AI architecture separating strategic reasoning from rapid motor control. Training draws on one of Europe’s largest real-world industrial datasets, augmented by simulation and human demonstrations. This approach allows skills learned in training to transfer directly into live industrial operations with high reliability. Human-Centered Design Agile One emphasizes safe and intuitive human-robot interaction. Visual cues include bright colors, expressive eyes, proximity sensors, and a chest-mounted information display. The design prioritizes comfort, predictability, and trust in close human collaboration. Production and Outlook Full-scale production is planned for early 2026 at a new facility in Bavaria. Agile Robots will retain full in-house control over hardware manufacturing to ensure quality and scalability. Why This Matters Agile One signals a shift from isolated industrial robots to collaborative humanoids capable of integrating into existing workflows. By combining Physical AI, dexterity, and human-friendly design, this platform points toward factories where flexibility, resilience, and productivity scale together—without redesigning the workplace around machines. I share daily insights with 37,000+ followers across defense, tech, and policy. If this topic resonates, I invite you to connect and continue the conversation. Keith King https://lnkd.in/gHPvUttw

Yesterday, I shared ten ideas on the crossing paths of augmented humans and humanized robots. If you missed it, here’s the post: https://lnkd.in/gSEx4MNw Over the next few days, I’ll go deeper into each concept, starting with a big one: Synthetic Theory of Mind: Teaching Robots to Get You What will it take for robots to go beyond following commands and actually understand us? The next leap in robotics isn’t more compute. It’s empathy. We need a new kind of intelligence: A Synthetic Theory of Mind Engine is a system that lets machines infer our beliefs, emotions, intentions, and mental states. This isn’t sci-fi anymore. China recently introduced Guanghua No. 1, the world’s first robot explicitly designed with emotional intelligence. It can express joy, anger, and sadness and adapt behavior based on human cues. The vision: emotionally aware care, especially for aging populations. ... and as Scientific American reports, researchers are now building AI models that simulate how people think and feel, essentially teaching machines to reason about our inner world. We’re witnessing the first generation of emotionally intelligent machines. So, what can a Synthetic Theory of Mind Engine do? Imagine a robot that can: ⭐ Detect confusion in your voice and rephrase ⭐ Notice emotional fatigue and pause ⭐ Adapt its language based on what you already know ⭐ Predict what you’re about to need before you say it To do this, it builds a persistent mental model of you. One that evolves with every interaction; making collaboration more intuitive and aligned. In healthcare, education, customer support, and even companionship, the future of robotics isn’t just about capability. It’s about alignment with our goals, our states, and our humanity. We're not just building smarter agents. We’re building partners who can make us feel seen, understood, and supported. 2–3 years: Expect early pilots in eldercare, education, and social robotics 5–7 years: Emotionally aware, intent-sensitive agents in homes, hospitals, and teams If you're working on cognitive robotics, LLM + ToM integration, or human-aligned AI, I’d love to connect and collaborate.