Augmented Reality Use Cases and Challenges

What if you could show a patient exactly how their medication works inside their body—in real-time, right in front of their eyes? That's not science fiction. That's augmented reality transforming pharma and medical devices TODAY. I've been tracking AR's explosive growth across healthcare, and the results are staggering. When Bayer first used AR at a medical conference, booth engagement time jumped from under 2 minutes to 10 minutes. Johns Hopkins surgeons are performing spinal surgeries with AR-guided precision. One pharma company is saving an estimated $27 million per factory using AR training solutions. Here's what's really exciting me about AR in pharma: 🎯 Sales & Marketing Revolution: Instead of complex brochures, reps can now demonstrate drug mechanisms in 3D. Patients can literally see how treatments will help them. 🔬 Manufacturing Excellence: AR overlays guide operators through complex processes, reducing errors and ensuring consistency across billion-dollar production lines. 👨⚕️ Medical Training Transformed: Students can practice procedures without risk, and surgeons gain "x-ray vision" for unprecedented precision. 💊 R&D Acceleration: Virtual molecular simulations are speeding drug discovery—just look at how mRNA vaccines developed at record speed during COVID. But here's the kicker: Over 50% of physicians WANT AR to learn about diseases and conditions. The demand is already there. Yes, there are challenges—costs, regulatory hurdles, capability gaps. But the companies moving first are seeing tangible ROI while their competitors are still figuring out the basics. The pharmaceutical industry has always been about improving human health. AR isn't just a cool tech toy—it's the next evolution of how we educate, engage, and heal. The question isn't whether AR will transform pharma and medical devices. It's whether your organization will lead the transformation or watch from the sidelines. What's your experience with AR in healthcare? I'd love to hear how you're seeing this technology impact patient outcomes. #Pharma #HealthTech #MedicalDevices #DigitalHealth

AR museums (Ghost in the Shell Tokyo) 🤩 This is the XR use case I've been waiting to see scaled properly. KDDI, STYLY, Inc., and XREAL just launched a Ghost in the Shell AR exhibition in Tokyo running hundreds of headsets simultaneously for visitors. Not a one‑time demo. Not a press event. An actual repeatable, operational experience. Anyone who's worked in XR knows how rare that is. Most projects get stuck in pilot mode forever because the operational side – device management, indoor positioning, guest experience at scale – never quite comes together. This one did. And that's what makes it interesting to me, not the IP or the visuals, but the fact that someone figured out how to run it as a real product. If this becomes a blueprint, museums and cultural spaces could finally treat XR as part of the core visitor journey rather than a side attraction they pull out for opening night. Would you visit an AR‑enhanced exhibition like this, or do you prefer the traditional experience? Genuinely curious! 😊 #XR #AR #ImmersiveTech #LocationBasedXR #ExperienceDesign #Museums

Augmented Reality Recently I have been using an Augmented Reality (AR) Platform to align my total knee replacements and I have a few early comments. 1.     Reason for using: I have been individualizing my total knee replacement surgeries for 17 years, beginning with a randomized controlled trial of kinematically aligned vs mechanically aligned cases. Our published RCT referenced in my LinkedIn Homepage showed mean knee alignment 2 degrees more femoral valgus and 2 degrees more tibial varus produced higher patient outcomes. However, it is the planning and execution of each individual plan rather than the mean alignment that allows the better outcomes. For many years, I have been able to accurately reproduce my pre-op plans by using patient specific instrumentation (PSI). This requires a special MRI, data transmission to Belgium, on-line planning, fabrication of the patient guide, shipping and sterilization of the guide for surgery. There are quite a few extra steps here. Over the last 5 years, I tried to obtain a surgical robot, but our efforts were unsuccessful primarily due to cost but also concern of the size of the robot in our smaller OR rooms. Two years ago, I began looking at augmented reality to replace our patient specific guides with a cost effective, accurate system which could eliminate the extra steps we were taking with our PSI guides. 2.     Implementation: We selected an FDA approved system, and carried out an extensive educational process for Residents, OR staff, and sterile processing. We began using the AR system on our PSI cases, meaning our backup included PSI guides and also conventional instruments if needed. We have one extra small tray of 9 instruments, in addition to the A/R glasses and camera worn on our Stryker hood system. We broadcast the A/R display to one of our OR monitors so everyone in the room can view what the surgeon is seeing on the heads up display. 3.    Quality control: At the time of surgery, we measure each of the bone cuts with calipers to confirm the resections. Our particular AR system has a navigated Control Tool to compare the bone cut angles with the pre-operative plans. Post-operatively, our PACS imaging system allows us to measure our x-rays to see if we have achieved our pre-operative plans. 4.    Early impressions: As expected there was a definite learning curve which early on did increase our OR time. We learned that the most efficient use of the AR glasses was to calibrate them after the sterile Stryker Helmet shield was in place. We made some adjustments to the use of the stylus for the bone resections. The manual stylus will soon be replaced with a digital stylus on this system, which I believe is a good idea. Our x-rays have been excellent, a couple examples are attached. The long leg film is from a case this week, planned only with standard xrays and a long leg radiograph. The post op films exactly matched our preop plan. #total knee replacement #kneealignment #Pixeemedical

The Convergence of AI and AR: It's hard to ignore where Apple's Vision Pro is heading. Imagine this. You're walking down a bustling city street, your Apple Vision Pro headset perched comfortably (if not heavily) on your face. Suddenly, a virtual assistant pops into view, offering real-time translations of foreign language signs, providing step-by-step directions to your destination, and even suggesting the best lunch spots based on your dietary preferences. This isn't science fiction—it's the future Apple is envisioning (and announced last week) with its expansion of AI capabilities to the Vision Pro headset. ▪️ The AI-AR Fusion: Crazy Together Apple's move to bring its "Apple Intelligence" suite to the Vision Pro is like adding rocket fuel to an already powerful engine. But why is this fusion so important? Let's break it down: a) Enhanced User Experience: AI can make AR interactions more natural and intuitive. b) Personalization: Your headset could learn your preferences over time, tailoring experiences to you. c) Real-time Processing: AI can help process complex AR environments faster, reducing lag. ▪️ The Challenges: It's Not All Rainbows and Unicorns Some hurdles to overcome: • Cost: At $3,499, the Vision Pro isn't exactly an impulse buy. Will AI features justify the price tag? • Battery Life: More AI processing could mean shorter usage times between charges. • Privacy Concerns: With AI analyzing our visual world, capturing everything - big issues. ▪️ The Bigger Picture: This Goes Way Beyond Apple Apple isn't alone in this AI-AR gold rush. Microsoft's HoloLens and Meta's Quest are also exploring similar territories. ▪️ The Ethical Implications (Especially as these devices get smaller) As we strap these AI-powered windows to our faces, we need to ask some tough questions: • How will constant AI-assisted vision change our perception of reality? • Could AI-AR combinations exacerbate digital addiction? • What safeguards are needed to prevent misuse of this powerful technology? ▪️ The Road Ahead While Apple's AI features are unlikely to be available on the Vision Pro this year, their announcement signals a clear direction for the future of wearable tech. This reminds me of Arthur C. Clarke's famous quote: "Any sufficiently advanced technology is indistinguishable from magic." With AI-powered AR, we're about to step into a world that might make all of these tools feel magical. What do you think? Are you ready to embrace this AI-AR future, or does it make you want to retreat to a cabin in the woods? I think AI will finally make AR the opportunity we've imagined.

AR faces a challenge that most developers never encounter… reality refuses to stay still. A storefront changes its display weekly. A stadium transforms from a baseball field to a concert venue overnight. Lighting shifts from harsh noon sun to soft evening glow. Crowds ebb and flow, creating constantly changing occlusion patterns. Traditional computer vision often expects consistency. It matches predefined patterns against stable environments. But real spaces are fluid, dynamic, and unpredictable. We learned this early in our journey while testing at major venues. Out-of-the-box visual positioning systems (VPS) often perform well in static, controlled conditions but struggle when confronted with even minor real-world changes…. like a sudden holiday decoration or unexpected weather shifts. This forced us to rethink AR from the ground up. Instead of trying to catalog every possible variation, we built systems that understand the fundamental structure of spaces. They recognize when change is meaningful versus superficial. Today, our AR experiences maintain accuracy even as environments evolve. They can adapt to seasonal transitions, temporary changes, and dynamic lighting conditions without requiring constant updates. Perfect isn't real. Real is what matters.

I'm thrilled to share the latest episode of Connected in 3D: Real-Time Solutions & Digital Twins! This week, I had the incredible opportunity to sit down with Paul Davies, Technical Fellow of Immersive Technologies at Boeing. We dive deep into how Boeing is revolutionizing manufacturing with XR—from reducing defects by 80% on the 767 tanker wiring project to enabling first flights through AR innovation. 🎙️ Here are some key highlights: 1️⃣ How Boeing started exploring AR for high-value manufacturing use cases in satellites as early as 2006, even before headsets were practical. 2️⃣ The game-changing impact of AR on reducing production time and defects with real-world case studies from defense, commercial, and service sectors. 3️⃣ Insights into scaling AR and digital twin solutions across Boeing’s diverse programs while overcoming challenges like IT integration and hardware limitations. 4️⃣ The importance of being platform-agnostic and how Unity XR is helping Boeing future-proof their immersive applications. 💡 Paul also shares some amazing advice for manufacturers starting their journey with XR: “Start small, solve a well-known problem exceptionally, and scale from there. Prove the value and watch leadership embrace the tech organically.” This is a must-listen for anyone curious about the future of XR and its tangible impact on global industries. 🌍 👉 Tune in now: https://lnkd.in/gm3gc2Bz https://lnkd.in/g-7Q64ki https://lnkd.in/g9EthpJr I’d love to hear your thoughts on how your organization is leveraging XR or exploring digital twins! Let’s discuss in the comments. #XR #DigitalTwins #Innovation #Manufacturing #Boeing #ConnectedIn3D #AR #SpatialComputing