About
Articles by Philip
Activity
34K followers
-
Philip Koopman shared thisAutonomous vehicle safety is won or lost on the so-called edge cases. Recent recalls and news reports have made it start to feel like we might be losing. But we can get back on track if we acknowledge some hard truths about what edge cases really are and what it takes to beat them. The pieces are already on the playing board, but we’re not getting straight talk about what it takes to win. As long as we see business pressure to continue scaling up robotaxi service, we’ll see increasing problems from edge cases. Success will require the humility to acknowledge that remote assistants are a safety-critical part of the edge case handling process. We need to see a more proactive approach to treating edge case incidents as central to safety rather than as publicity gaffes to be quashed. Additionally there is more hard work required to figure out how to get edge case retraining to be effective in practice for end-to-end machine learning. Edge cases are not an embarrassment should be swept under the rug. They are the core problem that must be addressed to have safe autonomous vehicles. And they are here to stay. Read the full story here: https://lnkd.in/ddspsDbw
-
Philip Koopman shared thisPart 2 of Junko Yoshida's response to the physician Pro-AV open letter that calls for robotaxi scale-up. As with Part 1 Junko quotes from a who's who of experts, concluding: 🔹Industry organizations and academics are concerned with the legislative issues pressed in the medical community’s open letter. They feel that state and local concerns are being brushed to the side. 🔹Safety experts see danger in proposing to prove safety with an overly simplistic numeric approach — by dividing the number of crashes by miles driven. They see that net statistical harm is only one aspect of acceptable safety. 🔹The analysts in academia worry about uncertainty that scaled AV deployment is alreadying bringing to the real world. 🔹The industry and academia agree on the need for a uniform way of collecting appropriate AV safety data, and maintaining that data in a form that can be used effectively. 🔹Neither the industry groups nor the professors are advocating accelerated AV deployment. 🔹The impact of remote operations and E2E machine learning on AVs should not be downplayed. Read the full article here: https://lnkd.in/gm9tsACVThe AV Scaling Dilemma: Federal vs. State Regulations and AV Safety DataThe AV Scaling Dilemma: Federal vs. State Regulations and AV Safety Data
-
Philip Koopman shared thisAcceptable safety involves meeting threshold requirements of stakeholders, not optimizing risk. Full episode here: https://lnkd.in/gqSn_QJK
-
Philip Koopman shared thisTesla robotaxi speculation confirmed and documented. In-depth reporting with inside sources. Use of remote drivers. Targeted training for deployment areas. Near misses. Some not-misses. Misleading data claims. Based on interviews with safety experts and those working for Tesla. Turns out data label staff have seen a lot. Well worth a read.Why Tesla’s AI trainers don’t trust its self-driving tech – or its safety statsWhy Tesla’s AI trainers don’t trust its self-driving tech – or its safety stats
-
Philip Koopman shared thisRobotaxi companies love to criticize human drivers for risk hotspots while claiming only net statistical safety should apply to them. Risk hot spots -- identifiable clusters in incidents or loss events. Full episode here: https://lnkd.in/gRRcqcNK
-
Philip Koopman shared thisA paper with an interesting hypothesis: AI use diminishes rationality in organizations. It hinges on bounded rationality and satisficing decision-making, so I just couldn’t resist reading it. Tons of important observations in the discussion, with lessons that go beyond organizational management. Really nicely done. From the abstract: “four interconnected mechanisms through which algorithmic systems erode the richer capacity: proxy optimization, which substitutes measurable targets for meaningful goals; automation bias, which compresses human judgment around algorithmic defaults; decontextualization, which strips cases of the particularity that good decisions depend on; and the suppression of reflexive judgment, which prevents organizations from questioning their own premises. These mechanisms produce cumulative organizational consequences: weakened accountability, diminished contestability, and reduced capacity for adaptive learning. The argument is not that organizations should abandon AI, but that they should stop treating it as a rationality upgrade. AI is a capable tool for narrow computational tasks and a poor substitute for organizational judgment. The organizations most at risk are those that have forgotten the difference.” The Optimization Trap: How Artificial Intelligence Diminishes Decision Rationality in Organizations, D. Jemielniak. https://lnkd.in/g99ew2a4
-
Philip Koopman reposted thisPhilip Koopman reposted thisWHAT THE MD-AUTONOMOUS VEHICLE DEBATE RISKS MISSING: Raising the profile of questions about the safety of AV technology and missed opportunities over the years (both large and small) to make our roads safer has value. But this debate risks losing sight of the central roles that properly channeling human incentives and workability of legal systems play. 1.--We saw one such incentive when Waymo decided to suspend freeway operations to address risks from accumulated water on roads. This prudent step helped preserve franchise value. And it did not need a law or regulation. 2.--Some economic incentives, however, need a law crafted for the situation. Our current system falls short. Simple tort reform would bolster economic incentives for companies to deploy safe AVs without getting into the weeds about statistical measures of safety (e.g. projecting fatalities over miles driven, positive risk balances) or missed alternatives (e.g. lower speed limits, stronger DUI laws, better road design) or identification of the technology minimally necessary to ensure an appropriate level of safe operation (e.g. a vision only systems versus systems that also use lidar or radar). 3.--We need a simple, non-technical, statute (ideally federal) that imposes negligence liability on the AV maker when its ADS performs in a manner below the standard to which we hold human drivers. If we would find the human driver negligent (and thus liable for an accident), then the law should find the computer driver negligent. We attribute this liability to the AV maker because the AV maker is the party who can improve performance of the ADS. 4.--Lay judges and juries have domain expertise with evaluating behavior in auto accident situations that transfers to evaluating AV performance. Sensors and recording devices on AVs allow for responsible reconstruction of accidents to help courts make judgments. 5.--Courts do not have expertise needed to evaluate complex technology. Importantly, there are not enough qualified experts to staff more than a few product liability cases per year, if even that, to help them. The product liability regime (as opposed to a negligence regime) is not practically administrable, even if the technology allowed for identification of defects in ADS. 6.--The negligence liability must rest with the AV maker so the plaintiff has a single point of recovery. The lead AV maker can sort out allocation of liability among itself and vendors by contract. 7.--Human drivers still purchase auto insurance, and insurance companies pay claims, the old-fashioned way based on determinations of negligence liability. AV maker's will either purchase insurance or, more likely, self-insure given their confidence in the safety of their products. 8.--A workable incentive system allows AV makers freedom to do whatever they want against the backstop of a federal regulator who can issue recalls to address moral hazards and satisfy demands for public accountability. QED
-
Philip Koopman shared thisNew Podcast Episode: This week Phil & Junko talk about what it means for Autonomous Vehicle to be safe enough. At a high level, in addition to net statistical safety, acceptable safety also requires an absence of risk hot spots. It’s not about perfection; it’s about acceptably safe: 🔹As safe as a human driver on a net statistical basis (accounting for which car, which driver, which road, etc.) 🔹Avoiding reckless driving 🔹Avoiding shifting risk onto vulnerable populations 🔹Absence of unreasonable risk (AUR) 🔹Standards conformance 🔹Avoiding negative externalities (e.g., not blocking fire trucks) 🔹Accountability for breaking road rules 🔹Meeting all stakeholder safety requirements A single-metric approach such as crashes per mile does not work due to multiple stakeholder concerns. A multi-constraint satisfaction approach is the way to go. The bottom line: acceptable safety is meeting the threshold requirements of all your stakeholders. https://lnkd.in/gTJ3uDpK
-
Philip Koopman shared thisICYMI: Junko Yoshida responds to the Slotkin/Topol pro-AV Open Letter with an outstanding exercise in journalism. Junko's summary of what the letter claims: 🔹No other cause of death at the scale (of traffic fatalities) would be met with so little urgency. 🔹Autonomous vehicles safety data has reached a threshold that the medical community cannot responsibly ignore. 🔹Waymo is the only AV company publishing the data clinicians need to evaluate health impacts. 🔹A call to action is that states and municipalities should not resist AV companies’ efforts to deploy. What she learned talking to Bryan Reimer, Missy Cummings, David Zipper, Bryant Walker Smith, William Wallace, Philip Koopman, Greg Brannon, Eric Teoh, and Michael Brooks: 🔹AVs can save lives only in the long run. 🔹With more than 280 million vehicles on the road, it will take decades to see significant benefits from the technologies. 🔹The U.S. is a global outlier. Europe has already shown how to dramatically reduce road deaths without autonomous vehicle technology. 🔹A comprehensive Safe System Approach already offers widely available solutions that can effectively reduce road deaths in the short term. 🔹Yet, the promise of fully driverless vehicles has too often distracted the industry from safety improvements that could help drivers today. 🔹As robotaxis proliferate, a big, unanswered question remains. Can safety scale in parallel with accelerated operations? 🔹The brittleness of autonomous vehicles is real. The AV industry is still learning every day about how self-driving cars can go wrong. I truly appreciate medical professionals advocating for improved road safety. But the letter is quite one-sided in claiming AV safety victory prematurely while dismissing both concerns and more proven ways to improve road safety that for some reason they are choosing not to spend their advocacy efforts on. It is not yet time to pronounce a public health mandate for AV adoption. The jury is still out. The authors and signatories would do well to take seriously what the safety advocates interviewed in Junko's piece have to say. The open letter's call for improved data collection could be useful, especially if that data goes beyond what Waymo is collecting to be more stakeholder-inclusive and independent of manufacturers. (Safety advocates before them have made similar calls with detailed proposals that are not referenced.) But IMO their fodder to support federal preemption is an AV industry wish-list item that has no place in a serious safety advocacy piece. This is part 1 of a piece that has already stirred up a lot of debate. Can't wait for part 2! https://lnkd.in/gNiWz37T
-
Philip Koopman liked thisPhilip Koopman liked thisA recently posted article by Phil Koopman (see link in first comment) gives a great write-up about edge case handling and the current state of public-road AVs, which rely on remote human supervisors. To paraphrase a couple important points: for practical purposes there are an infinite number of edge cases, and the context needed to handle them safely in the real world may include not just the environment, but also knowledge of local culture and events. And something that AI currently is not good at is “being aware enough to know that something doesn’t feel right, and reducing risk exposure to buy time to figure out how to muddle through.” I thought the below image in this article was very informative. It highlights how data distributions can relate to each other, as shown in a two-dimensional occurrence space. An extension of this is that changes in distribution will also occur (due to AI model, data, and environment drift). If you read the above and/or the referenced article, I’d be happy to hear your feedback!
-
Philip Koopman liked thisPhilip Koopman reminds us that, in autonomous vehicle safety, edge cases are opportunities to learn what needs fixing and should be addressed rather than dismissed as bad publicity. #AutonomousVehicles 🚗 #SafetyEngineering 🛡️ #EdgeCases 🔍
-
Philip Koopman liked thisAgreed, EOC was especially relevant this year. My session on formal methods challenges the status quo in software development. A must-see considering the attack on embedded systems with the latest LLMs and how they are able to find defects when even just reviewing the binaries of an embedded system.Philip Koopman liked this7 new sessions from this year's Embedded Online Conference are now available on-demand. Free to watch (no paid pass needed). The lineup: • Containerized builds for embedded teams (Shawn Prestridge, IAR) • Formal methods with Ada SPARK (Mark Hermeling, AdaCore) • Turning your embedded device into a platform (Axel Wolf, SEGGER Microcontroller) • Verifying RTOS best practices at runtime (Johan Kraft, Percepio®) • Turning a FRDM board + Zephyr shell into a zero-code CAN/I2C bench tool (Eli Hughes, NXP Semiconductors) • Fuzzing monolithic firmware (Samuel Lichtenheld, Metalware, Inc.) • Modern unit testing with GoogleTest (Ricardo Camacho, Parasoft) Watch any of them at https://lnkd.in/escpTwfK. #EmbeddedSystems #Firmware #RTOS #EmbeddedSoftware
Experience & Education
-
Carnegie Mellon University
View Philip’s full experience
See their title, tenure and more.
or
Publications
-
Embodied AI Safety: Reimagining safety engineering for artificial intelligence in physical systems
See publicationEmbodied AI (eAI) uses artificial intelligence based on machine learning to interact with the physical world. We are already seeing eAI deployed in the real world in robotaxis, smart medical devices, household robots, and other applications. However, everyone is struggling with the safety of these devices: how to design for safety, how to evaluate safety, and how to think about whether any particular eAI system is acceptably safe.
This book provides a foundation for thinking about the…Embodied AI (eAI) uses artificial intelligence based on machine learning to interact with the physical world. We are already seeing eAI deployed in the real world in robotaxis, smart medical devices, household robots, and other applications. However, everyone is struggling with the safety of these devices: how to design for safety, how to evaluate safety, and how to think about whether any particular eAI system is acceptably safe.
This book provides a foundation for thinking about the topic of eAI safety that is accessible to a non-specialist technical audience. Robotaxi safety is used as a concrete example. Early chapters provide an introduction to safety engineering, cybersecurity engineering, machine learning technology, and human/computer interaction. Later chapters cover eAI safety challenges in the wild, the complexities of establishing what risks might be acceptable, and open challenges in eAI safety. A proposal for reimagining safety engineering responds to the huge disruption that eAI technology creates when applying traditional computer-based system safety approaches. In the end, what we need are ways to build justifiable trust in eAI safety.
Chapters:
1. Introduction
2. Safety engineering concepts
3. Cybersecurity engineering concepts
4. Machine learning concepts
5. The role of humans in eAI safety
6. eAI safety issues in the wild
7. Acceptable risk
8. Reimagining safety engineering
9. Open challenges for building safe eAI
10. Justifiable trust for safety-critical eAI
11. Conclusions -
Understanding Checksums and Cyclic Redundancy Checks
See publicationThis book gives practical, comprehensive answers to common questions about checksums and CRCs. Descriptions are based mainly on intuitive rather than mathematical explanations for both algorithmic operation and limitations, improving accessibility to non-specialists. Coverage includes single-sum checksums, dual-sum checksums (e.g., Fletcher checksum, DualX, DualXP), the new Koopman checksum, Cyclic Redundancy Checks (CRCs), and system-level usage considerations.
For decades much of the…This book gives practical, comprehensive answers to common questions about checksums and CRCs. Descriptions are based mainly on intuitive rather than mathematical explanations for both algorithmic operation and limitations, improving accessibility to non-specialists. Coverage includes single-sum checksums, dual-sum checksums (e.g., Fletcher checksum, DualX, DualXP), the new Koopman checksum, Cyclic Redundancy Checks (CRCs), and system-level usage considerations.
For decades much of the practical use of checksums and CRCs was based, at least in part, on folklore. This book provides a solid, comprehensive foundation for addressing core issues such as the comparative fault detection effectiveness of each technique, insight into speed differences, intuitive explanations for how speed-up techniques work, which CRC polynomial you should use for any particular situation, and source code examples for each approach.
This is the most comprehensive treatment of checksums and CRCs to date. The emphasis is on intuitive explanations and empirical validation of insights for practical application of these
Chapters:
Introduction
Decimal checksum examples
Checksum operation and terminology
Checksum fault model
Single-sum checksums
Dual-sum checksums (Fletcher, Adler, DualX)
Koopman checksum
Checksum plus parity for HD=4 (KoopmanP, DualXP)
Cyclic Redundancy Check (CRC)
CRC effectiveness
Other checksum considerations
System-level considerations
Resources
Conclusions
Appendix: Good CRC Polynomials
Glossary -
The UL 4600 Guidebook
See publicationANSI/UL 4600 is the most comprehensive standard for highly automated vehicle safety, applying to any vehicle in which a human driver can take their eyes off the road. It provides a way to check the completeness and correctness of a safety case that spans a broad range of concerns related to safety, including design, deployment, and lifecycle support. There is a special emphasis on computer hardware and software, as well as operational concepts and interaction with other road users. While other…
ANSI/UL 4600 is the most comprehensive standard for highly automated vehicle safety, applying to any vehicle in which a human driver can take their eyes off the road. It provides a way to check the completeness and correctness of a safety case that spans a broad range of concerns related to safety, including design, deployment, and lifecycle support. There is a special emphasis on computer hardware and software, as well as operational concepts and interaction with other road users. While other relevant standards can and should be used as well, UL 4600 provides an umbrella to make sure things don’t get missed for assuring safety.
This book, written by the author of the original UL 4600 standard proposal, serves as a high-level guided tour. Early chapters provide historical context, a description of the distinctive UL 4600 prompt element approach, a discussion of key terms, and how a safety case works in the context of the standard. Then comes a chapter-by-chapter tour of UL 4600, explaining overall concepts and how all the pieces fit together for each area covered by the standard, from safety cases to hazard analysis to assessment. This book will help technical readers prepare for diving into the nitty gritty of the standard, as well as provide a more accessible discussion for those who want to understand what UL 4600 covers at a higher level. The last chapter provides pointers to further information, including how you can view the current version of UL 4600 for free.
This is a comparatively short (about 100 pages of main content) trade paperback (6"x9") discussion of a much longer, fairly complex standard. So think of it as a tour guidebook and not a textbook. -
How Safe is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety
See publicationThe most pressing question regarding autonomous vehicles is: will they be safe enough? The usual metric of "at least as safe as a human driver" is more complex than it might seem. Which human driver, under what conditions? And are fewer total fatalities OK even if it means more pedestrians die? Who gets to decide what safe enough really means when billions of dollars are on the line? And how will anyone really know the outcome will be as safe as it needs to be when the technology initially…
The most pressing question regarding autonomous vehicles is: will they be safe enough? The usual metric of "at least as safe as a human driver" is more complex than it might seem. Which human driver, under what conditions? And are fewer total fatalities OK even if it means more pedestrians die? Who gets to decide what safe enough really means when billions of dollars are on the line? And how will anyone really know the outcome will be as safe as it needs to be when the technology initially deploys without a safety driver?
This book is written by an internationally known expert with more than 25 years of experience in self-driving car safety. It covers terminology, autonomous vehicle (AV) safety challenges, risk acceptance frameworks, what people mean by "safe," setting an acceptable safety goal, measuring safety, safety cases, safety performance indicators, deciding when to deploy, and ethical AV deployment. The emphasis is not on how to build machine learning based systems, but rather on how to measure whether the result will be acceptably safe for real-world deployment. Written for engineers, policy stakeholders, and technology enthusiasts, this book tells you how to figure out what "safe enough" really means, and provides a framework for knowing that an autonomous vehicle is ready to deploy safely. -
Better Embedded System Software
See publicationThis book distills the experience of more than 90 design reviews on real embedded systems into a set of bite-size lessons learned in the areas of software development process, requirements, architecture, design, implementation, verification & validation, and critical system properties.
Each chapter describes an area that tends to be a problem in embedded system design, symptoms that tend to indicate you need to make changes, the risks of not fixing problems in this area, and concrete ways to…This book distills the experience of more than 90 design reviews on real embedded systems into a set of bite-size lessons learned in the areas of software development process, requirements, architecture, design, implementation, verification & validation, and critical system properties.
Each chapter describes an area that tends to be a problem in embedded system design, symptoms that tend to indicate you need to make changes, the risks of not fixing problems in this area, and concrete ways to make your embedded system software better. Each of the 29 chapters is self-sufficient, permitting developers with a busy schedule to cherry-pick the best ideas to make their systems better right away.
Courses
-
Dependable Embedded Systems
18-849
-
Distributed Embedded Systems
18-649
-
Embedded System Engineering
18-348
-
Embedded System Software Engineering
18-642
Projects
-
UL 4600
-
See projectOriginator and member of voting Standards Technical Panel for UL4600: Standard for Safety for the Evaluation of Autonomous Products
Languages
-
English
-
Organizations
-
Life Member IEEE, Emeritus IFIP WG 10.4
-
Other similar profiles
Explore more posts
Explore top content on LinkedIn
Find curated posts and insights for relevant topics all in one place.
View top content