E26 Making Embedded AI Safe: Functional Safety in Embedded ML Systems

Ryan and Luca welcome back Sebastian Boblest and Marco Mader Mendes from Bosch to discuss the challenges of deploying AI in safety-critical embedded systems. The conversation explores two key aspects: ensuring the generated code itself is safe (following standards like ISO 26262), and guaranteeing the trained model behaves correctly in production (covered by ISO PAS 8800).
The guests share practical strategies including fallback algorithms, model monitoring, redundancy approaches, and the importance of constraining what AI systems can do. They discuss real-world challenges teams face—from organizational resistance to unrealistic expectations about quick AI wins. Sebastian and Marco emphasize that while getting to 80% accuracy is often fast, reaching production-ready robustness requires significant effort. The episode offers a grounded look at what it actually takes to deploy ML safely in automotive, aerospace, and other domains where failures can threaten human life.
Key Topics:
• [03:30] Two dimensions of functional safety: safe code generation (ISO 26262) vs. model behavior guarantees (ISO PAS 8800)
• [08:45] Code generation strategies: MISRA compliance, fixed memory layouts, and multi-variant verification
• [15:20] Practical safety approaches: fallback algorithms, output limiting, and model monitoring for out-of-distribution detection
• [22:10] Fuzzing ML systems and the importance of trying to break your model during development
• [25:40] Redundancy strategies: multiple models, sensor diversity, and majority voting for higher safety levels (ASIL C/D)
• [32:15] Industry trends: proliferation of NPUs on tiny devices and the challenge of opaque hardware accelerators
• [36:50] Common struggles: organizational resistance, unrealistic expectations, and the 80-to-99% problem in AI deployment
Notable Quotes:
"You have to really think what you do. There's no off-the-shelf strategy. You have to really think about how you ensure that everything is working correctly." — Sebastian Boblest
"We as AI engineers are not 'hey everything has to be replaced by AI.' Maybe it's better to have a hybrid model where you still trust physical equations that have been working for 100 years." — Marco Mader Mendes
"You can solve problems with AI to the 80% level very quickly. People are super thrilled with these quick results. But the last 10 to 20% to get to product level—that gets quite tough." — Sebastian Boblest
Resources Mentioned:
• ONNX Functional Safety Subgroup - Initiative defining an ONNX subset for functional safety applications, largely driven by French aerospace companies
• ISO 26262 - Automotive functional safety standard covering software development in safety-critical applications
• ISO PAS 8800 - Standard addressing how to ensure AI models behave correctly after training and deployment
• IEC 61508 - Top-level standard for software development in functional safety-critical applications across industries
• MISRA Standard - Coding guidelines defining what to do and not do in safety-critical embedded software
• Embedded AI Coder (ETAS) - Bosch's code generation tool for deploying ML models on embedded systems with functional safety support
• Agile Embedded Podcast Slack (join.slack.com/t/agileembeddedslack/shared_inv…) - Community discussion channel where you can reach Ryan and Luca, with a dedicated sub-channel for Embedded AI topics