Billion-parameter brains in pocket-sized chips: The local AI revolution

In this episode, we explore how engineers are embedding powerful AI directly into hardware – no cloud connection required.

Michaël Uyttersprot from Avnet Silica and Cedric Vincent from Tria Technologies reveal how they run ChatGPT-quality language models on resource-constrained embedded devices. What once demanded data centre infrastructure now fits onto chips with just 2GB of RAM.

The conversation covers the technical challenges of cramming billion-parameter models into embedded systems, real-world applications from conference rooms to industrial robotics, and the three compelling reasons driving this shift: data privacy, power efficiency, and cost control.

Michaël and Cedric discuss hardware platforms from AMD, NXP, and Qualcomm, explain techniques like quantisation and mixture of experts, and demonstrate applications including a vintage telephone box that lets you call avatars from different time periods.

Tune in to learn why the future of AI might not be in the cloud at all – and what that means for industries from manufacturing to healthcare.

#AI #LLM #embeddedsystems #IoT #privacy #wetalkiot

Summary of this week's episode:

What makes large language models special

Why run LLMs locally on embedded devices

Real-world applications: Vision LLMs and OCR

Technical deep dive: How to fit billions of parameters into tiny devices

Understanding temperature: Making AI creative or accurate

Industries moving fastest: OCR, security, and robotics

Future applications: Robotic arms and time series analysis

The biggest technical hurdle: Power consumption

Advice for engineers: Start with llama.cpp