From SIMD to CUDA with TornadoVM

An airhacks.fm conversation with Michalis Papadimitriou (@mikepapadim (twitter.com/mikepapadim)) about:
GPU acceleration for LLMs in Java (java.com/en) using tornadovm.org,
evolution from CPU-bound SIMD (en.wikipedia.org/wiki/Single_instruction,
_multiple_data) optimizations to GPU memory management,
Alfonso's original Java port of llama.cpp using SIMD and Panama (openjdk.java.net/projects/panama) Vector API (openjdk.org/jeps/338) achieving 10 tokens per second,
TornadoVM's initial hybrid approach combining CPU vector operations with GPU matrix multiplications,
memory-bound nature of LLM inference versus compute-bound traditional workloads,
introduction of persist and consume API to keep data on GPU between operations,
reduction of host-GPU data transfers for improved performance,
comparison with native en.wikipedia.org/wiki/CUDA implementations and optimization strategies,
JIT (en.wikipedia.org/wiki/Just-in-time_compilation) compilation of kernels versus static optimization in frameworks like developer.nvidia.com/tensorrt,
using LLMs like Claude to optimize GPU kernels,
building MCP servers for automated kernel optimization,
European Space Agency using TornadoVM in production for simulations,
upcoming Metal backend support for Apple Silicon within 6-7 months,
planned support for additional models including Mistral (mistral.ai) and huggingface.co/google/gemma-7b,
potential for distributed inference across multiple GPUs,
comparison with python.org and C++ (en.wikipedia.org/wiki/C%2B%2B) implementations achieving near-native performance,
modular architecture supporting en.wikipedia.org/wiki/OpenCL PTX and future hardware accelerators,
challenges of new GPU hardware vendors like tenstorrent.com focusing on software ecosystem,
planned quarkus.io and github.com/langchain4j/langchain4j integration demonstrations

Michalis Papadimitriou on twitter: @mikepapadim (twitter.com/mikepapadim)