TornadoVM: The Need for GPU Speed

An airhacks.fm conversation with Michalis Papadimitriou (@mikepapadim (twitter.com/mikepapadim)) about:
starting with Java (java.com/en) 8,
first computer experiences with Pentium 2,
doom (en.wikipedia.org/wiki/Doom_(franchise)) 2 and Microsoft Paint (en.wikipedia.org/wiki/Microsoft_Paint),
university introduction to Object-oriented programming (en.wikipedia.org/wiki/Object-oriented_programm…) using Objects First and bluej.org IDE,
Monte Carlo simulations (en.wikipedia.org/wiki/Monte_Carlo_method) for financial portfolio optimization in Java,
porting Java applications to en.wikipedia.org/wiki/OpenCL for GPU acceleration achieving 20x speedup,
working at Huawei on GPU hardware,
writing unit tests as introduction to TornadoVM (tornadovm.org),
working on FPGA (en.wikipedia.org/wiki/Field-programmable_gate_…) integration and Graal (graalvm.org) compiler optimizations,
experience at OctoAI (octo.ai) startup doing AI compiler optimizations for TensorFlow and PyTorch models,
understanding model formats evolution from ONNX (onnx.ai) to GGUF (github.com/ggerganov/ggml/blob/master/docs/ggu…),
standardization of LLM inference through Llama models,
implementing GPU-accelerated Llama 3 inference in pure Java using TornadoVM,
achieving 3-6x speedup over CPU implementations,
supporting multiple models including Mistral (mistral.ai) and working on chat.qwen.ai 3 and deepseek.com,
differences between models mainly in normalization layers,
GGUF becoming quasi-standard for LLM model distribution,
TornadoVM's Consume and Persist API for optimizing GPU data transfers,
challenges with OpenCL deprecation on macOS and plans for Metal backend,
importance of developer experience and avoiding python.org dependencies for Java projects,
runtime and compiler optimizations for GPU inference,
kernel fusion techniques,
upcoming integration with github.com/langchain4j/langchain4j,
potential of Java ecosystem with Graal VM and Project Panama (openjdk.org/projects/panama) FFM for high-performance inference,
advantages of Java's multi-threading capabilities for inference workloads

Michalis Papadimitriou on twitter: @mikepapadim (twitter.com/mikepapadim)