From the One Billion Row Challenge to a Multi-Threaded Parquet Library

An airhacks.fm conversation with Gunnar Morling (twitter.com/gunnarmorling) about:
returning guest discussion four years after episode "#174 Kafka Connect CLI,
JFR Unit,
OSS Archetypes and JPMS" (airhacks.fm/#episode_174),
the One Billion Row Challenge (1BRC) (github.com/gunnarmorling/1brc) as a Java (java.com/en) community coding challenge,
aggregating a 13 gigabyte semicolon separated file of one billion temperature measurements per weather station,
baseline single threaded idiomatic Java at five minutes versus a 1.5 second winning entry,
two orders of magnitude speedup,
parallel processing scaling near linearly across cores,
memory mapped files via the Foreign Function and Memory API (openjdk.org/jeps/454) for arbitrarily large files,
reading eight bytes at once into a long,
SIMD (en.wikipedia.org/wiki/Single_instruction,
_multiple_data) parsing via the incubating Vector API (openjdk.org/jeps/469),
GraalVM native binaries (graalvm.org/latest/reference-manual/native-image),
no third party libraries and single source file rule,
results (jaxenter.de/java/java-trends-frameworks-91786) on par with C,
Thomas Würthinger and the GraalVM (graalvm.org) team participation,
HTMLDB (github.com/gunnarmorling) a Java 25 (openjdk.org/projects/jdk/25) executable shebang script storing records as semantic HTML pages using the built in XML parser,
agents reading XML better than JSON (json.org),
Hardwood (hardwood.dev) a minimal dependency multi threaded Java library and CLI for reading and writing Apache Parquet (parquet.apache.org) files,
columnar versus row based formats,
efficient encodings including delta encoding and bit packing,
trading CPU for disk size,
range requests to read specific columns from object storage like S3 (aws.amazon.com/s3),
the dependency heavy Parquet Java (github.com/apache/parquet-java) library pulling in the Hadoop (hadoop.apache.org) stack,
zero mandatory dependencies with optional modules,
a hand written AWS Signature Version 4 (docs.aws.amazon.com/IAM/latest/UserGuide/refer…) signer using the JDK HTTP client (docs.oracle.com/en/java/javase/25/docs/api/jav…) instead of the AWS SDK (aws.amazon.com/sdk-for-java),
verifying the signer against the AWS TCK test suite,
owning code with LLMs,
optional dependency for IAM instance profiles (docs.aws.amazon.com/IAM/latest/UserGuide/id_ro…),
Zstandard (facebook.github.io/zstd) and Snappy (github.com/google/snappy) compression as domain specific dependencies,
Java LLM code generation quality and import handling,
AIRails.dev skills based on Boundary Control Entity (en.wikipedia.org/wiki/Entity-control-boundary),
spec driven development storing the spec in package-info (docs.oracle.com/javase/specs/jls/se25/html/jls…) Java Doc using EARS notation (alistairmavin.com/ears),
the code review pyramid as a review skill,
GraalVM WebAssembly (graalvm.org/latest/reference-manual/wasm) web image compiling a Java text user interface to analyze Parquet files in the browser,
an agent skill for Parquet analysis in natural language,
Apache Iceberg (iceberg.apache.org) and S3 Tables (aws.amazon.com/s3/features/tables),
Apache Flink (flink.apache.org) integration,
the shift in the reuse versus own trade-off with LLMs,
Agent zSmith a zero dependency Java agent harness,
var over explicit types

Gunnar Morling on twitter: @gunnarmorling (twitter.com/gunnarmorling)