A Billion Data Points Per Second

An airhacks.fm conversation with Christofer Dutz (linkedin.com/in/christofer-dutz) about:
porting the Apache PLC4X (plc4x.apache.org) drivers to Rust (rust-lang.org) and adding wrappers for Go,
python.org and typescriptlang.org,
Apache IoTDB (iotdb.apache.org) as a time series database,
what separates a time series database from a relational one: flat columnar data with the timestamp as primary key,
aggregate queries like the average temperature over three months grouped by hours,
why a select star query makes no sense in a time series database,
Apache Cassandra (cassandra.apache.org/_/index.html) as a time series store and its missing aggregation efficiency,
the history of IoTDB at Tsinghua University (tsinghua.edu.cn/en) in Beijing,
Apache incubation in 2018 and top level in 2020,
InfluxDB (influxdata.com) and TimescaleDB (tigerdata.com) as the better known competitors,
the split of the project into Apache TsFile (tsfile.apache.org) as storage format and IoTDB as query engine,
writing TsFile on the PLC and shipping it over MQTT (mqtt.org) instead of polling a signal every few milliseconds,
lossless compression to 10% and lossy compression to 1% of the original size,
reducing the number of tags,
network throughput and CPU load on the shop floor,
TsFile SDKs for Java (java.com/en),
C,
Rust and Python,
TsFile as a dataset format on Hugging Face (huggingface.co),
integration with Apache Spark (spark.apache.org) and Apache Hadoop (hadoop.apache.org),
the Shanghai Metro (en.wikipedia.org/wiki/Shanghai_Metro) replacing a twelve node Cassandra (cassandra.apache.org) cluster with a single IoTDB node,
ingest of a billion data points per second,
Timecho (timecho-global.com) as the company providing commercial support,
TimechoDB as IoTDB with authentication and query load distribution,
pure Java with a Java 8 baseline moved to Java 17,
a private fork updated to Java 17,
the jakarta namespace (jakarta.ee/blogs/javax-jakartaee-namespace-eco…) and current Spring Boot (spring.io/projects/spring-boot) versions,
Quarkus (quarkus.io),
reflection and the GraalVM (graalvm.org) closed world assumption,
config node,
data node and AI node,
gRPC (grpc.io) between the nodes even inside one virtual machine,
singletons instead of dependency injection,
CDI (jakarta.ee/specifications/cdi/4.1) and dropwizard.io/en/latest in the pom,
running IoTDB in process,
a setup.exe install compared to a Cassandra cluster with docker (en.wikipedia.org/wiki/Docker_(software)) and kubernetes.io,
starting an inference process in process instead of over HTTP,
github.com/AdamBien/lightmetal and github.com/AdamBien/zsmith as an example,
Vidocq (vidocq.dev) as a CDI implementation without external dependencies,
the JDBC driver (docs.oracle.com/en/java/javase/21/docs/api/jav…),
the REST API and an Apache Drill (drill.apache.org) plug-in,
push queries that call back whenever the query produces a new result,
hierarchies of aggregation levels with per level retention from milliseconds on the shop floor to years for business analysis,
the tree identifier of company,
site,
line,
machine and machine part,
the table model added for users who think in table algebra,
data types,
missing updates,
delete and automatic purging by retention,
replication so queries run on a second instance,
the built-in MCP server (modelcontextprotocol.io) giving Claude (claude.com/product/overview) access to an IoTDB instance,
the AI node running models inside the database,
data gravity,
a KNX (knx.org) house logging into IoTDB on a Raspberry Pi (raspberrypi.org),
an IoTDB app running in process on a Bosch Rexroth ctrlX (apps.boschrexroth.com/microsites/ctrlx-automat…) PLC

Christofer Dutz on linkedin: linkedin.com/in/christofer-dutz