# Kafka

> Distributed event streaming platform

- HTML version: https://robbiepalmer.me/technologies/kafka
- Website: https://kafka.apache.org

Kafka's core abstraction is a distributed, replicated commit log. That makes a
write-ahead log a useful model: records become durable in the log, and consumers can replay them to
rebuild derived state. In a conventional database, the WAL usually sits behind mutable tables and
indexes as an internal recovery mechanism. Kafka turns that arrangement inside out by making the
durable log the primary interface and treating tables, indexes, and materialized views as downstream
projections.

The log defines ordering within each partition. It does not prescribe the topology of the system
around it. Applications can use Kafka to drive an acyclic pipeline, a workflow with feedback loops,
a state machine, a cyclic state graph, or another processing model entirely.

## Ideas this technology builds on or exposes

- [Commit Log](https://robbiepalmer.me/ideas/commit-log.md): An ordered, append-only record of accepted state changes that can be replicated or replayed to reconstruct derived state.
- [Stream/Table Duality](https://robbiepalmer.me/ideas/stream-table-duality.md): A stream can describe the changes to a table, while a table represents the latest state produced by those changes.
- [Write-Ahead Log (WAL)](https://robbiepalmer.me/ideas/write-ahead-log.md): An append-only record written durably before the corresponding data changes, allowing a system to recover committed state after failure.

## Projects using this technology

- [Python UDFs for Confluent Platform](https://robbiepalmer.me/projects/confluent-python-udfs.md)
- [Automated PII Detection](https://robbiepalmer.me/projects/pii-detection.md)
- [Real-Time Multi-Camera Video Analytics](https://robbiepalmer.me/projects/real-time-multi-camera-video-analytics.md)

## Blog posts

- [Uniting the Machine Learning and Data Streaming Ecosystems - Part 1](https://robbiepalmer.me/blog/2023-03-28-uniting-machine-learning-data-streaming-1.md) — 2023-03-28
- [Uniting the Machine Learning and Data Streaming Ecosystems - Part 2](https://robbiepalmer.me/blog/2023-04-04-uniting-machine-learning-data-streaming-2.md) — 2023-04-04
- [How To Automatically Detect PII for Real-Time Cyber Defense](https://robbiepalmer.me/blog/2023-05-23-automatically-detect-pii-real-time-cyber-defense.md) — 2023-05-23

## Used professionally at

- Confluent — Customer Innovation Engineer II

---

Markdown index of this site: https://robbiepalmer.me/llms.txt