# Python UDFs for Confluent Platform

> Bringing the Python Machine Learning ecosystem to Kafka via UDFs for ksqlDB and Kafka Connect

- HTML version: https://robbiepalmer.me/projects/confluent-python-udfs
- Status: completed
- Started: 2022-09-01
- Technologies: Python, Kotlin, Kafka

# Vision

The goal was to break down the barrier between the Java-based Data Streaming ecosystem (Kafka) and
the Python-based Machine Learning ecosystem. By enabling Python User Defined Functions (UDFs) within Confluent Platform,
I aimed to empower ML Engineers to bring their models directly to the data stream,
without needing to learn Java or manage complex microservices.

# Problem Statement

* **Ecosystem Gap**: Kafka, ksqlDB, and Kafka Connect run on the JVM, while the richest ecosystem for ML and Data Science is in Python.
* **Fundamental Incompatibility**: Existing solutions like GraalVM or Jython couldn't support the C-extensions required by critical Python libraries (NumPy, pandas, TensorFlow, PyTorch).
* **Accessibility**: ML Engineers and Data Scientists were effectively locked out of the powerful data management tooling in Confluent Platform because they didn't know Java.

# Methodology

Inspired by Apache Flink's architecture, I developed a bridge to unite these two worlds:

* **Wheel Contract**: Required user-provided Python code to be packaged as a wheel that complied with the UDF interface.
* **Generated Bindings**: Used the wheel's interface to generate Kotlin bindings that mapped JVM calls to the Python entry points.
* **PemJa Bridge**: Used [PemJa](https://github.com/alibaba/pemja) to invoke an embedded CPython interpreter from the JVM through JNI. This retained native CPython support without relying on a Python implementation written for the JVM.
* **Bundled Runtime**: Bundled a Python runtime for deployment alongside ksqlDB.
* **Rich Ecosystem Support**: Because it used a real C-based Python interpreter, the solution supported the full breadth of the Python ecosystem, including all major ML libraries.

# Architecture

The system separated build-time binding generation from runtime record
exchange. The build used the wheel's UDF interface to map Kotlin calls to Python
entry points. The deployment packaged the wheel and generated bindings with a
Python runtime alongside ksqlDB.

## Build and deployment

```mermaid
flowchart LR
Wheel["Interface-compliant wheel<br/>user code + Python entry points"]
Generator["Binding generator<br/>reads the UDF interface"]
Bindings["Generated Kotlin bindings<br/>Python entry-point mapping"]
Runtime["Bundled Python runtime"]
Deployment["ksqlDB deployment<br/>wheel + bindings + Python runtime"]

Wheel -->|"interface contract"| Generator
Generator -->|"generates mapping"| Bindings
Bindings -->|"package with wheel"| Deployment
Runtime -->|"deploy alongside ksqlDB"| Deployment
```

## Runtime record exchange

```mermaid
flowchart LR
Kafka["Kafka records"]

subgraph Worker["ksqlDB worker"]
subgraph JVM["JVM"]
Host["ksqlDB UDF or<br/>Kafka Connect execution"]
Bindings["Generated Kotlin bindings<br/>Python entry-point mapping"]
PemJa["PemJa bridge<br/>JNI boundary"]
Host <--> Bindings
Bindings <--> PemJa
end

subgraph Python["Bundled CPython runtime"]
EntryPoint["Wheel loader<br/>mapped Python entry point(s)"]
Dependencies["Python packages<br/>native C extensions"]
Dependencies --> EntryPoint
end

PemJa <-->|"UDF arguments + result"| EntryPoint
end

Kafka <-->|"records"| Host
```

## Architecture in text

1. The user packaged Python code and its entry points in a wheel that complied
   with the UDF interface.
2. A build-time generator read that interface and emitted Kotlin bindings that
   mapped JVM calls to the Python entry points.
3. The deployment packaged the wheel and generated bindings with a Python
   runtime alongside ksqlDB.
4. A Kafka record entered ksqlDB or Kafka Connect in the JVM worker. The host
   called the generated binding for the mapped Python entry point.
5. PemJa crossed the JNI boundary and invoked that entry point in the embedded
   CPython interpreter. The wheel loader supplied the user code, while the
   bundled environment supplied Python packages and native C extensions.
6. The function result crossed the same boundaries in reverse and returned to
   ksqlDB or Kafka Connect for continued stream processing.

# Impact

* **Democratization**: Opened the Confluent ecosystem to a massive new audience of Python developers and ML engineers.
* **New Capabilities**: Enabled ML-driven functions (like real-time inference or complex data transformations) directly within SQL queries and Connect pipelines, which were previously impossible or impractical in pure Java.
* **Commercial Strategy**: Positioned as a key differentiator for Confluent Platform, directly addressing the "Python gap" in the Enterprise data streaming market.

---

Markdown index of this site: https://robbiepalmer.me/llms.txt
