Event-driven & streaming

Streaming is where integration and data architecture meet. Done well, a Kafka backbone replaces brittle nightly batches with observable, near-real-time flows that teams across an organisation can build on without routing every change through a central queue.

What I do

• Streaming backbones on Kafka. Apache Kafka on Cloudera (CDP), AWS MSK, Azure Event Hubs, with Schema Registry governing evolution so producers and consumers can move independently.
• Stream processing. Kafka Streams, Spark Streaming and Kinesis for the transforms and aggregations between raw events and usable data.
• Event-driven patterns that survive production. Outbox, idempotency, event sourcing, and the observability to know something is wrong before a consumer does.

Evidenced by

• Integration platform: event-topic ingress and egress on one runtime, with per-step run state so a half-finished flow is visible, and a measured answer to whether serverless could carry it.
• SAP event backbone: a broker-based backbone (SAP Advanced Event Mesh over AMQP 1.0) with the subscription model, delivery guarantee and dead-letter behaviour agreed as a contract before either side wrote code.
• Kafka data-product platform on Cloudera: 20+ productised data streams across 30+ source systems, with schema governance and domain ownership.
• Cloud Gateway: event-driven integration adapters (SNS/SQS, Lambda transforms) inside a cross-cloud API platform.

Background: Kafka in depth (Connect, Schema Registry, Streams), plus NiFi, Hive and Spark, across 20+ years of data systems.