Event-driven & streaming

Streaming is where integration and data architecture meet. Done well, a Kafka backbone turns brittle nightly batches into reliable, observable, near-real-time flows that teams across an organisation can build on without coordinating every change through a central queue.

What I do

• Streaming backbones on Kafka. Apache Kafka and Confluent Cloud, AWS MSK, Azure Event Hubs — with Schema Registry governing evolution so producers and consumers can move independently.
• Stream processing. Kafka Streams, Spark Streaming and Kinesis for the transforms and aggregations that sit between raw events and usable data.
• Event-driven patterns that survive production. Outbox, idempotency, and event sourcing — and the observability to know when something is wrong before a consumer does.

Evidenced by

• Confluent Kafka data-product platform — 20+ productised data streams across 30+ source systems, with schema governance and domain ownership.
• Cloud Gateway — event-driven integration adapters (SNS/SQS, Lambda transforms) within a cross-cloud API platform.

Background: deep Kafka (Connect, Schema Registry, Streams), plus NiFi, Hive and Spark across 20+ years of data systems.