Confluent Kafka data-product platform

A major UK telecom operator. Client abstracted for confidentiality; technical scope as delivered.

Context

The organisation needed to turn streaming data across 30+ source systems into reliable, reusable data products that analytics teams could depend on — with clear ownership and schema evolution that wouldn't break consumers downstream.

What I built

A Confluent Kafka data-product platform:

• 20+ productised data streams, each domain-owned rather than centrally bottlenecked.
• Schema governance via Schema Registry, so producers and consumers evolve independently.
• Hive LLAP and Spark 3 for query acceleration over the streamed data.
• A cross-team ownership model so domains, not a central queue, owned their products.

Impact

• 20+ data products in operation across the business.
• 30+ source systems integrated under one governed model.
• Domain-oriented ownership that scaled without a central gatekeeper.

Role & stack

Data engineer and technology architect (Accenture CTA group) — designed the platform and guided cross-border delivery teams.

Stack: Apache Kafka (Confluent Cloud), Schema Registry, Hive LLAP, Spark 3, NiFi, Hadoop, Elastic Stack.

→ See also Event-driven & streaming and Data & lakehouse.