← Back to Projects

Global Event Streaming Engine

A geo-distributed event streaming system handling millions of concurrent connections, acting as the nervous system for a global IoT sensor network.

GoKafkaScyllaDBgRPCKubernetes

Architecture Overview

Designed to ingest telemetry from widespread IoT devices, requiring aggressive optimization at the ingestion layer.

Core Technologies

  • Ingestion: Highly optimized Go microservices terminating TLS at the edge.
  • Data Pipeline: Apache Kafka with multi-region replication for disaster recovery.
  • Storage: ScyllaDB for time-series metrics, providing high throughput writes.

Challenges & Solutions

Cross-region latency and high egress volumes were major hurdles. We implemented a federated architecture where edge clusters handle local ingestion, aggregate data, and then send compressed batches to the central data lake. This reduced cross-region egress costs by 45% and improved edge responsiveness dramatically.

Performance Metrics

  • Throughput: 2.5M+ events per second at peak load.
  • Edge Latency: < 15ms p99.
  • Reliability: 99.999% SLA achieved.