Distributed systems
Event-Driven Notification System
Event-driven notification platform built with Python, FastAPI, Celery, Redis, PostgreSQL, and Docker. Designed for high reliability with idempotency, retry logic, dead-letter queues, and horizontal scalability.
Overview
Project overview
Event-driven notification platform built with Python, FastAPI, Celery, Redis, PostgreSQL, and Docker. Designed for high reliability with idempotency, retry logic, dead-letter queues, and horizontal scalability.
Engineering highlights
- I
Built an event-driven notification platform that sends email, SMS, and webhook notifications to up to 1,000 recipients per event, using separate Celery queues so each channel can scale independently.
- II
Made fan-out recoverable by saving every notification to PostgreSQL before dispatching worker tasks, preventing work from being lost when the dispatcher or a worker crashes.
- III
Prevented duplicate sends with Redis idempotency checks, PostgreSQL unique constraints, and row-level locking, keeping delivery correct even when Redis is unavailable or two workers receive the same task.
- IV
Added channel-specific retries with exponential backoff and jitter, along with a PostgreSQL-backed dead-letter queue for failed notifications and manual or bulk replay.
- V
Built a scheduled reconciliation process that recovers overdue retries and tasks left in progress by crashed workers, using FOR UPDATE SKIP LOCKED to allow multiple sweep workers to run safely in parallel.
- VI
Added atomic Redis rate limiting, HMAC-SHA256 webhook signing, and SSRF protection for outbound webhook requests.
- VII
Built a React and TypeScript dashboard for queue monitoring, delivery analytics, failure inspection, and dead-letter replay.
- VIII
Containerised the platform with Docker Compose and covered the API, workers, retry behaviour, and service logic with more than 200 tests.