Designing Event-Driven Systems That Survive Production

Event-driven systems fail in slow, embarrassing ways. The interesting failures are not the broker outages — those are loud and recoverable. The expensive ones are silent: a missing idempotency key here, an at-least-once consumer assumed exactly-once there, a projection drift you only notice when finance reconciles month-end.

Three patterns that earn their keep

On every serious system I have shipped, the same three patterns show up: deterministic projections, a transactional outbox, and replayable consumers. None of them are exciting. All of them prevent the kind of incidents that erode trust in the platform.

• Deterministic projections — every consumer can rebuild its state from the journal.
• Transactional outbox — your database and your broker stay in lockstep.
• Replayable consumers — recovery turns from heroics into a routine.

The transactional outbox, in 12 lines

sql
BEGIN;
INSERT INTO orders (id, total, status)
VALUES ($1, $2, 'created');

INSERT INTO outbox (aggregate_id, type, payload)
VALUES ($1, 'OrderCreated', $3);
COMMIT;

-- A relay process drains `outbox` and publishes to the broker.
-- The database and the broker can never disagree about what happened.

Without an outbox you eventually publish events you have not committed, or commit events you have not published. Both fail audits, and both are nearly impossible to detect from inside the application.

“Production rewards systems that are dull on bad days.”

What I would tell a team starting today

Design for the boring failure modes first. Treat exactly-once as a property of your business logic, not your broker. Make replay a first-class operation, not a panic button. The throughline: dull is a feature.