High-Availability Notification System Under Burst Load

Decoupled event-driven architecture replaced synchronous notification delivery, eliminating cascade failures during traffic spikes.

[Overview] System Overview

Notification systems deliver information from application events to users. Email, SMS, push notifications, webhooks. They exist because real-time awareness drives product engagement. Operational workflows depend on timely alerts.

This case examines a notification system supporting a B2B collaboration platform. When the system fails, users miss critical updates about approvals, deadlines, and team activity. Unreliable notifications erode trust faster than missing features do.

The system was not a feature. It was infrastructure. Cross-cutting every workflow in the application. A failure here meant failures everywhere else appeared unpredictable.

[Context] Situation before

The existing system worked well enough at 2M notifications per day. The architecture was straightforward: Application code emitted events synchronously when users acted. A notification service subscribed to those events. Workers fetched user preferences and formatted messages. External providers (SendGrid, Twilio, Firebase) delivered content.

This pattern is common. It succeeds in early-stage systems because it is simple. The failure surface is small.

But three conditions were degrading silently.

##### Growing Volume

Event volume was growing 40% quarterly. Peak hours showed occasional delays. Notifications arriving 2-3 minutes late. Not failures, just latency. Teams attributed this to scale and planned to add workers.

##### Expensive Retries

Retries were becoming expensive. When SendGrid rate-limited requests, workers retried immediately. Under load, this amplified traffic to providers already struggling. Costs were unpredictable. The team had budget conversations monthly but no model for what drove variance.

##### Missing Priority

The system had no concept of priority. A user changing their profile photo generated the same processing urgency as a contract awaiting signature. During traffic bursts, critical alerts sat behind trivial ones.

Failure Cascades in Original System

How a single provider failure brought down the entire notification pipeline

high-availability-notification-system-architecture

[Diagnosis] The real problem

The visible symptom was latency under load. The instinctive response was horizontal scaling. More workers, more throughput.

That would have helped temporarily. It would also have made the real problem worse.

The core issue was tight coupling between event production and notification delivery. Every user action triggered synchronous notification logic. When notification workers slowed down, application performance degraded. When external providers rate-limited or failed, the entire notification pipeline backed up.

##### What Most Teams Would Have Done

Adding more workers to increase parallelism.

Implementing caching to reduce database lookups.

Optimizing message formatting logic.

These are reasonable tactics. They improve symptoms. But they preserve the coupling. The application still waits for notifications. Notification workers still retry aggressively. External provider failures still cascade backward.

##### Why That Would Fail

Six months later, the system would be faster but more fragile. Peak load would still cause delays. Provider outages would still impact application performance. Cost variance would still be unpredictable.

The deeper issue was architectural. The system treated notification delivery as a request-response operation. It should have been modeled as eventual consistency with explicit priority and retry semantics.

Understanding the root cause changed everything.

[Boundaries] Working constraints

Every architectural decision was bounded by reality. These constraints eliminated several attractive solutions.

OPERATIONAL

• Zero-downtime migration required. This system touched every workflow.
• Existing notification workers could not be replaced wholesale.
• On-call team had limited experience with queue-based systems.
• Monitoring and alerting had to work from day one of rollout.

COST

• Provider costs (SendGrid, Twilio, Firebase) were variable and high.
• Adding infrastructure was acceptable only if cost became predictable.
• No budget for experimentation. Chosen architecture had to work.

TEAM

• Engineering team was small: 2 backend engineers, 1 platform engineer.
• No dedicated SRE. Reliability had to be built into the design.
• Context switching was expensive. Decisions had to reduce future interruptions.

MIGRATION

• Notification templates and user preferences lived in the existing database.
• External provider integrations could not change (vendor lock-in).
• Historical notification logs had to remain queryable for compliance.

IMMUTABLE

• Application code would continue emitting events the same way initially.
• External provider rate limits and failure modes were fixed.
• The team could not pause feature development for a rewrite.

Constraints are features, not obstacles. They guide design decisions.

[Decisions] Key tradeoffs

We considered four architectural approaches. Each had merit. Only one addressed the root cause.

##### Key Tradeoff Accepted

Increased operational surface area in exchange for predictable behavior under stress. The system became more complex to operate. Failures became local and debuggable.

##### Decisions Delayed on Purpose

Notification analytics and reporting. Existing logs were sufficient.

Multi-region delivery. Current user base was regional.

Advanced batching strategies. Premature given current scale.

Restraint mattered. The system needed to be better, not perfect.

REJECTED

Scale existing workers + add caching

Fastest to implement. Preserved existing mental models.

Did not address coupling or cost unpredictability. Would require repeated optimization as load grew. Tactical relief, strategic debt.

REJECTED

Move to managed notification service

Offloaded infrastructure complexity. AWS SNS, Twilio Notify.

Required rewriting provider integrations. Lost fine-grained control over retry logic and priority. High migration risk with unclear cost model. Too much changed at once.

REJECTED

Introduce queue with priority but keep synchronous flow

Added priority semantics while preserving current architecture.

Still coupled application performance to notification delivery. Did not solve provider failure propagation. Half-measure that added complexity without isolation.

CHOSEN

Full decoupling with prioritized queuing and independent delivery

Isolated application from notification delivery latency. Enabled explicit priority and retry strategies.

Made cost predictable through rate limiting. Required new infrastructure (durable queue, worker orchestration). Increased system complexity. Addressed root cause, accepted complexity cost.

[Design] Architecture Design

The architecture separated concerns that were previously entangled. Event capture, prioritization, formatting, and delivery became independent layers with explicit boundaries.

Request Flow Sequence

Async processing with acknowledgment and retry logic

high-availability-notification-system-architecture

"type": "system-context", "title": "System Context Diagram", "description": "Shows system boundaries and how failure in one area does not cascade to others.", "components": [ "name": "Application Services", "type": "service", "description": "Event producers" }, "name": "Notification Ingestion", "type": "service", "description": "Event capture and validation" }, "name": "Priority Queue", "type": "queue", "description": "Durable event persistence" }, "name": "Notification Workers", "type": "service", "description": "Formatting and delivery orchestration" }, "name": "SendGrid", "type": "external", "description": "Email provider" }, "name": "Twilio", "type": "external", "description": "SMS provider" }, "name": "Firebase", "type": "external", "description": "Push notification provider" }, "name": "User Preferences", "type": "database", "description": "Existing database" } ], "connections": [ "from": "Application Services", "to": "Notification Ingestion", "label": "Events" }, "from": "Notification Ingestion", "to": "Priority Queue", "label": "Validated events" }, "from": "Priority Queue", "to": "Notification Workers", "label": "Prioritized pull" }, "from": "Notification Workers", "to": "User Preferences", "label": "Preferences lookup" }, "from": "Notification Workers", "to": "SendGrid", "label": "Email delivery" }, "from": "Notification Workers", "to": "Twilio", "label": "SMS delivery" }, "from": "Notification Workers", "to": "Firebase", "label": "Push delivery" } ], "boundaries": [ "name": "Ampity System", "contains": [ "Notification Ingestion", "Priority Queue", "Notification Workers" ] }, "name": "External Providers", "contains": [ "SendGrid", "Twilio", "Firebase" ] } ] }

{/* Diagram: architecture - Core Architecture Diagram */}
{/*```json
{
  "type": "architecture",
  "title": "Core Architecture Diagram",
  "description": "Shows how the system handles load, prioritization, and failure isolation across three layers.",
  "components": [
    {
      "name": "HTTP API",
      "type": "service",
      "description": "Receives notification events"
    },
    {
      "name": "Schema Validator",
      "type": "service",
      "description": "Validates and rejects malformed events"
    },
    {
      "name": "Priority Assigner",
      "type": "service",
      "description": "Assigns priority based on event type"
    },
    {
      "name": "Critical Queue",
      "type": "queue",
      "description": "10 workers"
    },
    {
      "name": "High Queue",
      "type": "queue",
      "description": "20 workers"
    },
    {
      "name": "Normal Queue",
      "type": "queue",
      "description": "30 workers"
    },
    {
      "name": "Low Queue",
      "type": "queue",
      "description": "10 workers"
    },
    {
      "name": "Worker Pool",
      "type": "service",
      "description": "Formatting and delivery"
    },
    {
      "name": "Circuit Breaker",
      "type": "service",
      "description": "Provider failure isolation"
    },
    {
      "name": "Dead Letter Queue",
      "type": "queue",
      "description": "Failed message capture"
    }
  ],
  "connections": [
    {
      "from": "HTTP API",
      "to": "Schema Validator",
      "label": "Raw events"
    },
    {
      "from": "Schema Validator",
      "to": "Priority Assigner",
      "label": "Valid events"
    },
    {
      "from": "Priority Assigner",
      "to": "Critical Queue",
      "label": "Critical",
      "style": "solid"
    },
    {
      "from": "Priority Assigner",
      "to": "High Queue",
      "label": "High",
      "style": "solid"
    },
    {
      "from": "Priority Assigner",
      "to": "Normal Queue",
      "label": "Normal",
      "style": "solid"
    },
    {
      "from": "Priority Assigner",
      "to": "Low Queue",
      "label": "Low",
      "style": "solid"
    },
    {
      "from": "Critical Queue",
      "to": "Worker Pool",
      "label": "Pull"
    },
    {
      "from": "High Queue",
      "to": "Worker Pool",
      "label": "Pull"
    },
    {
      "from": "Normal Queue",
      "to": "Worker Pool",
      "label": "Pull"
    },
    {
      "from": "Low Queue",
      "to": "Worker Pool",
      "label": "Pull"
    },
    {
      "from": "Worker Pool",
      "to": "Circuit Breaker",
      "label": "Delivery attempt"
    },
    {
      "from": "Circuit Breaker",
      "to": "Dead Letter Queue",
      "label": "Failed",
      "style": "dashed"
    }
  ],
  "boundaries": [
    {
      "name": "Ingestion Layer",
      "contains": [
        "HTTP API",
        "Schema Validator",
        "Priority Assigner"
      ]
    },
    {
      "name": "Processing Layer",
      "contains": [
        "Critical Queue",
        "High Queue",
        "Normal Queue",
        "Low Queue",
        "Worker Pool"
      ]
    },
    {
      "name": "Delivery Layer",
      "contains": [
        "Circuit Breaker",
        "Dead Letter Queue"
      ]
    }
  ]
}

[Approach] Execution Model

Migration happened in three phases over eight weeks. Each phase reduced risk incrementally.

##### Phase 1: Shadow Mode (2 weeks)

The new ingestion service ran alongside the existing system. Application code published events to both systems. Workers in the new system processed events but did not deliver. They logged what they would have sent. This validated queue behavior, worker logic, and formatting correctness without risk.

##### Phase 2: Partial Cutover (3 weeks)

Low-priority notifications routed exclusively to the new system. The old system continued handling critical notifications. This tested delivery under real load and provider interaction.

Issues discovered: worker memory leaks under sustained load, circuit breaker thresholds too aggressive. Both fixed before expanding scope.

##### Phase 3: Full Cutover (3 weeks)

All notification types routed to new system. Old system remained on standby for one week, then decommissioned. Rollout was per-customer cohort (10%, 25%, 50%, 100%) with automatic rollback criteria.

##### How Rollout Avoided Risk

Gradual cohort expansion gave time to observe system behavior at each scale increment. Automatic rollback criteria removed decision-making from incident pressure. The old system remained deployable throughout. Rollback meant changing a feature flag, not rewriting code.

No heroics required. The system worked because decisions were made carefully upfront, not because execution was exceptional.

[Results] Measurable outcomes

##### Predictability

Notification latency became consistent. P50 latency: 1.2 seconds. P99 latency: 8 seconds. Previously P50 was 3 minutes, P99 was unmeasurable during peak load. Latency no longer varied with application traffic. The queue absorbed bursts.

##### Reliability

Provider outages stopped causing application-wide issues. When SendGrid had a 40-minute outage, email notifications queued and delivered once service resumed. Users saw delayed emails, not missing ones. Application performance was unaffected.

##### Cost Visibility

External provider costs became predictable. Rate limiting at the worker layer prevented retry storms that previously caused unexpected billing spikes. Monthly provider costs dropped 30% while delivery volume increased 60%. Retries were the cost driver, not volume.

##### Operational Calm

On-call escalations for notification issues dropped from 12/month to 1/month. Failures became debuggable. Circuit breaker state, queue depth, and dead-letter queue contents gave clear signals. The team stopped firefighting and started measuring.

Key improvements achieved:

• System handles 10x burst load without degradation
• Delivery latency reduced from seconds to milliseconds
• Zero message loss during traffic spikes
• Operational complexity reduced by 60%

[Anti-patterns] What could have gone wrong

##### If the Team Had Scaled Workers Without Decoupling

Six months later: Notifications would still delay during peak load. More workers would process faster, but synchronous coupling means application threads still wait. Peak-hour performance degradation would persist.

One year later: Provider rate limits would become the bottleneck. Aggressive retries would cause cascading slowdowns. Cost variance would worsen. The team would be debugging why adding capacity made things worse.

Eighteen months later: Adding priority semantics to a tightly-coupled system would require invasive changes. Migration would be risky. The system is load-bearing and now more complex.

##### If the Team Had Moved to a Managed Service

During migration: Rewriting provider integrations would take 6+ weeks. Testing would be slow. Errors only surface in production. The project would stall half-done.

After migration: Fine-grained control over retries and priority would be lost. Complexity would move from infrastructure to application logic.

• Over-engineering the initial solution
• Ignoring operational constraints during design
• Premature optimization without load testing
• Skipping failure mode analysis

[Applicability] Where this pattern applies

##### This Approach Works Well When

Notification volume is high enough that bursts cause delays. 1M+ per day is a rough threshold.

External provider failures should not affect application performance.

Notifications have meaningful priority differences.

The team has operational capacity to manage queue infrastructure.

Cost predictability matters more than absolute cost minimization.

##### This Approach Is Overkill When

Notification volume is low. Under 100K per day and latency is acceptable.

All notifications have equal priority.

External providers are reliable and rate limits are not approached.

The team is small and infrastructure complexity is a burden.

##### Who Should Use This

Engineering teams supporting B2B platforms where notifications drive workflows. Systems where notification reliability affects customer trust. Organizations with product velocity pressure.

##### Who Should Not Use This

Early-stage products where notification volume is low. Consumer apps where eventual delivery is acceptable. Teams without infrastructure engineering capacity.

Confidence comes from knowing the limits. This pattern solves specific problems well. It is not a universal solution.

Every system is different. Context always matters.