KnowledgeCafé

The Anatomy of a Black Friday Flash Sale

Amazon Engineering
engineering amazon blackfriday reliability scalability

The 10:00:00.000 AM Stampede

It’s Black Friday. A top-tier smartphone normally costing $1,200 is listed as a Lightning Deal for $499. There are exactly 50 units available.

Across the country, hundreds of thousands of users sit watching a countdown timer. 10:00:00.000 AM hits. In a single microsecond, 10,000 hyper-focused buyers slam the “Buy Now” button.

From a user’s perspective, the expectation is binary, fair, and instantaneous: they either secure the item or see a “Sold Out” badge. But behind the screen, a massive distributed system is hit by a catastrophic spike in traffic. If the system fails, two nightmarish scenarios await:

  1. Overselling: 75 people are charged for 50 items, resulting in support tickets, refunds, and lost brand loyalty.
  2. System Collapse: The entire checkout pipeline grinds to a halt under the weight of database locks, ruining the experience for everyone on the platform.

How do systems operating at massive scale handle this extreme concurrency without losing data integrity or crashing? Let’s trace the journey of that request down the stack.

1. The Immediate Bottleneck: The Distributed Transaction Trap

To understand why large-scale engineering platforms handle flash sales differently, we have to look at how traditional relational databases (like MySQL or PostgreSQL) treat data.

In a standard e-commerce application, processing a purchase requires absolute data consistency. This is usually achieved via ACID transactions. When a user clicks “Buy Now,” the backend runs a routine similar to this:

START TRANSACTION;

SELECT inventory_count 
FROM products 
WHERE id = 'phone_123' 
FOR UPDATE;

-- Application checks if inventory_count > 0. If yes:
UPDATE products 
SET inventory_count = inventory_count - 1 
WHERE id = 'phone_123';

INSERT INTO order_items (order_id, product_id, user_id) 
VALUES ('ord_999', 'phone_123', 'usr_456');

COMMIT;

Notice the phrase FOR UPDATE. This tells the database to place an exclusive write lock on that specific row in the products table. No other database connection can read or modify this row until the current transaction completes (COMMIT).

Why This Fails at Scale

When 10,000 concurrent requests hit the database for the exact same product row, the system acts like a funnel narrowed to the width of a single atom.

graph TD
    %% Define Theme Elements for Dark Mode Compatibility
    classDef requestStyle fill:#f9f9f9,stroke:#333,stroke-width:2px,color:#000;
    classDef lockStyle fill:#ffcccc,stroke:#cc0000,stroke-width:2px,color:#000;
    classDef waitStyle fill:#e1f5fe,stroke:#0288d1,stroke-width:2px,color:#000;

    Reqs[10,000 Concurrent Requests] --> Pool[Database Connection Pool]
    
    subgraph Engine [Database Transaction Layer]
        Lock[Request 1: Acquires Exclusive Row Lock]:::lockStyle
        Wait1[Request 2: Blocked / Waiting...]:::waitStyle
        Wait2[Request 3: Blocked / Waiting...]:::waitStyle
        Wait3[Request 10,000: Blocked / Waiting...]:::waitStyle
    end
    
    Pool --> Lock
    Pool --> Wait1
    Pool --> Wait2
    Pool --> Wait3

    style Pool fill:#fff,stroke:#333,stroke-width:2px,color:#000;
    style Reqs fill:#fff,stroke:#333,stroke-dasharray: 5 5,color:#000;

Request 1 acquires the lock, decrements the inventory, inserts an order, and commits. This takes maybe 10 to 20 milliseconds. Meanwhile, the other 9,999 requests are queued up, consuming database connections and memory.

As the queue grows, the database quickly runs out of available connections. Thread pool exhaustion leaks backward into the application servers. Latency spikes from milliseconds to tens of seconds. The payment gateway begins timing out, and eventually, the classic “504 Gateway Timeout” page appears. The database hasn’t just slowed down—it has completely stalled under lock contention.

2. The Elegant Solution: In-Memory Token Buckets

To handle a flash sale gracefully, engineers must decouple inventory reservation from the heavy, disk-bound order creation process. We don’t need a persistent relational database to tell us if an item is gone; we just need a hyper-fast, atomic counter in memory.

This is where in-memory data stores like Redis come into play. Redis is single-threaded by design at its core event-loop level, meaning it handles operations sequentially without requiring complex distributed locking algorithms. It can easily process over 100,000 operations per second with sub-millisecond latencies.

The Virtual Token Strategy

Before the flash sale kicks off, an administrative process populates an in-memory cache with exactly 50 “tokens” for the item. This can be modeled as a simple integer counter or an atomic list in Redis.

When a user clicks “Buy Now,” the request bypasses the relational database completely and talks directly to the Redis layer. The application runs an atomic operation—like DECR (decrement)—against the product’s inventory key.

graph TD
    %% Node Styling
    classDef gateway fill:#e0f7fa,stroke:#00acc1,stroke-width:2px,color:#000;
    classDef success fill:#e8f5e9,stroke:#4caf50,stroke-width:2px,color:#000;
    classDef reject fill:#ffebee,stroke:#f44336,stroke-width:2px,color:#000;

    Client[10,000 Microsecond Requests] --> LB[API Gateway / Load Balancer]:::gateway
    LB --> Inventory[Distributed Inventory Service]:::gateway
    
    subgraph Redis [In-Memory Gatekeeper]
        CounterA[Redis Counter: 50]
        CounterB[Redis Counter: 0]
    end

    Inventory -->|Requests 1 to 50| CounterA
    Inventory -->|Requests 51 to 10,000| CounterB

    CounterA -->|Atomic DECR >= 0| Token[Success: Token Issued]:::success
    CounterB -->|Atomic DECR < 0| Fail[Direct Cache Drop]:::reject

    Token --> Checkout[Proceed to Checkout]:::success
    Fail --> SoldOut1[Immediate 'Sold Out' Screen]:::reject

    style Client fill:#fff,stroke:#333,stroke-dasharray: 5 5,color:#000;
    style CounterA fill:#fff,stroke:#333,stroke-width:2px,color:#000;
    style CounterB fill:#fff,stroke:#333,stroke-width:2px,color:#000;
    style Redis fill:#f5f5f5,stroke:#9e9e9e,stroke-width:1px,stroke-dasharray: 5 5,color:#000;

Because Redis handles this atomically, it serves as an ultra-high-speed gatekeeper:

  • The first 50 requests successfully decrement the counter (from 50 down to 0) and receive a confirmation token.
  • The remaining 9,950 requests attempt to decrement the counter, see that the value has fallen below zero, and are immediately rejected.

The rejected users receive a clean, lightning-fast “Sold Out” message within milliseconds. Their requests never touch the core database, preserving system stability for everyone else.

3. The Lifecycle of a Flash Sale Request

Securing an in-memory token does not mean the user has bought the item; it means they have reserved the right to buy it. They now have a short window (e.g., 5 or 10 minutes) to complete the checkout process, supply billing details, and process payment. This separation of concerns turns a synchronous bottleneck into an asynchronous, resilient workflow.

High-Level System Architecture

graph TD
    User[User Client] -->|1. Click Buy| API[API Gateway]
    API -->|2. Reserve Token| Redis[(Redis Inventory Cache)]
    
    Redis -->|3a. Success Token Issued| API
    Redis -->|3b. Denied Out of Stock| API
    
    API -->|4. Push Order Event| MQ[Message Queue / SQS]
    MQ -->|5. Consume Event| Worker[Order Processing Service]
    Worker -->|6. Write Order| DB[(Relational DB / Aurora)]
    Worker -->|7. Charge Card| Pay[Payment Gateway]
  • Token Allocation: The API gateway routes the request to the inventory service, which decrements the Redis counter.
  • Asynchronous Handshake: If the token is acquired, the application generates a unique reservation ID and pushes an order creation event into a high-throughput message queue (like Apache Kafka or AWS SQS).
  • Decoupled Writes: Dedicated background worker instances consume messages from the queue at a controlled pace. They slowly write the records to the relational database and call the external payment gateway, safely isolated from the initial traffic spike.

4. Dealing with the Fallout: Saga Pattern & Compensating Transactions

What happens if a user claims 1 of the 50 tokens, but their bank declines the credit card 3 minutes later? Or what if they close their browser tab and let their reservation window expire? If we simply throw that token away, we end up under-selling our stock. We’d sell only 44 phones instead of the available 50, leaving inventory stranded in the warehouse. Because we decoupled our database from our cache, we cannot use a traditional database rollback to fix this. Instead, we implement the Saga Pattern using Compensating Transactions.

Orchestrating the Rollback

A Saga is a sequence of local transactions. If one step fails or times out, the Saga coordinator executes a series of compensating steps in reverse order to undo the changes.

sequenceDiagram
    autonumber
    participant Client as User Client
    participant Inventory as Inventory Service (Redis)
    participant Order as Order Service (DB)
    participant Payment as Payment Gateway

    Client->>Inventory: Request Reservation
    Inventory-->>Client: Token Reserved (Stock = 49)
    Client->>Order: Create Pending Order
    Order->>Payment: Process Payment
    Note over Payment: Payment Fails / Timeout
    Payment-->>Order: Decline Transaction
    Order->>Order: Mark Order as Cancelled
    Order->>Inventory: Release Token (Compensating Action)
    Inventory-->>Inventory: Increment Counter (Stock = 50)
  • The Trigger: The Payment Service attempts to bill the customer, but the transaction fails.
  • The Local Compensation: The Order Service updates the state of the order from PENDING_PAYMENT to FAILED_COMPENSATION.
  • The Cache Restoration: An asynchronous event is dispatched back to the Inventory Service. The service performs an atomic increment (INCR) on the Redis counter, lifting it back from 0 to 1.

Instantly, the single phone is placed back into the available pool, ready to be grabbed by the next user refreshing the page.

5. Architectural Takeaways

-Designing for ultra-high concurrency requires shifting your engineering philosophy away from traditional patterns.

  • Favor Eventual Consistency over Absolute Consistency: Don’t force your entire system to stay perfectly synced via heavy locks in a single database. Keep your fast edge layer (Redis) eventually consistent with your durable system of record (the relational database).
  • Protect the Core via Admission Control: Throw away invalid or unsuccessful requests as early as possible. Let your memory cache filter out the 9,950 users who didn’t get a token, ensuring your core databases and payment pipelines only process high-value, valid traffic.

Convert Synchronous Spikes into Asynchronous Flows: Use message queues to smooth out sudden traffic peaks. It doesn’t matter if 10,000 people click a button at the exact same microsecond, as long as your background workers can systematically process those orders at a sustainable rate over the next few minutes.

Enjoyed this deep dive? Staying updated on architectural shifts shouldn’t be a full-time job. At Knowledge Cafe, we act as your personal research team, filtering the fluff from the world’s top engineering blogs to bring you the insights needed to build the next generation of resilient systems.

Subscribe to Knowledge Cafe here