A market data cache is sharded by symbol hash across 16 nodes. One symbol takes 30 percent of all reads, so its shard saturates while the other 15 sit at 20 percent CPU. Give the fix that requires no resharding, then describe how you would move to 32 nodes with no downtime if you had to.

A market data cache is sharded by symbol hash across 16 nodes. One symbol takes 30 percent of all reads, so its shard saturates while the other 15 sit at 20 percent CPU. Give the fix that requires no resharding, then describe how you would move to 32 nodes with no downtime if you had to.

Approach: Note that hashing spreads keys and not load, find a way to add capacity for the single hot key without changing the key space, then stage the reshard as a dual write with a reversible cutover.

Replicate the hot key rather than move it: keep the shard map and serve that one symbol from a small replica set or from a short lived cache in front of every client, which removes the saturation without touching the key space. Hashing distributes keys uniformly and says nothing about load, so a single key carrying 30 percent of reads saturates one hot shard however many shards exist, and adding nodes does not help because the key still lives on exactly one of them. For a read heavy workload the cheap fixes are a client side cache with a short time to live, which converts 30 percent of reads into one refresh per client per interval, and read replicas for the few keys above a measured load threshold rather than a hand written list. Key splitting into symbol#0 through symbol#7 also works and costs a fan out on write. When a move to 32 nodes is genuinely required, run it as a dual write cutover: publish the new shard map beside the old, write to both, backfill the new map while comparing, move reads one shard at a time behind a per shard flag that can be reversed, and only then stop writing the old map. Splitting each existing shard in two keeps the copying to half of each shard rather than rehashing everything.

Follow-up: How do you keep the client side cache from serving a price staler than a stated bound?

Key concepts: hot shard, read replicas, key splitting, dual write cutover.