What happened

In earlier PlanetScale posts, the router had to join customers and orders itself because each table was sharded by its own id.

The fix: shard orders by customer_id with the same hash mapping as customers. In Neki that pattern is a shard group.

Think of a warehouse aisle stocked for one customer’s order: you don’t walk the whole building for every line item.

A shard group is the rule that says which rows live on the same aisle so joins and transactions stay local.

In the data topology, a group names a shard index (for example by_customer on customer_id with type xxhash) and key ranges across physical shards.

Tables like orders, items, and addresses inherit that index; customers uses column id but the same hash. One customer’s orders, line items, and addresses land on one shard.

Lookup tables such as countries or currencies are not customer-scoped. Park them on a single shard and many joins become cross-shard.

Neki marks them as reference tables: every shard in the listed groups holds a full copy. MoveTables seeds the copies; then the router fans writes to all of them.

A product catalog usually wants a SKU key. You add another group (catalog with by_sku), often reusing the same physical shards with a different cut of keyspace.

A main group holds database-wide metadata on one shard — sequences, schema info, and other topology-wide state.

Shard indexes can hash a SQL expression — lower(sku) or left(order_id, 4).

That keeps mixed-case SKUs from different vendors together, or pulls a store prefix out of a composite order id.

Default algorithm is xxhash: uniform spread across a 64-bit keyspace.

Neki also supports range (integer ranges mapped to shards — handy for legacy geo splits) and modulo (textbook value % N).

Modulo is predictable, but changing the modulus moves nearly every row. That is usually a poor fit for growing production systems.

A lookup by orders.id when the table is sharded on customer_id fans out unless you help the router.

A global secondary index (GSI) is a maintained lookup table from order id to customer_id.

The first single-shard read resolves the shard key; the second fetches the orders row. You backfill the lookup before enabling the index.

Placement is not forever. The Reshard workflow changes shard key or shard count when query patterns outgrow the migration-day design.

The same tooling can shard an unsharded table for the first time, or give catalog and customer data different physical clusters when load stops being symmetric.

Why it matters

Teams often treat sharding as “split the big table.”

The real cost is the share of queries that touch multiple shards.

Each fan-out means the router gathers results.

Worse: a multi-shard transaction loses atomic commit and a shared snapshot. Neki commits each shard separately.

While a customer–orders join stays local, you keep familiar Postgres semantics.

Once related rows drift apart, both latency and correctness get more expensive at once.

Shard groups make the trade-off explicit: which key owns the hot path, which tables sit beside it, what is duplicated as a reference, what lives in another group.

That is data architecture, not YAML cosmetics.

Reshard lowers the fear of picking the “wrong” key forever, but it does not erase move cost.

Start from the queries that pay for latency, not from a tidy entity list.

In practice

Write down the hot paths — customer orders, cart, checkout — before you assign tables to groups.

Otherwise the schema splits neatly by entity, and production pays for cross-shard joins.

1. Find the key shared by the busiest queries and transactions; every table that carries that key is a candidate for one group.
2. For the root entity (customers.id), inherit the default index with a different column so the hash matches child customer_id values.
3. Mark owner-less lookups as reference tables, or joins against them will spray across shards.
4. Give a separate domain (catalog by SKU) its own group; share physical shards now or split later.
5. Prefer xxhash; use range or modulo when the app already shards that way or you need an explicit geo cut — modulo at scale is usually a trap.
6. For lookups that are not the shard key, add a GSI and backfill the lookup before enabling it.
7. Plan for Reshard: changing key or shard count is a normal workflow, not an incident.

Takeaway

In Neki, a shard group is the unit of co-location: the same key hash keeps related rows on one Postgres cluster.

Design starts from the queries and transactions that must stay local — not from the table list.

Reference tables, extra groups, expressions in indexes, GSIs, and Reshard are how you fit the topology to how the product actually runs.

If joins already wander across shards, find which key should keep the hot path together before you add another cluster “just in case.”