Skip to main content

Sharding

Discord-PXE has evolved from one proven singleton Gateway connection into a local pool of isolated shard contexts.

The current stable hardware baseline can bring 128 local shard contexts to READY on one physical machine using the BSP scheduler.

Two different limits

128 local slots is the runtime's per-node capacity model. It is not the same thing as Discord's global shard count, and it does not imply that every deployment should use 128 shards.

Global vs local shard ownership​

Keep these values separate:

global shard count  = total Discord shard count for the bot
local shard start   = first global shard owned by this node
local shard count   = number of contexts owned locally

A node can therefore own a contiguous subset of a larger global plan.

Example:

global total: 256
node A: start 0,   count 128
node B: start 128, count 128

Per-shard state​

Each local shard context owns or references independent state for:

  • global shard ID
  • global shard total
  • lifecycle state
  • Gateway sequence
  • heartbeat timing / ACK tracking
  • Identify bucket
  • TCP context
  • TLS context
  • WebSocket context
  • Discord context
  • RX/TX buffers
  • reconnect/backoff
  • lease generation/deadline
  • session/resume data
  • last error

Isolation includes transport memory, not just Discord-level IDs.

Stable BSP scheduler​

The current default scheduler advances active shard state machines on the bootstrap processor.

For each shard, it progresses roughly:

lifecycle
-> TCP
-> reliable TCP
-> TLS
-> WebSocket / Discord Gateway

Global admission and Identify arbitration are also coordinated centrally.

This path is hardware-proven through a complete 128-shard READY run.

Experimental SMP scheduler​

The runtime also contains AP workers that can process disjoint subsets of shard contexts.

The intended ownership rule is:

AP worker 1 -> one subset
AP worker 2 -> another subset
AP worker 3 -> another subset
BSP         -> global admission / arbitration / RX coordination

The first SMP hardware test exposed shared temporary state in crypto code and shared hardware TX assumptions.

Private shard TLS context does not guarantee re-entrant crypto

Each shard owns separate keys/session state, but some lower-level crypto primitives still use global scratch buffers. Parallel AP execution therefore needs serialization or per-CPU/per-context scratch.

Identify scheduling​

Discord's Identify max_concurrency is server-provided session-start state. It should not be guessed from:

  • CPU count
  • local shard count
  • AP worker count
  • a convenient test value

Until authenticated runtime /gateway/bot discovery is implemented, local shell plans force:

max_concurrency = 1

Connection admission​

A shard is allowed to connect only when the runtime is ready to carry it through the Identify schedule.

Important gates include:

  • shard networking enabled
  • DNS READY
  • current connection/Identify wave permits it
  • TCP context idle
  • shard not FATAL

This avoids opening a large backlog of sockets before they are allowed to Identify.

Manual plans​

A full local test:

shardplan 128 0 128
shardstart

A smaller plan:

shardplan 8 0 8
shardstart

Inspect current state with:

shards

Singleton takeover​

The singleton path is still valuable as the control case.

Before sharding takes ownership, it proves:

  • DNS
  • TCP
  • TLS/X.509
  • WebSocket
  • Discord authentication

shardstart requires DNS READY before stopping the singleton path.

Reconnect and backoff​

A transient TCP/TLS failure moves only the affected shard into BACKOFF.

The runtime can then:

  1. preserve resumable Discord state when valid
  2. reset TCP/TLS/WebSocket state
  3. wait for the reconnect deadline
  4. rebuild the connection
  5. return the shard to READY

Hardware testing has demonstrated transient shard recovery in the stable BSP path.

Lease expiry​

For controller-style ownership testing:

shardlease 30

When the lease expires:

  • shard-owned traffic is disabled
  • local sessions are invalidated
  • TCP transport is aborted
  • shard states return to assigned/inactive ownership state

That prevents stale nodes from believing they own shards indefinitely.

Guild routing​

Discord maps guilds to global shard IDs with:

(guild_id >> 22) % total_shards

The local runtime can then determine whether that global shard falls inside its current leased range.

Current boundaries​

Hardware-proven​

  • 128 local contexts
  • isolated per-shard transport/Gateway state
  • stable BSP scheduler
  • 128-shard READY bring-up
  • transient failure/backoff/reconnect

Implemented but still experimental​

  • AP/SMP shard-state execution
  • serialized shared crypto under SMP
  • broader controller integration

Still missing for production-style operation​

  • runtime /gateway/bot discovery
  • full session-start-limit handling
  • complete Resume strategy across close/error cases
  • authenticated controller transport
  • durable per-device identity

For the deeper state-machine view, see Gateway Sharding and Scheduler & Lifecycle.