Gateway Sharding

Discord-PXE's sharding work is not merely "connect the same bot multiple times."

Every Gateway session needs independent protocol state, and Discord imposes global rules that must still be respected when those sessions are distributed across one or more physical machines.

Primary sources:

• legacy_shards.inc
• legacy_shard_gateway.inc
• legacy_shard_discord.inc

Per-shard Gateway state

The shard descriptor directly owns or points to state for:

• global shard ID
• total shard count
• sequence number
• heartbeat interval
• heartbeat deadline
• heartbeat sent count
• heartbeat ACK count
• heartbeat misses
• Identify bucket
• session pointer
• reconnect counters
• flags indicating Identify/READY/resume state

This state cannot be global once two Gateway sessions run simultaneously.

Sequence ownership

Discord dispatch events include a sequence number used for resume.

The runtime stores that value on the shard context and marks it valid with a flag.

On reconnect, the lifecycle code only preserves the sequence when a valid resumable session also survives.

That avoids sending a resume attempt with a stale or meaningless sequence.

Heartbeat scheduling

The Gateway Hello provides a heartbeat interval.

Instead of letting every shard send its first heartbeat at exactly the same relative instant, the sharded runtime deliberately spreads the first heartbeat across the interval.

The current scheduling derives a deterministic factor from the shard ID.

Conceptually:

shard 0   ┐
shard 1   │
shard 2   ├─ spread across the advertised interval
...       │
shard N   ┘

This avoids a local burst when many shards start together.

ACK tracking

The shard flags include an AWAITING_ACK state and the descriptor tracks:

• heartbeats sent
• heartbeat ACKs
• heartbeat misses

The acceptance test requires live ACKs, not merely a READY dispatch.

That catches cases where the connection looks online but the event loop/heartbeat path is not actually healthy.

Identify concurrency

Discord provides a max_concurrency value that defines Identify buckets.

The local runtime stores an Identify bucket per shard:

bucket = shard_id % max_concurrency

The local scheduler tracks which buckets have been used in the current Identify wave.

Code example of Identify bucket scheduling

NASM original

mov eax, [rdi + SHARD_OFF_ID]
xor edx, edx
mov ecx, [rel shard_identify_concurrency]
test ecx, ecx
jnz .bucket_div
mov ecx, 1
.bucket_div:
div ecx
mov [rdi + SHARD_OFF_IDENTIFY_BUCKET], edx

This is the actual bucket-assignment step. The scheduler later combines the stored bucket with per-wave usage state before allowing Identify.

Execution order

1. Read the shard's global ID.
2. Read Discord's current Identify `max_concurrency`.
3. Normalize zero concurrency to one bucket.
4. Divide shard ID by bucket count.
5. Use the remainder as the shard's Identify bucket.
6. During a wave, permit Identify only when that bucket has not already been consumed.

Rust

fn identify_bucket(shard_id: u32, max_concurrency: u32) -> u32 {
    shard_id % max_concurrency.max(1)
}

fn can_identify(shard_id: u32, max_concurrency: u32, used: &[bool]) -> bool {
    let bucket = identify_bucket(shard_id, max_concurrency);
    !used[bucket as usize]
}

C

uint32_t identify_bucket(uint32_t shard_id, uint32_t max_concurrency) {
    if (max_concurrency == 0)
        max_concurrency = 1;

    return shard_id % max_concurrency;
}

int can_identify(uint32_t shard_id, uint32_t max_concurrency,
                 const uint8_t *used) {
    return !used[identify_bucket(shard_id, max_concurrency)];
}

Python

def identify_bucket(shard_id: int, max_concurrency: int) -> int:
    return shard_id % max(max_concurrency, 1)

def can_identify(shard_id: int, max_concurrency: int, used: list[bool]) -> bool:
    return not used[identify_bucket(shard_id, max_concurrency)]

A zero or missing concurrency value is normalized to one bucket.

Why Identify cannot be local-only

If one PXE machine owns shards 0–15 and another owns 16–31, both machines are still identifying the same bot.

Therefore:

per-machine scheduling != global Discord scheduling

For multi-node operation, the controller exposes an Identify claim endpoint that verifies ownership and coordinates attempts by bucket.

See Controller Protocol.

Identify vs Resume

A reconnect should not always send Identify.

If the shard retains:

• a valid session ID
• a valid sequence
• resume URL/session state

the reconnect path can attempt resume instead.

Fresh Identify is reserved for a session that cannot be resumed or for an initial connection.

READY

READY transitions the shard toward the healthy online state and records that the Gateway session became active.

But the project intentionally does not treat READY as the only acceptance criterion.

For a multi-shard test, READY could still occur even if some lower-level state is accidentally shared.

Guild routing

Discord routes guilds to shards using:

(guild_id >> 22) % total_shards

The bare-metal runtime implements this calculation, and the controller exposes a matching routing endpoint to locate the node that currently leases the resulting shard.

This connects application-level guild routing to physical node ownership.

Close codes and fatal state

Some Gateway close codes indicate configuration or authorization problems rather than transient transport failure.

Those cases should not reconnect forever.

The shard lifecycle supports a fatal state so a bad configuration can stop one shard while leaving unrelated shards alive.

That is especially important at scale: one bad shard must not cause a global reboot loop.

Shard state is broader than Gateway state

A shard can be:

ASSIGNED
STARTING
READY
BACKOFF
FATAL

while internal protocol state separately tracks TCP/TLS/WebSocket/Gateway progression.

This separation keeps the high-level scheduler simple.

Two-shard acceptance

The built-in test starts two private shard contexts on one machine and intentionally quiesces the old singleton connection for that boot.

The test then verifies both contexts independently make progress through:

TCP
TLS
WebSocket
Hello
Identify
READY
heartbeat ACK

The test is designed to prove context isolation before larger local shard counts are attempted.

Why scale 2 → 4 → 8 instead of 1 → 128

A failure at 128 concurrent sessions is difficult to localize.

A failure at 2 is much easier to classify:

• shared buffer
• bad local port
• reused TLS workspace
• scheduler bug
• Identify timing bug
• session-state leak

The staged sequence turns scaling into a verification process rather than a stress-test lottery.

Initial heartbeat deadline scheduling

Discord allows the first heartbeat to occur at a jittered point inside the negotiated interval. Discord-PXE spreads first-heartbeat deadlines across twenty deterministic positions derived from the shard ID.

NASM original

Context before the block

Symbol / register	Meaning
RBX	Owning shard context.
SHARD_OFF_HEARTBEAT_MS	Heartbeat interval supplied by Discord Hello.
SHARD_OFF_ID	Global shard ID.
tsc_cycles_per_ms	Runtime calibration used to convert milliseconds into TSC ticks.
SHARD_OFF_HEARTBEAT_DEADLINE	Absolute TSC deadline for the next heartbeat.

mov r9d, eax                              ; R9D = advertised heartbeat interval in milliseconds.
mov eax, [rbx + SHARD_OFF_ID]             ; EAX = global shard ID.
xor edx, edx                              ; Clear high dividend before DIV.
mov ecx, 20                               ; Split one interval into twenty deterministic slots.
div ecx                                   ; EDX = shard_id % 20.
inc edx                                   ; Convert remainder 0..19 into factor 1..20.

mov eax, r9d                              ; EAX = heartbeat interval again.
imul eax, edx                             ; Multiply interval by this shard's factor.
xor edx, edx                              ; Clear high dividend for second division.
mov ecx, 20                               ; Divisor = number of jitter slots.
div ecx                                   ; EAX = deterministic delay in milliseconds.

test eax, eax                             ; Did integer division round delay down to zero?
jnz .have_ms                              ; No: keep calculated delay.
mov eax, 1                                ; Yes: force minimum delay of 1 ms.

.have_ms:
mov r8, [rel tsc_cycles_per_ms]           ; R8 = calibrated TSC ticks per millisecond.
test r8, r8                               ; Is timing calibration available?
jz .bad                                   ; No: cannot create a reliable TSC deadline.

imul rax, r8                              ; RAX = delay expressed in TSC cycles.
mov r9, rax                               ; Preserve cycle delay while reading current TSC.

rdtsc                                     ; EDX:EAX = current time-stamp counter.
shl rdx, 32                               ; Move high TSC half into bits 63:32.
or rax, rdx                               ; RAX = complete 64-bit current TSC.
add rax, r9                               ; RAX = absolute first-heartbeat deadline.

mov [rbx + SHARD_OFF_HEARTBEAT_DEADLINE], rax ; Persist deadline on this shard.

Execution order

1. Read the heartbeat interval and global shard ID.
2. Map the shard ID into one of twenty deterministic jitter slots.
3. Scale the heartbeat interval by that slot and clamp to at least 1 ms.
4. Convert the delay from milliseconds into TSC cycles.
5. Read the current TSC.
6. Store the absolute first-heartbeat deadline on the shard.

Rust equivalent

fn schedule_first_heartbeat(
    shard: &mut ShardContext,
    cycles_per_ms: u64,
    now_tsc: u64,
) -> bool {
    if shard.heartbeat_ms == 0 || cycles_per_ms == 0 {
        return false;
    }

    let factor = (shard.id % 20) + 1;
    let delay_ms = ((shard.heartbeat_ms * factor) / 20).max(1);

    shard.heartbeat_deadline =
        now_tsc + u64::from(delay_ms) * cycles_per_ms;

    true
}

C equivalent

int schedule_first_heartbeat(struct shard_ctx *shard,
                             uint64_t cycles_per_ms,
                             uint64_t now_tsc) {
    if (!shard || !shard->heartbeat_ms || !cycles_per_ms)
        return 0;

    uint32_t factor = (shard->id % 20u) + 1u;
    uint32_t delay_ms = (shard->heartbeat_ms * factor) / 20u;
    if (!delay_ms) delay_ms = 1;

    shard->heartbeat_deadline =
        now_tsc + (uint64_t)delay_ms * cycles_per_ms;
    return 1;
}

Python model

def schedule_first_heartbeat(shard, cycles_per_ms, now_tsc):
    if shard.heartbeat_ms == 0 or cycles_per_ms == 0:
        return False

    factor = (shard.id % 20) + 1
    delay_ms = (shard.heartbeat_ms * factor) // 20
    delay_ms = max(delay_ms, 1)

    shard.heartbeat_deadline = now_tsc + delay_ms * cycles_per_ms
    return True

Why deterministic instead of random?

The purpose is burst spreading, not cryptographic unpredictability. A deterministic mapping makes startup behavior reproducible in serial logs while still avoiding every shard firing at the same instant.

Success condition

Each shard receives a nonzero TSC deadline somewhere inside its advertised heartbeat interval, distributed by shard ID.

Gateway READY and heartbeat-ACK transitions

These helpers are intentionally tiny because Gateway state changes should be visible and local to one shard descriptor.

NASM original

State fields touched

Field	Meaning
HEARTBEAT_ACKS	Total ACK opcodes observed for this shard.
SHARD_FLAG_AWAITING_ACK	Set after sending a heartbeat; cleared by ACK.
HEARTBEAT_MISSES	Consecutive missed-ACK count.
SHARD_STATE_READY	High-level shard lifecycle state.
SHARD_FLAG_READY	Orthogonal flag that records READY observation.
SESSION_GEN	Monotonic generation counter for successful Gateway sessions.

legacy_shard_gateway_note_ack64:
    test rcx, rcx                              ; Validate shard-context pointer.
    jz .bad                                    ; Null context cannot record an ACK.
    inc qword [rcx + SHARD_OFF_HEARTBEAT_ACKS] ; Count this heartbeat acknowledgement.
    and dword [rcx + SHARD_OFF_FLAGS], ~SHARD_FLAG_AWAITING_ACK ; Clear outstanding-ACK flag.
    mov dword [rcx + SHARD_OFF_HEARTBEAT_MISSES], 0 ; Healthy ACK resets miss streak.
    xor eax, eax                               ; EAX = 0 success.
    ret                                        ; Return to Gateway parser.

.bad:
    mov eax, 1                                 ; EAX = 1 failure for invalid context.
    ret                                        ; Return failure.

legacy_shard_gateway_mark_ready64:
    test rcx, rcx                              ; Validate shard-context pointer.
    jz .bad                                    ; Null context cannot become READY.
    mov dword [rcx + SHARD_OFF_STATE], SHARD_STATE_READY ; Promote lifecycle state.
    or dword [rcx + SHARD_OFF_FLAGS], SHARD_FLAG_READY ; Record READY as orthogonal flag.
    inc qword [rcx + SHARD_OFF_SESSION_GEN]    ; Advance successful-session generation.
    xor eax, eax                               ; EAX = 0 success.
    ret                                        ; Return to caller.

Execution order

1. On heartbeat ACK, increment the ACK counter.
2. Clear the awaiting-ACK flag.
3. Reset the heartbeat-miss streak.
4. On READY, promote the shard lifecycle state to READY.
5. Set the READY flag and increment the session generation.

Rust equivalent

fn note_ack(shard: &mut ShardContext) {
    shard.heartbeat_acks += 1;
    shard.flags.remove(ShardFlags::AWAITING_ACK);
    shard.heartbeat_misses = 0;
}

fn mark_ready(shard: &mut ShardContext) {
    shard.state = ShardState::Ready;
    shard.flags.insert(ShardFlags::READY);
    shard.session_generation += 1;
}

C equivalent

void note_ack(struct shard_ctx *shard) {
    shard->heartbeat_acks++;
    shard->flags &= ~SHARD_FLAG_AWAITING_ACK;
    shard->heartbeat_misses = 0;
}

void mark_ready(struct shard_ctx *shard) {
    shard->state = SHARD_STATE_READY;
    shard->flags |= SHARD_FLAG_READY;
    shard->session_gen++;
}

Python model

def note_ack(shard):
    shard.heartbeat_acks += 1
    shard.flags.discard("AWAITING_ACK")
    shard.heartbeat_misses = 0

def mark_ready(shard):
    shard.state = "READY"
    shard.flags.add("READY")
    shard.session_generation += 1

READY is not enough for acceptance

The built-in sharding acceptance path also checks private workspace identity, authenticated TLS, and heartbeat ACKs. READY proves the Gateway dispatched an event; it does not by itself prove two shard contexts are truly isolated.

Success condition

A healthy shard reaches SHARD_STATE_READY, records the READY flag, increments session generation, and continues receiving heartbeat ACKs without accumulating misses.

Source

• legacy_shards.inc
• legacy_shard_gateway.inc
• legacy_shard_discord.inc
• SHARDING.md
• TESTING.md