GLM-5.3-Flash on 2× DGX Spark

We took GLM-5.3-Flash (320B total / 18B active, NVFP4) from nothing to a serving endpoint across two NVIDIA DGX Sparks linked by a 200 Gb/s RoCE fabric. It is now the local model behind Marble as glm-5.3-flash.

Why GLM-5.3-Flash and not GLM-5.3

A fit check decided this. FP8 weights run roughly 1 byte per parameter, and the two Sparks offer ~242 GiB of pooled unified memory:

Flash is also the more interesting artifact: MIT-licensed, natively multimodal (image + video input), 1M-token context declared, and it was Z.ai's first model to combine sparse and linear attention.

Architecture

Serve configuration — the flags that define the recipe:

The speculative drafter is the headline optimisation. Decode ladder on this hardware:

Figures for the code prompt from the recipe authors' harness, for scale. Our own real-prompt measurements are lower and quoted in full below.

Measured performance

Single stream, temperature 0, measured after boot on 28 Sept 2026. Throughput on this model tracks how predictable the output is, because predictable text drafts better — so a single number without the prompt mix is meaningless.

Tool calling verified — a weather prompt correctly emitted get_weather({"location": "Boston"}). KV cache came up at 788,977 tokens, which is 3.01× concurrency against the 262K limit.

Two bugs worth writing down

# Broken: tab is IFS whitespace -> empty first field shifts everything
while IFS=$'\t' read -r sha size fn; do
  [ -z "${fn:-}" ] && continue      # skips silently
done < manifest.tsv

# Rule: never split a TSV with IFS=$'\t' read when a field can be empty.
# Split in python, or write a sentinel for empty fields.

Fix, now applied per rank:

case "$NODE_RANK" in
  0) HOST_IP=169.254.111.113; GID_INDEX=3 ;;   # head
  1) HOST_IP=169.254.154.250; GID_INDEX=4 ;;   # worker
esac
...
  -e NCCL_IB_HCA=rocep1s0f1 -e NCCL_IB_GID_INDEX=$GID_INDEX

# /home/rusty/nccl-test.sh <rank>  -- run on both nodes, worker first
docker run --rm --gpus all --network host --ipc host --shm-size 8g \
  --device /dev/infiniband:/dev/infiniband --ulimit memlock=-1:-1 --cap-add IPC_LOCK \
  -e NCCL_NET=IB -e NCCL_IB_HCA=rocep1s0f1 -e NCCL_IB_GID_INDEX=$GID \
  -e NCCL_IB_ADDR_FAMILY=AF_INET -e NCCL_IB_ADDR_RANGE=169.254.0.0/16 \
  -e NCCL_SOCKET_IFNAME=enp1s0f1np1 -e RANK=$RANK -e WORLD=2 \
  --entrypoint python3 "$IMG" -c '...dist.all_reduce()...'

Timeline

Discovery: the network was the real constraint

The 200 Gb/s fabric is not the bottleneck — the house uplink is. The head's wired port negotiates at 100 Mbps, and both wired and WiFi independently cap near 11 MB/s. Because the two interfaces are separately capped, splitting the shard manifest across both doubled throughput:

Parallelising across both nodes would not have helped — they share one internet uplink.

Operating it

# Launch (on the head). Worker rank 1 goes FIRST, then head rank 0.
ssh spark-be0f
bash /home/rusty/relaunch.sh

# The script does, in order:
#   1. docker rm -f vllm_glm53        (both nodes)
#   2. sudo -n /usr/local/bin/spark-prep   (both nodes)  <- drop_caches + swappiness=0
#   3. launch worker rank 1, sleep 25, launch head rank 0
#   4. poll  curl -sf http://127.0.0.1:8000/health

Boot takes ~25 minutes, not the ~15 the upstream docs imply:

Harmless noise during boot: No available shared memory broadcast block found in 60 seconds means it is compiling, not hanging. The drafter also warns it ignores multimodal embeddings — expected, since DFlash2 drafts on text only.

Explicitly deferred

Two optional accelerations exist that we chose not to install for the first boot, to keep the day-0 variable count down:

• RoCE all-reduce bundle — worth +5–18% aggregate throughput. The launcher logs RoCE all-reduce OFF (NCCL) and falls back cleanly. Optimisation, not correctness.
• 1M-token context — declared by the model, but the recipe authors report 1M starved the KV pool and crashed their fleet three times at TP2. 262K is the shipped setting; the TP4 sibling recipe is the lane where 1M lives.

Open items

• Stale model entries. mistral-small-4 and qwen3.6-35b now point at servers we deliberately stopped. GLM replaces both and they cannot coexist — it holds both nodes' unified memory.
• Default model. The process default is still qwen3.6-35b, a dead endpoint, so a fresh session with no model set would fail. Needs repointing at the CLI/default layer, not the catalog.
• Full re-hash in progress. A background sha256 pass over all 191 GB was still running at time of writing.

Caveats

This is day-0 silicon with a community recipe. The stack needed two fixes before it would boot, and the upstream recipe itself required adaptation for this fabric. It is stable and in production use for our purposes, but treat it as a system that needs its operator to know these specific failure modes — the NCCL smoke test and the /health rule are not optional ceremony, they are what makes a 25-minute boot debuggable.

Thanks to the community work this builds on: the tonyd2wild TP2 + DFlash2 recipe, NVIDIA's NVFP4 checkpoint, LibertAI and RedHatAI quantization work, and the DGX Spark forum threads that mapped the SM121 crash modes.