GLM-5.3 Flash on 4× H100: a 320B Model at 73 Tokens per Second

GLM-5.3 Flash is one of the strongest open-weight models you can download today: 321 billion parameters, 18 billion of them active per token, MIT licensed. The official vLLM recipe serves it on eight H200s.

We wanted to know what it takes on less. We rented a machine with four NVIDIA H100 SXM5 on Paralon, 80 GB each, and got the model answering at 73 tokens per second for one user and 181 tokens per second in total for eight. This post is the route that worked, the two that did not, and every command, so you can do the same.

The setup

Which version fits in 320 GB

Four H100s give you 320 GB of VRAM. The model's weights have to fit with room left for the KV cache, and that rules out most of what Z.ai publishes:

The FP8 checkpoint is what vLLM and SGLang would serve, and it is the reason the official recipe starts at eight H200s. On Hopper with four cards the practical route is a 4-bit GGUF, and a GGUF means llama.cpp.

Ollama: "sharded GGUF" is a dead end

Our rental came with Ollama and Open WebUI already running, so the first try was the obvious one:

ollama pull hf.co/unsloth/GLM-5.3-Flash-GGUF:UD-Q4_K_XL

Error: pull model manifest: 400: {"error":"This repository only contains
sharded GGUF files. Ollama does not yet support pulling sharded GGUF via the
registry; please download the shards and merge them locally with
`ollama create` ..."}

Hugging Face caps a single file at 50 GB, so every large GGUF comes in shards and this applies to all of them, not just this one. The suggested workaround is to download 200 GB and then write another 200 GB merged copy. llama.cpp reads the shards as they are, so we went straight to it. Ollama's search box in Open WebUI will not find the model either: it only searches Ollama's own library.

Step 1: build llama.cpp with CUDA

The rental image ships the CUDA runtime but not the compiler (that keeps it small and quick to start). Installing the toolkit takes a few minutes:

apt-get update && apt-get install -y cmake build-essential git wget
wget -q https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb && apt-get update && apt-get install -y cuda-toolkit-12-8
export PATH=/usr/local/cuda-12.8/bin:$PATH

cd /workspace && git clone --depth 1 https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=90 \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-12.8/bin/nvcc
cmake --build build -j --target llama-server llama-bench

CMAKE_CUDA_ARCHITECTURES=90 builds for Hopper only, which cuts the compile time to a fraction of a build for every architecture.

Step 2: download the shards

pip install -U huggingface_hub
hf download unsloth/GLM-5.3-Flash-GGUF --include "UD-Q4_K_XL/*" --local-dir /workspace/glm53

That is six files: a 9 MB header shard, four of 46 to 47 GB and a 2.6 GB tail. Run it in tmux so a closed browser tab does not stop it. The machine had 11 TB of free disk, so space was never a question; check yours with df -h /workspace before you start.

Step 3: measure

F=$(ls /workspace/glm53/UD-Q4_K_XL/*00001-of-*.gguf)
./build/bin/llama-bench -m $F -ngl 99 -fa 1 -p 512,4096 -n 128 -r 3

You point llama.cpp at the first shard and it finds the rest. Loading 186 GiB into four cards takes a few minutes on the first run.

73 tokens per second from a 320B model is faster than anyone reads. A 4,096-token prompt, a long file or a few pages of documentation, is read in under two seconds. The variance across the three runs is almost nil.

The reason it is this fast is the architecture: only 18B of the 321B parameters work on each token, and an H100 reads its memory at over 3 TB/s. A dense model of the same size would be several times slower on the same cards.

Step 4: serve it, and measure eight users

./build/bin/llama-server -m $F -ngl 99 -fa on -c 65536 --parallel 8 \
  --host 127.0.0.1 --port 8081 --jinja

--parallel 8 opens eight slots that share the 65,536-token context, so each request gets 8,192 tokens. Give each slot more by raising -c; there is VRAM for it. --jinja uses the chat template inside the GGUF, which GLM needs for tool calls and its reasoning format.

We sent the same 400-token request 1, 4 and 8 times at once through the OpenAI-compatible endpoint:

With eight users each one still gets 23 tokens per second, which reads as a normal chat. These numbers are end to end through the HTTP server, prompt included, so the single request lands a little under llama-bench's 73.

VRAM with the server up and eight slots allocated:

About 188 GiB of 320 GB. There is room for a much longer context, or for the UD-Q5_K_XL quant if you want to stay closer to the original weights.

Row split does not work for this model

llama.cpp spreads a model across GPUs by layer by default: each card holds a slice of the layers and they take turns. The alternative, -sm row, splits every layer across all cards so they work at the same time. For GLM-5.3 Flash it fails at load:

llama_model_load: error loading model: device CUDA0 does not support split buffers

So layer split it is. That is also why throughput grows less than linearly with users: llama.cpp is built for one machine and a few users, not for hundreds of concurrent requests. For the single-user and small-team case it is measured above, and it is quick.

Step 5: use it from Open WebUI

If you rented with the Open WebUI template, the chat UI is already open in your browser. Point it at the server:

1. Admin Panel → Settings → Connections, switch OpenAI API on.
2. Add a connection: URL http://127.0.0.1:8081/v1, any API key (llama-server does not check it unless you start it with --api-key).
3. Save and reload. The model shows up in the picker under its file name.

Open WebUI itself listens on port 8080 in this template, which is why the server above runs on 8081. Starting llama-server on 8080 fails with couldn't bind HTTP server socket, and anything you send to 8080 is answered by Open WebUI instead (we saw a 405 Method Not Allowed before we noticed).

From your own code, the same endpoint speaks the OpenAI API: over SSH with a port forward, or from a notebook in the same machine.

What to take away

• GLM-5.3 Flash runs on 4× H100. Not in FP8, as the official recipe does, but in Unsloth's 4-bit GGUF, 186 GiB, with room to spare.
• 73 tok/s for one user, 23 tok/s each for eight. Enough for a private assistant, a coding agent or a small team.
• Use llama.cpp, not Ollama, for any GGUF in shards. Point it at the first file.
• Build for sm_90 and leave -sm row alone for this architecture.
• Nothing here needs a contract. The machine is billed by the minute, the weights download at data-center speed, and when you stop the rental the billing stops with it.

The same four cards are on the rent page, with JupyterLab, SSH, or the Open WebUI and Ollama template ready when the machine starts.