Serving GLM-5.3 at 607.8 tok/s across two Nvidia DGX stations with vLLM
First reported by Developer.nvidia ·
Serving large language models locally is now possible on multi-node setups, bypassing cloud service costs for massive AI deployments.
NVIDIA has detailed a method for serving the GLM-5.3 NVFP4 model, a 753 billion parameter mixture-of-experts model, across two NVIDIA DGX Stations using vLLM. The model requires approximately 464 GB of GPU memory, exceeding the capacity of a single GB300 GPU. By connecting two DGX Stations with ConnectX-8 RDMA and employing tensor parallelism (TP=2) via vLLM, the model weights and computation are split across both stations. This setup allows for a single OpenAI-compatible endpoint. Benchmarking on the SPEED-Bench Coding dataset showed a peak throughput of 607.8 tokens per second at a concurrency of 16, and 385.0 tokens per second at concurrency 32, with a mean request latency of 453.5 seconds. The process involves hardware setup, model downloading, and running vLLM in a distributed configuration.
This setup demonstrates a practical solution for deploying extremely large AI models locally, leveraging distributed computing across multiple high-performance stations. The use of vLLM with tensor parallelism is key to managing models that exceed single-GPU memory, offering an alternative to cloud-based inference which can incur significant ongoing costs.
The achievement of 607.8 tokens/sec throughput on a 753B parameter model indicates a significant step forward for on-premises AI infrastructure. This approach directly impacts organizations that require high levels of data privacy or cost control for their AI workloads, enabling them to harness frontier-scale models without relying on external providers.
AI-written summary. May contain errors.