News & Updates
10 min
One Model Name, Three Engines: vLLM on Linux, vLLM on WSL and llama.cpp on Apple Silicon Behind One Endpoint
The same model ID is now served by vLLM on Linux GPUs, vLLM on Windows/WSL, and llama.cpp on Apple Silicon Macs, and the caller cannot tell which one answered. Here is what had to be made equal for that to be true, the tokens-per-second we measured on an M1 Ultra and an M4 Max next to the GPUs, the 27B experiment that failed first and the rules it left behind, and a self-update bug that only exists when your inference server is a child process.
inferencellama.cppvllm