amBrain

vLLM vs Ollama: one machine for one user, or shared GPUs for many

Published Sep 23, 2026Facts checked: Sep 23, 2026

Short answer

Use Ollama when one developer, or a handful of internal users, need a model on a laptop or a single workstation: it installs natively on macOS, Windows and Linux[1], and by default each loaded model handles one request at a time[2]. Use vLLM when a shared endpoint serves many users at once: in Red Hat's August 2025 test on one A100, vLLM reached "a peak of 793 TPS compared to Ollama's 41 TPS", where the 41 is Ollama in its default configuration[7]. Both are open source, Ollama under MIT and vLLM under Apache-2.0[1][4], so the decision is who runs the server and at what load.

On this page

Two servers built for different jobs

Ollama is an MIT-licensed tool that pulls and runs models by name, with install commands for macOS, Windows and Linux, an official ollama/ollama Docker Hub image, and the "llama.cpp project founded by Georgi Gerganov" credited as the inference backend[1]. It binds to 127.0.0.1 on port 11434 by default[2]. Out of the box each loaded model processes one request at a time (OLLAMA_NUM_PARALLEL, default 1), up to 512 requests wait in a queue (OLLAMA_MAX_QUEUE), and beyond that the server answers with a 503 "server is overloaded"[2]. The default context window depends on GPU memory: 4k tokens below 24 GiB, 32k from 24 to 48 GiB, and 256k from 48 GiB[3].

vLLM is an Apache-2.0 inference and serving engine, "Originally developed in the Sky Computing Lab at UC Berkeley", with contributions "from over 2000 contributors"[4]. It has been a PyTorch Foundation-hosted project since May 2025[5]. Its design is about sharing GPUs between many requests: PagedAttention for key-value cache memory, continuous batching, chunked prefill and prefix caching, and "Tensor, pipeline, data, expert, and context parallelism for distributed inference"[4]. Its GPU installation page lists "OS: Linux" and "Python: 3.10 -- 3.13", and states that "vLLM does not support Windows natively"[6].

The one published benchmark, and who published it

Every throughput and latency number below comes from a single article: Red Hat Developer, 8 August 2025, by Harshith Umesh. Red Hat has an interest in the result — the same page advertises "Discover how Red Hat AI Inference Server, powered by vLLM, enables fast, cost-effective AI inference", a product built on the winner[7].

The setup was a "Single NVIDIA A100-PCIE-40GB GPU" with vLLM 0.9.1 serving meta-llama/Llama-3.1-8B-instruct and Ollama 0.9.2 serving llama3.1:8b-instruct-fp16, at "concurrency levels tested from 1 up to 256", driven by GuideLLM version 0.2.1. vLLM reached "a peak of 793 TPS compared to Ollama's 41 TPS", with lower P99 latency "(80 ms vs. 673 ms at peak throughput)". That 41 TPS is default Ollama, which the article describes as "configured to handle a maximum of four requests in parallel"; the tuned run with OLLAMA_NUM_PARALLEL=32, "the highest stable value for our NVIDIA A100 GPU", is reported separately and only up to a concurrency of 64[7].

Two reasons to re-run it rather than quote it. Both servers have moved on since those versions, and the default the article assumes, four parallel requests, is not what Ollama's current FAQ documents, which is one[7][2]. Repeat the test on your own versions, models and prompt lengths before you rely on the ratio.

Comparison table

The two self-hosted columns are the subject of this page; the hosted column is there to keep the third option in view. Whether to host anything at all is the earlier question, answered in a hosted API against a self-hosted LLM; the managed side has its own page, alternatives to Azure OpenAI.

Ollama, vLLM and a hosted API, by what each publishes
Ollama (self-hosted)vLLM (self-hosted)Hosted API (for example Ollama Cloud)
Code and data ownershipMIT licence[1]; models and prompts stay on your machineApache-2.0[4]; PyTorch Foundation-hosted[5]; prompts stay on your serversThe provider processes prompts; Ollama states "We do not use them to train models."[8]
Cost modelFree software; you pay for the machineFree software; you pay for GPUs — for example $6.98 an hour for one Azure Standard_NC40ads_H100_v5, "1 Hour", East US[9], a size "powered by NVIDIA H100 NVL GPU" with one 94 GB accelerator[10]Ollama Cloud: Free $0; Pro "$20 / mo. or $200/yr" with "$60 of usage credits per month"; Max "$100 / mo." with "$300 of usage credits per month"; Team "$500 / mo." with "$1,000 of usage credits per month, shared across the team"; Enterprise is Custom. "Usage beyond the included credits draws from a shared team balance billed as you go." gpt-oss:120b is listed at $0.15 per million input tokens, $0.014 cached and $0.60 output[11]
Time to launchInstall command or the Docker image, then ollama run <model>[1]A Linux GPU host with Python 3.10–3.13[6], then vLLM's OpenAI-compatible server[16]An API key[8]
Lock-in: API and formatsOwn REST API plus "a subset of the OpenAI API"[13]; imports Safetensors weights and GGUF files[14]"OpenAI-compatible API server, plus Anthropic Messages API and gRPC support" and "200+ model architectures on Hugging Face"[4]Cloud models have retirement dates: "Switch models before the retirement date. Downloaded local models are not affected."[8]
CustomisationContext length per model[3], server settings such as OLLAMA_NUM_PARALLEL and OLLAMA_MAX_LOADED_MODELS[2], and imports from Safetensors or GGUF[14]"Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and more", multi-LoRA, structured outputs and speculative decoding[4]The provider's model list and settings
Compliance and access controlLocal by default: 127.0.0.1 on port 11434[2]; the OpenAI-compatible endpoint "requires an API key value, but Ollama ignores it", so remote access needs a proxy[13]--api-key "only authenticates requests to endpoints under the /v1, /v2, and /inference path prefixes", /invocations stays unauthenticated, and the docs recommend a reverse proxy[16]The provider's terms and regions
Latency and scaleOne request per model by default; "Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH"[2]Throughput rises with concurrency: 793 TPS against default Ollama's 41 in Red Hat's test, though "above 16" concurrent users vLLM's inter-token latency began to rise while Ollama's "remained stable and low"[7]Shared capacity: 1 concurrent request on Free, 3 on Pro, "10 concurrent requests" on Max and Team[11]
Hardware"supports Nvidia GPUs with compute capability 5.0+", AMD via ROCm, Metal on Apple devices and Vulkan[15]NVIDIA, AMD and Intel GPUs, x86, ARM and PowerPC CPUs, plus plugins for TPU, Gaudi, Apple Silicon and others[4]Not your concern, and not your choice
Exit costLow: open source, OpenAI-style endpoints[13]Low: open source, OpenAI-compatible endpoints[16]Re-test on another model when a cloud model retires[8]

Stay on Ollama if…

  1. 1.Developers need models on their own machines. Install commands for macOS, Windows and Linux and an official Docker image[1], with Apple GPUs reached through Metal[15], make it the shorter path from download to answer. Red Hat's own conclusion places Ollama in "local development, prototyping, and single-user applications"[7].
  2. 2.Only a few people use it at the same time. With one request per model by default and OLLAMA_MAX_QUEUE at 512[2], a personal assistant or a small internal tool works without tuning.
  3. 3.You switch models often. Ollama loads models on demand, keeps them in memory for five minutes and defaults OLLAMA_MAX_LOADED_MODELS to "3 * the number of GPUs"[2].

Consider moving from Ollama to vLLM if…

  1. 1.Users wait in line. Symptom: the first token arrives later with every extra user, and busy hours end in 503 "server is overloaded" answers[2]. Red Hat saw Ollama's time to first token rise sharply with load because "incoming requests had to wait in a queue"[7].
  2. 2.Raising parallelism runs out of memory. Symptom: every increase of OLLAMA_NUM_PARALLEL needs more GPU memory, since "Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH"[2]. In Red Hat's test, OLLAMA_NUM_PARALLEL=32 was "the highest stable value for our NVIDIA A100 GPU"[7].
  3. 3.The model outgrows one GPU. Symptom: a larger model or a longer context no longer fits in the memory of one card. vLLM spreads inference with "Tensor, pipeline, data, expert, and context parallelism for distributed inference"[4].

Migration path in four steps

  1. 1.Pin the model and the weights. Note the exact model, quantisation and context length you run in Ollama, then pick the matching artefact on the vLLM side, which serves "200+ model architectures on Hugging Face"[4]. Red Hat's benchmark, for instance, paired meta-llama/Llama-3.1-8B-instruct on vLLM with llama3.1:8b-instruct-fp16 on Ollama[7] — the same model, two artefacts.
  2. 2.Stand up vLLM beside Ollama. Both speak OpenAI-style APIs: clients pointed at Ollama's http://localhost:11434/v1/[13] move to vLLM's /v1 endpoints by changing the base URL[16].
  3. 3.Load-test with your own prompts. Use realistic concurrency and prompt lengths. GuideLLM, which Red Hat ran at version 0.2.1[7], is Apache-2.0 and maintained inside the vLLM project[17].
  4. 4.Put a proxy in front and keep Ollama for laptops. Add authentication, rate limits and logging at a reverse proxy, as the vLLM docs advise[16], and leave Ollama where it fits: on developers' machines.

Cost drivers of hosting your own

What a self-hosted LLM server costs, and the public reference point for each line
Cost driverWhat drives itPublic reference point
GPU memoryWeights plus key-value cache for every active requestOllama: "Required RAM will scale by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH"[2]; vLLM manages the same cache with PagedAttention[4]
GPU hours or hardwareModel size and peak concurrencyAzure Standard_NC40ads_H100_v5: $6.98 an hour, East US, pay as you go[9]; the NCads H100 v5 series is "powered by NVIDIA H100 NVL GPU" with one 94 GB accelerator[10]
Model fitParameter count and quantisationOpenAI states gpt-oss-120b "fit into a single 80GB GPU (like NVIDIA H100 or AMD MI300X)" and gpt-oss-20b runs "within 16GB of memory"[12]
Access controlProxy, authentication, network isolationNeither server's built-in key check is enough on its own: Ollama ignores the key value[13], vLLM's covers only three path prefixes[16]
Load testing and evaluationYour prompts at your concurrency, rerun on every upgradeGuideLLM is free and Apache-2.0[17]; the labelled test set is your work
OperationsUpgrades, monitoring, on-callNo public reference; scoped per project

Frequently asked questions

On throughput and P99 latency in the published test, yes: "a peak of 793 TPS compared to Ollama's 41 TPS" and "(80 ms vs. 673 ms at peak throughput)", on one A100 with an 8B model[7]. Not on every metric, though. The same article reports that "At very high concurrency (above 16), vLLM's ITL began to rise, while Ollama's remained stable and low" — Ollama keeps inter-token latency flat because it "throttles requests, keeping its active workload small and predictable at the expense of making many users wait (high TTFT)"[7]. The article's one "at any concurrency level" statement is about capacity in the tuned run, reported up to a load test concurrency of 64: "Ollama, despite the tuning, saw its performance plateau and was unable to match vLLM's capacity at any concurrency level"[7].

For low concurrency, yes, with care. Ollama binds to 127.0.0.1 on port 11434 by default[2] and its OpenAI-compatible endpoint "requires an API key value, but Ollama ignores it"[13], so put a proxy with authentication in front of it, and size OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE for your peak[2].

vLLM ships an "OpenAI-compatible API server, plus Anthropic Messages API and gRPC support"[4], documented endpoint by endpoint[16]. Ollama "supports a subset of the OpenAI API" at the base URL http://localhost:11434/v1/, with /v1/chat/completions, /v1/completions, /v1/models and /v1/embeddings[13].

"vLLM does not support Windows natively" — the docs list "OS: Linux" and point to "the Windows Subsystem for Linux (WSL)"[6]. Apple Silicon appears among vLLM's hardware plugins[4]. Ollama installs natively on macOS, Windows and Linux[1].

Pick by who will operate it after launch, since both servers need upgrades, monitoring and a proxy you maintain; AI on your own servers lists the decisions to make first.

About amBrain

  • We have taken an LLM integration to production inside a client's FinTech perimeter: extracting and normalising unstructured broker and venue notices — corporate actions, instrument and margin changes — into structured records the trading system consumes.
  • The client keeps full ownership of the product and the code, except our reusable components.

Disclosure: this page is published by amBrain. Its quantitative comparison rests on one benchmark whose publisher has an interest in the outcome — the Red Hat Developer article advertises "Discover how Red Hat AI Inference Server, powered by vLLM, enables fast, cost-effective AI inference", a product built on the server that wins the test[7]. Every third-party fact on this page links to its source, and prices were read on 2026-09-23. vLLM, Ollama and the other product and company names are trademarks of their respective owners and are used only to identify their products. No vendor named here reviewed or endorsed this page.

Sources

  1. [1]GitHub, ollama/ollama repository page (the About sidebar and the LICENSE file give the MIT licence — the README body carries no licence statement; README install commands for macOS, Windows and Linux, the ollama/ollama Docker Hub image and the credit to the "llama.cpp project founded by Georgi Gerganov"). Official documentation · checked Sep 23, 2026
  2. [2]Ollama, FAQ (bind address and port, OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE, keep-alive, OLLAMA_MAX_LOADED_MODELS). Official documentation · checked Sep 23, 2026
  3. [3]Ollama, Context length (default context window by GPU memory). Official documentation · checked Sep 23, 2026
  4. [4]GitHub, vllm-project/vllm repository page (the About sidebar and the LICENSE file give the Apache-2.0 licence — the README body carries no licence statement; README origin, contributor count, parallelism, quantisation, API and hardware lists). Official documentation · checked Sep 23, 2026
  5. [5]PyTorch Foundation, PyTorch Foundation Welcomes vLLM as a Hosted Project, May 7, 2025. Press · checked Sep 23, 2026
  6. [6]vLLM, GPU installation requirements ("OS: Linux", "Python: 3.10 -- 3.13" and the Windows Subsystem for Linux note). Official documentation · checked Sep 23, 2026
  7. [7]Red Hat Developer, Ollama vs. vLLM: A deep dive into performance benchmarking (by Harshith Umesh; the same page advertises Red Hat AI Inference Server, which is built on vLLM — the benchmark's publisher sells a product based on the winner), Aug 8, 2025. Third-party source · checked Sep 23, 2026
  8. [8]Ollama, Cloud (data handling and model retirement). Official documentation · checked Sep 23, 2026
  9. [9]Microsoft, Azure Retail Prices API, Virtual Machines in East US matching H100 (the response carries armSkuName, retailPrice 6.98, unitOfMeasure "1 Hour", armRegionName "eastus" and productName "Virtual Machines NCadsH100v5 Series" — and no GPU specification). Vendor's own page · checked Sep 23, 2026
  10. [10]Microsoft, NCads H100 v5 sizes series ("powered by NVIDIA H100 NVL GPU"; Accelerators (Qty.) 1 and Accelerator-Memory (GB) 94). Official documentation · checked Sep 23, 2026
  11. [11]Ollama, Pricing (plan prices, included usage credits, concurrency per plan and the per-model token rates). Vendor's own page · checked Sep 23, 2026
  12. [12]OpenAI, gpt-oss-120b model card. Official documentation · checked Sep 23, 2026
  13. [13]Ollama, OpenAI compatibility (supported endpoints, base URL and the ignored API key). Official documentation · checked Sep 23, 2026
  14. [14]Ollama, Importing a model (import from Safetensors weights and from a GGUF file). Official documentation · checked Sep 23, 2026
  15. [15]Ollama, Hardware support (NVIDIA compute capability list, AMD via ROCm, Metal and Vulkan). Official documentation · checked Sep 23, 2026
  16. [16]vLLM, OpenAI-Compatible Server (supported APIs and the limits of --api-key). Official documentation · checked Sep 23, 2026
  17. [17]GitHub, vllm-project/guidellm repository page (Apache-2.0, maintained in the vLLM project). Official documentation · checked Sep 23, 2026

Related

Free project intro call, 30 minutes.

We reply within 24 hours.

Book a Call