Generative AI on Your Own Servers: How Much GPU Memory a Private Language Model Needs

To run a generative AI (GenAI) model on your own servers, add up two things: the model's weights and the memory taken by the requests it is answering at the same moment. At the usual 16-bit precision, the weights need about 2 GB of GPU memory per billion parameters, so a model with 32.8 billion parameters needs about 66 GB before it answers anyone. Every request then adds memory in proportion to its length, and that part decides how many requests the server can handle at once.

What takes up GPU memory when a language model runs?

A language model usually runs on GPUs, processors built for the kind of arithmetic models do, and the model has to sit in the GPUs' own memory while it works. That memory holds:

• The weights, the numbers the model learned in training, loaded once and kept there
• The KV cache, the model's working memory for each request it is answering
• Working space for the serving software

The weights are fixed once you pick a model and a precision. The KV cache grows with the number of requests answered at the same moment and with their length. Both can be worked out before you buy anything.

How much memory do the model's weights take?

A model's size is counted in parameters, and precision is the number of bits used to store each one. Hugging Face's guide to running models efficiently, read on 27 October 2026, notes that “Almost all models are trained in bfloat16 nowadays”, a 16-bit format that takes 2 bytes per parameter. Its rule of thumb: “Loading the weights of a model having X billion parameters requires roughly 2 * X GB of VRAM in bfloat16/float16 precision”, where VRAM means the GPU's memory.

The guide's own example is Llama 2 70B, at about 140 GB for the weights alone. Qwen3-32B, an open-weight model released under the Apache 2.0 licence, has 32.8 billion parameters according to its model card, so its weights take about 66 GB. Open-weight means the maker publishes the model files for anyone to run.

Quantisation stores the weights in fewer bits. At 8 bits a parameter takes 1 byte, and at 4 bits half a byte, so the same model needs about 33 GB or a little over 16 GB. The Qwen team publishes an 8-bit version, Qwen3-32B-FP8.

The Hugging Face guide says weights “can be quantized to 8-bit or 4-bits without a significant loss in performance”. It also warns that 4-bit quantisation “can in practice often lead to different results” than 8-bit or 16-bit, and adds: “It is up to the user to try it out.” Run your own test questions through any compressed version before you rely on it.

How much memory does each request add?

While the model writes an answer, it keeps values for every token it has read and written so far, so that it does not have to work them out again for each new word. A token is a piece of a word. These stored values are the KV cache, short for key-value cache. A 2023 post by the makers of vLLM, an open-source program for serving models, notes that the cache “Takes up to 1.7GB for a single sequence in LLaMA-13B.”

You can work out the size for your own model from its published settings. For each token, the cache holds two sets of numbers, keys and values, for every layer. Qwen3-32B has 64 layers and keeps 8 attention heads of 128 numbers each in its cache, according to its model card and configuration file. At 2 bytes a number, that is 2 × 64 × 8 × 128 × 2 bytes, about 262 KB per token.

A request of 8,000 tokens, such as a long question with passages found in your documents plus the answer, therefore holds about 2.1 GB until the answer is finished. The model's full native context of 32,768 tokens holds about 8.6 GB.

Count the requests being answered at the same moment, not the people with access. Staff using an assistant spend most of their time reading answers and typing questions, while a system that works through a backlog of documents can keep every GPU busy all night.

How many GPUs does that mean?

Data-centre GPUs come with a fixed amount of memory. Nvidia's specification pages, read the same day, list 80 GB for the H100 in its SXM version and 94 GB in its NVL version, 141 GB for the H200 and 48 GB for the L40S.

Serving software keeps part of each GPU free. vLLM's documentation describes its setting as “The fraction of GPU memory to be used for the model executor” and gives 0.92 as the default in its current version. On one 94 GB H100 NVL, that is about 86 GB. The 16-bit Qwen3-32B weights take about 66 GB of that. Roughly 21 GB remains for the KV cache and the software's other working memory, which is room for fewer than ten requests of 8,000 tokens at the same moment.

With the 8-bit weights, about 54 GB remains, enough for about 25 such requests. On an 80 GB H100, the 16-bit weights leave about 8 GB, room for fewer than four.

When the weights do not fit on one GPU, the model is split across several GPUs in the same server. vLLM's documentation says that “if the model is too large for a single GPU but fits on a single node with multiple GPUs, use tensor parallelism”, which is its name for this split. A 70-billion-parameter model at 16-bit precision does not fit on one 94 GB GPU, and on two of them about 33 GB remains for requests.

These figures tell you whether a model fits. How fast it answers depends on the GPUs, the length of the answers and the load, and only a test on your own requests can show that.

Should we rent GPUs in the cloud or buy servers?

Renting first is a cheaper way to find out what you need. Microsoft's public price list for Azure, read the same day, gives these pay-as-you-go prices for Linux virtual machines in its East US region:

• One 94 GB H100 NVL (Standard_NC40ads_H100_v5): $6.98 an hour, about $5,100 for a month of 730 hours
• Two 94 GB H100 NVL (Standard_NC80adis_H100_v5): $13.96 an hour, about $10,200 a month
• Eight 80 GB H100 (Standard_ND96isr_H100_v5): $98.32 an hour, about $71,800 a month

These are list prices for paying by the hour, and reserved capacity is sold on other terms. Prices change, so get quotes from more than one provider for the setup your tests point to.

Servers of your own make sense when the load is steady and you know which model you will run, or when your rules require the hardware to stay in your own building. The cost then includes the people who keep those servers running, which the article on running an LLM system without an ML team covers.

How do we size the system for our own use?

• Test one or two candidate models on real questions from your work, and keep the one whose answers pass
• Estimate how many requests the system will answer at the same moment in its busiest hour, and how long a typical request is, counting the question, any passages found in your documents and the answer
• Work out the memory for the weights and the KV cache as above, and leave a margin
• Rent GPUs for a few days and run a load test with your own prompts at that peak
• Buy or reserve capacity once the test is done

vLLM includes a command for such tests, vllm bench serve. It can report percentiles of “ttft” and “tpot”, the time to the first token and the time per output token. Decide before the test how long your staff will wait for the first words of an answer, and judge the results from the busiest hour.

Who can set this up?

Your infrastructure team can run the GPU servers if it already looks after servers of its own, and the hardware or cloud vendor can help with the machines. Connecting the model to your documents, sign-in and permissions, and testing its answers, is the work of whoever builds the system around the model.

The firm or team you hire should show you the arithmetic: which model and precision, how much memory goes to the weights and how much to the KV cache, what load the test used and what it measured. Ask for those numbers before you approve a hardware budget.

Where does amBrain fit?

The one language model project amBrain describes in public is this: “We have taken an LLM integration to production inside a client's FinTech perimeter: extracting and normalising unstructured broker and venue notices — corporate actions, instrument and margin changes — into structured records the trading system consumes.”

This article is not a case study and describes no client work. The quoted line does not say which model that system uses, on what hardware or where it runs. The figures above are calculations from published specifications and price lists, not measurements from that project or from any other amBrain work.

The companies, models and products named here are cited from their published pages, and the article claims no relationship between amBrain and any of them. It gives no amBrain prices, timelines or terms.

amBrain has been building software since 2019. It works in three formats: full delivery, a dedicated team, or engineers embedded in your team. Its line on ownership is “The client keeps full ownership of the product and the code, except our reusable components.” If amBrain is on your shortlist, ask it for the same arithmetic as any other firm.