Local LLM memory: Leave room for the conversation

You are setting up a local assistant to read a long document. Before choosing a model, you compare its download size with the memory available on your machine. That comparison leaves out the conversation itself: inference also needs attention caches and working memory, and increasing the context changes the requirement.¹²³

Treat “the model loads” as the first observation in a capacity test. Check whether the intended document and response fit while the applications you normally use remain open. The distinctions below explain what to measure; they are not results from a hardware benchmark.⁴⁵

Start with a budget that includes the request

Weights store the model’s learned parameters. Quantization reduces their storage precision to lower memory requirements, with methods that try to preserve accuracy. A parameter count therefore needs a precision and an actual model artifact beside it before it tells you much about storage.⁶

For planning, estimate memory as resident weights + KV cache + temporary computation buffers + runtime allocations. Account separately for other applications. This is an accounting guide, not a universal sizing equation. CPU and GPU placement changes which pool holds each part, and loading can have a different peak from generation.¹⁵³

For the weights alone, parameter count × bits per weight ÷ eight estimates bytes under the idealized assumption that every parameter uses the same packed precision. Real formats can retain modules at a different precision and store quantization constants. Use the downloaded artifact and runtime reports to refine that estimate; the formula excludes cache and working memory.⁶⁷

The cache belongs to the text being processed

An autoregressive transformer generates tokens using earlier context. Its key-value cache, usually called the KV cache, holds attention keys and values from processed tokens so the model can reuse them. These are request-dependent tensors, separate from learned weights.¹

That distinction explains why a small weight file can accompany substantial memory use during a long interaction. Dynamic caches grow as generation proceeds. Static caches preallocate a maximum size, so a large reservation can exist before the conversation fills it. Sliding-window and chunked attention limit growth on the layers that use those mechanisms; the same rule does not apply to every architecture.⁴

For a conventional full-attention cache, a useful estimate considers retained tokens, concurrent sequences, cached layers, KV heads, head dimensions, and cache precision. Hold the architecture and precision fixed when reasoning about changes in token count. Comparing models by parameter count alone leaves those cache dimensions unspecified.¹⁴

Plan space for the answer as well as the prompt. A document that fills the intended context leaves little room for a continuation in that same window. Use the model’s tokenizer and actual request format when preparing a representative input, including instructions and chat history.¹²

Context length and parallel requests share the budget

Ollama documents that larger context settings increase memory requirements. Its FAQ also explains that parallel requests expand the effective context allocation for a model. Several requests can share loaded weights while needing additional request state; multiplying the entire model size by the user count is a poor substitute for inspecting the server configuration.⁸²

Start a personal assistant with the concurrency you need. For a shared service, reproduce overlapping long requests before promising capacity. A waiting request and an actively processed request are different states: Ollama can queue work when it cannot load another model, and exposes separate controls for parallel processing and its queue.⁸

Check the allocated context and CPU/GPU placement with ollama ps. Set context explicitly when comparing configurations. The FAQ and context guide currently describe different defaults, which makes a remembered default an unreliable basis for a repeatable comparison.⁸²

Weight precision and cache precision are separate decisions

A quantized model does not establish the precision of its KV cache. llama.cpp exposes separate key and value cache type settings, while Ollama documents its own cache quantization control. Confirm support in the backend you run rather than inferring cache behavior from the weight filename.⁸⁹

Lower precision needs a quality decision tied to the job. Hugging Face describes quantization as reducing precision while preserving as much accuracy as possible; Ollama warns that cache quantization affects quality according to model and task. Neither statement supplies a universal acceptable quality loss.⁶⁸

Keep a few representative tasks with answers you can assess: extracting document details or following a required output format, for example. Compare the same model and prompt format while changing one precision setting. Judge missing facts and instruction failures alongside memory savings. These are suggested evaluation tasks, not benchmark results.

Cache quantization also has a runtime cost. Transformers warns that it can hurt latency for short contexts when sufficient GPU memory is already available. Enable it to address a measured constraint, then check whether the resulting behavior suits the task.⁴

Understand the memory number on the screen

In a PyTorch CUDA application, live tensors and memory reserved by the caching allocator are different measurements. PyTorch provides memory_allocated() for tensor allocations and memory_reserved() for allocator-managed memory, with corresponding peak counters. Reserved but unused memory can still appear occupied in nvidia-smi.⁵

Read these figures alongside overall device use. Do not add allocated and reserved figures as separate costs: the categories overlap. empty_cache() releases unused cached blocks, while tensors still in use remain allocated. It cannot make the current weights and live cache disappear.⁵

Those API names apply to PyTorch. A llama.cpp session needs its own runtime reports. Leave measured headroom for initialization and working buffers rather than applying an unexplained fixed overhead percentage to every backend.⁹³

CPU offloading changes where the work happens

llama.cpp supports CPU/GPU hybrid inference, and its GPU-layer setting controls how many layers may reside in VRAM. Transformers also offers KV cache offloading, moving most layer caches to CPU memory and fetching them as needed. These mechanisms move different objects; record which one you enabled.⁹¹⁰⁴

System RAM and dedicated VRAM need separate budgets. Offloading can make a configuration possible, but transfers and the selected execution path affect performance. Transformers explicitly describes a possible throughput penalty for cache offloading. This article supplies no slowdown multiplier for your machine.⁴³

Diagnose the point at which memory runs short

The table gives starting hypotheses, not diagnoses. Change one setting and inspect the relevant measurement before attributing a failure to a particular component.

Observation	Component to investigate	Useful next comparison
Failure during loading	Weights, placement, loading peak3	Smaller artifact or different placement
Failure with longer inputs	Context allocation and KV cache42	Same model with a shorter context
Failure during overlapping requests	Parallel request allocation8	Same workload processed sequentially
GPU use stays high after a request	Live tensors versus allocator reservation in PyTorch5	Compare allocated and reserved memory
It runs after offloading but feels too slow	Transfer and execution costs43	Same task with less offloading

Record the model artifact, runtime version, context setting, cache precision, placement, and active concurrency with each observation. Measure loading, prompt processing, and generation rather than saving only an idle screenshot. Loading and generation exercise different allocations.¹³

Choose the configuration that completes your own long request at acceptable quality and response time. Keep that request as a repeatable trial when changing the runtime or model. The parameter label can identify a candidate; the observed workload establishes whether it fits.

References

Footnotes

1. Hugging Face Transformers — How attention caching works and the dimensions of cached tensors: https://huggingface.co/docs/transformers/en/cache_explanation ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

2. Ollama context guide — Context allocation, memory requirements, and inspecting the running configuration: https://docs.ollama.com/context-length ↩ ↩² ↩³ ↩⁴ ↩⁵

3. Hugging Face Accelerate — Loading peaks, device maps, runtime memory, and offloading: https://huggingface.co/docs/accelerate/en/concept_guides/big_model_inference ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

4. Hugging Face Transformers — Dynamic, static, quantized, and offloaded cache behavior and limitations: https://huggingface.co/docs/transformers/en/kv_cache ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷ ↩⁸

5. PyTorch CUDA documentation — Tensor allocation, allocator reservation, peak counters, and empty_cache: https://docs.pytorch.org/docs/2.14/notes/cuda.html ↩ ↩² ↩³ ↩⁴ ↩⁵

6. Hugging Face Transformers — Quantization overview, precision choices, and accuracy tradeoffs: https://huggingface.co/docs/transformers/en/quantization/overview ↩ ↩² ↩³

7. Hugging Face bitsandbytes guide — Quantization constants, module precision, and CPU offloading representation: https://huggingface.co/docs/transformers/en/quantization/bitsandbytes ↩

8. Ollama FAQ — Parallel requests, memory placement, queueing, and KV cache quantization: https://docs.ollama.com/faq ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶

9. llama.cpp server documentation — Separate GPU-layer placement and K/V cache type controls: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md ↩ ↩² ↩³

10. llama.cpp — Official overview of CPU/GPU hybrid inference: https://github.com/ggml-org/llama.cpp/blob/master/README.md ↩