Correction — September 25, 2026: The previous parameter count omitted the FFN gate projection and used LayerNorm instead of RMSNorm. The count and memory examples below have been corrected; concurrency figures are illustrative memory budgets, not measured throughput.

The headline specification of a Large Language Model (LLM)—be it 7B, 70B, or 175B—is its parameter count. This single number is the most significant indicator of its potential capability, its operational cost, and the hardware required to run it. This article provides a technically rigorous breakdown of the Transformer architecture, showing precisely how these billions of parameters are calculated and how they translate into real-world performance and resource consumption.

The Anatomy of a 7B Model: A Parameter Audit

A "parameter" in an LLM is a trainable variable, typically a weight or a bias in a matrix, that the model learns during training. The "7B" signifies a total of seven billion such parameters. To understand where they come from, we will perform an audit using a standard 7B model architecture, like Llama 2 7B, as our blueprint.

Core Hyperparameters:

Vocabulary Size (vocab_size): 32,000 tokens
Model Dimension (d_model): 4096 (This is the width of the vectors that flow through the model)
Number of Hidden Layers (num_hidden_layers): 32 (This is the depth of the model)
Feed-Forward Intermediate Size (ffn_intermediate_size): 11008
Number of Attention Heads (num_attention_heads): 32

1. The Token Embedding Layer

This layer converts discrete token IDs into dense vectors. It is a single weight matrix.

Calculation: vocab_size * d_model
Example: 32,000 * 4096 = 131,072,000 parameters

2. The Transformer Blocks (The Inference Core Engine)

The bulk of the parameters reside in the stack of 32 identical Transformer Blocks. The total is num_hidden_layers multiplied by the parameters per block. Let's dissect one block.

A. Multi-Head Self-Attention (MHSA) The attention mechanism consists of four primary weight matrices:

Three matrices for generating the Query, Key, and Value vectors (W_q, W_k, W_v).
One output projection matrix (W_o) to combine the results from all attention heads.

Each of these matrices has a shape of d_model x d_model.

Calculation: 4 * (d_model * d_model)
Example: 4 * (4096 * 4096) = 4 * 16,777,216 = 67,108,864 parameters

B. The Feed-Forward Network (FFN) Llama 2 uses a gated SwiGLU FFN with three weight matrices: an up projection, a gate projection, and a down projection. All three must be counted.

Calculation: 3 * d_model * ffn_intermediate_size
Example: 3 * 4096 * 11008 = 135,266,304 parameters

C. RMS Normalization Each block has two RMSNorm scale vectors, each with 4096 elements and no learned bias.

Calculation: 2 * 4096 = 8,192 parameters per block

Total Parameters Per Block: 67,108,864 + 135,266,304 + 8,192 = 202,383,360

Total Across All Blocks: 32 * 202,383,360 = 6,476,267,520

3. The Final Prediction Head (LM Head)

For this architecture, the output projection is separate from the token embedding matrix.

Calculation: 4096 * 32,000 = 131,072,000

Grand Total Calculation

Component	Parameters
Token embeddings	131,072,000
32 Transformer blocks	6,476,267,520
LM head	131,072,000
Final RMSNorm	4,096
Total	6,738,415,616

That is approximately 6.74 billion, conventionally named a 7B model. See Meta's reference implementation for the three FFN projections, RMSNorm, and separate output head.

To reproduce the count:

python

vocab, width, intermediate, layers = 32000, 4096, 11008, 32
attention = 4 * width * width
ffn = 3 * width * intermediate
norms = 2 * width
total = 2 * vocab * width + layers * (attention + ffn + norms) + width
assert total == 6_738_415_616
print(total)

Architectural Choices and Their Impact on Capability

These hyperparameters are deliberately chosen based on Scaling Laws—empirical research that links a model's size, dataset size, and computational budget to its final performance.

d_model (Width): A larger d_model allows each token's vector to hold more rich and nuanced information. It directly increases the model's representational density.
num_hidden_layers (Depth): Depth enables the model to build a hierarchical understanding of language. Early layers may handle syntax, mid-layers may process semantics, and final layers can infer abstract concepts like intent. Deeper models excel at complex reasoning.
ffn_intermediate_size: The FFN's expansion factor is critical for a model's ability to store and access factual knowledge learned during training. Increasing this is a highly parameter-efficient way to boost a model's "intelligence."

From Architecture to Execution: Hardware and Resource Planning

Understanding the architecture allows us to precisely calculate the resources required for deployment.

1. Static Memory: Loading the Model Weights

This is the fixed memory (VRAM for a GPU, RAM for a CPU) or disk space needed just to load the model. It depends on the numerical precision used for the parameters.

Precision	Bytes per Parameter	Calculation for 7B Model	Required VRAM/Disk
FP32 (Full)	4 bytes	`7e9 * 4`	~28 GB
FP16 / BFloat16	2 bytes	`7e9 * 2`	~14 GB
INT8 (Quantized)	1 byte	`7e9 * 1`	~7 GB
INT4 (Quantized)	~0.5 bytes	`7e9 * 0.5`	~3.5 GB

This calculation dictates the minimum VRAM a GPU must have. A 7B model at FP16 precision will not fit on a GPU with 12GB of VRAM.

2. Dynamic Memory: The KV Cache and Concurrency

During inference, the model generates a Key-Value (KV) cache for every token in the input sequence. This cache prevents re-computation of attention for previous tokens and is the primary consumer of dynamic memory.

KV Cache Size Formula: 2 * num_layers * d_model * context_length * bytes_per_element
This memory cost is per user request.

Illustrative memory budget: Assume a 24 GiB GPU budget, 4 GiB for quantized weights and metadata, and another 4 GiB reserved for runtime overhead and activations. This leaves 16 GiB for KV cache.

For Llama 2 7B's 32 KV heads of dimension 128, at 4096 cached tokens:

FP16 cache: 2 * 32 * 32 * 128 * 4096 * 2 = 2,147,483,648 bytes = 2 GiB per request.
INT8 cache: the same calculation with one byte per element gives 1 GiB per request, before quantization metadata.
Memory-only upper bounds: 8 requests with FP16 cache, or 16 with an INT8 cache under these assumptions, before additional cache metadata.

These are not promises about concurrent users or throughput. Runtime support for cache quantization, temporary allocations, input/output length, batching, and latency targets determine practical capacity. Generation also grows the cache. Benchmark your chosen inference engine with representative requests.

For architectures with grouped-query attention, use num_kv_heads * head_dim instead of d_model in the cache formula. Weight precision and cache precision are separate choices.

3. Computational Performance: CPU vs. GPU

The core computation in a Transformer is the General Matrix Multiply (GEMM) operation.

GPU (Throughput Engine): A GPU is a massively parallel processor with thousands of cores (e.g., Tensor Cores on NVIDIA hardware) designed specifically to accelerate the multiply-and-add operations that constitute a matrix multiplication. It can perform trillions of these operations per second.
CPU (Latency Engine): A CPU has a small number of powerful cores designed for complex, sequential tasks. While it can perform matrix multiplication, it cannot match the parallelism of a GPU, so latency and throughput depend strongly on hardware, memory bandwidth, model size, and the inference engine.

Running a 7B model on a CPU is feasible using frameworks like llama.cpp, but it is suitable for development or non-real-time tasks. A GPU often improves throughput; benchmark both options against your latency and cost targets. The model's width (d_model) is the primary driver of computational density, while its depth (num_layers) is the main contributor to sequential latency.

Conclusion: From Theory to Practical Engineering

This deep dive demonstrates that the "7B" headline figure is far more than a marketing number; it is the direct outcome of a series of deliberate architectural choices. We have shown that the overwhelming majority of these parameters are concentrated in the Multi-Head Self-Attention and Feed-Forward Network components, which are replicated across dozens of layers to give the model its analytical depth.

The key takeaway is the direct and calculable relationship between these abstract hyperparameters and the concrete realities of deployment.

num_params dictates the static VRAM and disk footprint.
d_model drives the computational intensity (FLOPs).
num_layers creates sequential latency.
The KV Cache, a function of depth, width, and context length, governs the maximum user concurrency.

By understanding these fundamental calculations, engineers can move beyond treating LLMs as black boxes. They can precisely forecast hardware costs, optimize performance through techniques like quantization, and design robust, scalable systems that balance capability with efficiency. This architectural literacy is no longer optional—it is essential for building the next generation of intelligent applications.

Continue with LLM capacity and cost planning, quantization, and context windows.