Model Parameters (Weights)

What is it?

Parameters are the internal numbers a model learns during training — the 'weights' that store what it knows and determine its outputs. Model size is often quoted as a parameter count.

Explain like I'm 5

Why was it created?

A model needs somewhere to store what it learns. Parameters are that storage — adjustable numbers tuned so the model maps inputs to good outputs.

Where is it used?

• Describing model size (e.g. 7B, 70B)
• Estimating memory and cost
• Comparing models
• Fine-tuning discussions

Why should developers care?

Parameter count is the headline number for model size, hinting at capability, memory needs, and cost — so it's worth knowing what it actually means.

How does it work?

Each connection in a neural network has a weight (plus biases); these are the parameters. Training adjusts them via gradient descent so the model's outputs match the data. At inference the fixed parameters transform inputs into predictions.

Real-world example

A '7B' open model has ~7 billion learned weights; loading it needs roughly 14GB at 16-bit precision, which is why quantization matters.

Common use cases

• Sizing hardware for a model
• Comparing capability roughly
• Planning quantization/distillation
• Understanding fine-tuning scope

Advantages

• Simple headline for model scale
• More parameters can mean more capacity
• Guides memory/cost planning
• Comparable across models

Disadvantages

• Count alone doesn't equal quality
• Bigger = costlier to run
• Ignores data and training quality
• Can mislead (MoE active vs total)

When should you use it?

When reasoning about a model's size, memory footprint, or rough capability tier.

When should you avoid it?

Don't judge a model purely by parameter count — data quality, training, and evals matter more.

Alternatives

Related terms

Interview questions

Beginner

• What are model parameters?
• What does '7 billion parameters' describe?

Intermediate

• Why does parameter count affect memory usage?
• Why isn't more parameters always better?

Senior

• How do total vs active parameters differ in MoE models?
• How does precision (fp16/int8/int4) change the memory a parameter count implies?

Common misconceptions

• "More parameters always means smarter" — training data and method matter as much or more.
• "Parameter count equals memory in GB" — it depends on numeric precision.

Fun facts

• Model names like '7B' or '70B' refer to parameter counts in billions.
• Quantization stores each parameter in fewer bits to cut memory dramatically.

Timeline

• 2018-2020 — Parameter counts jump from millions to billions
• 2020s — Efficiency (data, MoE, quantization) rivals raw count

Learning resources

• Deep Learning (book)

Quick summary

Parameters are the learned weights that store a model's knowledge and set its outputs; parameter count is the common measure of model size.

Cheat sheet

• Learned weights = the model's knowledge
• Count (e.g. 7B) ≈ model size
• Memory depends on precision
• Count ≠ quality

If you remember only one thing

Further reading

• Deep Learning (book)