Inference
What is it?
Inference is the process of using a trained AI model to make predictions or generate output, as opposed to training it.
Explain like I'm 5
Why was it created?
It's simply the 'using' phase of a model's life. The term distinguishes running a finished model from the separate, costly job of training it.
Where is it used?
- Serving AI predictions in apps
- Generating LLM responses
- Real-time recommendations
- Any live use of a trained model
Why should developers care?
Inference is where AI delivers value and where most ongoing cost and latency live, so it matters for anyone deploying models.
How does it work?
A trained model takes new input and runs it through its fixed, learned parameters to produce an output. Unlike training, the model's weights don't change during inference.
Real-world example
When you send a prompt to an AI assistant, the running model performs inference to generate the reply, using weights set during training.
Common use cases
- Live predictions and generation
- Serving models at scale
- Real-time AI features
- Batch scoring of data
Advantages
- Delivers the model's value
- No weight updates needed
- Can be optimized for speed
- Runs continuously in production
Disadvantages
- Ongoing compute cost
- Latency to manage
- Hardware demands for large models
- Scaling to many users is expensive
When should you use it?
Whenever you put a trained model to work on new inputs.
When should you avoid it?
Not avoidable for using a model; the cost-saving levers are optimization and right-sizing.
Alternatives
Related terms
Interview questions
Beginner
- What is inference?
- How does it differ from training?
Intermediate
- Why does inference cost add up?
- Do weights change during inference?
Senior
- How do you reduce inference latency and cost?
- How does quantization help inference?
Common misconceptions
- "Inference and training are the same" — training learns the weights; inference just uses them.
- "Inference is free once trained" — running a model at scale has real, ongoing compute cost.
Fun facts
- For popular models, total inference cost over time can far exceed the one-time training cost.
- Techniques like quantization speed up inference by using smaller number formats.
Timeline
- 2010s — Inference optimization becomes a major focus as models deploy widely
Learning resources
Quick summary
Inference is using a trained model to generate predictions or output, where AI delivers value and where ongoing cost and latency concentrate.
Cheat sheet
- Using a trained model
- Weights stay fixed
- Where value + cost live
- Optimize for speed and price