AI & Machine Learning · Foundational

Inference

Using a trained model to make predictions or generate outputs on new data.

Definition

Inference is the production phase of the ML lifecycle. It can be real-time (synchronous, low latency), batch (large offline jobs), asynchronous or streaming. For LLMs, inference cost and latency are driven by token counts, model size and serving optimizations such as batching and KV caching.

Related terms