What is Inference?

Inference is the process of using a trained AI model to generate predictions or outputs from new inputs. In the context of LLMs, inference means sending a prompt to the model and receiving a generated response. Inference is distinct from training: training adjusts model weights using data, while inference uses fixed weights to produce outputs. Inference cost, speed (latency), and throughput are major considerations for AI deployment. Optimization techniques include quantization, speculative decoding, batching, KV-cache optimization, and purpose-built inference hardware like Groq LPUs. The term "inference-time compute" refers to giving models more processing time to reason.