Inference

Inference is the process of running a trained model on new input to produce output. In production AI systems it accounts for the majority of ongoing compute cost.

Latency depends on time to first token and generation speed; both are affected by model size, context length, batching and hardware.

Common cost levers include caching repeated prefixes, routing simple requests to smaller models, and constraining output length.