Temperature (LLM Parameter)
Temperature is a generation parameter that controls how random or deterministic an LLM's output is — near 0 makes the model consistently pick its most likely next token, while higher values let it sample less-likely tokens more often for more varied, less predictable output — and it's one of the first things QA needs to control for in testing.
A non-zero temperature is exactly why a prompt can return a different answer every time it's run, which breaks the basic assumption most testing relies on: that running the same input twice gives the same output. Before writing any assertion against an LLM's output, it matters whether the feature under test runs at temperature 0, which is deterministic-ish and testable with exact or near-exact matching, or a higher temperature, which requires semantic or fuzzy assertions instead.
Setting temperature to 0 for test runs is a common practice specifically to make tests reproducible, but it's not a complete fix — even at 0, some model providers' infrastructure, like batching or hardware non-determinism, can still produce slightly different outputs across runs, and testing at temperature 0 doesn't validate how the feature behaves at whatever temperature it actually runs at in production.
The right approach is usually to test functional correctness — does it follow instructions, avoid errors — at low temperature for reproducibility, and separately test output quality and safety, like tone, creativity, and guardrail adherence, at the production temperature setting, since that's the behavior users will actually see.
Example
{
"model": "gpt-4",
"temperature": 0,
"messages": [{ "role": "user", "content": "Summarize this ticket." }]
}Setting temperature to 0 in a test run for reproducible, near-deterministic output
