> GPU floating-point operations aren't always perfectly deterministic
What?
GIF
"temperature = 0 means the LLM gives the same output every time"
not necessarily
- temperature 0 makes decoding greedy, picking the highest-probability token
- GPU floating-point operations aren't always perfectly deterministic
- batching and parallel execution can slightly
