What is LLM evaluation?
LLM evaluation measures the quality, accuracy and safety of a large language model’s outputs for a specific use case.
Evaluations, often called evals, run a model or prompt over a set of test inputs and score the outputs using rules, reference answers, rubrics or another model acting as a judge.
For product teams, evals work like a regression suite for AI behaviour: they show whether a new model, prompt or retrieval change made things better or worse.
Example
A team scores 300 support questions for grounding before switching models. The new model is faster but fabricates policy details in 4% of answers, so they adjust the prompt before rollout.
Best practices
- Create a representative, versioned test set
- Choose scoring methods that match the task
- Run evals on every prompt or model change
- Review a sample of judged outputs by hand
Ship without fear.
Start testing free in minutes, or let a QAShift engineer deliver your first morning report within 5 days.