Evaluator for RAG output that judges, from the query and response alone, how directly and accurately the answer deals with what the user asked.
Also called RelevanceEvaluator.
Read more: Microsoft Learn
In the Ultra Transcenders books
Each book explains Relevance evaluator in context, with comparison tables and the common traps.
Terms in this definition
- RAG
Technique for answering from private or recent information: relevant content is fetched from your own data, typically a search index, and inserted into the prompt as grounding so the model can respond with citations.
- Chat message roles
Labels on chat messages: instructions go under system, the person's input under user, the model's previous answers under assistant, and results returned by a called tool under tool (or function).