Dayward AIWeChat Mini Program
RAG in 14 Days: From Retrieval to Trustworthy Answers
Week 2 · D8🔒 LockedAbout 5 hours

Evaluation First: Building a Golden Set, Computing Recall and Ranking Metrics, Using a Model as Judge for Faithfulness

Every optimization in week two needs to be backed by data, so build evaluation first. Today construct a golden question-answer set from scratch, implement three retrieval metrics — recall, Mean Reciprocal Rank, and Normalized Discounted Cumulative Gain — use a model as judge for faithfulness and answer relevance, then run a baseline report on week one's system.

Today's goals 0/3

Sign in to tick these off and save your progress.

Today's Goals

  1. Construct a golden set from real corpus data, and explain why it must include unanswerable questions and multi-hop questions
  2. Implement recall, Mean Reciprocal Rank, and Normalized Discounted Cumulative Gain, and explain what each is sensitive to
  3. Design a model-judge prompt and verify it's trustworthy with manual spot checks, rather than trusting its scores unconditionally

Week one got the system running and left one question unanswerable throughout: is it any good? Today installs that scale. Come back and tick off the three goals.

Plain-Language Walkthrough

No scale, no dieting

Somebody resolves to lose weight, switches to salads in month one, starts running in month two, and gives up late-night snacks in month three. Are they thinner after three months? They cannot say — there is no scale in the house. They can only judge by their trousers feeling looser, and that judgment is remarkably accurate on days when their mood is good.

At the end of week one your retrieval question-answering system is in that state. D4 changed the chunking, D5 tuned the index parameters, D7 fused two retrieval paths, and at each step it felt like it should be better. But if I ask now whether raising the vector path's weight from 0.6 to 0.8 makes it better or worse, all you can do is guess.

A scale's value is not telling you your current weight but turning "did this change help" into an answerable question. So week two's first day performs no optimization — it builds the scale first. From today, questions like "should we add reranking," "how large should chunks be," and "do we need query rewriting" have exactly one legitimate answer: run the evaluation and read the numbers.

A scale needs two things: a fixed set of subjects and a set of rules for reading it. The former is the golden set and the latter the metrics. Unfortunately neither is available off the shelf — public benchmark sets are all Wikipedia question answering with no relation whatsoever to your company knowledge base; and metrics come with a pile of ready-made names, the wrong choice among which lets the numbers rise while the user experience falls.

One last thing that gets skipped most easily: building the scale has a cost. Labeling answer documents for 20 questions took me two hours in the lab. Many teams skip it on the grounds of getting something working first, and three months later the system has twenty versions and none of them can say what it is good at — that did not save two hours, it turned three months of work into something unverifiable.

Sign in and link WeChat to read

Sign in, then link your WeChat mini program account on the account page. Linked accounts share one free article per day. Unlock additional articles with rewarded videos in the mini program.

Mini program QR code

Scan in WeChat to open Dayward AI. Open the same course and chapter, choose the rewarded video, then refresh this page after completion.

Interview questions

  • You need to build an evaluation set from scratch for a RAG system over a company knowledge base. How would you do it, and how many questions are enough?让你从零给一个公司知识库的 RAG 系统建评估集,你会怎么做?多少题才算够用?
    Common in ChinaCommon overseasIntermediate#evaluation#golden-set#rag

    🔒 Unlock this chapter to see the answer keys.

  • Recall, mean reciprocal rank, and normalized discounted cumulative gain - which failure mode does each one catch first, and what do you miss by watching only one?召回率、平均倒数排名、归一化折损累计增益,这三个检索指标分别在什么故障下会先掉下来?只盯一个会漏掉什么?
    Common in ChinaCommon overseasIntermediate#retrieval-metrics#evaluation#ranking

    🔒 Unlock this chapter to see the answer keys.

  • What systematic biases does an LLM judge have when scoring RAG faithfulness, and how do you detect them and prove your judge is trustworthy?用模型当裁判来评 RAG 的忠实度,有哪些系统性偏差?你怎么发现它们、又怎么证明你的裁判可信?
    Common in ChinaCommon overseasDeep dive#llm-as-judge#evaluation#faithfulness

    🔒 Unlock this chapter to see the answer keys.

  • Why must a RAG evaluation set include questions the corpus cannot answer, and what does leaving them out hide?RAG 的评估集里为什么一定要放语料里没有答案的问题?不放会掩盖什么?
    Common in ChinaCommon overseasBasic#evaluation#abstention#golden-set

    🔒 Unlock this chapter to see the answer keys.

Comments