Week 1 · D4🔒 LockedAbout 4 hours
Iteration and Evaluation: Small Test Sets, A/B Testing, Version Control, Common Anti-Patterns
Turn "this version feels better" into "this version got three more of ten test cases right": build a small test set, run an A/B comparison, version your prompts, and recognize the most common anti-patterns.
Turn "this version feels better" into "this version got three more of ten test cases right": build a small test set, run an A/B comparison, version your prompts, and recognize the most common anti-patterns.
Interview questions
How do you build a test set for a prompt? How would you choose ten samples, and where do the expected answers come from?怎么给一个提示词建测试集?十条样本该怎么挑,标准答案从哪来?
Common in ChinaCommon overseasBasic#evaluation#test-set🔒 Unlock this chapter to see the answer keys.
How do you version prompts? How does it differ from versioning code, and what is your first move when production misbehaves?提示词版本化怎么做?它和代码版本管理有什么不同?线上出问题你先做什么?
Common in ChinaCommon overseasIntermediate#prompt-versioning#evaluation🔒 Unlock this chapter to see the answer keys.
Is using a model to grade another model's output reliable? When is it acceptable, and when must a human look?用模型给模型打分靠谱吗?什么时候可以用,什么时候必须人工看?
Common in ChinaCommon overseasIntermediate#evaluation#llm-as-judge🔒 Unlock this chapter to see the answer keys.
