Productionizing and Retrospective: Writing Evals for Tools, Versioning, Publishing to npm and a Registry, Observability, and a Capstone Project
Turn a server that runs into a server you can maintain long-term: write an eval set for your tools, manage description changes as breaking changes, add tracing and metrics, and turn the week into one portfolio entry.
Today's Goals
- Write a re-runnable eval set for a tool, and name the three categories of cases it should cover
- Judge whether a change is breaking, and explain why a description change counts as one too
- Add tracing and key metrics to a server, and say which one to check first when something goes wrong
The last day. After six days you hold a server that runs, works remotely, can be connected by your own client, and has been through a security checklist. Today answers the final question: how does it survive the next six months. Come back and tick off the three goals above.
Plain-Language Walkthrough
Tools need evals too
Start with an incident so real it is almost boring.
A server had been live three months with not a line of tool code changed when somebody edited search_docs's description from "search the internal documentation library by keyword" to "search documents" — on the grounds that it was too wordy. Nothing alerted after the deploy: unit tests green, zero server errors, clean logs. Two weeks later somebody reported that the assistant had got dumber lately and kept saying it did not know when asked about documentation.
The cause was that the model stopped selecting the tool. It never once failed a call, because it was never called.
That is MCP tools' most counter-intuitive property: whether the model selects it goes wrong earlier than whether the tool itself is correct, and no traditional test covers the former. A unit test checks that given arguments produce the right output; break the description and the arguments never reach your function at all.
So a tool needs another layer of testing, generally called an eval. Its shape is simple: give one user sentence and a tool table, see which tool the model picked (or that it picked none), and compare against the expectation.
Interview questions
How do you evaluate whether an MCP tool is any good, and what categories of test cases would you design?怎么评估一个 MCP 工具做得好不好?你会设计哪几类测试用例?
Common in ChinaCommon overseasIntermediate#eval#tooling🔒 Unlock this chapter to see the answer keys.
When versioning an MCP server, what counts as a breaking change, and why is editing a tool description more dangerous than renaming a field?给一个 MCP 服务端做版本管理时,什么样的改动算破坏性变更?为什么说改一句工具描述比改一个字段名更危险?
Common in ChinaCommon overseasIntermediate#versioning#tooling🔒 Unlock this chapter to see the answer keys.
A user reports that one of your MCP tools is 'always getting it wrong' in production. In what order do you investigate?线上有人反馈某个 MCP 工具「总是调不对」。你按什么顺序排查?
Common in ChinaCommon overseasIntermediate#observability#debugging🔒 Unlock this chapter to see the answer keys.
