問題文
A team is building its first evaluation set for a customer-facing agent. They have six months of production transcripts. What is the most useful way to use them?
選択肢
- Use every transcript, so the evaluation set is as large as possible.
- Select real requests that represent the distribution of what users actually ask, and record the expected outcome for each.
- Ask the model itself to generate a set of synthetic questions in the same style, because generated questions can be produced in far greater volume than the transcripts allow and they avoid any concern about handling real user data.
- Use only the transcripts where the agent failed, since those are the cases that need fixing.