Design a retrieval-augmented assistant
Hard45 minFree, no account
Easy to demo, hard to evaluate, and the failure mode is a confident wrong answer with a citation.
The question
Design an assistant that answers employee questions from 50,000 internal documents, with citations.
Say how you would evaluate it and how you would stop it answering confidently when it should not.
Functional
- Natural-language question, grounded answer with citations.
- Respect document permissions per user.
- Say "I do not know" rather than guess.
Non-functional
- First token within 2 seconds.
- Documents change daily.
- A confidently wrong answer is worse than no answer.
45:00Commit to an answer before you open the solution. Reading it first teaches you to recognise good answers, which is not the skill being tested.
Stuck?
0 of 3 hints takenThe worked solution
written by a person · not a gradeScore yourself
0 of 5 marked- Structure-aware chunking with a stated reason20
- Hybrid retrieval with reranking15
- Permissions enforced as a retrieval filter20
- Evaluated retrieval, grounding and answer quality separately30
- Structural abstention rather than prompt instructions alone15
We run no AI here and nothing on this page grades you. The score is yours, and the useful number is the one you get on the same problem a month from now, cold.
kept in this browser only