· intermediate
Evaluating and Operating LLM Systems
What happens after the demo works: measuring whether a change helped, bounding what a compromised model can do, controlling cost, and surviving the model version being retired underneath you.
Written in house, from the work rather than from a syllabus. Every example is one we would use ourselves, and every quiz question explains why its answer is right.
What you’ll learn
- Build an evaluation set and grade with something better than impressions
- Use an LLM as a judge while knowing what it is biased toward
- Bound what a prompt injection can accomplish, since prompting cannot prevent it
- Reduce cost substantially without quietly degrading quality
- Build on a component that returns a different answer each time
- Recognise when a deterministic solution is the correct answer
Before you start
- · Some exposure to using an LLM through an API or a chat interface
- · Building with LLMs covers RAG, embeddings and agents, and pairs well with this