LLM Builder: Zero to Shipped Agent
Move from prompting to RAG, evals, tools, agents, deployment, and failure analysis.
Learn how to build a small AI app that works outside a demo.
Learning track
Build path.
Ship one useful AI assistant with a clear task, test set, retrieval boundary, and failure log.
Prerequisites
- basic Python
- basic web literacy
- willingness to test with boring examples
- 01 Prompting as interface design
Write prompts with role, task, context, output contract, and examples.
Turn a messy note into a structured student resource. - 02 RAG without superstition
Understand chunks, retrieval, citations, stale data, and source trust.
Build a scholarship finder over a small verified corpus. - 03 Evals before vibes
Create small pass/fail cases and regression examples.
Evaluate the scholarship finder on 20 realistic questions. - 04 Tools and agents
Use tools only where the model needs external action or exact computation.
Add a deadline checker and source verifier. - 05 Ship and observe
Deploy with logs, known limitations, and a feedback loop.
Publish a read-only demo with a failure diary.
Checkpoints
- Can explain what the model knows
- Can show citations
- Can run evals
- Can name failure modes
- Can improve from feedback
The system you will build
The assistant needs a defined task, sources it can cite, a test set, and a failure log.
What To Build
A good first build is narrow: a scholarship helper over verified sources, a student FAQ assistant, or a small research-note organizer. Keep the first version read-only until source, privacy, and failure modes are visible.
Each module ends in a commit, a screenshot, a failing example, and one paragraph explaining what changed in your judgment.
Sources & further reading
External resources can change. Check the linked source for current requirements and details.