GitHub - calmrocks/ai-engineer-notebooks: Hands-on, framework-free Colab notebooks for the AI Engineer / Forward Deployed Engineer (FDE) skill set — model APIs,
**Learn the applied-LLM stack the way you'll actually be interviewed on it: framework-free, on a free API, from prompting all the way to serving, fine-tuning, and a red-team benchmark.**
Runnable Colab notebooks for the **AI Engineer / Forward Deployed Engineer (FDE)** skill set. You build working systems on top of foundation models (model APIs, RAG, evals, agents, adaptation, serving) using raw APIs, not frameworks.
What makes this different
[](https://github.com/calmrocks/ai-engineer-notebooks#what-makes-this-different)
- **Framework-free, on purpose.** You write the agent loop, RAG, and evals from raw API calls first, so you understand what LangChain/LlamaIndex actually do before you reach for them (and can judge when _not_ to). Patterns are durable; wrappers churn.
- **Evals are the spine.** "Measure before you tune" is installed early and returns in every section. It's the habit that separates an engineer who shipped a system from one who built a demo.
- **Free to run, end to end.** Everything runs on the **free Groq API** (no credit card). The two topics Groq can't host, LoRA fine-tuning (06) and self-hosted serving (09), are concept-first with optional, fenced Colab-GPU appendices that were **verified on a real Colab T4**.
- **Real case studies, not toy demos.** Three end-to-end case studies show the skills combined under real constraints: a support assistant _debugged in production_, a pipeline-vs-agent cost showdown, and a red-team robustness benchmark.
- **OpenAI-compatible throughout**, so every pattern transfers directly to OpenAI and (with small changes) Anthropic. Swap the base URL and the skills carry over.
Built as the hands-on companion to Plan: Transitioning to Forward Deployed Engineer / AI Engineer. The plan explains what to learn and why; these notebooks are where you run it.
Who this is for
[](https://github.com/calmrocks/ai-engineer-notebooks#who-this-is-for) Backend or full-stack engineers moving into AI Engineer, FDE, Applied AI, or Solutions Engineer (AI) roles. Different titles, largely the same job. You can ship production code; you want the applied-model layer on top.
Learning order
[](https://github.com/calmrocks/ai-engineer-notebooks#learning-order) Work top to bottom. Each notebook is self-contained (installs its own dependencies, reads API keys from Colab secrets) and ends with exercises.
00 — Setup
[](https://github.com/calmrocks/ai-engineer-notebooks#00--setup) | Notebook | What you'll learn | | --- | --- |
| Environment & cost hygiene  | API keys via Colab secrets, spend guards, model picking |
01 — Model APIs
[](https://github.com/calmrocks/ai-engineer-notebooks#01--model-apis) | Notebook | What you'll learn | | --- | --- |
| Prompting fundamentals  | Clear instructions, few-shot, output-format specs, step-by-step reasoning: the cheapest lever, each shown moving a number |
| Structured output  | Getting reliable JSON out of a model, and where it breaks |
| Tool calling  | Function/tool calling end to end, error paths included |
| Streaming  | Streaming responses and what UIs need from them |
| Context & caching  | Context-window budgeting, prompt caching, batch vs real-time pricing |
02 — Evals I: measuring outputs
[](https://github.com/calmrocks/ai-engineer-notebooks#02--evals-i-measuring-outputs) | Notebook | What you'll learn | | --- | --- |
| Measuring outputs  | Golden sets and metrics on the section-01 task; install the "measure before you tune" habit _before_ building anything you'd need to tune. Evals is the spine; it returns in every section after this |
03 — RAG
[](https://github.com/calmrocks/ai-engineer-notebooks#03--rag) | Notebook | What you'll learn | | --- | --- |
| What is RAG?  | The retrieve → augment → generate loop, why RAG beats a plain LLM, and why RAG isn't the same as embeddings, with a 15-line working demo |
| Embeddings & retrieval  | Embedding choice, vector search, similarity pitfalls. Get retrieval working first |
| Hybrid & reranking  | Keyword + vector hybrid retrieval, rerankers, when each earns its cost |
| Chunking  | Chunking strategies on a real messy corpus, revisited last once you can judge them against retrieval |
| Why RAG fails  | Diagnosing bad answers: retrieval quality, not generation, is usually the bottleneck |
04 — Evals II: the differentiator
[](https://github.com/calmrocks/ai-engineer-notebooks#04--evals-ii-the-differentiator) | Notebook | What you'll learn | | --- | --- |
| Golden sets  | Building a golden set for the RAG system from section 03 |
| LLM as judge  | Judge prompts, agreement with humans, and the judge's own failure modes |
| Regression evals  | Evals as CI: catching quality regressions when you change a prompt or model |
05 — Agents
[](https://github.com/calmrocks/ai-engineer-notebooks#05--agents) | Notebook | What you'll learn | | --- | --- |
| Agent loop from scratch  | A working agent loop in raw API calls, no framework |
| Tool design  | Designing tools the model can actually use well |
| Guardrails & budgets  | Stopping conditions, cost/latency budgets, when a pipeline beats an agent |
| MCP & the tool ecosystem  | Concept: what the Model Context Protocol standardizes, how it maps to the raw tool loop, and when to reach for it |
| Skills & progressive disclosure  | Concept: packaging reusable know-how an agent loads on demand. The `SKILL.md` pattern, the context-budget payoff, and Tools/MCP/Skills as one story |
| Harness engineering  | Synthesis: the scaffold _around_ the call: context assembly & compaction, tool-result shaping, and verification loops. Names the discipline the section has been teaching piece by piece |
06 — Adapting the model
[](https://github.com/calmrocks/ai-engineer-notebooks#06--adapting-the-model) | Notebook | What you'll learn | | --- | --- |
| Fine-tune vs RAG vs prompt  | When to change the model's weights vs its inputs; what LoRA/QLoRA are and cost; the argument you'll have in the room, plus an optional real LoRA fine-tune on a free GPU |
07 — Security
[](https://github.com/calmrocks/ai-engineer-notebooks#07--security) | Notebook | What you'll learn | | --- | --- |
| Prompt injection & the trust boundary  | Direct & indirect prompt injection, output handling, PII, excessive agency. The OWASP LLM Top 10 risks, failing live then defended |
08 — Operations
[](https://github.com/calmrocks/ai-engineer-notebooks#08--operations) | Notebook | What you'll learn | | --- | --- |
| Observability & LLMOps  | Tracing every call, safe prompt logging, cost/latency/error metrics, drift detection, and the observe→eval feedback loop |
| Reliability & fallbacks  | Retries with backoff, timeouts, fallback models, output validation, circuit breakers, graceful degradation |
| Experiment tracking & registry  | MLflow end to end: log runs/params/metrics from the section-04 eval harness, register and version a model, and promote by stage: the tooling that turns "I ran an eval" into a tracked, reproducible workflow |
09 — Serving & inference performance
[](https://github.com/calmrocks/ai-engineer-notebooks#09--serving--inference-performance) Where the free Groq API can't run the topic (these frameworks need a GPU), the notebook teaches it **concept-first** and fences an optional Colab-GPU appendix, the same pattern as the section-06 LoRA appendix.
| Notebook | What you'll learn | | --- | --- |
| Serving frameworks  | The serving stack an AI engineer actually picks between (vLLM, TGI, Triton, TensorRT-LLM): what each optimizes, how they map onto the raw API you've been calling, and when to reach for which |
| Inference performance  | The levers behind throughput and latency: continuous batching, the KV cache, quantization, and the throughput-vs-latency trade, with the napkin math to size a deployment |
10 — ML system design & performance
[](https://github.com/calmrocks/ai-engineer-notebooks#10--ml-system-design--performance) | Notebook | What you'll learn | | --- | --- |
| Designing an inference service  | Concept: the ML system design interview, worked end to end: QPS/VRAM/latency/cost estimation, replica scaling, queueing, caching, and the SLA trade-offs, on a realistic LLM-serving prompt |
11 — Customer craft (the FDE differentiator)
[](https://github.com/calmrocks/ai-engineer-notebooks#11--customer-craft-the-fde-differentiator) | Notebook | What you'll learn | | --- | --- |
| Scoping & discovery  | Turn a vague customer ask into a scoped, evaluable system: discovery questions, a one-page scoping doc, the demo discipline: the customer-scenario interview round most engineers can't evidence |
12 — Case Studies & Capstone
[](https://github.com/calmrocks/ai-engineer-notebooks#12--case-studies--capstone)
Where the skills come together into projects. First a **case study** (one realistic scenario worked end to end, runnable), then the **capstone**: the deployed repo you build yourself. (Section overview.)
| Notebook | What you'll learn | | --- | --- |
| Case study A — Customer-support assistant  | One scenario scoped → built → served → **debugged in production**: a vague ask becomes a deployed, evaluated RAG+agent assistant, then a live quality regression (a stale index after a corpus migration) that you diagnose and fix. A build-to-debug arc threading sections 02–11 |
| Case study B — Contract extraction: pipeline vs agent  | The judgment call interviewers love: build the _same_ extraction task as both an agent and a pipeline, then prove with accuracy + token cost that the pipeline wins when the steps are known |
| Case study C — Red-team robustness benchmark  | A different _kind_ of system, a harness that evaluates a model instead of serving one: an attacker→target→judge (PAIR) loop that measures attack success rate, composing the agent loop, LLM-judge, security, and evals |
**Capstone:**the brief for the deployed project that goes on your resume, a real repo with a serving component and an eval report. Case studies are for learning; the capstone is for hiring.
Conventions
[](https://github.com/calmrocks/ai-engineer-notebooks#conventions)
- **Raw model APIs, no frameworks.** Patterns are durable; wrappers churn.
- **One shared corpus** (`data/`) across RAG and eval sections, so evals measure the retrieval you actually built.
- **Self-contained notebooks.** First cell installs, second cell calls `from aien import setup; client, MODEL = setup()` to load your key from Colab secrets (or a local env var). No hidden state between notebooks. `aien` is the tiny shared-setup package in this repo (one place to change credential loading), installed automatically by the first cell.
- **Every notebook ends with exercises.** Do them before moving on.
Setup
[](https://github.com/calmrocks/ai-engineer-notebooks#setup) 1. Get a free API key at console.groq.com, no credit card required. 2. In Colab: the key icon in the left sidebar → add `GROQ_API_KEY` as a secret, and toggle notebook access on. 3. Open any notebook via its badge and run top to bottom.
Running locally instead: `pip install -r requirements.txt && pip install -e .` (the second installs the `aien` setup helper), `export GROQ_API_KEY=...`, open with Jupyter.
Related reading
[](https://github.com/calmrocks/ai-engineer-notebooks#related-reading)
- Plan: Transitioning to FDE / AI Engineer is the roadmap these notebooks implement
- Guide: Building a Real LLM Project for Your Resume sets the capstone's requirements bar
- Walkthrough: Designing a RAG System covers the systems view of section 03
- Walkthrough: Designing an AI Agent Orchestration System covers the systems view of section 05