Build Fast with AI logo
RoadmapProjectsCareer PathsCheckpoints
Start Level 0
Learning path
★Roadmap overview0Environment & Foundations1LLM Fundamentals & APIs2Building Simple GenAI Applications3RAG Systems & Intelligent Agents4Production Systems & Deployment5Advanced GenAI Techniques6Choose Your Specialization Path
Resources
Projects & notebooksCareer pathsCheckpoint tasks
  1. Home
  2. GenAI Roadmap
  3. Level 4
4Level 4 of 6 · Advanced

Level 4: Production GenAI Systems

Level 4 turns a working prototype into a service you can leave running. You wrap the app in FastAPI, containerise it with Docker, move slow LLM calls to background jobs, add logging and tracing, and catch prompt regressions with Promptfoo before they ship. Expect about 30 hours over 3 to 4 weeks.

By the Build Fast with AI team·Published April 10, 2026·Updated November 19, 2026

Duration
3-4 weeks
Difficulty
Advanced
Study time
~30 hours
Checkpoints
7 tasks
On this page
  1. What you will learn
  2. Topics and tools
  3. Checkpoint tasks
  4. Notebooks and projects
  5. FAQ
  6. Related guides

Outcomes

What you will learn in Level 4

  • Build production-grade APIs
  • Deploy and scale applications
  • Monitor and debug production systems
  • Benchmark models and trace production runs
  • Optimize costs at scale
  • Implement robust testing strategies

Syllabus

Topics and tools in Level 4

  1. 01FastAPI for building APIs
  2. 02Docker containerization
  3. 03Database integration (PostgreSQL, MongoDB)
  4. 04Asynchronous processing with Celery
  5. 05Logging and monitoring
  6. 06Promptfoo-based regression testing
  7. 07Rate limiting and load balancing
  8. 08Cost optimization strategies
  9. 09Testing and CI/CD pipelines
  10. 10Security best practices
Tools:FastAPIDockerPostgreSQLCeleryPromptfooOpikGitHub Actions

Practice

Level 4 checkpoint tasks

Finish these 7 tasks before moving on. Open a task to see its steps.

  1. 1

    FastAPI service

    2 hours
    • Expose your Level 2 or Level 3 app as a FastAPI endpoint
    • Validate requests and responses with Pydantic

    Time: 2 hours

  2. 2

    Logging and monitoring

    2 hours
    • Log every LLM call with latency, tokens and cost
    • Add tracing so you can replay a bad response

    Time: 2 hours

  3. 3

    Unit and eval tests

    2 hours
    • Unit-test the code that does not call the LLM
    • Add a Promptfoo suite that runs your key prompts on every change

    Time: 2 hours

  4. 4

    Dockerize

    1.5 hours
    • Write a Dockerfile for the service
    • Run the service and its database together with Docker Compose

    Time: 1.5 hours

  5. 5

    Deploy to the cloud

    2 hours
    • Deploy the container to a managed cloud platform
    • Add a health check and load secrets from the environment

    Time: 2 hours

  6. 6

    Cost and performance

    2 hours
    • Cache repeated calls
    • Route easy requests to a cheaper model
    • Compare cost per request before and after

    Time: 2 hours

  7. 7

    Production checklist

    3 hours
    • Add rate limiting, retries with backoff and input guardrails
    • Write a short runbook for when the model provider goes down

    Time: 3 hours

Build

Notebooks and projects for Level 4

5 runnable notebooks, all free to open in Google Colab.

Promptfoo GPT-5.4 Evaluation Cookbook

Notebook

Compare frontier models with cost, latency, and red-team checks using Promptfoo.

IntermediateLevel 1–4#promptfoo#evals

GPT-5.4 Cookbook

Notebook

Ship structured outputs, agent tools, and long-context workflows with GPT-5.4.

IntermediateLevel 2–4#openai#gpt-5.4

Opik LLM Evaluation and Monitoring

Notebook

Track traces, metrics, and evaluation runs for LLM and agent workflows.

IntermediateLevel 3–4#opik#monitoring

AgentOps Evaluation

Notebook

Add replay analytics, benchmarking, and monitoring to multi-agent systems.

IntermediateLevel 3–6#agentops#monitoring

LLM Benchmark Comparison

Notebook

Compare frontier models across reasoning, coding, and speed-focused tasks.

AdvancedLevel 4–6#benchmarking#comparison
16 more project ideas to build yourself

Production-Ready API

Brief

Build a FastAPI-based API for serving an LLM application at scale.

AdvancedLevel 4#fastapi#api

Monitoring & Observability Dashboard

Brief

Create monitoring tools to track performance, costs, and errors in production.

AdvancedLevel 4#monitoring#observability

Async Job Processing System

Brief

Implement a scalable async job queue for long-running LLM tasks.

AdvancedLevel 4#async#celery

AI-Powered Code Editor

Brief

Full-stack application with AI code completion and intelligent suggestions.

AdvancedLevel 2–5#app-dev#frontend

Real-Time Collaborative AI App

Brief

Build a collaborative app with WebSockets and AI features supporting multiple users.

AdvancedLevel 2–4#app-dev#websockets

Mobile AI Assistant

Brief

Create a mobile app (iOS/Android) powered by GenAI with offline capabilities.

AdvancedLevel 3–5#app-dev#mobile

Production SaaS Product

Brief

Build and deploy a complete SaaS application with AI features, billing, and analytics.

AdvancedLevel 4–5#app-dev#saas

Scalable RAG Platform

Brief

Engineer a large-scale RAG system supporting millions of documents and high QPS.

AdvancedLevel 3–5#engineer#rag

LLM Serving & Inference Platform

Brief

Build infrastructure for serving multiple LLMs with caching, batching, and optimization.

AdvancedLevel 4–5#engineer#llm-serving

Distributed Multi-Agent System

Brief

Design and implement distributed agents orchestrating complex workflows.

AdvancedLevel 3–5#engineer#agents

AI System Optimization Framework

Brief

Create tools for benchmarking, profiling, and optimizing AI application performance.

AdvancedLevel 4–5#engineer#optimization

Novel RAG Architecture Research

Brief

Research and implement innovative RAG approaches with published benchmarks.

AdvancedLevel 3–5#research#rag

AI Product Market Research

Brief

User and market research for positioning an AI product.

IntermediateLevel 2–4#product#research

Competitive Benchmarking Study

Brief

Analyze competitive AI products and create detailed feature benchmarks.

IntermediateLevel 3–4#product#competitive-analysis

AI Product Metrics Framework

Brief

Design the metrics and analytics framework for an AI product.

AdvancedLevel 4–5#product#metrics

AI Business Case Analysis

Brief

Develop detailed business cases for AI product investments.

AdvancedLevel 4#product#business

FAQ

Level 4 questions, answered

What does production-ready mean for an LLM app?

It means the app has tests, logging, tracing, rate limiting, retries, cost tracking and a way to catch prompt regressions before deploy. A demo that works once is not production-ready; one you can debug at 2 a.m. is.

How do I test an LLM application?

Test deterministic code with normal unit tests, and test prompts with an eval suite such as Promptfoo that checks outputs against assertions on a fixed dataset. Run the suite in CI so a prompt change cannot silently break answers.

How do I handle LLM API rate limits?

Retry with exponential backoff, queue slow or bulk work in background jobs with Celery or a similar queue, and spread load across models or providers. Track tokens per minute so you hit the limits in testing, not in production.

How much does it cost to run a GenAI app?

Cost depends mostly on tokens per request and which model you call. Log cost per request from day one; caching repeated calls and routing simple requests to a smaller model are the two quickest ways to bring it down.

Do I need Kubernetes to deploy a GenAI app?

No. A single Docker container on a managed platform handles most early-stage GenAI apps. Learn Kubernetes when you run several services with real traffic, which the AI Engineer path covers.

Read next

Related guides

  • What is inference?
  • AI guardrails
  • What is AI hallucination?
  • AI engineer interview questions for freshers
← PreviousLevel 3: RAG Systems & Intelligent AgentsNext →Level 5: Advanced GenAI Techniques

On this page

  • What you will learn
  • Topics and tools
  • Checkpoint tasks
  • Notebooks and projects
  • FAQ
  • Related guides
Build Fast with AI logoBuild Fast with AI

A free, project-based roadmap for learning generative AI, from your first API call to production systems.

Levels

  • Level 0: Environment & Foundations
  • Level 1: LLM Fundamentals & APIs
  • Level 2: Building Simple GenAI Applications
  • Level 3: RAG Systems & Intelligent Agents
  • Level 4: Production Systems & Deployment
  • Level 5: Advanced GenAI Techniques
  • Level 6: Choose Your Specialization Path

Career paths

  • AI App Developer Roadmap
  • AI Engineer Roadmap
  • ML Researcher Roadmap
  • AI Product Manager Roadmap
  • All projects and notebooks
  • Checkpoint tasks

Keep learning

  • AI concepts explained
  • AI glossary
  • AI project ideas
  • AI interview prep
  • Blog
  • Notebooks on GitHub

© 2026 Build Fast with AI. All rights reserved.

Privacy PolicyTerms