Get in Touch
 Duration 35 hours

Course Outline

Introduction and Diagnostic Foundations

  • Examining failure modes in LLM systems and identifying common Ollama-specific challenges
  • Setting up reproducible experiments and controlled testing environments
  • Exploring the debugging toolkit: local logs, request/response captures, and sandboxing techniques

Reproducing and Isolating Failures

  • Techniques for developing minimal failing examples and specific seeds
  • Managing stateful versus stateless interactions to isolate context-related bugs
  • Addressing determinism, randomness, and controlling nondeterministic behavior

Behavioral Evaluation and Metrics

  • Applying quantitative metrics such as accuracy, ROUGE/BLEU variants, calibration, and perplexity proxies
  • Conducting qualitative evaluations through human-in-the-loop scoring and rubric design
  • Implementing task-specific fidelity checks and defining acceptance criteria

Automated Testing and Regression

  • Creating unit tests for prompts and components, along with scenario and end-to-end tests
  • Building regression suites and establishing baselines with golden examples
  • Integrating Ollama model updates and automated validation gates into CI/CD workflows

Observability and Monitoring

  • Utilizing structured logging, distributed traces, and correlation IDs
  • Tracking key operational metrics including latency, token usage, error rates, and quality signals
  • Setting up alerting, dashboards, and SLIs/SLOs for model-backed services

Advanced Root Cause Analysis

  • Tracing through graphed prompts, tool calls, and multi-turn conversational flows
  • Performing comparative A/B diagnosis and conducting ablation studies
  • Managing data provenance, debugging datasets, and resolving dataset-induced failures

Safety, Robustness, and Remediation Strategies

  • Applying mitigations such as filtering, grounding, retrieval augmentation, and prompt scaffolding
  • Employing rollback, canary, and phased rollout patterns for model updates
  • Conducting post-mortems, documenting lessons learned, and fostering continuous improvement loops

Summary and Next Steps

Requirements

  • Substantial experience in developing and deploying LLM applications
  • Proficiency with Ollama workflows and model hosting practices
  • Solid working knowledge of Python, Docker, and fundamental observability tools

Target Audience

  • AI Engineers
  • MLOps Professionals
  • QA teams overseeing production LLM systems

Number of participants


Price per participant

Upcoming Courses

Related Categories