DeepEval in CI: Gating, Cost, Flakiness, and the Practices That Separate Real Suites From Demos

Table of Contents

1. The pipeline
2. Cost control
3. Flakiness: your tests are now probabilistic twice
4. Gate PRs or report-only?
5. Good practices vs bad practices: the consolidated list
6. Where DeepEval stops

Written against DeepEval v4.1.0 (July 2026). Part 3 chose the metrics; this post wires them into a pipeline you can trust.

An eval suite that only runs on your laptop is a science project. But moving LLM evals into CI raises problems unit tests never had: every run costs real API money, takes minutes instead of milliseconds, and can fail without anything being wrong because the judge is itself a probabilistic model. This post covers the mechanics (GitHub Actions, caching, flags), then the judgment calls (gate or report?), then the consolidated do/don’t list for the whole series.

The pipeline

Because Part 2’s suite is plain pytest underneath, CI integration is one step: run deepeval test run and let its exit code do the talking — it exits non-zero when any test falls below threshold, so it gates a PR “like any other test step” (CI/CD docs). The official example uses Poetry; here’s the same shape with uv:

# .github/workflows/evals.yml
name: LLM evals

on:
  pull_request:
    branches: [main]

jobs:
  evals:
    runs-on: ubuntu-latest
    timeout-minutes: 15          # LLM calls hang; never rely on the default 6h
    steps:
      - uses: actions/checkout@v4

      - name: Install uv
        uses: astral-sh/setup-uv@v5

      - name: Install dependencies
        run: uv sync

      - name: Run DeepEval suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          CONFIDENT_API_KEY: ${{ secrets.CONFIDENT_API_KEY }}   # optional: cloud reports
        run: uv run deepeval test run evals/ -c -n 4

Notes that earn their place through pain:

• timeout-minutes is not optional. Slow judge calls hang pipelines; practitioner guides consistently recommend explicit 10-15 minute timeouts (DeepEval Review 2026).
• Judge API key comes from CI secrets; DeepEval reads it from the environment exactly as locally.
• CONFIDENT_API_KEY is optional. With it, results also land on Confident AI as shareable reports and score trends; without it, everything still runs and gates locally — nothing else changes (docs).

The flags that matter

deepeval test run (docs) adds eval-specific machinery on top of pytest:

Also worth wiring in: hyperparameter logging, so every score is attached to the app configuration that produced it —

import deepeval

@deepeval.log_hyperparameters
def hyperparameters():
    return {"model": "gpt-4.1-mini", "prompt_version": "support-v7", "top_k": 3}

When scores move, the first question is always “what changed?” — this answers it.

Cost control

Order of magnitude first: a third-party 2026 review measured roughly $2.50-$5.00 per 500 evaluations with GPT-4o as judge, depending on prompt length (aitestingguide.com). A 30-golden suite with 5 metrics is 150 judge interactions — pocket change per run, but multiplied by every PR push, every retry, every developer’s local loop, it becomes a line item. Levers, in order of impact:

1. Cache (-c) everywhere. Unchanged test-case/metric pairs shouldn’t be re-billed.
2. Tier your suites. PRs run the 20-30 goldens that gate; a nightly job runs the full dataset with --ignore-errors and reports. You rarely need 500 goldens’ worth of signal to approve a prompt tweak.
3. Right-size the judge. A cheaper judge (e.g., a mini-class model) for PR loops, the strong judge for nightly/release runs — accepting the docs’ warning that weaker judges score noisier.
4. Local judge for dev loops. An Ollama-served model makes the inner loop free (integration docs); just don’t let a small local judge be the merge authority.
5. Cap verbosity. include_reason=True is worth its tokens for debugging; on stable high-volume suites you can turn it off.

Flakiness: your tests are now probabilistic twice

Non-determinism enters at two points — your app may answer differently per run, and the judge may score the same answer differently per run. Treating either as ordinary test flakiness (retry until green) destroys the suite’s meaning. The mitigations, roughly in order:

• Lock G-Eval evaluation_steps instead of regenerating from criteria each run — the single biggest variance cut for custom metrics (G-Eval docs).
• Set judge temperature=0 where you control the model object (e.g., OllamaModel(..., temperature=0)).
• Leave headroom between typical scores and thresholds. If healthy runs score 0.85 +/- 0.05, a 0.8 threshold flakes weekly; 0.7 catches real regressions only. Use the repeat flag to measure that variance instead of guessing.
• Escalate deterministic criteria to DAG metrics — hard-coded leaf scores, LLM only branches (DAG docs).
• strict_mode=True for absolute invariants (“never leaks PII”) where any violation is 0 — binary scores can’t wobble.
• Aggregate where it makes sense. A gate on “suite pass rate >= 90%” is far more stable than 30 individual all-must-pass gates, at the cost of pytest-level granularity.

Gate PRs or report-only?

The framework is built to gate — assert_test failures block merges, and Confident AI’s own guidance frames this as preventing regressions from reaching production. But gating a PR on a probabilistic score is a commitment: every flake now blocks a human. A decision rule that holds up:

Gate when the metric is stable (you’ve measured variance with repeats), the threshold has headroom, the dataset is curated and small, and the metric guards something you’d genuinely block a release for (faithfulness on a legal-adjacent bot). Report-only when the metric is new, the suite is large/noisy, or you’re still calibrating thresholds — run it, publish scores to the PR or dashboard, let humans decide. A strong pattern: start every new metric in report-only for a few weeks, watch its variance and its correlation with real defects, then promote it to a gate with an evidence-based threshold. Gates should be earned, not declared.

Good practices vs bad practices: the consolidated list

Where DeepEval stops

Fair-witness closing, because no tool review should read like an ad. Independent comparisons (aiml.qa 2026 benchmark) note that DeepEval is metric-driven rather than a red-teaming harness (adversarial testing is promptfoo’s turf, or Confident AI’s separate DeepTeam project), runs one model per test run where promptfoo does native multi-model matrices, and offers limited production observability — which is why “most mature teams run two frameworks in parallel”: DeepEval (often plus RAGAS) at development time, an observability platform (Langfuse, Arize Phoenix, LangSmith) in production. Note the vendor’s own comparison page claims a broader edge — weigh who’s talking.

For the job this series set out — pytest-native, CI-gated, developer-owned evaluation of an LLM application — DeepEval is, as of v4.1.0, the most natural fit for a Python QA team: the skills you already have (pytest, fixtures, CI, thresholds, flake management) transfer almost one-to-one. The new skill is the one no framework installs for you: treating scores as evidence to be calibrated, not truth to be obeyed.