🧪Deepeval
confident-ai/deepeval · homepage ↗
The LLM Evaluation Framework
⭐ Very popular: 18k stars, gaining about 137 a week
View on GitHub ↗repo profile
momentum
durability
bus factor = how many people it takes to cover more than half the commits (6 months). 1 is a solo project; higher means the work is spread across a team. top-author share is the single busiest author's slice of those commits.
since we covered it
why it's a big deal
- Gives teams a way to unit-test LLM applications the way pytest tests code, running assertions against metrics like faithfulness, answer relevancy, and hallucination.
- Covers common failure modes for RAG pipelines, agents, and multi-turn conversations, so teams can catch regressions before shipping.
- Integrates with pytest and CI/CD via a deepeval test run command, letting evaluation gate builds automatically.
under the hood
- Written in Python and runs metrics locally using LLM-as-judge approaches and NLP models rather than a required hosted service.
- Ships prebuilt metrics including G-Eval for custom criteria and DAG for graph-based deterministic scoring, plus RAG, agent, and conversation metric sets.
- Supports component-level and end-to-end evaluation through @observe decorators and an evals_iterator for dataset-driven runs, and connects to frameworks like LangChain, LlamaIndex, CrewAI, and OpenAI Agents.
Radar summary, generated from the project's public sources
star history
- PR#9 7k 2025-05-28
- now 18k + 11k since first covered
curve is sampled from GitHub's star history, plus our own daily readings since we covered it; the dashed stretch is before we first covered it, the solid line since. figures at coverage are the numbers we printed then (approx.), current count is live.
understory
Deep in the understory, shipping hard, still flying under the radar.
- output, commits & releases
- clout, star velocity
output = commits & releases; clout = star velocity, both 0 to 100 monthly indices; the gap where output runs above clout is the understory. The understory →
covered in
-
Pytest-Inspired Framework for LLM Evaluation
similar projects
compare these →- 🩺 iFixAi
leaner, 9k stars
Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
9k ACTIVE - 🧩 TensorZero
Rust
TensorZero is an open-source LLMOps platform that unifies an LLM gateway, observability, evaluation, optimization, and experimentation.
12k OFF THE RADAR - 🖥️ Open WebUI
8.5× the stars
User-friendly AI Interface (Supports Ollama, OpenAI API,...)
149k ACTIVE
comments
Sign in with GitHub to add your blip on Deepeval.