🔭

LLMWatch

GitHub ↗

The Business Problem

Teams ship prompt or model changes without knowing if answer quality quietly got worse, cost crept up, or latency regressed — until a user complains.

The Solution

A plug-and-play LLM observability platform — a two-line callback wraps any LangChain or LangGraph agent and tracks cost, latency, and hallucination rate automatically.

The Result

Fails a pull request automatically if answer quality drops below threshold, before it reaches production
Live dashboard with 6 KPI cards and 4 real-time charts for cost, latency (P50/P95/P99), and hallucination rate
Supports multi-model pricing across OpenAI, Anthropic, Groq, and Gemini with configurable budget alerts

Architecture

Agent Call -> Callback Handler -> Cost / Latency / RAGAS Scoring -> Dashboard + CI Gate

⚙️ Technical Details For engineers & technical reviewers — click to expand

Fails a PR automatically if answer quality drops below threshold — no more shipping a quietly-worse prompt change. A plug-and-play LLM observability platform: a drop-in callback handler wraps any LangChain or LangGraph agent in 2 lines and instantly tracks cost per query, latency percentiles (P50/P95/P99), and hallucination rate via RAGAS. Includes a live Streamlit dashboard with 6 KPI cards and 4 real-time charts, plus a GitHub Actions workflow that auto-runs evaluation on every PR. Supports multi-model pricing (OpenAI, Anthropic Claude, Groq, Gemini) with configurable budget and latency alerts.

LangChain LangGraph RAGAS LangSmith SQLite PostgreSQL Streamlit GitHub Actions Python

Have questions about this project? Ask my AI assistant for details.

Ask AI about this →