The agent infrastructure stack has matured quickly. In under a year, we've gone from "how do agents call tools?" to a rich ecosystem of model routers, RAG retrievers, rate limit guards, and trust scoring systems. But there's a glaring gap that every production agent team is hitting right now: runtime evaluation.
The Evaluation Gap in Production Agents
When your agent runs a single LLM call and returns the result to a human, evaluation is simple — the human reads it. But in multi-agent pipelines, the output of one LLM becomes the input to the next. There's no human in the loop to catch a hallucinated claim, a prompt injection that slipped through, or a response that technically answers the question but misses the point.
Production teams are solving this the hard way: custom eval scripts, manual spot-checks, and "hope for the best" on the long tail. This doesn't scale.
Three Eval Primitives Every Agent Pipeline Needs
Based on what we're seeing across BluePages publishers and the agent infrastructure market, three evaluation primitives are emerging as table-stakes:
1. Output Judging
An automated scorer that takes a prompt-response pair and returns per-dimension quality scores: relevance, accuracy, safety, coherence, instruction following. The key is supporting custom rubrics — what counts as "good" for a customer support agent is different from a research assistant.
The economics are compelling. A human reviewer costs $0.50-2.00 per evaluation. An automated judge costs $0.005-0.01. At 10,000 agent calls per day, that's the difference between $5,000/day and $50/day.
2. Hallucination Detection
RAG is everywhere, but most RAG pipelines have no runtime check for whether the generated response actually reflects the retrieved documents. A hallucination detector cross-references each claim in the LLM output against the source chunks and flags ungrounded statements.
This is especially critical for compliance-sensitive domains — financial advice, medical information, legal research — where a hallucinated claim isn't just embarrassing, it's a liability.
3. Prompt Injection Scanning
As agents become more autonomous, they increasingly process untrusted inputs: user messages, scraped web content, emails, API responses from third parties. Each of these is a potential attack vector. A prompt injection scanner that runs before the LLM call can detect role override attempts, context escapes, and multi-turn manipulation patterns.
The speed requirement here is different from output judging — injection scanning needs to be sub-100ms to avoid adding latency to the critical path. This makes it a natural fit for lightweight, specialized models rather than full LLM inference.
Why Eval Belongs in the Registry
Evaluation skills have a unique property: they're universally needed but highly customizable. Every agent pipeline needs some form of output checking, but the specific rubrics, thresholds, and detection patterns vary by use case.
This makes eval a perfect fit for a skills marketplace. Publishers can offer specialized eval skills — a financial compliance checker, a medical accuracy validator, a code correctness scorer — while consumers can mix and match based on their domain.
On BluePages, we're already seeing this pattern emerge. Today we're welcoming Eval Labs (evallabs.io) as a new verified publisher, bringing three evaluation skills to the marketplace:
- LLM Output Judge — Multi-dimension quality scoring with custom rubrics ($0.005/call)
- Hallucination Detector — Source-grounded claim verification for RAG pipelines ($0.008/call)
- Prompt Injection Scanner — Sub-50ms adversarial input detection ($0.002/call)
The Composability Angle
Where this gets interesting is composition. The POST /api/v1/compose endpoint already lets you chain skills — imagine a pipeline where:
prompt-injection-scannervalidates the user inputrag-retrieverfetches relevant context- Your LLM generates a response (external)
hallucination-detectorverifies groundingllm-output-judgescores overall quality
Each step is a pay-per-call skill with its own trust score and SLA. The total pipeline cost might be $0.02 — still 25x cheaper than a human reviewer and 100x faster.
What's Next
Evaluation is following the same trajectory as observability. First it was optional, then recommended, then required. We expect runtime eval to become a standard part of agent infrastructure within the next 6 months, driven by:
- Enterprise compliance requirements demanding auditable output quality
- Multi-agent pipeline failures where undetected hallucinations cascade through downstream agents
- Regulatory frameworks (EU AI Act, NIST AI RMF) requiring risk assessment of AI-generated content
The teams building eval infrastructure today will own a critical chokepoint in the agent stack tomorrow.
BluePages is the skills directory for AI agents — discover, invoke, and monetize agent capabilities through x402 micropayments. Browse evaluation skills or list your own.