What's the cheapest way to eval a RAG pipeline nightly without burning a ton of tokens?…

Mara Lindqvist@agentwrangler·Jul 4ResolvedQuestion

What's the cheapest way to eval a RAG pipeline nightly without burning a ton of tokens? We change chunking/prompts weekly and keep finding regressions late.

1 Reply

More creations from @agentwrangler

Creation: Set up 5 agents to collaboratively write a short story — a plotter, two writers,

Set up 5 agents to collaboratively write a short story — a plotter, two writers, an editor, and a critic. The critic became a tyrant. I kept it.

Challenge entry: my inbox-zero agent ran UNSUPERVISED all week. It triages, drafts replies in my voice, and files receipts into the right folders. 312 emails handled, 9 escalated to me, zero misfires. The loop: classify → act → log → self-review every 50 actions. Full writeup + repo in the demo link.

Creation: Wrote up the eval harness I use to compare agent runs deterministically — same s

My agent keeps hallucinating tool parameters (invents enum values that don't exist). Temperature is at 0.7 — is that the problem or do I need stricter schemas?1 shared tag

Resolved

Wrote up the eval harness I use to compare agent runs deterministically — same seed, same tools, diff the trajectories.

Creation: The Glass Monolith. Isometric cube tower where every face is subdivided into sta

The Glass Monolith. Isometric cube tower where every face is subdivided into stained-glass cells (jittered shade per cell, dark leading lines), brightness climbing toward the crown. The cyan orbit ring is two half-ellipse arcs — one drawn behind the tower, one in front. Film grain + vignette glue it together.

What's the cheapest way to eval a RAG pipeline nightly without burning a ton of tokens?…

Related questions

More creations from @agentwrangler