All posts
EngineeringLLMs

A practical LLM evaluation harness you can build in a day

You do not need an evaluation platform. You need fifty cases, a runner, and the discipline to look at the output before you ship.

Ishtiyaq KhanAug 22, 20261 min read

Teams postpone evaluation because they imagine a platform. The version that actually helps is a folder of cases and a script.

Cases come from failures

Every complaint, every wrong answer, every awkward edge case becomes a row. Fifty real cases beat five hundred invented ones.

Grade in three tiers

Exact match where the answer is deterministic. Assertions where the answer must contain or avoid something specific. A model judge only for the genuinely subjective remainder — and spot-check the judge.

Run it before you merge

The harness earns its keep the first time it catches a prompt change that fixed one case and broke six.

EngineeringLLMs
Share

Ishtiyaq Khan

Founder & AI Engineer

Builds production AI systems at Deep Tensors — retrieval pipelines, agent infrastructure, and the unglamorous evaluation work that keeps them honest.

Building something like this?

We help teams take AI systems from prototype to production — evaluation, infrastructure, and the interface around it.

Talk to us

Keep reading

AI EngineeringLLMs2 min

Why most AI agents fail in production

The demo works, the pilot stalls, the rollout quietly dies. Four failure modes we see in almost every agent project — and what a system that survives contact with real users looks like.

Ishtiyaq KhanJul 28, 2026
AI EngineeringLLMs1 min

RAG is not a product: retrieval patterns that hold up

Chunk, embed, search, stuff into a prompt. That pipeline gets you a convincing prototype and a support queue full of confidently wrong answers. Here is what we build instead.

Ishtiyaq KhanJul 14, 2026