Agentic Search vs Vector Search for Coding Agents: We Ran the Eval — Braintrust

Same accuracy. Four times the cost. What happened when Braintrust pitted vector search against agentic search for a coding agent. Jess Wang, developer advocate at Braintrust, starts with the basics of evals: why shipping on vibes fails, the four parts of an eval (dataset, task, scorer, experiments), and how observability and evals form a flywheel. Then she runs a real one. Using fix PRs from Microsoft's TypeScript Go repo, she has Claude Code find the buggy code with either agentic search (grep, find, reading files) or vector search over embeddings, scored by the repo's own test suite. Both reached the same accuracy, but vector search cost about four times as much because chunks lacked the surrounding context and the agent kept searching. In this talk: • What evals are made of, and why they're a team sport • Building an eval dataset from real merged fix PRs • Forcing Claude Code into vector-only search, and fixing orphaned subprocess traces • Why vector search found nearby code but missed the connective tissue SPEAKER Jess Wang, Developer Advocate, Braintrust LinkedIn: https://www.linkedin.com/in/jessica-wang-a1aa15ba/ X: https://x.com/daRubberDuckiee YouTube: https://www.youtube.com/@theRubberDuckiee LINKS Braintrust: https://www.braintrust.dev/ CHAPTERS 0:00 Intro 0:42 What evals are 1:02 Don't ship on vibes 1:27 What evals answer 2:22 The sycophancy rollback 3:02 The four parts of an eval 4:32 Comparing experiments 4:52 Querying traces in plain English 5:47 Observability 6:02 The eval flywheel 7:47 Evals are a team sport 8:42 The experiment: agentic vs vector search 9:32 How vector search works 10:42 How agentic search works 11:22 Building the dataset 12:17 Running both in Claude Code 13:17 Fixing orphaned traces 14:37 Scoring with the test suite 14:57 Results: same accuracy, 4x the cost 15:22 Why vector search fell short 16:27 Why it cost more 16:47 Caveats and best practices Recorded at the AI Engineer World's Fair 2026 in San Francisco. Subscribe for more talks from the engineers building with AI. AI Engineer: https://ai.engineer YouTube: https://www.youtube.com/@aiDotEngineer X: https://x.com/aiDotEngineer LinkedIn: https://www.linkedin.com/company/aidotengineer/ #LLMEvals #CodingAgents #AIEngineer