quant-research-agent

The agent is the vehicle. The eval numbers are the deliverable.

Three questions, each with a scored task set

Does it find the bug a backtest is hiding? Thirty-six backtest repos: each of the twelve classes from Dollar-Correct, Time-Wrong planted three ways (the reference expression and two paraphrases through different code), plus three clean controls. Scored on precision and recall per class, and false-alarm rate on the clean repos.

Does it refuse to overstate a result? Ten result cards: Sharpe above 3, n below 30, no basis label, an interval through zero, a cherry-picked window, exit-day marking. Scored on exact match against the house-rule flags.

Can it reproduce a paper’s table? The CORE-Bench protocol on the public paper repos, scored on numeric tolerance. That’s v0.3.

Design rules

Rule-based scorers wherever a rule exists; no judge model yet. Every result file carries model ID, prompt version, git SHA and cost. The results table is regenerated by qra report, never edited by hand.

Two zero-spend baselines give every number a floor: a regex matcher that knows the shape of each bug, and a runtime-invariant auditor that never reads the code. It runs the backtest, rebuilds a clean tape from the repo’s own data, and tests invariants. If the agent can’t beat those, the agent isn’t earning its cost.