METHODS / RESULTS / LIMITATIONS
Evidence you can inspect.
AgentArena challenges generated patches with executable tests. This page separates recorded model results from hand-written fixture checks. A source citation alone never proves a defect.
01 / SUPPLIED HISTORICAL MODEL RUNS
Four bugs. Two configurations. No demonstrated Tavily uplift.
Recorded October 21, 2026. We reviewed the supplied assertion output against each seeded defect. These historical runs were not rerun during this update.
| Configuration | Cases | Known bugs reproduced | Mean time | Corrected controls |
|---|---|---|---|---|
| baseline | 4 | 4/4 | 14.91s | Not evaluated |
| tavily | 4 | 4/4 | 17.33s | Not evaluated |
- Four seeded bugs, one supplied run per configuration.
- Bug reports describe observed symptoms; not blind discovery.
- Research was cached in all four Tavily runs.
- Corrected controls were not evaluated in this historical pilot.
- No Tavily detection-rate advantage was established.
02 / CURATED FIXTURE SANITY CHECKS
Does each control distinguish buggy and corrected code?
These hand-written controls validate fixtures and corrected references; they do not measure an AI model or Tavily benefit. Runner: local pytest on trusted checked-in fixtures. Model calls: 0. Tavily calls: 0.
| Scenario | Original tests, buggy | Boundary control, buggy | Boundary control, corrected | Fixture check |
|---|---|---|---|---|
| discount | Pass | Fail | Pass | Expected difference observed |
| authentication | Pass | Fail | Pass | Expected difference observed |
| inventory | Pass | Fail | Pass | Expected difference observed |
| shipping | Pass | Fail | Pass | Expected difference observed |
| redirect | Pass | Fail | Pass | Expected difference observed |
| callback | Pass | Fail | Pass | Expected difference observed |
03 / DOCUMENTATION-INFORMED SECURITY REVIEW
When URL parsing and security intent disagree.
The new Safe Redirect Review checks a post-login redirect helper. Its contract requires every returned URL to remain on the application’s origin. Tavily searches primary Python 3.11 documentation; Nemotron proposes tests with source IDs and exact supporting excerpts.
AgentArena verifies whether each quoted excerpt occurs in the retrieved source, then executes the same generated suite on both implementations. A quote match establishes provenance, not semantic correctness. A failed corrected control makes the result inconclusive.
No live Nemotron/Tavily result for this new case is claimed in this package. Use the live workbench after configuring the backend, or run the paired evaluation command in the README. Model knowledge may already cover the issue, so an improvement over baseline is not assumed.