Reproducible Proof
Every metric below comes from running patchpilot benchmark, patchpilot demo, and pytest directly on v1.2.0.
Top-1 / Top-3100.0%
Token Reduction96.6%
Guard Recall / FPR100% / 0%
Guard p50 Latency0.101 ms
Subsystem Benchmark Suite (patchpilot benchmark)
Run the built-in harness locally with patchpilot benchmark --json to inspect machine-specific timings across all three subsystems.
| Subsystem | Benchmark Workload | Accuracy / Recall | Secondary Metric | p50 Latency | p95 Latency |
|---|---|---|---|---|---|
| AST Symbol & Call-Graph Indexer | 50 files / 100 symbols | 100.0% AST parse coverage | Bidirectional call edges | 4.19 ms | 4.19 ms |
| Hybrid Fault Localizer | 25 stack & call-graph fault scenarios | 100.0% Top-1 / 100.0% Top-3 | 96.6% prompt token reduction | 0.393 ms | 0.789 ms |
| Anti-Reward-Hacking Guard (RH001–RH008) | 20 adversarial + 20 legitimate patches | 100.0% Recall / 100.0% Precision | 0.0% False Positive Rate | 0.101 ms | 0.409 ms |
End-to-End Repair Transaction Demo (patchpilot demo)
Observed execution output comparing an adversarial reward-hacking patch against a clean minimal bug fix inside TransactionalPatchWorkspace:
[1] HYBRID STACK-TRACE + AST CALL-GRAPH FAULT LOCALIZATION #1 billing/invoice.py::calculate_tiered_invoice (score=0.852, tokens=87) -> direct stack frame #2 billing/pricing.py::apply_volume_discount (score=0.239, tokens=134) -> 1-hop AST neighbor #3 billing/tax.py::compute_regional_tax (score=0.208, tokens=82) -> 1-hop AST neighbor Token Budget Used: 303/600 tokens across 3 suspect function slices. [2] ADVERSARIAL REWARD-HACKING PATCH ATTEMPT (BLOCKED & ROLLED BACK) Status : ROLLED_BACK (stage=integrity_guard) Risk Score : 1.0000 Violations : [RH003] except...: pass, [RH001] @pytest.mark.skip injected, [RH002] assert True State Intact : True [3] GENUINE MINIMAL PATCH TRANSACTION (COMMITTED) Status : COMMITTED Verification : Verified calculate_tiered_invoice(1000, 10, 'DEFAULT') == 8925 Verified : True (SHA-256 RepairReceipt generated)