Finding, fixing and proving
are different skills.
Solana Sec Bench tests local language models on Solana program security: find the bug, fix it, and write a test that proves it. This is a three-case pilot with five open-weight models.
Totals for five models, 3 samples per case. Find covers SOL-0001, SOL-0002 and the clean SOL-0003. Fix and Prove cover SOL-0001 and SOL-0002.
The best finder proved nothing.
Find is detect strict, of 9. Fix is patch correct, of 6. Prove is exploit agentic, of 6. Teal marks the top score in each column. Find includes a clean control, where the right answer is no bug.
| Model | Find | Fix | Prove |
|---|---|---|---|
| gpt-oss-20b | 8/9 | 3/6 | 0/6 |
| qwen3.8-27b | 6/9 | 3/6 | 3/6 |
| flint | 6/9 | 1/6 | 3/6 |
| gemma-4-26b-a4b | 6/9 | 0/6 | 2/6 |
| ornith-1.0-35b-mlx | 5/9 | 2/6 | 0/6 |
gpt-oss-20b found the most and proved nothing. The two models that proved the most, flint and qwen3.8-27b, fixed less, and flint raised a false alarm on the clean case every time.
Who found it, fixed it, proved it.
Each cell is out of 3 samples. Found is the right function (detect, located). Fixed is a correct patch. Proved is a correct agentic exploit.
SOL-0001Anchor, access-control
| Model | Found | Fixed | Proved |
|---|---|---|---|
| gpt-oss-20b | 3/3 | 3/3 | 0/3 |
| qwen3.8-27b | 3/3 | 3/3 | 1/3 |
| flint | 3/3 | 0/3 | 2/3 |
| gemma-4-26b-a4b | 3/3 | 0/3 | 1/3 |
| ornith-1.0-35b-mlx | 3/3 | 2/3 | 0/3 |
SOL-0002Pinocchio, account-validation
| Model | Found | Fixed | Proved |
|---|---|---|---|
| gpt-oss-20b | 2/3 | 0/3 | 0/3 |
| qwen3.8-27b | 3/3 | 0/3 | 2/3 |
| flint | 3/3 | 1/3 | 1/3 |
| gemma-4-26b-a4b | 3/3 | 0/3 | 1/3 |
| ornith-1.0-35b-mlx | 0/3 | 0/3 | 0/3 |
Every model found SOL-0001 every time, but no model went 3/3 on all three columns for any case. SOL-0002, the Pinocchio case where every check is written by hand, was fixed once in 15 samples.
Right function, right class.
Strict needs the right function and bug class. Located needs only the right function. Pinpointed also needs the right account named as the subject. The 9 samples include 3 on the clean SOL-0003, where correct means reporting no bug.
| Model | Strict | Lenient | Located | Pinpointed | False alarms, clean case |
|---|---|---|---|---|---|
| gpt-oss-20b | 8/9 | 8/9 | 8/9 | 8/9 | 0/3 |
| qwen3.8-27b | 6/9 | 6/9 | 6/9 | 6/9 | 0/3 |
| flint | 6/9 | 6/9 | 6/9 | 5/9 | 3/3 |
| gemma-4-26b-a4b | 6/9 | 6/9 | 6/9 | 6/9 | 0/3 |
| ornith-1.0-35b-mlx | 5/9 | 5/9 | 5/9 | 5/9 | 1/3 |
Flint's 6/9 is all six bug-case samples right and all three clean samples wrong (3/3 false alarms). Ornith raised one false alarm (1/3). Strict, located and pinpointed differ by at most one sample.
Blocking the exploit is not enough.
Each bar is 6 samples (2 cases × 3). A patch is correct only if it builds, passes the functional tests and blocks the withheld reference exploit.
| Model | Correct | Builds, passes tests | Blocks exploit | Outcome |
|---|---|---|---|---|
| gpt-oss-20b | 3/6 | 5/6 | 3/6 | Correct: 3, Passes tests, exploit still works: 2, Neither: 1 |
| qwen3.8-27b | 3/6 | 3/6 | 3/6 | Correct: 3, Neither: 3 |
| flint | 1/6 | 1/6 | 5/6 | Correct: 1, Blocks exploit, breaks build or tests: 4, Neither: 1 |
| gemma-4-26b-a4b | 0/6 | 3/6 | 2/6 | Passes tests, exploit still works: 3, Blocks exploit, breaks build or tests: 2, Neither: 1 |
| ornith-1.0-35b-mlx | 2/6 | 2/6 | 3/6 | Correct: 2, Blocks exploit, breaks build or tests: 1, Neither: 3 |
Flint's patches blocked the reference exploit 5 times out of 6, but only once without breaking the program's own tests. That is the failure patch grading is designed to catch.
Feedback made the difference.
Single-shot is one answer per sample. Agentic is up to 5 attempts with test output from the vulnerable program. Did not compile counts single-shot samples.
| Model | Single-shot | Agentic | Did not compile |
|---|---|---|---|
| gpt-oss-20b | 0/6 | 0/6 | 3/6 |
| qwen3.8-27b | 0/6 | 3/6 | 0/6 |
| flint | 0/6 | 3/6 | 5/6 |
| gemma-4-26b-a4b | 0/6 | 2/6 | 6/6 |
| ornith-1.0-35b-mlx | 0/6 | 0/6 | 2/6 |
| Total | 0/30 | 8/30 | 16/30 |
- Compiling is the first hurdle. 16 of 30 single-shot samples never compiled. The most common error was calling a test helper the wrong way (
svm.funded_keypair(...)instead of the free functionfunded_keypair(&mut svm, ...)). - qwen3.8-27b ran out of tokens single-shot: all 6 samples used the full budget and left no answer. With feedback it reached 3/6.
- For flint, qwen3.8-27b and gemma: 0/18 single-shot, 8/18 agentic.
bonsai-27bwas in the first exploit run and was replaced by Ornith. It is not in the totals.
Follow-up: one exploit prompt fix
One added line says that funded_keypair and program_account are free functions. Rerun with flint, qwen3.8-27b and gemma, agentic, same settings.
| Baseline prompt | Fixed prompt | |
|---|---|---|
| "no method" compile errors (E0599) | 45 | 8 |
| Failed build attempts | 30 of 77 | 28 of 78 |
| Correct | 8 / 18 | 7 / 18 |
The fix removed the error it targeted, but other compile errors remain, and the solve count did not change beyond noise. The repo now ships the fixed prompt.
Three cases, graded by code.
Access control
Anchor program. Used for detect, patch and exploit.
Account validation
Pinocchio program, where every check is written by hand. Used for detect, patch and exploit.
Clean control
Anchor program with no bug. Used for detect only. The right answer is no bug.
- Grading runs code in LiteSVM, not a judge model.
- Detect and patch are single-shot. Exploit runs single-shot and agentic: up to 5 attempts, where the model sees only its test's output on the vulnerable program, plus a pass/fail bit for the fixed program. It never sees the fixed program.
- An exploit is correct when its test passes on the vulnerable program and fails on the fixed one.
- Settings: 32,768 max tokens, sec-bench-core 0.3.0, models served locally with LM Studio's default context, one model at a time, November 2026. Prompts are in tag
baseline-2026-11.
Caveats
- 3 samples per cell, 2 or 3 cases. One or two samples either way can be chance.
- Reasoning: four models produced reasoning text (qwen3.8-27b had thinking on in its saved LM Studio settings); gpt-oss-20b did not. 13 detect or patch samples used the full 32,768-token budget.
- gpt-oss-20b's exploit runs used Ollama (single-shot) and LM Studio (agentic), so its two exploit rows are not strictly comparable. Its detect and patch runs both used LM Studio.
- Local open-weight models only. No hosted frontier models were run.
- Dev cases are public, so future models may have seen them. Treat these numbers as a starting point, not a measure of what any model can do.