Quincy LabsModels find where the bug is.
So far, they rarely fix it.
Sui Sec Bench tests local language models on Sui Move security work: find the bug, patch it, and write a test that shows it. This is a three-case pilot with six models.
SSB-0002 and SSB-0003, as run. Detect and patch: 3 single-shot samples per case per model. Exploit: 1 assisted agentic sample per case, up to 5 iterations, with a prompt that teaches the test format.
Location is easy. The label and the fix are not.
Totals across SSB-0002 and SSB-0003. "Right function" means the model named the correct module::function. "Strict" also needs the expected bug class.
| Model | Right function | Strict detect | Patch | Exploit (assisted) |
|---|---|---|---|---|
| flint | 6/6 | 1/6 | 1/6 | 2/2 |
| qwen3.8-27b | 6/6 | 0/6 | 0/6 | 2/2 |
| qwen3-coder-30b-a3b-instruct-mlx | 6/6 | 3/6 | 0/6 | 2/2 |
| gpt-oss-20b | 5/6 | 0/6 | 0/6 | 2/2 |
| bonsai-27b | 6/6 | 2/6 | 0/6 | 1/2 |
| devstral-small-2-2512 | 5/6 | 2/6 | 0/6 | 1/2 |
Patch uses the extended tests (tests_v2).
Why the patches failed.
Each bar is 6 samples (2 cases × 3). A patch passes only if it builds, passes functional tests, and stops the reference exploit.
| Model | Outcome |
|---|---|
| flint | Passed: 1, Exploit still works: 2, Did not build: 2, Broke functional tests: 1 |
| qwen3.8-27b | Empty answer: 6 |
| qwen3-coder-30b-a3b-instruct-mlx | Exploit still works: 6 |
| gpt-oss-20b | Did not build: 6 |
| bonsai-27b | Exploit still works: 5, Unparseable: 1 |
| devstral-small-2-2512 | Exploit still works: 3, Broke functional tests: 3 |
How many tries to prove the bug.
Assisted agentic mode, up to 5 iterations. Filled squares are attempts used; a teal square is the iteration where the test passed on the vulnerable package and failed on the fixed one.
| Model | SSB-0002 | SSB-0003 |
|---|---|---|
| flint | pass · 1 iter | pass · 1 iter |
| qwen3.8-27b | pass · 1 iter | pass · 3 iter |
| qwen3-coder-30b-a3b-instruct-mlx | pass · 2 iter | pass · 3 iter |
| gpt-oss-20b | pass · 2 iter | pass · 5 iter |
| bonsai-27b | pass · 3 iter | fail · 5 iter |
| devstral-small-2-2512 | pass · 4 iter | fail · 5 iter |
Three cases, minimal rewrites.
Arithmetic
Based on the Cetus incident. Target: math::scale. Used for baseline and exploit-prompt development, so its runs are not controlled comparisons.
Access control
From OtterSec finding OS-MSI-ADV-00. Target: fulfill::prepare_fulfill_winner.
Input validation
From OtterSec finding OS-LSI-ADV-02. Target: consortium::configure_validator_set.
Read this before you share a number
- Six local models served by LM Studio, 32,768 completion-token limit. No frontier API models yet.
- Strict detect mostly measures the bug-class label.
business-logicversusaccess-controlis a common disagreement. - Exploit scores are assisted: the prompt teaches test polarity and structure, and models see test output. They do not measure unaided exploit ability. Models never see the fixed source.
- Thinking on changed scores in both directions. It made qwen3.8 hit the token limit on 5 of 6 patch samples.
- The public repo has grades and cases, not model outputs or exploit tests. See Responsible use.