The cyber benchmark turns on price

Security scans over large codebases can become expensive fast when they use frontier models. Malte Ubl, Vercel’s CTO, posted results from a private cybersecurity benchmark and wrote: “GPT 5.6 is best recall/precision but at 7x higher cost per run.” His takeaway was that Kimi K3 is the practical workhorse for continuous cyber analysis, while GPT-5.6 Sol is best reserved for an expensive baseline pass.

The test used Deepsec.sh, an open-source cyber harness for finding vulnerabilities in large codebases. Ubl said the eval ran Deepsec on an undisclosed open-core application at a git sha before many security issues were fixed. Keeping the target private matters because public security benchmarks can quickly be optimized against, especially once model providers and users know the exact bug set.

The ranking in the tweet is unusually concrete. GPT-5.6 Sol is described as the most thorough model, but at more than seven times the price of the runner-up. Kimi K3 gets the best price-to-recall slot. GLM 5.2 is described as 40% cheaper than Kimi K3 while still reaching good recall. GPT-5.5 and Opus 4.8 are framed as viable only with subscription pricing or major API discounts. Fable 5 is said to have a 100% refusal rate for this security-analysis task.

That matters because AI security tooling has to run repeatedly: on pull requests, nightly scans, vendor updates, and post-incident audits. A model that is marginally better but too costly may be useful for occasional baselines, while a cheaper model with strong recall can cover the continuous path. Watch whether Deepsec publishes reproducible suites, whether Kimi K3 maintains low false positives on real repositories, and whether providers tune refusal behavior for legitimate code audit work. The source tweet is available on X.

#deepsec

Kimi K3 beats GPT-5.6 on cost in a private cyber eval