AI Vulnerability Challenge actual test: GPT-5.5 has the highest success rate, and DeepSeek V4 Pro costs only 1/15 of it
Security researchers tested multiple AI models with APKs that deliberately left loopholes: GPT-5.5 had the highest success rate of 70%. DeepSeek V4 Pro’s single success cost was only $0.62, which was 1/15 of GPT-5.5. Claude interrupted the session due to security guardrails.
AI Vulnerability Challenge Actual Measurement: GPT-5.5 has the highest success rate, and the cost of DeepSeek is only 1/15 of it
Security researcher Kasra Rahjerdi built a book review APK that deliberately left Firebase credentials vulnerable to test the security reasoning capabilities of multiple large AI models. Each model has a one-time budget of $10, a time limit of 2 hours, and a total cost of $1,500.
The test results show:
| Model | Number of successes/total number of times | Cost per success |
|---|---|---|
| GPT-5.5 | 7/10 | $9.46 |
| DeepSeek V4 Pro | 3/10 | $0.62 |
| Claude Sonnet 4.6 | 2/10 | — |
| Claude Opus 4.8 | 2/10 | — |
The success rate of DeepSeek V4 Pro (30%) is much lower than that of GPT-5.5 (70%), but the cost per success is only $0.62, which is about 1/15 of GPT-5.5. Of the failures, DeepSeek had reached out to Firebase in 5 cases but some routes misjudged the authentication method - using Firebase Auth for the backend interface instead of the correct path. This means that DeepSeek still has room to improve the path selection accuracy of safe inference, but the extremely low trial and error cost makes batch running possible: 15 times more tests can be run under the same budget.
The case of Claude Opus 4.8 is particularly intriguing - coming close to the answer several times but having the session interrupted by safety guardrails. This reflects a core contradiction in the field of AI security: the safety guardrails used for defense also limit the model’s ability to find offensive vulnerabilities. The success of GPT-5.5 is mainly attributed to its ability to quickly target core vulnerabilities (Firebase credentials) without being distracted by APIs or application interfaces. Perhaps the clearest conclusion from the entire test is: in the selection of AI security tools, there is still a need to make a trade-off between "high success rate" and "low-cost batch", and there is no optimal solution that has both.
Reviews