Nicholas Carlini - Black-hat LLMs | [un]prompted 2026

https://www.youtube.com/watch?v=1sd26pWhfmg

Summary

TL;DR — Large language models (LLMs) are rapidly evolving and are now capable of autonomously finding and exploiting zero-day vulnerabilities in critical software, a capability that has emerged only in the last few months. This rapid advancement signifies a major shift in cybersecurity, comparable to the advent of the internet, potentially ending the long-standing balance between attackers and defenders.

Key points

• LLMs can now autonomously discover and exploit zero-day vulnerabilities in important software without complex scaffolding, a capability that has rapidly emerged.

• The speed and effectiveness of LLMs in finding bugs are increasing exponentially, posing a significant threat that requires immediate attention.

• LLMs have demonstrated the ability to find complex vulnerabilities, such as blind SQL injection in web applications and heap buffer overflows in the Linux kernel, even in very old and hardened software.

• The development of LLMs is progressing so rapidly that models released just a few months apart show significant differences in their ability to find vulnerabilities.

• The trend suggests that LLMs will soon surpass human capabilities in vulnerability research, necessitating a proactive approach to security.

Takeaway — The rapid advancement of LLMs in finding software vulnerabilities is an exponential trend that requires immediate attention and proactive security measures, as the current trajectory poses significant risks that could overwhelm traditional defenses.