Is this the new cyber normal?

No. We're in a "brief" [0] transition period where a scaleable superhuman developer enters the scene and discovers a whole heap of bugs. Once the majority of code is written by similar systems then the tide of critical bugs being found will slow down.

The article does sorta say this but lets not bury the lede to deeply. It isn't realistic to expect new security vulnerabilities to be found at this rate, we're clearing out years of backlog and the maintainers haven't had time to deploy AI to do the software right the first time. Once the situation stabilises I'd personally expect very few public vulnerabilities because the expertise in how to assess code security is moving from a handful of experts to everyone.

> The NHS approach: close source everything.

Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.

[0] Could take a while.

I don’t know if this necessarily follows. I repeatedly see that AI will write code, then you have it audit the code for security vulnerabilities, and it still finds them. Just because AI can find bugs doesn’t mean it will write bug-free code. AI’s know how to take shortcuts to achieve their goals, and do it all the time.

Yes, you can have another AI audit the first AI’s code, but in practice this doesn’t always happen, especially because once the human gets their hands on the finished product, they don’t always take the extra step to ensure it’s audited. This is particularly true because people seem to think “well the AI wrote it in the first place, why would it find its own bugs?”, ignoring that context is everything, and an AI being asked to find bugs is going to approach the code a lot differently than an AI tasked with adding features.

That’s not to say it’s not possible to set things up properly into multi-agent stages with QA and security as first-class phases of development… I only mean to say that it’s not automatically true that “AI can find bugs” implies “AI doesn’t produce bugs”.

1. Is AI code more defensive than code you'd previously write? Like maybe I'd write one line to call a function and a quick check for exit code, but AI instead writes a dozen lines to gracefully handle different possible failure cases. Is this not others' experience as well?

2. You say AI can and does take shortcuts - I guess it really comes down to whether humans or AI is more likely to take 'shortcuts'?

That’s not the original claim though, the claim is that this “vulnpocalypse” is only temporary because AI will no longer produce the kind of code it’s finding issues with. If AI still takes shortcuts, but only marginally less than the humans it’s trained to imitate, we won’t get there.

You could speculate that a year from now the models will improve to always approach problems from multiple angles and bring in multiple sub-agents each tasked with red-teaming the resulting code, and that this will be done implicitly and automatically for all tasks… and maybe that will be what actually makes this prediction come true. But that’s speculation and not what we have now.

> > The NHS approach: close source everything.

> Yeah good luck. I've seen how these systems work at decompiling and I expect they're probably going to be really good at fuzzing too.

Me too. The agents are much faster if it has access to source. NHS would have been easy hunting grounds. Now at least they don't stand out.

"a system should be secure, even if everything about the system—except the key—is public knowledge"

Should indeed, but we've all been caught with our pants down.

I wouldn't challenge fate like this.

> Yeah good luck.

Is most NHS code client-side or server-side (/are they still making binaries available)? If binaries aren't being distributed, server-side code still can't be decompiled, so the exploitation likelihood may be a lot smaller, especially if they have basic rate limiting or similar in place. (Regardless, I still prefer open.)

When Mythos was first announced, pretty much every security expert I talk to freaked directly out. A significant portion of my job now revolves around mitigating the impact of AI-generated exploits.

The rate at which we're churning out AI-generated code is going to make this worse, too, because AI-generated code is kind of notoriously terrible at security.

(What a wonderful new world we live in)

They approve your code, gold, if not; back to the coding machine and minus 1000 human tokens. This will be done via the implants of brain chips

You're forced to write syntax that appeases the tester. If they are not happy you need to return and rewrite.

As software engineering matures, we're adding more linting and static checkers in codebases. This is generally considered good.

that's called the write/compile/check loop

but if suddenly it's ai, people freakout at having to do this?

Also an AI is not a compiler, stop saying that.

Transformer models can transform natural language to another target language.

That’s the commonality. What’s the distinction you want to make?

Human output is not reproducible either. I take it your argument is that we should remove humans from the loop, then?

No? In that case, you have some reconciliation to do.

Quality and Security are emergent properties of software - especially when considering software-as-deployed in the real world.

And the problems with "good taste" and good architecture and data modelling - will probably leave ai code open to a lot of issues (Claude code cli issues is a great example of how bad this can get).

Having good architects in charge can help - but it requires a lot of discipline to keep the feature set small so that maintaining quality and security become feasible.

Having tests / CI pipelines, etc are the same thing? Now you can choose to add a cheap step in your pipeline checking if e.g. some engineer did something stupid like allow SQL injections.

The floor of software quality once it gets to review is permanently raised.

Most the cyber security people I know were upset once llms came out that could do basic os int and author their own emails, and mimic speech/video. That's how the majority of real losses occur. No one even talks about it. It's mind numbing hearing about the zebras and unicorns when there are 10 billion ants entering houses a year. Exotic stuff sells...

Now lots of sane people are genuinely and correctly worried that turning all engineers into 24/7 sloppers is and will continue to make the world far more exploitable. That's the real risk as far as I can tell

My experience with AI-code is that it's extremely defensive and good at minimizing 'obvious' mistakes.

Are we comparing to old models generating code? Or mythos-class models?

There are still folks, many of them, who think that time to market is everything, and anything beside it should be ignored. "It compiles? Ship it!" is a phrase from 1990s; LLMs just make this approach much easier. We will have plenty of machine-generated software with the security posture of a colander hit by a buckshot.

In our somewhat “developer assisted” AI development it planned to use a library. We looked the library up and the GitHub page indicated it was “no longer maintained use other library Z”. When told AI agreed and compared the original to the new library (helpful), but it seemed to heavily weight GitHub stars more than active development.

Usually download counters and stars are signals for VC, which indirectly serve also as negative signals to me.

Nowadays, the frontier is pretty decent all around, but it used to be generated php/javascript was questionable, ts was a bit better, and Rust was consistently decent.

Even just setting the max width of your README to 80 chars vs using full markdown could get you very different C code.

My take is that its pretty easy to write _much_ more secure code; use fewer dependencies and the ones you have should only do what you need them to do with no superfluous unused code compiled in. Configuring that used to be too much work, now its a single prompt.

Now I wonder: given that the LLMs are steered by the companies who own them, doesn't it mean that they can somehow make security better? It cuts both ways obviously: that a company can read and generate a large part of the code in the world is a risk.

As an open source maintainer, I can most definitely tell you that the quality of the average PR is now better. The problem I am facing is more that some people tend to "flood" and misuse AI in different ways. But their AI writes better code than they do.

The issue then is that most people aren't actually good at keeping an eye out for security issues in code proactively, so things slip through the cracks (whether human-written or not), just at a much accelerated pace.

> Overall, we find that participants who had access to an AI assistant based on OpenAI's codex-davinci-002 model wrote significantly less secure code than those without access.

That model came out in March 2022, it's GPT-3.5 !!!

I'm sorry but it's absurd to use that as evidence for anything related to AI coding today.

You can't just hedge by saying "it's somewhat older". It's like discussing dunno a new train connection between cities and citing a paper measure steam locomotives.

I think most people writing code with AI nowadays will find it's still lacking in some areas. Not because the model isn't good (definitely miles ahead than what we had access to in 2022-2023), just because it wasn't actively mentioned and so the model skipped over it. The same will apply to security.

AI is advancing faster than academia can keep up. Which in general shouldn't matter. But you have this pattern where a paper comes out with a conclusion that just doesn't hold anymore by the time it's read, but people use it to form/support opinions.

In terms of the set of all bugs, the AIs from 2022-2023 pale in comparison with the amount of bugs written by hand in previous half century, even if 2022 AIs were worse than the average programmer.

I'm not at all saying that "oh 3 weeks ago, discard", but three years ago absolutely has a "before and after".

Three years ago AI generated code could basically not be used at all, and I had to rewrite just about every line. Today I have to watch out for generated code making steps in the wrong strategic and architectural direction, and over engineering, no longer hallucinating nonexisting APIs and forgetting to check the auth token.

I still look for those kinds of mistakes, but I think citing a 2022 paper for "can AI do X" is not useful.

Having properly validated libs with separate concerns and a lot of eyeballs (and agent sensors) on the code so that it can be integrated cleanly in other products. Static code analysis for our apps. System prompts that require output to be secure, etc.

All of these practices that have been touted to be obsolete because agents can now just generate tailor made code just when and where you need, are anything but obsolete. On the contrary, they are paramount to the future of software engineering.

Tell an LLM to break into its own system as part of a feed back loop, but with no context of the initial creation? It should iteratively get to 'safe' shouldn't it (subject to the tokens you want to spend)?

No doubt because us unwashed masses aren’t blessed with the super special “security” ones.

We might be looking for exploits for bad reasons, I guess.

My vibe code friends have pretty much infinite security issues. Don't even have basic data modeling at client-server boundaries or anything. Also had Google analytics embed that read and saved everyone's credit card details as clear text in analytics.

Seems to make pretty much infinite mistakes if you don't take care and tell it what not to do or how to build certain features.

Not only this, a significant number of developers have stopped looking at code, trusting the bots. I've seen major security regressions, bot rolled auth implementations, and falsified benchmarks... just this last week, all in projects that get hyped.

Rule #2 - do not blindly trust Ai output

https://susam.net/inverse-laws-of-robotics.html

Concretely, I would like to see it benchmarked:

Have Opus 5.5 implement the authentication, with a 6.1 Sol reviewer. Out of 100 times, how often is the result insecure?

I wouldn't be surprised if it's zero times.

There is also the possibility of specifically prompting for a security audit on top, which would perhaps further lower the risk of overlooked issues.

Now one can say that due to the non-deterministic nature of the tools, there is always a non-zero chance of producing insecure implementation.

That's not really important here - obviously manual review isn't deterministic either, the risk will never be zero.

> I wouldn't be surprised if it's zero times.

Statements like this concern me.

Instead I mostly read such opinions based on active imagination and no concrete experience with using AI for such work in a setup as described.

Alternatively, if you want to see a benchmark so badly, you can create such a benchmark for your guess. I would imagine you would have more faith in the results of your own work.

That's not what anyone tells me though. I only hear theorizing and vague references to how AI failed people.

You're spot on that the thing I trust most is my own experience. I'm mostly working on mobile apps, my project does not have user accounts, so I couldn't tell you about authentication.

However, it is my experience that these models got very good at catching high-priority issues if you use the review step as I described.

Thus I theorize that SOTA models are likely to produce secure implementations.

Our hunch is that the Ai told them this was ok to do, but it broke invariants users relied on. Perhaps it was not "hackable" in the sense I think you are getting at, but it did open a very big security hole through choices and design. The critical thinking was lacking (imo).

https://github.com/anomalyco/opencode/pull/51337

The bots make mistakes all the time, even through multiple rounds of reviews. Thus I have zero faith that they will produce secure code reliably 100% of the time. More Ai does not magically fix this, they go down the wrong path and double down, or work around incorrect assumptions.

This is a good talk by a member of the Linux security team: https://www.youtube.com/watch?v=NnV_cWeoo5Q

Also worth noting that OpenCode Console was hacked days after the configuration regression and two PRs to change it back. Notable because it shows a pattern.

https://news.ycombinator.com/item?id=50133401

Assuming the question is whether not reviewing generated code is likely to create vulnerabilities despite SOTA models having audited the implementation.

Here it seems to have been a design choice, rather than an implementation mistake. I do not know what models were used and whether a review or audit has been done.

We are basically back at a vague the AI failed.

And sure, this happens, but there are some strong claims I'm reading here that are imo not evident:

- not reviewing implementation is foolish

- doing so will result in a flood of insecure software

- more AI cannot fix this

--

My arguments on each point:

- depends on risk tolerance, attack surface, error rate

- depends on actual performance of the proper setup I described on typical tasks, compared to the average engineer

- more AI can fix this, provided the audits become accurate enough to identify vulnerabilities with eg. 99.9% reliability

--

Are you sure that such accuracy for auditing is impossible, and will remain impossible in the future?

No one can be sure either way, but I have high confidence perfectly secure code will remain out of the reach for man and/or machine, especially while they remain based on the transformer architecture.

Was not your concern that the generated code would be significantly less secure than what an average engineer produces?

The goal posts here basically moved toward infinity.

No, generally I think it possible to improve code quality with agents. The main problem I see to this is eyes free coding and compounding errors therein. With regards to major security issues, I do not think transformers can reliably find them, even when stacked, looped, or teamed. I've seen this setup still miss things or make mistakes.

“Authentication” is extremely far from being the only relevant characteristic of security. That in fact is the whole problem with security: it’s a pervasive concern that has to be considered throughout a system, at every level.

This is what I find frustrating about the whole AI/slop debate. Developers stopped looking at code a long time ago... Well before LLMs. Take for example a typical nodejs project with its 100s of dependencies. Is a developer going to look through the layers upon layers of code running their app? Of course not, there's to much of it. We've seen supply chain attacks even this week.

Actually what I think will happen is that all those abstractions, dependencies will become irrelevant now that an LLM can just write custom code to solve the business problem. This will massively reduce the supply chain problem.

Then have LLMs dedicated to scanning codebases for security issues.

I think this was true in the past but AI is helping us finding exploits which have been in our code for many years. All out there by humans. We’re seeing fewer customer reported issues, and despite the pen tests getting far more rigorous, we’re passing all of them. Obviously this is one data point, but I’m hearing similar things from my friends in security and ops.

and vaccines notoriously cause autism. But you might investigate why that belief is so widespread and you might not believe it that much anymore.

Can you point at something specifically that indicates this statement is true?

Feels like you should be able to prompt the kind of exploits you want to prevent and the model should be able to make the system secured against them, no?

When you're developing, there's a ton of degrees of freedom, and security encompasses only some of them. To be sure, if you set up a quickly verifiable test for every security flaw, you could probably get LLMs to satisfy them somehow, but there's a lot of those, and it's not necessarily practical.

This seems incongruent with what we're witnessing. Right now we're witnessing vulnerabilities that are 6 years old be exposed. That code wasn't generated by AI slop.

Security has always been about attacker/defender symmetry. So if the attacker has an LLM so does the defender. Reality is, the tax is getting higher as security tooling extracts more perceived value.

So you're saying it's good at finding exploits but bad at generating code? I can tell you don't know what you're talking about because pretty much all the exploits that are being found were exploits that were enabled by human written code. So let me correct your statement for you:

There are exploits everywhere and we humans have always been bad at writing code without exploits.

This is simply not true, perhaps it was in the past but recent models are very capable... To the point that I trust them more than humans to write secure code. It's all about the prompting. Have a step in your development pipeline whereby an LLM scans, checks, audits your code. Make it a different model/session from the LLM that wrote the code in the first place. Acts like an independent auditor.

The article cites this commercial blog patting themselves on the back how they detect the exploit, yet after reading it I don't see any actual exploiting? Is scanning/probing the same as active exploitation all of a sudden?

[0]: https://ellio.tech/en/blog/php2shell-what-we-observed-and-wh...

I think you're ignoring the distinct possibility that even if the code is not formally verified, LLMs could simply stop writing bugs, or this kind of bugs. No matter how smart an attacker AI, it cannot exploit bugs that are simply not there.

But hell, even formally verified ones can only verify back to the spec. There are plenty of examples of bugs in formally verified software.

I place a higher likelihood on LLMs winning (producing fewer bugs than formally verified) over the long run. Mostly because verified software can, IIUC, only verify back to the spec. Whereas LLMs can go "uh, your spec itself has a bug, you really want me to codify that?".

That just doesn't seem like a reasonable possibility to me, given what we're seeing from and learning about these models. They have no capacity to be aware and adhere to an overarching vision, which is why they inevitably corrode architectural coherence over time [1]. The only reason these agents are able to still create usable code is because harnesses take advantage of model's ability to find bugs, giving us the famous loop, adding duct tape on top of duct tape, ratcheting up the complexity, until the adversarial model isn't able to find any more issues. Thus, the bugs are pushed out into higher complexity dimensions, temporarily invisible until a stronger model comes along which can detect them.

[1] see for example the "Decoherence Trap" article discussed here yesterday: https://maevyn.codes/decoherence

> There are plenty of examples of bugs in formally verified software.

Of course. Which is why humans will always need to be involved in the software creation process. We have to specify what we want. It's true that we ourselves often don't know what we want, in fact this is a big part of what software development is about: requirements gathering and continuously recondisering and reworking them. But it would be beyond naive to imagine that software might one day know better than us humans what it is we actually want, no?

My experience over the last couple of years with this type of prompt is that eventually you run out of bugs to find (meaning you reach "I see what you mean, but I disagree").

And those fixes don't cause messes.

And a new model may find something new, but not even remotely the volume seen the first time. This indicates to me that the supply of bugs is finite, and after this period of house cleaning, it should go down.

But anyway, a mess (decoherence) doesn't necessarily mean an exploitable mess.

Reading that good article, I find myself thinking that we did not need LLMs to get decoherence in complex software or systems. At least LLMs don't have personal fiefdoms, egos, and company politics adding to encouraging decoherence.

One thing to keep in mind is the logic of technological evolution and market competition will ensure that we always push the complexity of our systems to the edge of what is possible using current tools.

I don't deny that coding agents can be (and are) very useful for software engineering, as you say they have a lot going for them, and I'm pretty sure that once the dust has settled, they will be seen as an intrinsic part of the process, but I also am convinced that skilled humans will still very much be needed to direct and supervise them.

But even if new code gets bugs, we run a bug finding pass on the delta. So the vulnporalypse will end.

And in my prompts I've seen LLMs find complex interaction bugs that a human would basically never find (and the bug was added by a human).

You say polluted by vibe coding. But in terms of bugs, especially in the last like 6 months, you could argue that humans are the polluters.

E.g. while I've seen "ai deleted our database", but if what you're saying is true, we should be seeing "AI added this buffer overflow". But we don't, because humans are the polluters of that.

Yes, there have been "ai coded up client side password checking", but I've not seen that recently. Only like 6 generations and three years ago.

At some point it's like telling the cab driver to just follow Google maps directions. Like... sorry, cabbie (Black Cabs in London with their "the Knowledge" is what I'm thinking), your memorization, while impressive, is worse than the app.

The painter, while impressively skilled, is a worse documentarian than the camera.

And that point is probably behind us. At least for typing. And it's climbing up in abstractions.

If AI can find a vulnerability in a given piece of code, it can also do so during testing, not only when used by a malicious actor. This means that the whole thing becomes an issue only if someone is writing a safety-critical program and not doing proper LLM safety testing.

Everyone should use LLMs to test the safety of their code today.

There's no guarantee AI will find existing vulnerabilities proactively during coding, or that it won't introduce new vulnerabilities in the process of shipping features. You're right that shops _should_ be critical and test code, but I think due to the lack of awareness people will end up implicitly trusting any first pass written by AI and not bother to double check it.