It’s all in the name
Mythos finds a curl vulnerability
Yes, as in singular one.
[…]
My personal conclusion can however not end up with anything else than that the big hype around this model so far was primarily marketing. I see no evidence that this setup finds issues to any particular higher or more advanced degree than the other tools have done before Mythos. Maybe this model is a little bit better, but even if it is, it is not better to a degree that seems to make a significant dent in code analyzing.
This corroborates my own experience that I reported over on the fediverse:
Having now seen Mythos vulnerability scan reports, I can say first-hand that the “OMG the sky is falling” narrative was hype. I'm not saying there’s nothing there, it does find things, even things here and there that humans have overlooked, and some fraction of its findings may be scary ones – but the only way in which Mythos is a step change over previous models is the signal-to-noise ratio of its results.
So far the only people who seem to remotely corroborate Anthropic’s own claims are the Firefox folk, who had 271 vulnerabilities identified during initial evaluation. A whole bunch of vulnerabilities, to be sure – but even they conclude as follows:
Encouragingly, we also haven’t seen any bugs that couldn’t have been found by an elite human researcher. Some commentators predict that future AI models will unearth entirely new forms of vulnerabilities that defy our current comprehension, but we don’t think so.
Neither do I. And their article’s central premise is that rather than unearthing some inexhaustible subterranean ocean of insecurity, the models are merely surfacing more from a limited well of vulnerabilities which will simply dry up faster as a consequence:
The defects are finite, and we are entering a world where we can finally find them all.