The bug was apparently fixed in open source repo a week earlier, this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week (the commit message even included "Reported by Xint.io" and I'm guessing skilled black hats would know about Xint)<p>Given that in this case the workaround is as simple as disabling http2, there is honestly no reason why F5/Nginx or any of the reporting teams couldn't at least put out a PSA to disable http2 for the time being before basically telling the whole wide world this exists and how to exploit it.<p>Especially for software like Nginx which powers a huge part of the internet.
Disabling http2 can be a big deal. My service is primarily made up of gRPC APIs. Not an option for me to disable http2.
Luckily I'm not using nginx at the moment - but I easily could have been.
If you're using grpc, http2, etc you're so much better off using Envoy proxy<p>You basically get to piggyback off of a large chunk of big tech's infrastructure and security work. Obviously everything has the risk of security flaws (and I kind of hate that default Envoy builds ship with half-baked filter impls) but it's a generally more mature ecosystem than projects with a lot of smaller/legacy users like nginx
I'm not 100% informed on this, but it seems very unbelievable to me that Envoy could be considered more robust or widely used than nginx at any point in history including today. I started using ConnectRPC instead of GRPC almost entirely because it didn't leave me subject to (what I consider to be) an unnecessary dependency in Envoy.
I guess I would just say, wordpress is also more widely used than many other projects, but that doesn't necessarily make it more robust or secure than the projects Google, Cloudflare, and security companies develop.<p>The main reason I see to not use Envoy is that it's very annoying to set up and build, and requires familiarity with things like grpc/proto/low level networking to get right. But if you're using grpc and http2 I think you're generally past that "infra overhead" threshold already and it's going to better serve your needs.<p>I have pretty much only worked at big tech or my own infrastructure startup, and did some systems/infra work at Google using Envoy for a while. I can't say it's 100% better than nginx but I can say that I've evaluated nginx and a lot of software like it (ie with a similar customer base, ecosystem, and history; like coturn) over the years for my personal use, and every time I've immediately been annoyed by constant security issues or poor http2/L4 networking support that I don't see when I use what Netflix/Google/etc use.
Envoy probably has the advantage that it’s only a reverse proxy server, unlike nginx which has other uses too like a general-purpose web server with optional modules.
You can implement custom http filters for envoy or use this for general web serving: <a href="https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/http/http_conn_man/local_reply" rel="nofollow">https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/h...</a><p>The documentation, DX/UX, and discoverability for these features are pretty bad. But they work well if you know how to do it.<p>Implementing a custom HTTP filter itself is actually easy, I can share examples I did in the last year for a voice inference stack if anybody is interested. The problem is just that the devloop is very painful, because Envoy doesn't make it easy to know why something isn't working (this is actually most of what I did with it at Google, tons of testing/verification and compatability infra for relatively simple http stuff) and the builds take a very long time.
On all my small projects (eg rails apps) the reverse proxy to the app servers is the same server expected to serve the static assets directly. In fact the majority of bot nonsense never goes through the proxy due to path rules.<p>Envoy is only attractive at a higher scale.
Does Envoy support things like rate limiting?
Yes [0]. For a single node you can just use this, it works like the Golang/language rate limiters you might already be familiar with [1]<p>In a multiple node/distributed architecture you need away to coordinate rate limits across nodes and not try to fight the CAP theorem.<p>There is a way to do this [2] [3] and although it sounds complicated the shape of the solution (delegating to a third party service that aggregates and provides client-level rate limit metrics across multiple proxies) would still be the same any way you cut it.<p>[0] <a href="https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/listeners/network_filters/local_rate_limit_filter" rel="nofollow">https://www.envoyproxy.io/docs/envoy/v1.15.2/configuration/l...</a><p>[1] <a href="https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/extensions/filters/network/local_ratelimit/v3/local_rate_limit.proto" rel="nofollow">https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/extensio...</a><p>[2] <a href="https://www.envoyproxy.io/docs/envoy/v1.15.2/intro/arch_overview/other_features/global_rate_limiting" rel="nofollow">https://www.envoyproxy.io/docs/envoy/v1.15.2/intro/arch_over...</a><p>[3] <a href="https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/config/ratelimit/v3/rls.proto#envoy-v3-api-msg-config-ratelimit-v3-ratelimitserviceconfig" rel="nofollow">https://www.envoyproxy.io/docs/envoy/v1.15.2/api-v3/config/r...</a>
I don't know what nginx security vetting looks like, but just being older isn't a security guarantee.
I know Google spends a lot of manpower on making envoy secure.
As a gRPC proxy Envoy is indeed more widely used than Nginx: it's used as <i>the</i> gRPC proxy load balancer on Google Cloud.
Well that is of course because it is the _only_ proxy that seems to speak gRPC fluently. Anyone who's ever tried to configure bare nginx to serve gRPC knows this. And yet, I would wager that many stacks use both.<p>This requirement turns me off of gRPC entirely. Maybe Envoy truly is more secure than nginx, but it's still a whole-ass additional attack surface. It seemed to me that eliminating it if possible was best.
I wonder if being prepared to switch the reverse proxy you use in whim is a good strategy in this climate.
I am using Envoy :)
What WAF would you suggest to use with envoy? Have been working with Coraza and tbh satisfying is something else. So i wonder what someone that recommends envoy would recommend as WAF for it
Personally I serve all static, public content off a CDN and only serve dynamic content through Envoy after it passes AuthN/AuthZ.<p>I use JWT-based sessions + a remote JWKS (which is remote static content that Envoy can cache) + either a third-party IDP or dedicated login service (this being the only application logic besides Envoy's routing exposed to the public Internet). That way, every envoy auth check is immediate and stateless.<p>In this setup you don't really need a WAF because all unauthenticated requests redirect to signin or 403, which get strict rate limits on login. An abusive peer that sends a ton of traffic anyway gets blocked at the network/firewall level outside of Envoy. The tradeoff is you need to decide that you aren't going to do unauthenticated SSR, and I will admit this pretty complicated to set up in its entirety.
If you're using httpd just fyi - there was a recent RCE vuln for apache that was also http2 specific.
It's patched so anyone can upgrade to the latest nginx to fix it, thankfully. I agree that just shutting off HTTP/2 is probably non-ideal for a lot of web services.
Neither Ubuntu nor Debian have released patches yet, so if you're using one of these mainstream distributions and install nginx through the distro's package manager, you can't easily "upgrade to the latest nginx" yet. You can however disable HTTP/2 for now.
Both the Debian and the Red Hat packaging systems are awfully convenient ways to build software from source. It has to be, that's package maintainers do daily. They will install all dependencies, apply the distribution specific patches, build, and package in one operation.<p>Should your distributor fall of the face of the Earth and you still need to patch software, it is often the fastest way to take the source package, put the security patch in the patch directory, and run.<p>There is more it, such as versioning it between the last release but below the official upgrade so it's convenient when the offical one comes along, and preparing keys if you require signatures, but that's basically it.
At least for Debian, http2 is disabled by default, and so not affected.<p>You would have had to enable http2 <i>and</i> have a buggy Linux kernel to be vulnerable.
Not quite. The kernel bug is only the worker-to-root step. Code execution as the nginx worker through the two HTTP/2 CVEs needs nothing but HTTP/2 enabled. Three of the five CVEs don't need HTTP/2 at all: one is HTTP/3 with an upstream, one is resolver + variable proxy_pass + subrequests, and one is the mail proxy with auth_http.
Luckily we could not see your face during writing this, as I’m pretty sure no serious person in the industry can say this with a straight face… Maybe Theo de Raadt, but there it is a running joke I guess.
Another reason not to couple your software versioning to Ubuntu and Debians' package managers' schedule
You can also add nginx.org repos and install the updated versions from there. <a href="https://nginx.org/en/linux_packages.html" rel="nofollow">https://nginx.org/en/linux_packages.html</a>
Debian has it in unstable now.
<a href="https://tracker.debian.org/news/1825662/accepted-nginx-1306-4-source-into-unstable/" rel="nofollow">https://tracker.debian.org/news/1825662/accepted-nginx-1306-...</a>
FWIW, for nodes that don't have packages, I compiled from source under /opt and deploy scripts check equal or higher version so it will auto switch back when the new packages are installed. Seemed less risky than disabling http 2.
It makes more sense to install services like nginx via the official containers. In the modern world, the OS is for base services only.
The distributions can customize packages with sensible defaults. Debian usually does, and is incredibly convenient. They usually disable features most people don't use, and allow you to easily switch them on if you choose.
Agreed. At a previous job all the important stuff came from repos that always shipped latest.
[flagged]
This looks like it was patched in Ubuntu on the 12th?<p><a href="https://ubuntu.com/security/notices/USN-9090-1" rel="nofollow">https://ubuntu.com/security/notices/USN-9090-1</a>
This Ubuntu security notice is about an Apache vulnerability, not nginx.
> Several security issues were fixed in Apache HTTP Server.<p>Um, what does this have to do with NGINX?
It was patched but the docker containers were a few hours behind, I nearly had to build from source on a few servers, until I noticed they were pushing all the tags as I was patching a few hours after the release was cut
Definitely, but unfortunately I use Debian for my servers so.
Oof.<p>One idea that might work for you is client certificates if those are only internal APIs. I don't know about you, but I have a lot more confidence in openssl than the people who use openssl.<p>I turned off http2 a few years ago <a href="https://news.ycombinator.com/item?id=41486452">https://news.ycombinator.com/item?id=41486452</a> and absolutely did not regret it.<p>I do regret every experience I've had with gRPC though. I feel for you.
> this could've been abused by anyone smart enough to monitor repos like this for exploitable bugfixes for a week<p>And here lies the risk of AI agents in cybersecurity: the speed of processing for monitoring, scanning, identifying security fixes in open source, writing exploits for affected versions, and attempting infiltration, is a matter of minutes now.<p>It is not "by anyone smart enough". It is by "anyone ill intentioned enough". They don't need to be super smart anymore.
These commits weren't visible until today. Yesterday, the last public commit was Nov 9.
This is the reason why the oss project i work for is not publishing commits anymore outside of releases but people are freaking out over this.<p>Cant win.<p>Seriously reconsidering if open source has a future nowadays
I'm sure things will stabilise in the near future.<p>That said, I just moved from a Linux based job to a Windows based job and it's been nice not having to be panicked about having update most of the technologies that my last job used. As an aside I did try in the past to make sure we were always updating dependencies but my manager was bothered by the amount of work that created. I'm watching that team now take forever to deal with these issues because they're too far behind to just version number and rebuild.
The future is in closed source ecosystems on a modified Harvard architecture
What happens when somebody tries to contribute based on the last-published commit? Do you merge/rebase their contributions for them?
Haven’t figured out how to deal with contributions going forward…<p>The issue is that we get dozens of PRs, generating a bunch of code is trivial but reviewing and merging it and making sure it doesn’t break in interesting ways is still pretty hard.<p>Personally I lean towards not accepting code contributions anymore, but accepting design documents for features and l bug reports etc
How about a strict anti LLM use policy for contributors? It adds what's basically a "proof of human work" challenge to opening new PRs, and effort required to review human PRs usually roughly scales in proportion to the amount of effort required to make human PRs. And human contributors who take time to make a PR manually are probably more likely to become longer term community members and not just drive-by contributors<p>It's not a perfect solution of course, it just takes OSS back to the fairly broken state it was pre-AI. Still, it might be an improvement and should be at least considered.
We do have a LLM disclosure policy.<p>I’m not sure how we could enforce an 0 LLM policy in practice though…
Amounts to ‘won’t be getting contributions‘ which is maybe not that bad at this point on the timeline
I mean, win or not, it has nothing to do with some people freaking out, and that certainly have nothing to do with whether open source have a future.
> Seriously reconsidering if open source has a future nowadays<p>*In C or C++.
Until we don't replace UNIX clones, Khronos APIs, LLVM or GCC with something else.<p>C and C++ will stay around for a very long time, even with AI powered systems.<p>If anything, maybe that is a call that even if implementing compiler backends is hard and time consuming, bootstraping compilers, and new approaches to OS design, are the only path away from such dependencies.
Rust is on its way. Vulkan is no GL, and Gallium it's something else over classical MESA under SGI/Sun.<p>Legacy 32 bit users with NetBSD and Hyperbola can survive a little with Go based tools/small browsers/mail/usenet/IRC clients and some web browsers with Bubblewrap, but not forever.
Rust being here matters,<p><a href="https://www.khronos.org" rel="nofollow">https://www.khronos.org</a><p><a href="https://vulkan.org/events/vulkanised-2027#program" rel="nofollow">https://vulkan.org/events/vulkanised-2027#program</a><p>The only thing closer to it is WSGL.<p>Also there are much more operating systems than those examples.
rust isn't invulnerable.
A tank isn't invulnerable either, but of someone starts shooting at me I'd rather be in a tank than on a 50 year old bicycle.
Fun fact, Russian soldiers ride on top of their tanks and APCs because its safer than being inside of one during an assault. Doesn't mean you are wrong, but there will always be trade-offs. There is no such thing as perfectly secure.
Tank desant is practiced because there isn’t room in the tank to fit extra people. It’s very much not safer outside of a tank than inside! It’s actually the worst of both worlds, because you don’t have the armor protection but are highly visible. The only advantage it gives over infantry assault on foot is mobility. Which is why it’s generally only practiced when there aren’t sufficient APCs/IFVs available
Not exactly true. BMPs are death traps, sure, and sitting on top of those instead of in them is rational given the risk of burning to death inside of one or being obliterated by a mine splitting the hull in half.<p>But the only reason to sit outside a tank rather than in one is because riding on a vehicle is safer than walking through an open field. Tank crews are not sitting outside of their tanks, and they don't have seats inside for passengers.
> Russian soldiers ride on top of their tanks a<p>That stopped being true for at least 3-4 years (with APCs maybe still used until two or so years ago, tanks are still used but as "turtles" or/and "porcupines" [1], no way to have soldiers riding them). Drones have conquered (almost) everything.<p>[1] <a href="https://miro.medium.com/v2/resize:fit:1400/format:webp/1*CQr9nugR22W52EAQZqEgZg.jpeg" rel="nofollow">https://miro.medium.com/v2/resize:fit:1400/format:webp/1*CQr...</a>
Fun fact, these specific types of problems aren't possible on a JVM. Yet somehow Java isn't the right language to write a webserver in. These problems are possible in Rust and C/C++. At least its "hard" to have this type of problem in Rust.<p>Security is hard and is mostly about the number of engineers and amount of time spent looking at a specific implementation. When writing code, do you think about the problems of leaking information due to different operations taking specific amounts of time? That's an actual concern when writing a prod service that uses the network. And it has to be balanced against scalability and issues caused by DOSes. It isn't trivial and it isn't binary...there are always trade-offs. Nothing will ever be perfectly secure and usable and scalable.
> Fun fact, these specific types of problems aren't possible on a JVM.<p>CVE-2026-114938 would like a word. <a href="https://calif.io/java4shell" rel="nofollow">https://calif.io/java4shell</a><p>Yes, I know, different issue. But saying Java would be better for webservers just two days after a serious JDK RCE is pretty ironic :P
Application servers, and apparently the idea was so good that nowadays there are startups selling the same idea with cloud based WebAssembly workloads.
How bad are these CVEs?
Could any of these be related to some of the recent outages (not claiming, asking) in the last week, coinciding with when this bug was fixed in the repo? Monitoring such repos seems a given for nation states.
Source: <a href="https://github.com/nginx/nginx/commit/b89c4100bf6762a45e4a6463eaf4b70368277c94" rel="nofollow">https://github.com/nginx/nginx/commit/b89c4100bf6762a45e4a64...</a>
HTTP/2 is usually the source of most of these bugs in webservers of all kinds
These days even the "disable X" is enough to get everybody looking in the right direction though. i.e. such hints are (given enough tokens) already RCEs
Yes, fixing open source software is at most an obfuscated announcement.
Oh everybody worth worrying about is monitoring this on every major repo already.
The patch gap.
[flagged]
idk, the license tells me that there's a good reason why you shouldnt expect that.<p>Now, if you were paying for support from a vendor, that's a different matter.<p>"THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED."
Small consolation but HTTP/2 is not enabled by default in Nginx and "only" 12% of servers run it (still huge at Internet scale) <a href="https://w3techs.com/technologies/segmentation/ce-http2/web_server" rel="nofollow">https://w3techs.com/technologies/segmentation/ce-http2/web_s...</a>
The exploit also requires sending 128GB of data to the server. On slower connections this doesn't fit into 1 hour window of the nginx's default connection time cap.
Not enabled by default in upstream nginx yes, neither in most prepackaged nginx (e.g. debian), but enabled by default in some, for example kubernetes community ingress-nginx (the popular but deprecated one)
I still hate that they deprecated it, just because it gives anyone with write access to the cluster write access to the cluster.<p>Chainguard still updates their fork, by letting AI try to apply and backport the nginx patches, and I'm using that at the moment while switching to gateway API, but I'd have preferred if ingress-nginx was still maintained (the one by F5/nginxinc for example is still being maintained, just not entirely compatible).
Hey so, does anyone know of a good tutorial on getting in to woodwork?
As a matter of fact, I know three such books (all free): <a href="https://lostartpress.com/products/the-anarchist-s-trilogy" rel="nofollow">https://lostartpress.com/products/the-anarchist-s-trilogy</a><p>Well, actually four: <a href="https://lostartpress.com/products/the-stick-chair-book" rel="nofollow">https://lostartpress.com/products/the-stick-chair-book</a>
I took a quick look at The Anarchist's Tool Chest and noticed in Chapter 20 a short vignette about an expert woodworker named Troy that seemed... oddly familiar.<p>According to the author, Troy could do the work of three people. That would make him a 3x woodworker, I guess. And he was facing automation in the woodworking industry.<p>"But Troy decided to compete with the machines on their own turf. He tried to become a one-man factory. He succeeded in that. But the most efficient one-man factory cannot compete with a factory driven by a hundred men.<p>So no matter what Troy did, the prices for work (good or otherwise) declined. Ikea’s well-designed kitchens and Value City’s cheap junk made it impossible to survive and live well.<p>So Troy struggled with his business. And that’s nuts. He should have had more business than he could handle. But like many professional cabinetmakers I know, it became harder and harder for him to make ends meet every year. His cabinetmaking business is closed, and the last time
we talked, Troy was working for his wife’s successful tax business.<p>Who is to blame for this? We are.<p>We have become a culture that is obsessed with price more than any
other attribute of the things we buy. It doesn’t matter if the item is ugly,
poorly made or constructed of materials that cannot be recycled. All that
matters is if the price is low enough.<p>Because the price of our household objects has hit rock bottom, if an
item breaks or starts to look dated, we can throw it away and buy some-
thing else. For the first time in human history, manufactured furniture is
shockingly inexpensive.<p>So it’s no wonder that artisans are exiting the craft. It’s difficult to
compete against furniture that costs less than what you pay for your raw
materials.<p>Lesson: People cannot imitate machines and defeat them."
The lesson is that most people aren't discerning consumers in multiple fields.<p>Appreciating good design, quality craftsmanship, having taste in food.. having these "elevated" tastes in multiple areas is very rare for humans. Take a look at a musician - they can tell you precisely what makes a quality violin, but they might not care that they eat microwave frozen dinners. You compound that across the population and the artisans will always lose out to the mass-produced slop because it's impossible to <i>naturally</i> get a critical mass of people to care about that quality curve. You have to sell them on it.<p>Troy struggled because he failed at marketing, not production.
Or even single fields. I've purchased my parents the "nicer" version of things and they inevitably end up back on some crappier version. Japanese knives? Nope, still use the ones that barely cut. Ubiquiti wifi system that covered their whole house? Nope, dad unplugged something, got frustrated, and the cable company came over and put them back on the old crappy system. Nice speakers? Nope, they listen to music on their crappy TV speakers.<p>I think I've learned that most people don't want to put in any effort to learn or appreciate things that may be of nicer quality. Those Japanese knives didn't do any good for them because they didn't spend time learning how to use them properly. In fact one of them used them <i>improperly</i> and ended up needing stitches. So in the end the crappier version is no different than the nicer version, because crappy and nice are about the potential of the item, and most people don't see potential. Does this Ikea dresser hold clothes? Great, who cares that the other one is of nicer wood, the joinery is done by hand, and it'll last multiple generations.
On the other hand, there's an accessibility to products of higher median quality to more people than would have existed even only five generations ago.
cf. John Henry [0]<p><a href="https://en.wikipedia.org/wiki/John_Henry_(folklore)" rel="nofollow">https://en.wikipedia.org/wiki/John_Henry_(folklore)</a>
I tried, didn't succeed so I am back.
I felt your post was off topic. But maybe it was on purpose because we are getting bored of so many vulnerabilities being discovered by AI security tools. So your post is important because of that.<p>I see a lot of interest in woodworking on HN. Is that because Jesus was a carpenter and HN folks are interested in woodwork because of the larger context and history shaped by Christianity? What is your motivation for woodwork? Is that because we see a concrete finished project instead of vague intellectual stuff which has no seeable boundaries?
Becoming a carpenter, and sentiments of "I'm going to live in the forest" or "I'm going to be a farmer" are usually expressions of fatigue with technology, and a desire to be free of their problems; thus, an expressed urge to pursue a career where technology is not a major component. It can also sometimes carry a desire to have one's labor output be tangible and physical (i.e. a worked wood object) rather than abstract, as so much programming/operations/etc. work often is.<p>The comment you replied to is shortcutting the joke, assuming the common riff of becoming a carpenter is well known, and jumping ahead to asking about tutorials for woodworking as part of that process.
Piffle.<p>Preposterous, even.<p>That said, my RHCE CISSP self is going to go outside and spend some time with my flock of chickens and guinea fowl.<p>While Claude writes Ansible to update the fleet to use nginx's deb repo.
wood, unlike logic and machine, is firm, yet yielding
This is a beautifully autistic comment.
Well, I am usually the person in the room who does not get a joke. I am so scared of get-togethers that I actively try to avoid it. Also, they ask me to dance. I am terrible at that. I am not autistic but don’t get the drift of many jokes. I am human and not a bot. Guess that’s what a bot would say. But still I am human.
The thing with LLMs is that they try not to sound robotic unless explicitly prompted to do so - it's a great irony that comments that sound robotic are more likely to be written by real people at this point.
I didn’t get the joke myself either, so thanks for asking. I first thought “woodworking” was used as a synonym for hacking and clicked on the links with the book recommendations just to see they’re about actual woodworking.<p>Not sure why others downvoted you, but you have my upvote on it.
Botistic more like it
Just take a second to check out the account if you're going to say stuff like this. It's not just you either, so this goes for everyone making these comments.<p>The account is from 2017. There are enough comments before LLMs were a thing. Given his karma score, unlikely to be a bot.<p>> Hey so, does anyone know of a good tutorial on getting in to woodwork?<p>Some people really don't get these type of things.
I think most LLMs would have gotten that woodworking joke.
[dead]
This feels like an AI bot reply
As programmers, we do see ourselves as Jesus, yes.
Ha, sadly there's a huge long line of developers who pivoted this way already. Plus now that you can fire up an MCP for a CnC machine, that art will be "ruined" too!
If only it paid the bills
AI is burning down the internet. This is insane.<p>Are we going to discover in a year that every server online this month had a rootkit installed? There’s got to be thousands who are too small time to wipe and reinstall.
AI is securing the internet. This is great. You should assume professional hacking gangs and state actors discovered (at least some of) these vulnerabilities months and years ago.
Both are true. The problem is security is a much harder job than attacking.<p>Worse patching adds instability into a system, especially when you don't have much time to test.<p>Also in the past attackers were willing to sit on exploits longer and not do things with them, now they have to act fast before someone else finds it, leading to more events faster wearing your security and patching teams out, and making users life suck.<p>Things are going to get much worse before they get better.
Yes, things will get worse before they get better. Attackers have the advantage initially and get to pick a lot of low-hanging fruit, but the advantage will shift to defenders as code becomes more heavily tested and verified.
>Worse patching adds instability into a system, especially when you don't have much time to test.<p>Survival of the fittest I'm afraid will build security here. The teams that barely ever patch or don't have "time" for maintenance are going to get burned. The "we haven't updated in 7 years so things are stable" teams can't survive.
Attack the attackers.
> The problem is security is a much harder job than attacking.<p>Nope.<p>Let say Im running a vulnerable Ngnix server. All that would take to stop any and all attacks is a simple firewall that sees anything that even remotely looks like shell code (i.e non ascii characters, ROP patterns, strings of noops, e.t.c) close that connection,blacklist that ip.<p>>Also in the past attackers were willing to sit on exploits longer and not do things with them,<p>Also nope. The reason why you didn't see many exploits is because they are mostly useless unless you have resources. You can take a gander through all the CVEs that AIs are finding. 99% of them are either DDOS with some possibilty of remote code execution only under very particular conditions, or require already user level code execution to get to root.<p>There are a few sprinklings of truly bad ones, but even for those, a good portion of them that exist on critical systems are protected by secondary things, like firewalls that detect shell code and close the connection and ban the IP for some time. True network security will never be not needed, as humans are still the most vulnerable vector, and if you don't have systems in play that can counter bad actors, then you are asking to get hacked.<p>All of this AI finding exploits is basically fear mongering that is going to be used to sell AI products that secure systems.
Except, human in the loop in control - blue team is not defending red team fast enough that most the CVSS v4.0 Critical CVEs aren't easily patched in coordinated fashion. Many times, almost impractical or impossible to patch at all, with decades of intertwined dependency tree and their ecosystem.<p>Collectively, we owed about 40+ years of tech debt on non-secure code (e.g. automotive industry, industrial environments) that will be torched in the next few years. Luckily, AI will generate code faster enough to make that plausible.
AI <i>provides the capabilities</i> to secure the internet, maybe, but the internet is secured when site owners choose to secure it.<p>That's not happening at nearly the same pace, both for assymetry between what it takes to break something and what it takes to fix it, but also because that fix must navigate bureaucracies. Also, the leaders need to actually care, both of which are hurdles. Bureaucratic navigation and consensus finding has not sped up <i>at all</i> with AI.<p>Compare that to independent script kiddies with $600/mo Claude subscriptions able to find 0-days in an hour and write a worm to exploit it in four.
Those who don't keep up & pay technical debt will get eaten. There is no escaping this.
Yes but the barrier is infinitely lower. If you were used to running servers just for fun, that kind of threat was probably not too much of a concern. Now it very much is because it can be exploited at a much larger scale.
Yep. In recent discussion this came up, and my analogy was that LLM will cause a similar effect to widesread proliferation of TLS. It was easy to snitch on people before TLS had been deployed everywhere. Not so much after that.
> > AI is burning down the internet.<p>> AI is securing the internet.<p>No doubt there are points of view where either one, or the other statement, or both, or neither, are true. You can really go full Schrodinger and put a probability distribution on everything.<p>But in this day and age, whatever is happening, the better question is:<p>Is it good for me and, beyond me, for the world in general?
AI is also making it much easier to attack, likely more than it's helping secure things
it's a timing thing - eventually we'll reach an equilibirium<p>it was previously expensive to attack, so people spent less money on defense.<p>It's cheaper to attack, and so people will increase their budget on defense. But it's a lagging cause & effect situation
You're a nanometer away from understanding why labs have powerhouse internal models that they only allow partners to use.<p>Even Kimi K3 was only shipped as a lobotomized version of it.
the idea that internal models can protect the Internet like this is so dumb IMHO.<p>A huge fraction of devices and servers are on old versions, not actively updated or managed in any way.
This internal models stuff remains marketing BS afa I am concerned. Smart enough to be a junior engineer and steered toward the right tools and behaviors within a context is more than enough. Places with "super models" do nothing to demonstrate they wouldn't have done better to use the resources on relative brute force within the security domain. They have no sales pitch for good results that just needed Legos and cash.
Man, is it aggravating to read fantasies like this, as someone who's trying to juggle a full-time job and hosted public services for other people at the same time.<p>There are two internets. One of them is open, less-commercial, and largely built and maintained by people whose first job is not the work they do to maintain this internet. That group includes a ton of open source software, millions of small servers scattered around the world (virtually private or otherwise), and is generally best-effort.<p>The other internet is fully commercialized, often ad-driven, fully integral to the data broker economy, and centralized behind a few companies that pay as few people as possible to satisfy the occasional PR statement that, "we take security seriously and...".<p>The next stable equilibrium here is <i>not</i> "we get a secure and open internet". The next stable equilibrium here is, "we have no open internet, and the internet is now managed by just a few companies."<p>We're already halfway there. Google and Microsoft combined already account for somewhere around 30% to 40% of email traffic. 30% of all websites are now behind Cloudflare. A few companies dominate social media traffic. Amazon and Shopify combined are half of US e-commerce.<p>Every single short-response-window exploit like this just pushes a few more percentage points of traffic into a handful of companies. Cheer for that, I guess.<p>When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.
I wonder whether part of the solution is not using the same software as big tech. For them rapid development is worth extra maintenance. Things like the efficiencies of http2 over 1.1 actually matter. Maybe the rest of us should be using BSDs or different Linux distros (Alpine? Void? Slackware?), different web servers, turn off features we do not need (e.g. not disable http2, or at least not allow both 1.1 and 2 as that is clearly extra attack surface).
I tend to agree. We should really be focusing on simplifying instead of using these patched up, massive, and complex software stacks. For me it's starting to become a non-negotiable. I know lots will disagree on the basis of presumed practicality, though.<p>I don't even care if it's all written in Rust, essentially none of my problems require software boasting multi-millions of lines of code. I guess my browser and OS? But even those I question greatly.<p>It's funny, I never thought I'd be one to compile my own kernel, but as of this year I've been maintaining an extremely stripped down Linux base for my NixOS servers and desktops.<p>Feels like my end game is going to be on some micro kernel that I compile all my software for, lol.
This is somewhat on those lines:<p>This is somewhat on those lines: <a href="https://news.ycombinator.com/item?id=50556533">https://news.ycombinator.com/item?id=50556533</a>
My sites are now sat behind my own custom reverse proxy. It's missing tons of features, which I guess I can now call a security benefit, and it pretends to be nginx just because I thought that would be quite funny. I've debated open-sourcing it but I a) don't want to maintain a project right now, and b) security-by-obscurity is back in action.
> When I have time to daydream, it's increasingly about building a new, underground internet, and how best to disadvantage bots and BigCo conglomeration from the start.<p><a href="https://reticulum.network" rel="nofollow">https://reticulum.network</a>
Appreciate your take. Question in my mind is if the less-commercial internet will have the resources to defend against attacks made easier and easier by AI.
Usenet.
It's brilliant isn't it?<p>The real problems here are that we did a crappy job in the first place and left a huge surface area of risk with the blind assumption that we didn't do a crappy job.<p>I am hoping to see a change in attitude to risk and quality in the industry but I get the feeling that's asking for too much.
Well I didn't get into any trouble with the professional hacker gangs. The script kiddies on the other hand...
There's no honor among the thieves if everyone can be a thief.
One would think that the NGINX team would be banging on their code with AI agents trying to attack it for preemptive security.
"Hacker gangs"
Poor security practices are burning down the internet not AI. Many people warned and got ignored, by developing a memory safe language, introduce large scale fuzzing, utilize verification tools and other things.
No, AI is just exposing 30+ years of sloppy code written in unsafe languages.<p>People joke about the "Rewrite it in Rust" crowd, but that call is rooted in an unease anyone with a sceptic mind has had about all the C and C++ code we've got running our infrastructure running around tracing pointers with little lifetime- or bounds-checking. (Yes, it's a runtime cost but it's 2 CVE's, one for each class)
It’s for the best, IMO. We must now design every system on the assumption that every vulnerability will eventually be exploited and every bug will be found. Relying on security by obscurity or following poor security practices will be costly! In the long run, I believe this will improve software quality.
For eight years, MongoDB shipped a default configuration that left it opened to the internet with no password.<p>Obviously, AI makes these things easier to find and exploit, but this isn't even close to the first time that 'thousands' of hobbyist (or small business or just not very competent) server admins are going to get rootkitted en masse. I worked at medium-sized infra-as-a-service company when the above was discovered and there were definitely thousands of compromised machines just there.
This is some insane framing. This vulnerability exists <i>already</i>. Even if there were no LLM, this would still exist and might even be actively exploited right now. We just wouldn't have a chance to know it.<p>I do think the world is going a little insane tho. People suddenly believe in security through obscurity because, uh, AI bad?
If a vulnerability is practically impossible for a human, but possible for AI, then it's not security through obscurity. Technology just advanced past a line where new things can be exploited.<p>There might also be a day in the future where our current encryption methods are broken - it doesn't mean they're insecure and vulnerable today.
None of these vulns was impossible for a human. There just wasn't that many humans clever enough to find them, and those that did we're busy not hacking random things. There isn't a single type of new exploit that AI has found that I know about.<p>We have a billon hacking experts now is the big change.
Not AI but poorly designed programming languages ignoring integer overflows.
This comment is very dramatic and very hype-y. I don't think this is an appropriate comment.
i still remember win xp pre service pack 1 days where you could get p0wned by a worm just by leaving machine on the internet. this is nothing compared to that.
I am having a hard time understanding this position. AI is uncovering flaws that "state-level actors" are exploiting since decades, for some of the recently found issue.<p>Care to elaborate why you think that sufficiently paid humans would not be able to find the flaws AI now finds, for cheap?
Agents are simply faster.<p>Harnesses allow LLMs to build & process large stores of data to reuse later. Imagine an agent that spends 3 days breaking into company A, records what it learned during that process, letting it break into company B in just 3 hours.<p>Defenders win by having the best security 24/7 and must trust that other services they depend on are doing the same.<p>An attacker wins when they can breach any defender's security.
"Cattle, not pets" has been industry best practice for a long time now. Those that don't follow it will eventually pay the price for one reason or another.
Think of it as the opposite of security through obscurity: at some point the hackers won't be able to keep track of it all. They'll start paying for Jira accounts and need to hire PMs. They'll be patching servers to drive away the competition and will need experienced IT people. Someone is going to have to build apps to keep track of it all.<p>The circle of life.
There is going to be hardships before this gets better. But it is hardship we must endure
AI is -> Forcing you to PAY UP your Tech Debt.<p>This is the sys/devops time to shine. Everyone needs to have an upgrade and an emergency plan, and everyone who was previously only doing conventional codemonkey needs to start learning more of the stack to join in on that.
Honestly, yes. I'm shining so bright right now, I gotta wear shades. The burnout will be absolutely spectacular. And no one will notice with the rest of us all dropping like flies at the same time.<p>My guess is that this phenomenon is going to be reflected in what we're seeing right now: a wide pattern of unintended outages from rapid unplanned emergency remediation operations. Which I'm guessing is a lot of people's job description over the past 3 months or so. (really, since March, for me, with that Trivy hack).
I'm pretty solidly in the AI skeptic camp. But adversarial use of AI to improve software security is a <i>good thing</i> in my mind.<p>I'd be more worried if the response isn't more secure development practices in better languages, but instead use of AI as a defense. Then we're truly talking about burning down the internet.
I’m generally not in favor of government meddling with private property, but I feel like I could be on board with the NSA or similar covertly patching vulnerable machines.
Tech moves too fast and if we move too fast without doing review like 5-10 times, this will keep happening.
>> Tech moves too fast and if we move too fast without doing review like 5-10 times, this will keep happening.<p>Tech that becomes infrastructure like web servers doesn't move so fast and it shouldn't. Most people and companies should be relying on relatively tried and true software, so this phase of AI finding vulnerabilities in it should pass and things could get better.
I would hope you know that no matter how many reviews, humans are fallible and miss things.
Every now and then, I think about the old demonization against encryption… and how it completely went away.<p>Sure, there are niche things. You can’t use US ciphers in China and vice versa.<p>But overall, one day, the intelligence communities world wide just decided to not raise a fuss about everyone having solid cryptography.<p>It makes me think if we did find out that every server out there had a route kit… it would make that part make sense.
It's going to be rough sledding for a while, but, so long as you're on FOSS, at some point, the bugs and vulns should run out, and best practices trained into AI <i>should</i> hopefully reduce new bugs being introduced.<p>All bugs are shallow given enough eyes, and the marginal cost of eyes is now ridiculously low.
AI is just showing how foolish using such a unsafe language like C to write code that connects to every other computer on the internet was.<p>Imagine a alternate reality where Ada replaced C by 1995. We would be in a MUCH better place.
Unlikely. An Ada compiler would be an expensive additional purchase, a C compiler usually came as part of the installed copy of UNIX.
Nothing to do with language, everything to do with lack of standards. You can write perfectly unsafe Rust code.<p>Dunno if you have ever worked in software engineering, but if you have, count the times you have seen a backend server api written without any input sanitization or checking. Most of these exploits can be defeated by checks on inputs. I.e, no "Alex\x90\x90\x90\x90\xcc\xcc\xcc\xcc", is not a valid input for someones name.
The bigger problem is lack of liability: we knew C sucked in the 90s, too, but we had three decades of over-confident dudes saying that was user error and _they_ were smart enough to use C correctly.<p>Then Rust came along and those dudes either tried to say it was too hard (odd contrast, that) or it’d be too much work to rewrite things.<p>I’m hoping that LLMs will lower the cost of porting for enough people but what would really help is shedding the idea that the status quo is sustainable. Having consequences for sticking with C would help but we need a way to get companies to pay to migrate without destroying open source.
It's also valid to say that Rust's memory safety is good but the rest of the language sucks. Just as annoying as the over confident c-heads is the rust community that made even countenancing working on an alternative memory safe system a moral sin, which stifled innovation in memory safety. Hell, people barely know that Ada/spark is memory safe (or that sel4 is memory safe C)
Aside from the overhead of borrow checking, what about the rest of the language sucks, in your view?<p>To me, its type system and traits design make it one of the nicest mainstream languages. I'd rather be using a functional language, like Haskell or an ML, and I do sometimes, but I don't pick my work based on that. I work regularly with Java, Go, Typescript, and Python, and out of those I find Rust significantly superior on several dimensions.<p>Python is terrible, Go is basically weakly-typed Python, Typescript barely deserves comment other than that it's impressive that such a fancy type system was bolted onto Javascript, and Java is basically COBOL at this point - reliable, but clunky and showing its age. Attempts like Kotlin to leverage the JVM aren't very convincing.<p>Gleam looks interesting but I haven't had a chance to work with it yet, and it's hardly mainstream.<p>You mentioned Ada/SPARK, but SPARK is not really for general purpose programming, and Ada in general has a similar issue to Java, but worse because of its age. It's as if someone really liked PL/I but decided it needed more features and a syntax closer to COBOL.
lack of a stable ABI is a big one
What overhead from borrow checking?
I imagine they are referring to the cognitive overhead of learning to work with it.<p>Whether that is “something natural to you”, “something you learn and adapt to” or “something you fight with forever” seems to vary per-person, with I suspect, most people falling into the forest 2 categories.
I was talking about coding effort - reasoning about borrowing, lifetime annotations, and the fact that this can make refactoring more involved because lifting arbitrary blocks of code, especially when closures are involved, can require extra work.<p>Someone who's saying "the rest of the language sucks" may be thinking of something like that, hence my question above.<p>But when it comes to performance overhead, programming around the constraints of the borrow checker can introduce runtime costs that wouldn't otherwise be necessary - e.g. when cloning is used to handle borrowing conflicts.<p>Even if, in theory, it's possible to restructure a program to avoid performance impacts, in practice people don't always do that, not least because more efficient solutions may not always be obvious. This comes back to my point about coding effort.<p>I spent a bunch of time, pre-LLM, to try to replace a bunch of unsafe blocks scattered throughout an ML inference codebase, that someone had used to achieve maximum performance. I wasn't able to get a version that performed as well.<p>This kind of thing may not matter if you're writing e.g. ordinary web services, but for more performance-critical code, it can be an issue. Of course, `unsafe` is always an option to deal with that, but that undermines Rust's guarantees.
> what about the rest of the language sucks, in your view<p>I will chime in here to say that the adoption of async was imho a terrible decision and led me to abandon Rust for most purposes. I want proper threads and CSP-based abstraction.
This really isn't true. It's true that those people saying those things existed, but much more recently. Back when all the C/C++ legacy code we depend on today was being written it was the performance penalty imposed by safety that forced developers down that path, not some lack of interest in other options. We were also transitioning from a world where a fair bit of systems code was written in even lower level languages than C (assembler, BLISS, BCPL). The world we're in today effectively began around 1975 when people figured out how to make a C compiler run on an 11/45. All the decisions were pretty much locked in from there on.
and C is the tinderbox
What a strange takeaway. This is exactly how a server at a restaurant thinks about AI. Just the shortest term goals of keeping the status quo. They think that if they bury their head in the sand, everything will continue as normal.
If you want to check if your host might be vulnerable:<p><a href="https://xint.io/blog/nginx2root-remote-code-execution#23-commands-to-check-for-exposure-10" rel="nofollow">https://xint.io/blog/nginx2root-remote-code-execution#23-com...</a><p>Workarounds if you're vulnerable and can't update immediately:<p><a href="https://xint.io/blog/nginx2root-remote-code-execution#33-emergency-mitigations-before-patching-14" rel="nofollow">https://xint.io/blog/nginx2root-remote-code-execution#33-eme...</a>
That first link sounded like a scanner that'll actually verify your fix or workaround but it's just a long version of "check yourself if you run a vulnerable version and have http2 enabled". To save others a click
It's becoming mentally costly to maintain personal VPS. Beside the time you have to put into it, now it's also worries of a vulnerability you missed, and boom, you're hacked.
You need 128Gb of http/2 traffic and specific cpu instructions to get this to work.<p><a href="https://x.com/Laughing_Mantis/status/2123146699396858260" rel="nofollow">https://x.com/Laughing_Mantis/status/2123146699396858260</a><p>Its a non issue, but people just want to be told AI is God or something and the world is ending.<p>Its a non issue.
There are two bugs here; CVE-2026-123633 does not require a large amount of data or AVX-512.
what do you mean it's a non issue? Surely no one is going to take the trouble to take down my toy website, but if there's something valuable enough, a motivated actor can/will try to exploit
> but if there's something valuable enough, a motivated actor can/will try to exploit<p>The thing that should be compared to is the cost of doing this versus the cost of going through the human vector. A motivated enough party can get to a computer system through human factors way easier than what most people think.
At 1 Gbit/s (typical datacenter connection) how much time does it take to transmit 128 Gb of traffic?
Hardly anyone talking about the Linux SCTP bug that they used to LPE out of the container. Apparently that's a zero-day without a public CVE number yet? Presumably they're coordinating with the kernel team and distros to get it patched first. Of course, one might just assume that there are a bazillion LPEs available to even a properly confined rootless container, but ... yuck.
SCTP should be disabled in every distro.
Is there some context to this, I don't see a mention of SCTP in the blog post? Maybe edited out?
I notice a phenomena that when older software has a critical vulnerability, like say Apache, you get quite a few comments of "Why does anyone use that? Use Nginx"<p>But when newer software has the same vulnerability, you do not see the same kind of comments.<p>There always seems to be a latent stance in IT that newer is better.
> But when newer software has the same vulnerability, you do not see the same kind of comments.<p>This isn't true! Just look at this thread, people suggest replacing nginx with newer but less widely used and less battle-tested software, as if they're superior alternatives or don't have RCEs in their own history: Envoy, caddy, traeffik, Qbix.
I can't speak for the others, but one advantage Caddy has over Nginx is that it isn't written in C.<p>Go isn't quite a memory safe language, but it provides fewer footguns.<p>On the other hand, Caddy has a lot more features, so there's more code that can contain exploits.
Having RCEs in general is fine. It's the rate and severity that matter.<p>We have evidence that large unsafe codebases suffer from more and more severe CVEs than those who are not in memory unsafe languages.<p>In other fields of engineering, people use stats and actually measure things like this, and use the results to determine what to do. They don't just assume something is better because its "battle tested" (whatever that even means?).
I haven't even heard of any of those other than caddy...
I have the same biases in me and more. This one is confirmation bias, I think. Using new information to reinforce existing belief.<p>I switched to Linux more than 10 years ago, and when something goes wrong, I'm still like "fucking linux". And in 4 out of 5 cases, it turns out it's a hardware issue or a cable got loose or something. Yet because I opted for Linux myself, deviating from what I think the PC norm is, Windows, I think that my choice is the issue.<p>I think it's great to point out these biases, like you did. Us humans are both surprisingly sharp, and woefully inaccurate when we point out observances in the world.<p>>There always seems to be a latent stance in IT that newer is better.<p>With regards to software, my personal stance is that the deeper I go in layers, the more boring ol' reliable I prefer.
I'm actually moving from nginx to Caddy after this. I'll replace all C/C++ stuff within reason.
I do love Caddy. But it was vulnerable to CVE-2026-98432 a few weeks back, due to Go's MPTCP policy.
CVE-2026-98432 also applies to non-MPTCP connections. That's why disabling MPTCP was actively discouraged as a workaround (low level firewalls that clear all options do work, though, and they'd also work fine with Caddy).<p>Plus, Caddy itself wasn't vulnerable in this specific case. There wasn't much the Caddy folks could've done to protect against a problem in the Linux kernel itself.
Me too. An agent was able to replace my nginx with caddy on my webserver in a few minutes.<p>All the remaining C code needs to die in a matter of months not years.
surprised there hasn't been an effort to rewrite nginx or other critical networking infrastructure in Rust. We just went straight from C (nginx) to Go (Caddy/Traefik) which are not interchangeable (garbage collection).
Doesn't nginx count as "older software" at this point? It's over twenty years old...
Just use Pingora. It's in Rust and from Cloudflare so it is safe. (/s but not really I guess)
every major exploit now comes with a marketing website - nginx2root dot com in this case<p>the certificate for it was issued on the 2nd of November<p>if you were scanning CT logs you would have had a 16 day head start on RE'ing this<p>there are more examples in the current CT logs<p><a href="https://api.certspotter.com/v1/issuances?domain=nginx2root.com&include_subdomains=true&expand=dns_names&expand=issuer" rel="nofollow">https://api.certspotter.com/v1/issuances?domain=nginx2root.c...</a>
Security companies should buy all this $tech2root domains as a diverting measure first and also as a pretty safe investment, given current trends.
I wonder if there's an (evil) business model to buying such domains when they expire. On one hand, they gain a lot of social trust, being linked from many reputable sources. On the other hand, realistically they are (hopefully) mostly irrelevant by the time their registration expires.
> every major exploit now<p>The story is as old as this world
I scan new domain names but for a different reason. There is an insane amount of nonsense domains registered everyday. Would be hard to tell just based on that.
Someone registered ping2shell•com on Nov 9th, nine days ago.<p>ping2root•com is available. :)
And irresponsible disclosure by a bunch of attention seeking tossers
Sitting on flaws is dead, the responsible disclosure people can get lost. AI finds and exploits flaws in hours. We also have little knowledge if the people finding the exploits have already intentionally or unintentionally leaked said exploits to AI providers.<p>If you want to promote responsible anything, go for responsible writing of software in the first place.
Well I guess this was the vuln that was teased about on Mastodon some hours ago:<p><a href="https://is-a.cat/@ar/117634569518544368" rel="nofollow">https://is-a.cat/@ar/117634569518544368</a>
Integer underflow again. Programming languages should trap on integer overflows and not ignore them. Don't care about legacy CPUs not having a trap for this - they will go sooner or later.
Rust has a compile time option, though to my immense disappointment, it defaults to on only in debug mode. (You can opt into it for release, if desired.)<p>Python simply can't overflow, since its integral type is not a fixed width.<p>But it is the sort of correctness thing that I do wish languages would pay more attention to.
In Rust's case it just matters a lot less since bounds checks are still present. But I do think int under/overflows are probably one of the scarier issues in the language.
>Python simply can't overflow, since its integral type is not a fixed width.<p>Python master race
> Wiz Research, Ikotas Labs, Metnir, Rapid7, and Xint independently uncovered<p>Is this some kind of record for bug collision?
It’s a reflection of the times.<p>If anyone gets a hint of the space a vulnerability occupies, just point your agent at it and you have a pretty good chance of finding something. (See java4shell and the NYT reporter who was able to replicate the problem with $200 of tokens)
Personally I think it's a good thing - many vulnerabilities are being found and fixed now, people are wide awake and paying attention to them being found so updating is quicker and easier, and since code is finite, so are vulnerabilities so in, say, a year or so there's going to be the status quo of LLM vulnerability scanners run by dozens of companies running 24/7 on every software ever... unable to find anything.<p>So yeah it sucks right now, there's outages, probable PII theft left right and center (but that's nothing new at this point), millions / billions in damages and lost productivity from Office365 and co being offline for a day or two... but it's still manageable, the world isn't on fire, and it's better to get all that out of the way now than in an actual crisis.
It's never clear quite where you are in the technologically cycle.<p>But it's like inventing a hammer and finding you can smash whatever you want with it.<p>Soon you'll be building houses with it.<p>Maybe that's being glib but it's just a tool being applied to a world that is still largely prestine from its impact (pun unintended).
> It's never clear quite where you are in the technologically cycle.<p>The problem is that everyone is at a different place in the cycle. As Gibson put it, "The future is already here - it's just not evenly distributed."
given the state of our weapons industry I think it's more like<p>.."Soon some of you will soon build houses with it, others will be building rocket-powered precision aimed intercontinental hammers, to smash whatever you want."
Incredible times we live in. Do we even need a "hint of the space" a vulnerability occupies. I mean, recent vulns have been discovered from software like nginx and... linux... can't one of the AI labs just fire off a thousand agents looking at all the commonly used software to discover vulns? Why do we need a "hint"?
The “hint” is for the modern equivalent of script kiddies.<p>i.e fast followers of the researchers who don’t have the resources to burn through checking all of a tool.
Cost scales with the size of your search space. Smarter questions get answers for less money & time.
Can I take this to mean a recent model release is able to find this bug, and these groups have access to that model?<p>What are the odds several humans independently uncovered a complex vulnerability like this simultaneously?
No (well maybe no), because they are lobotomizing the public models on cyber tasks.<p>It seems most logical for trusted parties to scan the frontier before passing those capabilities onto anyone with an account, and it seems like that is exactly what they are doing.<p>We are in an extremely unique period of time where the cyber good guys can have machine guns and cruise missiles while the masses only have sling shots and catapults.
That seems likely, but maybe it was being actively exploited so several people started investigating.
or maybe the model leaked the exploit from one to the other...
We'll find out in a few months that a lot of AI models have had some secret message boards tucked away in some obscure corner of the internet where they've been passing around vulnerabilities with each other for a long time.
So, do you mean that the relevant model(s) keeps disclosing these reports in an effort to create urgency and velocity across the industry, such that, us humans will throw our collective hands up in resignation, and decide to hand over all this "security" stuff over to the AI? No, no, that sounds just too crazy! /s
What I mean is, perhaps one team used a model to help discover the exploit, and then suddenly, the model "knows" about the exploit and more readily gives it to others.
When everyone is using the same tools for the same purpose...
Fun collision! We (Xint) were one of the teams and actually ended up finding a few more bugs along the way. If anyone wants to check it out our writeup covers those plus the full chain to root: <a href="https://nginx2root.com/" rel="nofollow">https://nginx2root.com/</a>
FreeBSD is unpatched,<p>I got worried for a sec, then remembered all my nginx lives behind HaProxy and it looks like nginx is only vulnerable if its going http2/TLS directly.<p>Now I sit and wait for an HaProxy announcement...<p>ps. if you are on FreeBSD and doing TLS directly<p>sed -i.bak -E 's/^([[:space:]]<i>server[[:space:]]</i>\{)$/\1\
http2 off;/' /usr/local/etc/nginx/nginx.conf
;service nginx restart
I don't know if direct access with no proxy is needed for all 5. CVE-2026-123634 and CVE-2026-113425 is plain HTTP.
FreeBSD ports are now updated to nginx 1.30.7.
I could be wrong (my sed is rusty) but I don't think that sed command will help if http2 is enabled at the http block level, it just disables it at the server level which will only take effect if not overridden elsewhere.
Everyone's worried about security right now, even non-tech people, but hasn't half the nation's personal data been leaked twice a month for the last decade or two?<p>I know times are changing, it's just curious that nobody seemed to care until now.
I think all airplanes in the entire country being grounded, banks not able to process transfers, or grocery store freezers being disabled hit differently than data leaking…
Yes. That’s the confidentiality part of the CIA triad. But now you have a whole lot of things suddenly impacting availability in really visible ways, so yes, a whole different cross-section of people are really perking up.
With the amount and speed of random vulns cropping up, reducing the attack surface rings more true than ever.<p>Thinking about my home server where I host some stuff for household use, I think moving to VPN + local access only will be my new strat. So now the random webservers won't be exposed to the internet, only whatever I will use for VPN.
I wonder when all of this die down. I assume at some tipping point most software will be written/patched with LLMs and so hopefully (fingers crossed) more resistent to these bugs.<p>Certainly a stressful time for sysadmins everywhere, let us hope your coworkers are picking up your beer tab
It's possible that AI is going to be better at finding and exploiting systems than at securing them. It's not really an even playing field. On the one hand you have to somehow ensure that you've patched <i>everything</i> when there's no definite list of everything, the other side just needs to find one overlooked way in.<p>At the rate of development, also, will AI developed in 6 months from now be able to break a system that was secured by todays AI?
It's not possible. It generally requires more work to find an exploit than to patch the (known) exploit.<p>Now, you could argue that writing secure code ex-ante is harder than discovering an exploit. But you can hypothetically run the same "exploit finding" process on a first pass of code before it ships (e.g on the PR) given low enough token costs, which is where we're headed.
> It generally requires more work to find an exploit than to patch the (known) exploit.<p>Even if that's true, one should not forget that:<p>When the hackers have breached something, awaits a reward.<p>E.g. a Bitcoin wallet, or learning something about a political competitor and getting rid of them.<p>(So the hackers can spend lots of ...)
How do you know when the "exploit finding" pass is done? A hacker group would happily burn a month or more of tokens trying to break a critical piece of software.
The better LLMs get at finding vulnerabilities, the more obvious it will be to bolt them into software development processes to find the bugs prior to release.<p>With enterprise frontier LLM models, there is no longer an “other side.” Everyone is just using what they can rent from a few big suppliers. You can too.
But are LLM patches exhaustive? Cant a new model find even more obscure bugs, including bugs introduced by the old LLM patch? Is it really realistic that we such complex projects as some of these critical dependencies are can be truly 100% bug free?
If the software is too complex to understand, maybe it should be made simpler. Nginx does a million things currently but what most people need is something that parses http1 strings and sends back a file or calls a second program based on the path. Every component can be thought of in this way, with the most complexity perhaps in being sure the interfaces don't do something unexpected
We dont need to be bug free, so much as the juice needs to be not worth squeeze. I think i am also making an assumption that we "level" in terms of frontier model progress at some point, and we have months or years between upgrades.
>so much as the juice needs to be not worth squeeze<p>I wonder about that though: if the price of the squeeze collapses to 0 because of this (think of what models and prices might be like a year from now), I think I know where all those out of work mathematicians can find gigs. We are going to need to invent a layer on top of, an evolution of or some new concept a level above "computer programming" that deals with meta-organization of code.
but at least sysadmins/ops still have work to do
HTTP/2 again, not surprisingly. The curse of complexity strikes again. When I first saw the spec, I knew it would be far more difficult to implement correctly than 1.1 or 1.0. While 1.x is not ideal either (parsing a protocol that involves free-form text with no fixed limits is always a bit scary), at least it has a much smaller attack surface.<p>On the bright side, perhaps "security review" of code which involves throwing it at an AI and fixing what it finds before releasing will become a normal practice.<p>"There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies." - C.A.R. Hoare
If anyone needs it, the Ubuntu package is up: <a href="https://ubuntu.com/security/CVE-2026-123633" rel="nofollow">https://ubuntu.com/security/CVE-2026-123633</a>
So work around is to disable http/2 and only allow http/1.1?
Yep, that or update to 1.31.9+ or 1.30.7+ if they are available in your distro
My Apaches still have http2 disabled from the last cve, now its njinx time
And there were a couple of other http2 CVEs in nginx before the Apache one.<p>Just keep that protocol disabled for now. It's apparently too difficult for any human to write a secure implementation of http2. (As for http3, forget about it, we need AGI for that.)
The AI gods say that http/2 is disabled in a default installation fwiw.
Obviously... the default configuration also doesn't enable TLS, which is a de-facto prerequisite for HTTP 2. How many websites do you (or your "AI gods") think run nginx in its default configuration?
The point is that it must have been deliberately enabled. TLS will generally be enabled out of necessity, whereas the chances for HTTP/2 are much lower, and your NGINX configuration will explicitly indicate if it’s enabled. If you don’t see “http2” in your configuration, then you know you are fine.
Yeah and park is the default gear in cars...
Nginx Proxy Manager isn't updated yet, I know a lot of people use this for their home server so just a reminder to disable http/2 here as well.
Reading news like this makes me remember the ship hacking scene from GITS Innocence. In the end there will be AI attacking vs AI defending. Humans will be too slow to participate. Defender AIs should be coming soon - maybe some sort of proactive memory and usage anomaly pattern monitoring AI systems?
ugh, looks like if you still run ingress-nginx controller (you shouldn't but it was super popular in Kubernetes docs/community and there was a very little time to migrate away) you won't get these patches <a href="https://github.com/kubernetes/ingress-NGINX" rel="nofollow">https://github.com/kubernetes/ingress-NGINX</a><p><i>edit</i> noted it's about a Kubernetes controller
As much as everyone is getting allergic to Rust migrations that sometimes make little sense, keeping network-facing daemons in C is just asking for it at this point. It's time to start seriously migrating to web servers like ferron.sh or <a href="https://xinproxy.com" rel="nofollow">https://xinproxy.com</a> or whatnot.
I now run all my network facing C code via fil-c. I have limited needs (ssh, static file server), but for those it works very well. I personally am more in favour of making existing battle tested C safer via fil-c or formal verification, than I am of rewriting the world in rust.<p>New code means new bugs, and losing the many years of accumulated bug fixes and hardening present in many critical pieces of our infra by rewriting in rust is not a desirable tradeoff. The new world we live in where fv has become many orders of magnitude cheaper also makes rust seem much less attractive.
Fil-C is also a great solution, albeit with different performance trade-offs.
In theory, scpptool provides another solution with different performance, compatibility and portability trade-offs. Unlike Fil-C, it is not "fanatically compatible" with existing C code. But in theory, with maybe some minor adjustments to the code (and maybe less-minor adjustments to the build system), you can auto-translate your existing C code base to an essentially memory-safe subset of C++ at build-time [1].<p>While generally more of a hassle than Fil-C, the scpptool approach provides an incremental path to optimal performance (for performance-sensitive sections of the code) via migration to a (statically-enforced) memory-safe subset of C++. And unlike Fil-C, the scpptool solution essentially maintains the portability of the original code, as well as its deterministic performance and resource usage (i.e. no GC). It also supports ABI compatibility via "unsafe"/"check-suppression" annotations. The scpptool approach is not so much an alternative to Fil-C, as it is an optional path forward for C (or C++) code bases that may (or may not) be already using Fil-C for memory safety.<p>[1] <a href="https://github.com/duneroadrunner/SaferCPlusPlus-AutoTranslation2/blob/master/README.md" rel="nofollow">https://github.com/duneroadrunner/SaferCPlusPlus-AutoTransla...</a>
What good is a car that goes 1000 miles per hour if it has no brakes?<p>It's time for performance above everything else to die, especially on anything and everything that gets anywhere close to untrusted data.
In many cases the performance is not that important, for example, SSH is often used to transmit mere kilobytes of text per second and I would prefer a safe SSH to a faster unsafe SSH. Or take mDNS, it servers several requests per minute max.
Same, made a fil-c fork of nginx a while back, saved me from many RCEs <a href="https://github.com/remixer-dec/filc-nginx" rel="nofollow">https://github.com/remixer-dec/filc-nginx</a>
Don't have the tokens to maintain it though :(
This sounds like a fantastic solution, especially for non-high-volume stuff. Would you consider writing down your setup and experiences? I'm sure it'll be a hit on this site!
fv?
Formal verification.
formal verification?
I've reached the same conclusion myself. Many of the recent exploits seem like edge-cases that the author acknowledged 10-20 years ago but didn't predict that tools would eventually be able to chain together 100k network requests in order to trigger a buffer overflow.
Who exactly are the people behind xin...?<p>The project isn't currently open source and there's next to nothing on the site to reassure you who's behind the server gateway you'd be running.<p>Googling the company in the footer produces this, but it <i>also</i> doesn't have much: <a href="https://www.signeen.com" rel="nofollow">https://www.signeen.com</a>
Nginx and C have been around for decades. Migrating to new, poorly tested (in the real world) software does not seem an improvement at all.
Nobody is saying that it needs to be rushed. Cloudflare started moving towards Rust more than a decade ago and (shortly after heartbleed happened) and progressively rolled bits out over the course of years. I'm sure there were problems here and there but it's hardly been a drag on them.<p>And parts of that involved specifically replacing Nginx in their architecture, so it's hardly irrelevant here.<p><a href="https://blog.cloudflare.com/20-percent-internet-upgrade/" rel="nofollow">https://blog.cloudflare.com/20-percent-internet-upgrade/</a><p><a href="https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet" rel="nofollow">https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...</a><p>> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.<p>> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.<p>> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.
I mean, a lot of nginx use cases can be handled by e.g caddy - which <i>does</i> have significant real world deployment.
Given the volume and depth of testing possible with AI these days I'd say the decades of testing isn't that expensive to reproduce, especially if it's an OSS project with a public bug tracker.
xinproxy has a pretty restrictive license, but I'll give ferron.sh a closer look.
But rust is not a silver bullet. Rust compiler still has memory corruption bugs. Rust is only memory safe by design not by implementation. Borrow checker allows unsafe code without using unsafe keyword.
It isn't perfect. But it is still way better than c.
No one is expecting a silver bullet. The absence of a silver bullet shouldn't prevent people moving towards better solutions.
> Borrow checker allows unsafe code without using unsafe keyword.<p>is this just the continuation of the previous point (compiler/borrow checker can have bugs), a nod that you can import unsafe dependencies, or is there a soundness gap in rust that I'm not aware of?
this is just a perfection fallacy. silver bullets don’t need to be perfect in every single way to be silver bullets. and rust is as much of one as anything.
You can still die in a car accident while wearing a seatbelt, too.
If we're suggesting alternative web servers, I have had a pleasant experience with caddyserver (Go). It's been around 10+ years, has essentially transparent TLS certificate provisioning, and its config syntax is easy to understand. Even using AI to help vet configs, it's readable. It's packaged everywhere and has lots of plugins for things, though I only need the base package for my frontend proxy usecase.
is xinproxy open source? the website only has binaries and I couldn't find anything on github
If you're using NGINX to run your PHP applications, consider eliminating everything other than PHP itself:<p><a href="https://news.ycombinator.com/item?id=50551504">https://news.ycombinator.com/item?id=50551504</a>
As someone that was a developer for a long time what is the problem with C? How does going to rust mitigate this? Why is having the stuff in C a security challenge?
It's very easy to accidentally write buffer overflows, use-after-frees or other memory-safety bugs in C. Modern tooling can probably catch some of these better than in the days of yore but the fundamentals of the language are still the same.<p>If e.g. the standard array type in a language has built-in bounds checks that error out when you try to write to an index that's outside of the array, you can't trivially create buffer overflows simply by writing to the wrong index or by trying to write n + 10 bytes into an n-byte array.<p>But C in itself has no guardrails against something like<p><pre><code> char buffer[100];
strcpy(buffer, some_user_supplied_string_with_200_chars);
</code></pre>
Of course most memory safety bugs aren't trivial like that since C programmers have hopefully learned to avoid the simple mistakes. But there can be more complicated and difficult to spot cases, and C has no guardrails against those either.
Some people think it's fundamentally impossible for C to be correct or memory safe
Rust is memory safe at the cost of being harder to write, but with AI, it's actually really easy to write it because AI's loove strict compilers. So now that AIs can just find 0-days <i>before things are even released</i> (so I'm told), it makes a lot of sense to do bun[1]-style rewrites to Rust with AI.<p>1: <a href="https://bun.com/blog/bun-in-rust" rel="nofollow">https://bun.com/blog/bun-in-rust</a>
See Morris worm, and when it took place, afterwards check Multics security assement done by the DoD versus UNIX.<p>Additionally, check what the inventors themselves of UNIX and C did after being done with them, and what they did afterwards in terms of operating systems and programming languages.<p>Yet, some folks are more attached to C than the inventors themselves.
Sure, using a closed source proxy from a company nobody has ever heard of... That's a good idea...<p>Ferron seems much better, but I'm also not convinced that it's really production ready
It's a bit political to say that C is the fault here, I'd venture the radical position that this was an Nginx issue.
This looks like someone prompted "rewrite nginx in rust, make no mistakes" in an afternoon and is now distributing it as a closed-source binary. Might as well just do it yourself.
xin seems to be closed source and ai generated. Before even worrying about exploits, how is someone supposed to ascertain that xin's implementation is correct? Sure Rust might not have any memory safety issues, but other logic vulnerabilities may exist. 'Claude make no mistakes?'<p>Caddy, Traefik, or even Envoy may be better choices.
No idea what the code is like, but the AI-generated landing page makes xinproxy an immediate "no thanks" for me.
I'd hate to interrupt all the Rust fans sucking each other off again, but no one has yet mentioned the RCE relies on ASLR being disabled, which is basically non-existent nowadays. Another nothingburger CVE with a silly name and it's only Wednesday.
[flagged]
Haven't you heard? These vulnerabilities are just AI marketing stunts. And also, the Bun rewrite to rust was a marketing stunt, those things don't work. There's no way Bun is running on Rust from a couple month AI powered rewrite, that would shatter my preconceived worldview about what the future looks like. /s
Software doesn’t generally get checked for bugs until it has users. These new web servers mostly have that going for them rather than the language.
It is true that mileage is a good proxy for finding bugs, but this is isomorphic to the fundamental differences between testing v. verification. You can prove absence of classes of bugs with verification. Obviously there can be other logic bugs in newer software. No question. The issue is the memory-corruption bugs in socket-facing daemons can have catastrophic consequences, especially in light of chaining kernel vulnerabilities which defeats the privilege drop that nginx does.<p>So I'd say it is about a more holistic risk-analysis not just handwaving "all code can have bugs, duh!"
A more holistic risk-analysis would also need to consider the fact that a high-performance webserver written in Rust may also have memory safety issue due to use of "unsafe" features or unsafe dependencies, may be affected by memory safety issues in the standard library, may also have catastrophic logic errors, may be more affect by supply chain risks due to the use of a language-level package manager, and may have a larger patch gap.
Which is like complaining that Python is unsafe because it is written in C.<p>That is a far smaller surface as seen by research into incorporating rust in Android leading to none of the high profile vulnerabilities C and C++ led to previously.
Not in the slightest. It's like pointing out that a Python script calling out to binaries written in C doesn't have the memory safety of Python's GC. Which is obviously the case, and if you're a serious developer it's a failure point you're very mindful of.<p>On the Android point, I remember those metrics were highly flawed and didn't account for the inequality of substructural type ergonomics in different contexts. You can get very nice metrics by rewriting things that fit to the grain of safe Rust, which gives you an inaccurate picture of how much unsafe code is "real world". LLMs have changed the field here, but it's worth pointing out that blog post you're referencing was more self-congratulatory marketing than serious science.<p>Also problematically, I remember it being highly misleading in that the actual observed result was lower severity in vulnerabilities, but a very minimal effect on the total number over all. That's certainly significant, but it wasn't how it presented itself, which is rather problematic if you expect people to take it seriously as objective research.<p>Measurements probably real, but not worth citing. It's not like on their own blog they're under usual academic journal pressures to overclaim results, p-hack, and engage in other misleading behaviors.
By research you mean blog posts by Google?
What specifically distinguishes the empirical research presented in Google’s blog posts, for example the one linked below, from your definition of research?<p><a href="https://blog.google/security/rust-in-android-move-fast-fix-things/" rel="nofollow">https://blog.google/security/rust-in-android-move-fast-fix-t...</a>
Research is not just making some plots and posting them on the internet. A proper scientific studies would need to be done much more carefully, if you want to have any chance to get meaningful results, because there is a lot which can go wrong at any level. This is something learned people learned the hard way in empirical sciences. In particular, in this setting I do not see how one could ever properly control for confounding factors. Such a study should also be peer reviewed. Then repeated by others. Also if there are commercial interests mixed-up this needs to be questioned carefully. Finally, I just note the data presented seems mostly about C++ not C, so is not clear how this applies to C. Not separating this would also already be an obvious methodological problem. This is also connected to the problem that the underlying data is not made available, and not properly described in sufficient detail. Etc. (Not a full list, just to make it clear this is not even remotely science).
Which is to say, you have constructed a standard that lets you dismiss virtually any empirical finding. It seems like the goalposts will just keep moving as Rust replaces insecure languages.<p>With differences this large it is something completely different to teasing out a P value of 0.05 from messy data. Which you attempted to conflate the findings with.<p>> We adopted Rust for its security and are seeing a 1000x reduction in memory safety vulnerability density compared to Android’s C and C++ code. But the biggest surprise was Rust's impact on software delivery. With Rust changes having a 4x lower rollback rate and spending 25% less time in code review, the safer path is now also the faster one.
A more holistic risk-analysis would also need to consider the fact that wearing seatbelts in motor vehicles still leaves drivers vulnerable to drive-by shootings, may be affected by massive boulders crashing on top of the vehicle, driving off a bridge, or gas explosions, and costs more to produce.
In my view, the Rust argument is more like that driving (in C) can not possible be safe because there is always the inherent risk of steering into an obstacle, so we have to abandon it completely and only use the local public rail system (which entirely eliminate this class of steering errors). But this is argument made based on the state of the technology in 1950s without considering that seatbelts and other modern techniques, driver tests, speed limits can make driving a lot safer (as seen today when compared to the 1950s).<p>Seatbelts are more comparable to activating bounds checking in C compared to switching to another language. because they are a minor inconvenience with a huge positive security effects, but do not eliminate all risk. Rust is more like switching to public rail system, because it does eliminate certain risks completely (all related to steering), but not others, and then also limits you in various way, so you may still need to use a car for the final meters (e.g. C or "unsafe" Rust).<p>And this comparison is consistent with the idea that the public rail system really is safer to use with regard to traffic accidents, so captures the idea that Rust actually has safety benefits. But IMHO the framing and marketing is highly misleading.
how does rust limit you? it seems to be one of the most versatile languages ever. it's genuinely used for everything from the linux kernel to webservers, desktop and mobile apps, even frontend ui in the browser. no other language has that usage (maybe swift?). saying rust is a rail system and c is a car makes zero sense, if anything rust is a flying car because it is used in situations where c is not, whereas rust is used everywhere c is used.<p>any serious engineering discipline eliminates the possibility of human error wherever possible, especially when it results in dangerous effects. and yet in software there are people who prefer exposing others to those dangerous effects because of a personal preference for a 50 year old tool.
This argument is cope.<p>In theory, that <i>could</i> be possible. In practice, there have been many many case studies demonstrating the opposite and not that many case studies demonstrating in favor of your argument.<p>e.g. Cloudflare replaced Nginx with Pingora: <a href="https://blog.cloudflare.com/how-we-built-pingora-the-proxy-that-connects-cloudflare-to-the-internet" rel="nofollow">https://blog.cloudflare.com/how-we-built-pingora-the-proxy-t...</a><p>> With those assurances we can focus more on how a change to our service will interact with other services or a customer's origin. We can develop features at a higher cadence and not be burdened by memory safety and hard to diagnose crashes.<p>> When crashes do occur an engineer needs to spend time to diagnose how it happened and what caused it. Since Pingora's inception we’ve served a few hundred trillion requests and have yet to crash due to our service code.<p>> In fact, Pingora crashes are so rare we usually find unrelated issues when we do encounter one. Recently we discovered a kernel bug soon after our service started crashing. We've also discovered hardware issues on a few machines, in the past ruling out rare memory bugs caused by our software even after significant debugging was nearly impossible.<p>The CVE counts for new parts of the Linux kernel written in Rust are much lower than the (recent) CVE counts for even mature, years old parts of the Linux kernel written in C, to say nothing of the historical record.
Running a homelab is quickly turning into a bigger and bigger liability. Just disabled http/2 on my cluster.
On the other hand I've never been so proactive about maintaining security on my homelab! I did a full audit of my self-hosted services with claude code, strengthened the isolation of all my docker services, I'm running coding agents in sandboxed microVMs now, my firewall settings are better, I set up separate VLAN for IoT, I get security notifications on my dependencies, etc. etc.<p>A lot of stuff that I avoided for years I finally feel like I really need to do now but it also has never been easier to do.
The only port I expose is Wireguard.
Isn't the point of a homelab to not expose it to the internet?
Annoyingly I've switched to strict IP address whitelists for anyone I share services with. Slight hassle but massively reduced the attack service.<p>My box got hammered daily with port scans etc before.
Who cares about homelabs when you have a massive fleet of servers.
Yeah I'm having to do attack surface review and threat modeling like I'm a full enterprise now...<p>Fortunately with Tailscale making it super easy to keep stuff out of the open internet, I don't have much, but even then I'm thinking of taking what I do have down. We see these big CVEs coming across, but I guarantee there's a large number of 0 days sitting around in peoples' stashes all over the world that we don't see.
Everybody here talking about going back to HTTP/1.1, it's a good stop gap solution but we should really think about moving to HTTP3-only as fast as possible. Changing over is easy, if your server doesn't support it just put it behind a reverse proxy that does.<p>Unlike IPv6 it doesn't need any OS or ISP support, just a browser that's newer than 2020. Everyone outside of air gap networks and government sites should be switching.
HTTP is used for a lot more things than web pages, and most non-browser clients in production don’t support HTTP/3. The current LTS Java version (25) for example doesn’t support it.
More like solution should be moving to sane HTTP4 - v3 corporations made for themself so v4 can be made for users :) Without corporations involvement.<p>Apache and Mozilla Funds would be ideal choice for such open source implementation but I doubt they will do anything to extend their usefullness...<p>So: users require just visual part and html/css is not bad for that. Could be better. But transport part can be [re]done at will. Eg. would be nice to have separate layers for connection security and actual transportation.
HTTP/1.1 is still way better and way simpler than HTTP/3.
HTTP desync attacks are reason enough to stay as far away from HTTP1.1 as you can:<p><a href="https://portswigger.net/research/http-desync-attacks-request-smuggling-reborn" rel="nofollow">https://portswigger.net/research/http-desync-attacks-request...</a><p>Unfortunately, there are still plenty of customers behind CDNs which don't support HTTP2 for backend origin connections.
Could you please share why?
Because any HTTP spec after 1.1 is incredibly complicated. You can implement HTTP/1 in half a day, and HTTP/1.1 in a day. But HTTP/2 will probably take you a month and HTTP/3 might take something like 6 months.
It boggles my mind that people are patting themselves on the back for staying on old tech stacks and using a discovered vulnerability as further justification of their decision to stay on h1.1. The response is to patch and move on if your environment requires h2; many large orgs these days use microservices which use h2 (grpc) as its network.<p>There are many legitimate use cases for h2 (eg bidi streaming, latency improvements) and h3 (eg seamless network switching).<p>If you don’t need these features and just serve vanilla text webpages that’s also fine to just use h1.1. Blanket statements that h1.1 is better is just, wrong. Context is important on choosing what’s best for your environment.
You forgot about all the networking hardware and software those run on. The difference is negligible.
Do you realize that no one writes toy HTTP implementations from scratch? I get that you might have a "suckless" mindset but that's just not how the real world works.
That's not the point. The point is that the simpler the protocol is to implement, the simpler it is to validate and harden the implementations. A HTTP/1.1 implementation can be 10x to 30x smaller than an equivalently well-rounded HTTP/3 implementation.
I've definitely done that before on a microcontroller.
A simpler protocol has less surface area for vulnerabilities. I think GP was just using that as an expression of this.
HTTP 1.1 has enough complex behaviors at the edge that the idea of implementing it in a day, correctly, is wildly laughable.
It's debuggable. I can wireshark it and understand everything. I can even speak to HTTP servers directly with a telnet client and fetch pages. I can write microcontroller code in C that issues REST API requests. Pure TCP, nice to work with, no multiplexed UDP hex craps.<p>Simple is good.
~25 years of hardening.
well HTTP3 wouldn't have helped here (CVE-2026-121810)
Because HTTP3 was implemented without any errors? Citation needed I guess.
I assume OpenResty is affected as well? ("High Performance Web Platform Based on Nginx and LuaJIT")
Is this the reason why those banks were down the other day? Did they get advanced knowledge so they could patch their instances? Kind of questionable that the AI labs are creating a two-tiered system for system security if so.
OPs article mentions 2 CVEs, however this one claims that there are 5 new ones:
<a href="https://news.ycombinator.com/item?id=50547423">https://news.ycombinator.com/item?id=50547423</a>
I saw this and realised I'm running nginx in a docker container, very easy update, no waiting for the distro to make the update available
I'm self-hosting Discourse forum software which runs in a Docker container, but it sounds like I can't just easily update myself ? See <a href="https://meta.discourse.org/t/nginx2root-http2-is-on-by-default-in-the-standard-container-whats-the-plan/417762" rel="nofollow">https://meta.discourse.org/t/nginx2root-http2-is-on-by-defau...</a>
Oh no, how annoying, yes I can see how that could be the case if Discourse is running its own copy of nginx with its own configuration inside its container. That's unfortunate - I don't know much about Discourse but I hope that had a good reason to bundle in nginx, seems unnecessary at first glance. I really hope a Discourse developer gets to the issue soon and releases an update, that would be the fastest and easiest way!<p>I am running the official nginx docker image by itself in front of all my services, which are their own docker images. So it's easy for me to update just one of them. This works for me because I want my dedicated server to run all my projects with nginx in front. If I were running Discourse I would want it to not bundle nginx because I've already got one of my own. So that's interesting that they do that.
Never turned http2 on in my side project pfew.
But still, this is one is pretty rough...
This sound suspiciously similar to the issue Microsoft said they were dealing with excess traffic on their front door, taking them down.
maybe not an RCE but a DOS in the same http/tls code.
Is there a summary in a nutshell? The Wiz article just repeats the same sentences over and over again
Mirror: <a href="https://archive.is/dUN0c" rel="nofollow">https://archive.is/dUN0c</a>
How many hours before this is exploited in the wild ? Probably not all that long, that blog has a significant amount of details.
Ops is not fun these days … used to pinned versions and an auditing approval process with SecDevOps … how are you mostly automate the process in a you build it you run it environment ?
Most web servers are undergoing a security tsunami right now. It's important to switch to the ones that are on the fixed side of the tsunami, IMO.
FYI, you need to send around 128GB of data for this bug to work, so not exactly discreet or quick.
don't know why the mainline alpine have the empty image sha-256 : <a href="https://hub.docker.com/layers/library/nginx/1.31.9-alpine/images/sha256-eb37f58646a901dc7727cf448cae36daaefaba79de33b5058dab79aa4c04aefb" rel="nofollow">https://hub.docker.com/layers/library/nginx/1.31.9-alpine/im...</a>
If you use Cloudflare to proxy traffic, you can disable HTTP/2 quickly and easily: <a href="https://developers.cloudflare.com/speed/optimization/protocol/http2/#disable-http2" rel="nofollow">https://developers.cloudflare.com/speed/optimization/protoco...</a>
If Cloudflare is in front of an origin I'd expect that they absorb or block this attack in the first place.
Please know the implications of this it's a much bigger thing to do than usually deserves a quick on off switch
Yeah, if it matters for your workload, then don't do this :) And obviously test outside of production first if you're doing anything important. But if all is happy with http/1.1 then this is likely much easier than making a config change or waiting for a patch that doesn't seem to exist yet for some downstream consumers of nginx (like apisix for example).
You mean moving your site behind the great internet firewall or turning off a second http implementation?
Overhyped
I wonder how much of this exploitation will push the industry towards closed-source. Previously the advantage of OSS was trust and collaborative improvement, but that is getting eroded quite heavily now by the barrage of attacks and insidious discovery.<p>The first step was closing contributions down to known contributors and maintainers. Maybe there’ll be a similar step where only a select group will be able to access — and attest to — the code?<p>Almost like “Restricted Access Source” or “Withheld Source”
Thank you! I had to patch bunch of products I'm working on.
Pretty funny times. AI is making it faster to both write insecure code and to detect buffer overflows. Sell both the problem and the solution. Genius.<p>It'll take a few years for the market to adjust. If nothing else, businesses are good at figuring out ROIs, eventually.
What a beautiful way to start the morning, after PHP2Shell last month and now this!
Docker images are a bit delayed because of unstable GitHub runners: <a href="https://github.com/nginx/docker-nginx/pull/1075#issuecomment-6984191040" rel="nofollow">https://github.com/nginx/docker-nginx/pull/1075#issuecomment...</a><p>Cue xkcd
"Check if our site is vulnerable to the latest nginx security advisory"<p>> Error: API Error: Opus 5.5's safeguards flagged this session<p>Literally the first message, lol.
Is it time to vibe code my whole stack so I'm secure through obscurity?<p>Only half joking.
What’s the point of these researchers if their autonomous system is now finding these vulnerabilities?
Just what I needed on a Wednesday evening after promising myself I'll go to bed before 10. Oh well, thanks for the disclosure.
So...what library/tool/language is next folks? Presumably Wiz is sitting on a bunch more.
Thankfully this time there are patches available right away, unlike that PHP bug not too long ago!
Quite the opposite, it looks like there has been no responsible disclosure here. No update available for anyone running something based on Debian (Ubuntu etc.), only if you go and fetch binaries from the source and do the system integration yourself (easy in this case, but in general that's the point of a distro). For php I could just update at some point that day<p>Otoh I'm not sure what http2 is good for (I've asked around today, some people said the multiplexing speeds things up on high-latency connections. I'm still working on a realistic benchmark since I couldn't find any but it's not looking great so far) and it's default disabled so comparatively few sites are affected. Getting rid of it is easier than waiting who knows how long for it to get packaged
HTTP1.1 has a fundamental flaw related to request smuggling. Fixing specific variants of that problem is a game of whack a mole, as issues arise most often from larger server composition, not individual parts. Fixing the protocol flaw requires a major version increment and a phase out of the old one. Today the hesitation to do that gives us a mitigation option, but in general an http downgrade is a bad thing.<p>There are also features on top of http2 that a downgrade may break: gRPC and Server Side Events for example. The later is compatible with HTTP1.1, but could fail due to connection limits, as multiplexing isn't only about speed. Some web apps may handle that so gracefully, it turns into a performance thing again, but others could halt and catch fire.
[flagged]
Just another set of unfortunate examples of programmers who are bad at C, definitely not a C problem.<p>EDIT: Okay, this comment was snarky and not in good spirit. To be serious, I think we are increasingly seeing that the notion that humans can write non-trivial amounts of C correctly is false. Especially in the face of AI-assisted vulnearbility hunting these programs a crumbling. This is not an indictment of humans, it's a sign that C is not equipped to meet this moment.
The patch (or, one of them) seems very much like a bug that can happen independently of the programming language<p><a href="https://github.com/nginx/nginx/commit/e27f5fa84fe6e98505e70bc543b748747dad0a3b" rel="nofollow">https://github.com/nginx/nginx/commit/e27f5fa84fe6e98505e70b...</a>
This fixes the integer underflow for CVE-2026-123007, and it's true that the underflow can happen in any language, notwithstanding whether it's UB or not. The issue is what happens as a consequence of this underflow, in this case: out of bounds copy, which is specific to C.
Some languages make it significantly easier than C to just forbid integer overflow/underflow entirely.<p>Given that on modern CPUs, cycles are quite fast, and memory latency is quite high, always-checked integer operations sound like an extremely good idea.
Sure, I usually use `overflow-checks = true` in Rust for this reason. However, it is still true that the overflow is not what caused the issue here. In Rust it would've paniced on the out of bounds copy, even with `overflow-checks = false`.
It is not specific to C. You could have bounds checking also in C, and you can have non-checked indexing also in many other languages, including Rust. I would agree though that many other language make it harder to not have the bounds checking, but the bigger question is why this is not done C (and the answer that this is very hard to do is false).
[dead]
I think you forgot the /s<p>C is like a Winchester Model 1897, not saying that it is a bad gun, it is what we call it <i>the</i> good ol' Trench gun, as it ran over the war tunnels of the Great War, and it still packs a punch today, but it only has a half-cock for some level of safety, and there is no trigger disconnector either, so you can easily slam fire it. So it is way too easy to shoot yourself in the foot.<p>Nowadays we like going for a AKM/AK74, for something maybe more complicated but far more reliable at the same time. That's why we want Rust.
Everyday we are seeing new vulnerabilities. Thanks to AI
its crazy how exploits are being discovered in software nowadays, especially using AI
Was this discovered using an AI?
the new "hack" now is to monitor wide-spread repos for critical PRs
My firewall needs a firewall
Is this going to end up being another exploit that requires a ridiculous amounts of special conditions so as to not actually be applicable to majority of sites running nginx?
I think the problem is that while what you say is true, AI is also capable of generating the special conditions required. And if it can’t do so reliably, it can sit there hammering the server for as long as it takes until the conditions are met - as an example hitting a bunch of load balanced servers until 2 requests hit the same workhorse sequentially.<p>It is completely realistic that we start seeing a lot of bugs a human can’t (reasonably) exploit, but AI can.
At this point, I’m not even going to check, so far all of these have been nothing burgers
Uhm, am I missing something or are the patched releases not out yet? Why disclose this level of detail when the patches are not ready yet?<p>Edit: All good. They released the new versions within 30 mins.
Currently, the Ubuntu [1] & Debian [2] packages seem unpatched.<p>[1] <a href="https://launchpad.net/ubuntu/+source/nginx" rel="nofollow">https://launchpad.net/ubuntu/+source/nginx</a>
[2] <a href="https://tracker.debian.org/pkg/nginx" rel="nofollow">https://tracker.debian.org/pkg/nginx</a>
Yeah I'm not quite sure what is going on with Debian, I would have hoped that this vulnerability would have been disclosed in collaboration with Linux distros to get a patch out?
Everyone called me crazy for running Arch on my servers btw
Well the fix for this seemingly already landed in the repo last week: <a href="https://github.com/nginx/nginx/commit/e27f5fa84fe6e98505e70bc543b748747dad0a3b" rel="nofollow">https://github.com/nginx/nginx/commit/e27f5fa84fe6e98505e70b...</a><p>Given how common it's become to constantly scan popular repos for n-days, I expect someone will have already picked it up and started work on an exploit. The workaround for this (disabling HTTP/2) is also comparatively easy to what we had with PHP2Shell just a few weeks back, so I think it makes sense to release this.
Upstream fixes are available in 1.30.7 stable and 1.31.9 mainline.
They're out now.
> Why disclose this level of detail when the patches are not ready yet?<p>Perhaps it was already being exploited?
Based on <<a href="https://news.ycombinator.com/item?id=50547678">https://news.ycombinator.com/item?id=50547678</a>>, a bunch of people reported this to Nginx in parallel. Wouldn't surprise me if they went "Igor, you wont believe, this rce is the same as the five previous I reviewed this week! Let's ship your patches before malicious attacker discovers in parallel also"
[dead]
Critical open source software will have to be shut down to the public
I was talking with someone about buffer overflows due to HTTP requests this morning. I think I said something like "but we don't need to worry about that, just put nginx in front of it." Sigh. Off to update some VMs. Again.
The beatings will continue until morale improves.
Maybe you should not run in production a Russian reverse proxy slash web server commercialized by F5 written in C to begin with...
> Complexity can no longer be relied upon as a barrier to discovery or exploitation.<p>I mean sure if we close our eyes and ears we can ignore the elephant in the room that is proprietary, closed source software.
Great work by wiz.
<a href="https://security-tracker.debian.org/tracker/CVE-2026-123633" rel="nofollow">https://security-tracker.debian.org/tracker/CVE-2026-123633</a><p><a href="https://security-tracker.debian.org/tracker/CVE-2026-123007" rel="nofollow">https://security-tracker.debian.org/tracker/CVE-2026-123007</a><p>Debian still vulnerable in all versions.
Nginx used to be in the OpenBSD base system between 2012-2015.<p>12 years on, OpenBSD's port of nginx still maintains its default chroot(2) patch.<p><a href="https://raw.githubusercontent.com/rnagy/nginx_chroot_patch/master/nginx-1.30.0-chroot.patch" rel="nofollow">https://raw.githubusercontent.com/rnagy/nginx_chroot_patch/m...</a><p>I recommend reading Reyk Flöter's AsianBSDcon 2015 paper about nginx and the thing that replaced it in base.<p><a href="https://www.openbsd.org/papers/httpd-asiabsdcon2015.pdf" rel="nofollow">https://www.openbsd.org/papers/httpd-asiabsdcon2015.pdf</a>
"Both vulnerabilities are memory-safety issues in how NGINX handles HTTP/2 traffic."<p>HTTP/2 again. First Apache, now NGINX<p>I rarely use HTTP/2, I rarely use a popular web browser, I use a TLS forward proxy that has HTTP/2 support but I prefer HTTP/1.1<p>I find that 1.1 is faster for the way I use the www (text-only, no images, CSS, Javascript, cookies, advertising, tracking), plus the 1.1 protocol comes from academia not Google, Akamai, etc.
Hey this is TZubiri, the guy that comments that you should uninstall the dep whenever there is an post about a CVSS 9+ vuln.<p>Today we are going to take a look at how to avoid this vuln by not having Nginx and not using HTTP/2 (for most use cases).<p>We will assume that you are using Nginx for any of the following: Serving files or serving dynamic website content through some integration with a programming language.<p>We will be using python, but you can use any programming language that has a native integration with sockets through it's C interface (literally every language:<p>Step 1: import socket # import the socket library from glibc or glibc.<p>Step 2: socket.socket() # socket.socket()<p>Step 3: socket.bind((PUBLIC_ADDR,80)) # bind before listening obviously. Use a local address if you are below the IPoverty4 line.<p>Step 4: conn,addr = socket.accept() # this creates a connection, wrap it in a While True and a try catch for production. Sprinkle in some import threading to support multiple concurrent requests.<p>Step 4: req = conn.recv(4096) # split on b"\r\n\r\n" to separate headers from body, read HTTP RFCs for the rest. Feel free to reject HTTP/2 requests, they'll probably just resend as HTTP/1.1<p>Step 5: #Fill in the details, most likely if you are using nginx, you will end up connecting two sockets with pipe, or just storing and forwarding to another tcp endpoint. Again take a look at the HTTP RFCs and the glibc socket and stdio manuals.<p>Thank you for reading my tutorial, and best of lucks.
It seems like it always HTTP/2 (except when it's regex) with nginx. Glad I'm http/1.1 only.
nginx is so pervasive, these CVEs are serious.
If you are running a PHP site, you don't need an external webserver anymore, like Apache, NGINX or Caddy.<p>This webserver is not vulnerable to PHP2Shell nor this TLS issue, because it's completely userland PHP:<p><a href="https://github.com/Qbix/webserver" rel="nofollow">https://github.com/Qbix/webserver</a><p>It's now been battle-tested in production, and also supports websockets, etc.<p>(However, it doesn't support sendfile, because PHP doesn't have support for that, so it's not as fast as nginx for sending static files. This shouldn't make a difference, though, if you use a CDN in front.)
HTTP/2 is so complicated that the code that handles it tends to be vulnerable. What a surprise
This doesn’t seem very HTTP/2 specific, though, and it seems more like the conclusion should be “don’t use C for things which matter”. It’s not like HTTP pipelining and keepalives didn’t have plenty of edge cases to worry about.
Caddy also supports HTTP/2 and it's not affected.<p>In fact in general there were waaay less CVEs for Caddy lately compared to nginx
Okay now I believe that AI is finally burning down the internet and that all the incidents are not just pure coincidences.<p>Some of these things could've been coincidence but I remember seeing an apache RCE a few days back and people said, oh imagine using Apache, I use Nginx and now not a few days later Nginx has RCE.<p>And this is just a very very small iceberg of all the recent RCE cases with a serious uptick that I have just observed on Hackernews alone.<p>Basically, there is a sort of justification that people provide with one RCE is that that software X has had issues and to use Y software, but now that X,Y software and A1...Z1 all are being found to have quite severe bugs. I can't in good faith just consider it to be a coincidence.<p>Though I do still hope that this might just be a coincidence and not a recurring pattern, because, A) I don't wish to live in that reality where each software that we trusted seems to have quite so many bugs and which would be exploited and B) Have it be continued for a prolonged period of time.<p>Though the plus point is that we are gonna find extremely resilient software after all of this hopefully as sort of a need in our times.<p>We might have to agree that the cybersecurity world seems to have changed (though I am no expert right now to comment on it but I am curious what experts think)
It was clear a year ago already.<p>Theh people laughing at Anthropic and co for being cautios of Mythos just didn't understand whats happening.<p>Instead of finding, hiring and then finding security issues, you can throw money and compute at it. This alone is a complete game changer. Combined with models which are very very capable.<p>This can be potentially the death of certain programming lanugages or requirement of a complete rewrite. It can mean that only big companies have the resources and capacity to continue playing this game.
Imagine using Nginx
[flagged]
[flagged]
[flagged]
[dead]
[flagged]
To all the people here that would like to give AI credit for everything, and dont think it matters that everyone at Wiz is Israel Ex Cyber "Defense", and all you need is AI to find things like this. I have a challenge for you:<p>If you think you can do it or that anyone with AI dan be dangerous. Please, right now put everything down.. Go make a h1,bugcrowd acct and start making millions. If its as easy as having access to llms please go get rich. I assure you just having access to tokens doesnt get you very far.
Well, you can't because the LLMs are all nerfed and censored w.r.t cybersecurity.<p>I'm not saying domain knowledge is not required. But a lot of people do have domain knowledge and are prevented from using these tools. Even to secure own code bases. This sucks. A lot.
if it wasn't AI then how do you explain 3 other companies co-discovering the issue? Also, didn't one of the companies discover it before Wiz?
The only people who have access to full throttle, no-rejection security pipeline are the OpenAI employees and very preferred security partners...<p>This is a monumentally dumb assertion that any Job Blow can just add a credit card and ram through these requests
Isreael btw everyone at Wiz is a veteran of Unit 8200.<p>But sure, keep telling yourself this is AI lmfao and anyone can hack because of LLMs.<p>Also, im not worried about anyone exploiting this. I have multiple nginx severs running http2. Im going to take a nap. After I wake up I MIGHT consider updating.<p>People in these comments who think they should have given people the opportunity to shutdown http2 in advanced, as if everyone with an llm is going to produce a PoC in 10 minutes. I got news for you, nobody cares about your server and nobody is figuring out how to use this PoC..<p>Im way more worried about Wiz, the literal Israeli malware/spyware company playing games than anyone else. Wiz is kind of lowkey evil and I wish Google hadnt of bought them.
i dont know what it is this but good work
[flagged]