As I mentioned in the Northflank breach post:<p>Unfortunately for infra startups, I think we're going to see a convergence back to hyperscalers as a result of incidents like this and railway.<p>There's no amount of "nicer to use" that can replace security, at least certainly when selling to enterprises<p>Also, a trick i've started doing is encrypting my env vars, then hard-code the decryption key in the code
As far as my personal life is concerned, I honestly don't think I'm going to 'build and deploy' anything to any SaaS that isn't directly in my control for a while.
damn, i've been thinking about the same recently. saas is dead, at least developer oriented saas for sure. the only things i rely on are my own self hosted git (not public to the web), and that's it. all other "developer oriented saas" has been replaced with a home made version with claude. probably not safe from hacking, but id rather take the risk of someone targeting ME specifically, than let all the "saas platforms" store all my data and get hacked every monday and friday
I am sorry, this is "Resolved"? How can they even consider marking this resolved in any situation. Yeah we exposed your env vars but we told you about it over email so it's all good... We dont know how or to what extend though.
The biggest case for building bespoke software is that you drop your attack surface area and it’s not as lucrative to hack one person. Don’t even expose it to the internet if you don’t need to and just run Tailscale.
Or do onprem or selfhosted. Surface area reduces drastically.
I was also referencing that using commercial software usually brings features you probably don’t use. Those features probably require new NPM packages so by building a bespoke software, you only build features you need.
this is often a worse time-to-patch
Tailscale got hacked recently as well, didnt they (?). I think they did, the whole system is poisoned, because I believe one of the vectors had to do with eMudhra certs, which again, is not an issue specific to them, or something they had any say on it... there is literally no escape these days it would seem.
tailscale got hacked recently? sources?<p>they had a priviledge escalation vulnerability that requires the attacker to already be a part of the tailnet, and more specifically allows a user-to-root escalation devices the attacker can tailscale ssh to.<p>> Not trying to play down the vuln (its important), but its not a free-for-all RCE that some ppl are acting like it is. [1]<p>eMudhra/TLS certificates have nothing to do with this (?).<p>[1] <a href="https://news.ycombinator.com/item?id=50266179">https://news.ycombinator.com/item?id=50266179</a>
That was Tailscale SSH and it’s not on by default
I think you've just explained why people rent infrastructure.
This is probably correct if you aren’t a target at all, e.g. you aren’t running a business
It’s bold to set status page incident to resolved.
If they fixed the exploit then what do you want them to do? Do they need to wait an arbitrary amount of time before setting it to resolved?
I don't disagree with you, but I think generally when the mistake involves compromised user data, the answer is a bit more emotionally driven. People get rightfully mad when things like that happens, and services usually keep the incident in a "Mitigated" status while they "work with their customers to ensure blah blah blah" or whatever. Everywhere I've personally worked the Mitigated->Resolved involves publishing a postmortem, repair items, and general assurances or safe guards based on learnings from the incident that the issue won't happen again. However, the industry as a whole doesn't have a standard for what the line between Mitigation and Resolution is.
To use a fearful, apt analogy: resolving the operational issue isn't just plugging the hole in the dam, its also fixing the damage by pumping the water out and then addressing the damages.
they could have explained a bit better with proper postmortem and steps taken to improve, before marking it as resolved
They are no longer exposing anything, so they no longer have an incident.<p>Their customers now have an incident that will be resolved once their secrets are rotated. Maybe they also need to do previous intrusion forensics… In any case, not triggers problem anymore.
I like the fancy gradient INCIDENT REPORT image as well. At least it looks fancy and AI-ey
This also happened to a trigger.dev competitor a few months ago, inngest.<p>I was able to patch before it was exploited but that definitely made me question the service.
This should not be marked as "resolved". "Mitigated" until some percentage of customers ack the email notification at least (track clicks, get teams on Zoom call, etc.).
Have any of these companies disclosed the actual vulnerabilities used to gain access to their systems?
Northflank mentioned they patched CVE-2026-98432. This appears to be similar, though not confirmed by trigger.dev yet
No. Which may mean they don't know.
Holy smokes, is security just not a priority at all anymore? Like in what world do we live in, that this is acceptable. And at the volume this has been happening, bad guys are having a FIELD DAY.
In my experience, small teams put security off because someone on the dev team has some security experience, or thinks they do. But it is never the priority, there is always something more urgent to ship, and nobody has real skin in the game because it is not their job. Then shit happens. They rush to hire a security team, and that team spends the next few years fighting the culture.
What makes you think security was ever a priority?
Security has never been a priority in the software world
So at this rate, how long before people start hiding use of AI in development of in their SaaS and services? or people really don't care, do they?
Given the recent Claude Code remote access tool, it would be wise to not install any new software, or use any package manager for a few weeks as they could have already been silently backdoored during the aftermath of that incident.
The very fact that Nextjs vibe projects need an external (and paid!) background jobs / async service show how much of an anti pattern that this framework and similar are. Just use a sensible stack from day one that can handle concurrency out of the box like Elixir or something with established open source background jobs library. Then you can worry less about this kind of paid 3rd party nonsense. Ditto on auth. I just can't fathom outsourcing these types or core functionalities, the operational, security and cost risks make very little sense when there's libraries that do this out of the box.
Yes, outsourcing these kinds of core functionality (your queue, your users table!) is insane to me.
You can run trigger in your own infra. That has very little to do with the libraries used for your web app
Trigger and friends exist precisely because Node is not a serious backend runtime so you have to reach for bolt-ons for every little thing. No matter if trigger is open source or managed, it's an anti-pattern. Erlang/Elixir, C#, Go etc all handle this sort of async natively, literally out of the box. Mature frameworks like Rails, Django have pretty easy to use officially libraries as well. This has been a solved problem for literally decades.
[flagged]
[flagged]