I don't have much experience with GitHub's CI offering. But if this is an accurate description of the steps you need to take to use it securely ... then I don't think it <i>can</i>, in fact, ever be used securely.<p>Even if you trust Microsoft's cloud engineering on the backend, this is a system that does not appear to follow even the most basic principles of privilege and isolation? I'm not sure why you would even <i>try</i> to build "supply-chain security" on top of this.
I would agree with this. I recently tried to figure out how to properly secure agent-authored code in GitHub Actions. I believe I succeeded in doing this[1] but the secure configuration ended up being so delicate that I don’t have high hopes of this being a scalable path.<p>Now, as other commenter pointed out, maybe this is just inherent complexity in this space. But more secure defaults could go a long way making this more secure in practice.<p>[1] <a href="https://github.com/airutorg/sandbox-action" rel="nofollow">https://github.com/airutorg/sandbox-action</a>
Out of curiosity, is there a build setup you have seen in the past that you think could be a good replacement for this complex GitHub CI setup? Asking for a friend ;)<p>Update: now I've finished reading the article, my impression is that complexity is mostly inherent to this problem space. I'd be glad to be proven wrong, though!
I think any of the webhook-based providers are better, because you can isolate your secrets. PRs go to a PR webhook that runs in an environment that just doesn’t have access to any secrets.<p>Releases go to the release webhook, which should output nothing and ideally should be a separate machine/VM with firewall rules and DNS blocks that prevent traffic to anywhere not strictly required.<p>Things are a lot harder to secure with modern dynamic infrastructure, though. Makes me feel old, but things were simpler when you could say service X has IP Y and add firewall rules around it. Nowadays that service probably has 15 IP addresses that change once a week.
The only binaries of uv in the world you can get that were full source bootstrapped from signed package commits to signed reviews to multi-signed deterministic artifacts are the ones from my teammates and I at stagex.<p>All keys on geodistributed smartcards held by maintainers tied to a web of trust going back 25 years with over 5000 keys.<p><a href="https://stagex.tools/packages/core/uv/" rel="nofollow">https://stagex.tools/packages/core/uv/</a><p>Though thankful for clients that let individual maintainers work on stagex part time once in a while, we have had one donation ever for $50 as a project. (thanks)<p>Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.<p>I am annoyed.
This is the market telling you what matters.<p>OpenClaw has been an outstanding success, it is providing people the ability to leak their keys, secrets, and personal data, and allowing people to be subject to an incredible number of supply chain attacks when its users have felt their attack surface was just too low.<p>Your efforts have been on increasing security and reducing supply chain attacks, when the market is strongly signaling to you that people want reduced security and more supply chain attacks!
>Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.<p>Unpaid volunteer hackers provide their work for free under licenses designed for the purpose of allowing companies like OpenAI to use their work without paying or contributing in any form. OpenAI wants to make the most money. Why would they spend any time or money on something they can get for free?
Not sure if you're fully over the context that openAI bought Astral - who "own" uv.
Yep. Permissive licenses, "open source", it's all just free work for the worst corporations you can think.
It's free work for anyone.
Seems like the most cynical take on OSS possible.<p>Like anything good you do an evil person could benefit from - is the solution to never do any good?
Never let the left hand know what the right hand is doing. I suppose it works both ways here, but the specific end user is not why people make code available, it’s in the hope of improving things, even just the tiniest bit.
I don't think you are annoyed. You have done this to produce a reproducible linux distribution which your partners sell support for.<p>I wouldn't find this annoying at all - I would expect to have to do this for hundreds of packages.<p>Without unpaid volunteers things like Debian do not exist. Don't malign the situation and circumstances of other projects, especially if they are your competitors.<p>Compete by being better, not by complaining louder.
(I’m the author of TFA.)<p>> All keys on geodistributed smartcards held by maintainers tied to a web of trust going back 25 years with over 5000 keys.<p>Neither the age nor the cardinality of the key graph tells me anything if I don’t trust the maintainers themselves; given that you’re fundamentally providing third-party builds, what’s the threat model you’re addressing?<p>It’s worth nothing that all builds of uv come from a locked resolution and, as mentioned in TFA, you can get signed artifacts from us. So I’m very murky on the value of signed package commits that come from a <i>different</i> set of identities than the ones actually building the software.
StageX does reproducible builds, so they are signed independently and can also be verified locally. I don't think it applies to Astral, but it's useful for packages with a single maintainer or a vulnerable CI, where there is only one point of failure.<p>But I also think it'd be nice if projects provided a first-party StageX build, like many do with a Dockerfile or a Nix flake.
> Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.<p>Didn't the acquisition only happen a few weeks ago? Wouldn't it be more alarming if OpenAI had gone in and forced them to change their build process? Unless you're claiming that the article is lying about this being a description of what they've already been doing for a while (which seems a bit outlandish without more evidence), it's not clear to me why you're attributing this process to the parent company.<p>Don't get me wrong; there's <i>plenty</i> you can criticize OpenAI over, and I'm not taking a stance on your technical claims, but it seems somewhat disingenuous to phrase it like this.
Yeah, I'll just establish for the record that we've been thinking about this for a long time, and that it has nothing to do with anybody except our own interests in keeping our development and release processes secure.
What are you using for signed reviews?
The private jet wont fuel itself now will it.
>Why is it a bunch of mostly unpaid volunteer hackers are putting more effort into supply chain security than OpenAI.<p>To be frank. Because more effort doesn't actually mean that something is more secure. Just because you check extra things or take extra steps that doesn't mean it actually results in tangibly better security.
FYI it was actually William Woodruff (the article author) and his team at Trail of Bits that worked with PyPI to implement Trusted Publishing.
I maintain `repomatic`, a Python CLI + reusable workflows. It bakes most of the practices from this post into a drop-in setup for Python projects (uv-based, but works for others too). The goal is to make the secure default the easy default for maintainers who just want to ship packages. Also addresses a lot of GitHub Actions own shortcomings.<p>But thanks to the article I added a new check for the fork PR workflow approval policy.<p>More at: <a href="https://github.com/kdeldycke/repomatic" rel="nofollow">https://github.com/kdeldycke/repomatic</a>
One (amongst other) big problem with current software supply chain is that a lot of tools and dependencies are downloaded (eg from GitHub releases) without any validation that it was published by the expected author. That's why I'm working on an open source, auditable, accountless, self hostable, multi sig file authentication solution. The multi sig approach can protect against axios-like breaches. If this is of interest to you, take a look at <a href="https://asfaload.com/" rel="nofollow">https://asfaload.com/</a>
I’m maybe not understanding here, but isn’t it the point of release attestations (to authenticate that the release was produced by the authors)?<p>[0] <a href="https://docs.github.com/en/actions/how-tos/secure-your-work/use-artifact-attestations/use-artifact-attestations" rel="nofollow">https://docs.github.com/en/actions/how-tos/secure-your-work/...</a>
The problem is nobody checks.<p>All the axios releases had attestations except for the compromised one. npm installed it anyway.
Artifact attestation are indeed another solution based on <a href="https://www.sigstore.dev/" rel="nofollow">https://www.sigstore.dev/</a> . I still think Asfaload is a good alternative, making different choices than sigstore:<p>- Asfaload is accountless(keys are identity) while sigstore relies on openid connect[1], which will tie most user to a mega corp<p>- Asfaload ' backend is a public git, making it easily auditable<p>- Asfaload will be easy to self host, meaning you can easily deploy it internally<p>- Asfaload is multisig, meaning event if GitHub account is breached, malevolent artifacts can be detected<p>- validating a download is transparant to the user, which only requires the download url, contrary to sigstore [2]<p>So Asfaload is not the only solution, but I think it has some unique characteristics that make it worth evaluating.<p>1:<a href="https://docs.sigstore.dev/about/security/" rel="nofollow">https://docs.sigstore.dev/about/security/</a><p>2: <a href="https://docs.sigstore.dev/cosign/verifying/verify/" rel="nofollow">https://docs.sigstore.dev/cosign/verifying/verify/</a>
> without any validation that it was published by the expected author<p>SPOF. I'd suggest use automatic tools to audit every line of code no matter who the author is.
Overall I believe this is the right approach and something like this is what's required. I can't see any code or your product though so I'm not sure what to make of it.
Here's the GitHub repo of the backend code:
<a href="https://github.com/asfaload/asfaload" rel="nofollow">https://github.com/asfaload/asfaload</a><p>There's also a spec of the approach at <a href="https://github.com/asfaload/spec" rel="nofollow">https://github.com/asfaload/spec</a><p>I'm looking for early testers, let me know if you are interested to test it !
If anyone from Astral sees this: at this level of effort, how do you deal with the enormous dependence on Github itself? You maintain social connections with upstream, and with PyPA... what if Github is compromised/buggy and changes the effect of some setting you depend on?
We talk to GitHub as well! You're right that they are an enormous and critical dependency, and we pay close attention to the changes they make to their platform.
> what if Github is compromised/buggy<p>What if? GitHub has is extremely buggy! I'm getting increasingly frustrated with the paper cuts that have become endemic across the entire platform. For example its not uncommon for one of our workflows to fail when cloning a branches of the repo they are running in.
The open source ecosystem has come very far and proven to be resilient. And while trust will remain a crucial part of any ecosystem, we urgently need to improve our tools and practices when it comes to sandboxing 3rd party code.<p>Almost every time I bump into uv in project work, the touted benefit is that it makes it easier to run projects with different python versions and avoiding clashes of 3rd dependencies - basically pyenv + venv + speed.<p>That sends a cold shiver down my spine, because it tells me that people are running all these different tools on their host machine with zero sandboxing.
Pinning github actions by commit SHA does not solve the supply chain problem if the pinned action itself is pulling in other dependencies which themselves could be compromised. An action can pull in a docker image as a dependency for example. It is effectively security theatre. The real fix is owning the code that runs in your CI pipelines. Or fork the action itself and maintain it as part of your infrastructure.
We do address this in the article! It's defense in depth, not theater.<p>We audit all of our actions, check if they pull in mutable dependencies, contribute upstream fixes, and migrate off using any action when we can.<p>(I work at Astral)
Shouldn't you always read & double-check the 3rd-party GitHub actions you use, anyway? (Forking or copying their code alone doesn't solve the issue you mention any more than pinning a SHA does.)
> It is effectively security theatre.<p>I disagree. Security is always a trade-off.<p>Owning, auditing, and maintaining your entire supply chain stack is more secure than pinning hashes, but it is not practical for most projects.<p>Pinning your hashes is more secure than not pinning, and is close to free.<p>At the end of the day, the line of trust is drawn somewhere (do you audit the actions provided by GitHub?). It is not possible to write and release software without trusting some third party at some stage.<p>The important part is recognizing where your "points of trust" are, and making a conscious decision about what is worth doing yourself.
With the recent incidents affecting Trivy and litellm, I find it extremely useful to have a guide on what to do to secure your release process.<p>The advices here are really solid and actionable, and I would suggest any team to read them, and implement them if possible.<p>The scary part with supply chain security is that we are only as secure as our dependencies, and if the platform you’re using has non secure defaults, the efforts to secure the full chain are that much higher.
This is a really great overview; what a useful resource for other open-source projects.
The entire paragraph about version pinning using hashes (and using a map lookup for in-workflow binary deps) reminds me that software engineers are forever doomed to reinvent worse versions of nixpkgs and flakes.<p>I don't even love Nix, it's full of pitfalls and weirdnesses, but it provides so much by-default immutability and reproducibility that I sometimes forget how others need to rediscover this stuff from first principles every time a supply chain attack makes the news.
Lengths people will go to rediscover Nix/Guix is beyond me
Isn't Nix just reinventing what Vesta did for software reproducibility decades earlier? <a href="https://vesta.sourceforge.net/" rel="nofollow">https://vesta.sourceforge.net/</a>
I don't see the connection though?
Nix provides declarative, reproducible builds. So, ostensibly, if you had your build system using Nix, then some of the issues here go away.<p>Unfortunately, Nix is also not how most people function. You have to do things the Nix way, period. The value in part comes from this strong opinion, but it also makes it inherently niche. Most people do not want to learn an entire new language/paradigm just so they can get this feature. And so it becomes a chicken and egg problem. IMHO, I think it also suffers from a little bit of snobbery and poor naming (Nix vs. NixOS vs. Nixpkgs) which makes it that much harder to get traction.
Nix, if not used incorrectly (and they really make it hard to use it, both correctly and incorrectly lol), gives you reproducible and verifiable builds.<p>Unfortunately I have to agree with the sibling comment that it suffers from poor naming and the docs are very hard to grok which makes it harder to get traction.<p>I really hate the idea of `it's all sales at the end of the day` but if Nix could figure how to "sell" itself to more people then we would probably have less of those problems.
If it doesn't work on Windows, it is not a full replacement.
Reading the paragraph on hash pinning and "map lookup files" (lockfiles) made me audibly sigh.
Earlier submission from author: <a href="https://news.ycombinator.com/item?id=47691466">https://news.ycombinator.com/item?id=47691466</a>
Super useful info... but I feel so tired after reading it
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
Created an agent skill based on this blog. Assessing my own repos now.<p><a href="https://github.com/backnotprop/oss-security-audit" rel="nofollow">https://github.com/backnotprop/oss-security-audit</a>