This is the right way to deliver software.<p>Produce working product first, validate the idea, stabilize the business, start generating profit, and then you can start optimizing your costs.<p>In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivial.
Its a yes if you do not know the domain space, query patterns well enough and also if the cost of optimization or time for optimization may have detrimental impact to business. In this case it most likely means that the crowd in the room did not anticipate much on this in early phases and no one in the room pointed these things out. The irony is that these performance and disk numbers are heavily discussed as a part of system design interviews.<p>> In fact optimization is by far the easiest part of the process because there are many system programming experts on this HN thread who consider these optimizations to be trivia<p>This is a misconception when you including roll out as a part of the change too, changing data once its running in production is hard, changing the data structure is even harder and when you talk about making changes in cache which is at the hot path its probably the hardest. Looking at the graph at the end it looks like it took them 4+ months to roll out the changes after optimization.
“changing data once its running in production is hard, changing the data structure is even harder”<p>100% agreement on this. There are a class of optimizations that can happen transparently. Those can happen at any time, and are fine to defer. Not all profiling and scalability improvements fall into this bucket. Some are very expensive to roll out, and ignoring these concerns can cause huge headaches down the line. Not fun to hear, but it’s definitely true. Even with LLMs, this can still be a huge challenge.
The fact that it took them 4 months to roll out does not mean this is the hard part: it's simply a coordinated rollout with incremental, staggered deployments and rate-limited migrations.<p>Changing data structures with that approach just takes its time as you avoid lock-step updates between components. Sure, by definition this type of development and deployment complexity is hard.<p>However, what I found the hardest is pushing engineers to adopt this evolutionary data structures mindset, and unless you do that right for the full team, someone will sneak in a backwards-incompatible change that blows the entire effort up.<p>So it is hard, but primarily for different-mindset-needed, and only then for technical complexity.
> This is a misconception<p>Pretty sure they were joking.
I do not think Cloudflare was a less-than-peers optimized product when they launched. This is one of their blog posts which describes taking one aspect even further.<p>I think Cloudflare became big only because they were so much more optimized than others that they offered some services for free that others were not offering. If running costs are high, you only burn (VC) cash and then you exit.
There is the entirely plausible option of the NSA indirectly bankrolling them to counteract the growing number of TLS connections. We terminate your TLS for free in our server and you don't have to change anything was a sweet deal for website operators _and_ those that want to look at unencrypted traffic. Given enough sustained funding they could undercut competitors and grow to what they are today.
First version was a three layer VM monster, with the main entrance done as HTTP proxy written in PHP.
Or optimize a bit earlier and prevent having to scale out to a bazillion systems.
The way I usually prevent having to scale out to a bazillion systems is never getting more than 10 users.
And that's why I charge $10,000,000/user/mo.
I wonder why Cloudflare didn’t think of this
The Art of Production
The art of premature optimizations
The art is in knowing how to write software that doesn't perform like shit without doing all the work of measuring and refining. If you can save $100k in hardware costs in a couple days by just knowing what you're doing, that optimization is not premature.
And yet when Prof. Donald Knuth wrote that in 1974 paper[1] it was in this context:<p>> "<i>The improvement in speed from Example 2 to Example 2a is only about 12%, and many people would pronounce that insignificant. The conventional wisdom shared by many of today's software engineers calls for ignoring efficiency in the small; but I believe this is simply an overreaction to the abuses they see being practiced by pennywise-and-pound-foolish programmers, who can't debug or maintain their "optimized" programs. In established engineering disciplines a 12% improvement, easily obtained, is never considered marginal</i>"<p>also:<p>> "<i>In the late 1960's we witnessed a "software crisis", which many people thought was paradoxical because programming was supposed to be so easy. As a result of the crisis, people are now beginning to renounce every feature of programming that can be considered guilty by
virtue of its association with difficulties. Not only go to statements are being questioned; we also hear complaints about floating-point calculations, global variables, semaphores, pointer variables, and even assignment statements. Soon we might be restricted to only a dozen or so programs that are sufficiently simple to be allowable</i>"<p>In a recent comment I mentioned a youtube interview with Rico Mariani, a performance engineer from Microsoft, and he said that he often got called into projects approaching their deadlines and not meeting their performance goals.<p>In one anecdote he spent a couple of hours with a team and showed how their design could never meat the goal even with the fastest disks, CPUs, memory, and network. And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization".<p>[1] <a href="https://dl.acm.org/doi/pdf/10.1145/356635.356640" rel="nofollow">https://dl.acm.org/doi/pdf/10.1145/356635.356640</a>
I believe this is the video - around 50:00 into the video - <a href="https://www.youtube.com/watch?v=48Rig6v-xYU&t=3000s" rel="nofollow">https://www.youtube.com/watch?v=48Rig6v-xYU&t=3000s</a>
> And commented how strange it is if they had spent a day at the start of the project whiteboarding out the design against hardware specs at the start of the project - and avoided months of wasted effort - that would be called "premature optimization".<p>Oof that hit hard. The last project I worked on suffered from a very similar disease, and it has really taken a toll on me psychologically. To work day in and day out on something that you <i>can prove</i> cannot work is unbelievably demoralizing. From an organizational standpoint, it makes a lot of sense to have an "internal consultant" who can deliver bad news like this. I tried to do it from the "inside" which was a huge mistake--got a negative performance review saying I had a "communication problem" because nobody wants to hear "negativity". I can come off online as kind of an asshole, so this may not seem credible, but I did actually deliver this news in a professional, measured manner. It's just that organizations are allergic to it, and their antibody response kicks in. You need someone who is not affected by the organizational hierarchy (or at least not that branch of the tree) to step in and deliver the bad news without fear of retaliation.
pro move. made my evening.
You're never going to get promoted with that attitude!<p>I'm joking...but not entirely. It sounds impressive on a promo packet when you say you've saved 100 TB of RAM / $$$ through whatever technique. But it sounds a lot less impressive when you say if this system grows to this size in x years, I will have saved 100 TB, especially when no one yet knows how large the system will really be in that time or what the cost of RAM will be. I dunno, maybe if you say that x years ago, I made a decision that now is saving us 100 TB, that's kinda impressive, but you're also getting credit for it x years after you did the work. It also doesn't have the implication that it must be inherently complex/hard because some other smart person chose the other way. And there is a bias to care more about recent accomplishments. So I don't really think it'd be valued the same at all.<p>Also, in general big tech (at least Google) prefers growing the userbase over improving efficiency. Periodically efficiency is rewarded, e.g. when RAM cost suddenly balloons or some big must-have feature has suddenly used up capacity planned for something else. You get rewarded for doing efficiency work on demand, not eagerly.<p>I once got a $100 peer bonus for finding 100,000 cores that were essentially stranded by an accounting error in another team's migration script.
Remember that everything has an opportunity cost. Running a lot of servers might cost $10 million annually, but if the product team had to choose between a project that would recoup $5 million of that vs. an opportunity to earn $50 million ARR for the same amount of work, the logical answer would be obvious.
In my experience optimizations actually preformed tend to have ridiculously high ROI because they are so rarely prioritized.<p>Better performance = saving money + better user experience.
Depends on how you measure your ROI and how it could be different from how your company measure their ROI.<p>The problem with optimizations is that you are competing in prioritization with other features. Reducing the baseline cost always has a limit of zero, while the upside from new features is infinite according to your leadership and investors, so it is very hard to argue against.<p>In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.
> The problem with optimizations is that you are competing in prioritization with other features.<p>I think companies often over-indulge in features nobody wants, needs, or cares about. I quit my previous company because they were forcing us to build something that had single digit weekly active users. It was utterly pointless, driven entirely by some half baked navel gazing harebrained ideas about what a "nontechnical user" might want. But nobody ever asked any real users.<p>I estimate the company probably blew the greater part of $10M on this bullshit, not counting opportunity cost.<p>> In general it's challenging to convince a non-tech crowd of the importance of addressing any tech debt unless you can demonstrate a tangible financial impact on the product, such as delayed contracts or customer churn.<p>People like that are problematic not just because they don't understand tech debt. They also don't understand products. There are shitloads of people in the industry who market themselves as some kind of mystical gurus, are able to deliver impressive monologues talking over everyone on the zoom call, but contribute nothing else than a sense of urgency and frustration. If you find yourself in their company, better to just leave.
That's assuming the ops team has infinite capacity.
You're assuming faster/more resource efficient software would result in the same ARR as the laggy slow one, but that's not a given.
It was already reasonably lean. If they had 10 bazillion systems, they now need somewhere between 6 and 8 bazillion systems.
You can build foundations that aren't extermely optimal but have future optimisations in mind.
> start generating profit, and then you can start optimizing your costs.<p>this assumes you can generate profit before you can get optimized - what if profit generation is only possible with optimized software? A lot of online MMO-style games tend to require such optimizations as they scale into the size required to generate profit.<p>Or, in the current era of ai, the cost of the capital investment is far exceeding the ability to generate profit off it. The optimization in how the resources gets used will be needed to cut the costs down, and allow increase in the scale of usage for the same hardware. That's where profits would lay.<p>Of course, in order to achieve any of this, you'd need the runway to survive until such times. A small scale operator won't have this runway, and so die before they can accomplish anything profitable (or get big by begging for investor money to grow large - as we've seen in the past 20 years of tech).
This assumes that you have plenty of cash to burn in the process, which is approximately correct for VC-backed ventures, and for offshoots of large corporations that play a lomg game.
This is true if you can scale out (ie you can add resources to your system). But for a robot for example, just adding a GPU can just flop your product completely: you need more battery, more weight, suddenly your unit economics is out of the window... Your next hardware iteration will be very slow to come and very expensive.
So here, you better not have a system wasting too much resources pretty early on after the prototype phase.
What I find surpring here is this being about DNS. Simple optimization should have been done maybe 1997. Letting it build up to 100 TB is noteworthy, but on the other hand for IT that's common. It's surprising but also totally expected...
You underestimate the cost of "optimization", sometimes it means actually rebuilding large parts of the system. I would not say it is the "easiest" part, but it is usually not what will kill your business though.
Not the right order if the optimization is a prerequisite for a positive business case. That happens more often than people think...
right or wrong, good or bad are all taste.<p>from a business perspective, this might be considered the only way, but it is not. at large volume scale it becomes more, but often large scale is lacking optimisations in the first place.<p>its not wrong in my eyes, but definitely not the only path to take.
>optimization is by far the easiest part of the process<p>Not if the whole thing is architected poorly but was a requirement of the hour so it became big. Then optimisation becomes an art, but definitely not the ‘easiest part of the process’
This reasoning assumes you have access to infinite runway. You don't.
Exactly, and you need to start turning a profit before the end of that runway. Even if that means running code that is suboptimal.
This reasoning is largely centered around the runway being finite. You obviously can't have costs so high you are making a huge loss, but also there's little value in improving margins past profitability until you actually have a stable segment of the market.
we are all perfectly smooth, round, and filled with an incompressible liquid
Every startup is one bet in a Martingale strategy played by the class of people who remain solvent when you bust.
Make it work, make it fast, refactor
> start generating profit, and then you can start optimizing your costs<p>Good thing they jumped on that as soon as they were profitable instead of burning cash. Oh wait...<p>I think a distinction to draw here is that Cloudflare had relatively large capital raises and were almost immediately profitable¹. They had the luxury of throwing away money. Judicious optimisation makes sense for scrappy start-ups, especially when trivial optimisations like these could easily be farmed off to an agent.<p>¹ <a href="https://timeline.www.cloudflare.com/" rel="nofollow">https://timeline.www.cloudflare.com/</a>
Only if you have loads of capital
> Produce working product first, validate the idea, stabilize the business, start generating profit,<p>not everybody is so lucky to be able to go in that order? The first part requires upfront capital/investment?
[dead]
This is why system programming still matters.<p>Looks like they're missing the obvious optimisation of putting the record data right after the CacheEntry members instead of allocating memory separately though. But that might just be me as a C-programmer talking and not be all that easy in Rust.
For the curious, this is <i>technically</i> possible in Rust using a dynamically sized type [1], but in practice is difficult and doesn't really play nice with the rest of the language. The nomicon entry concludes with "Yes, custom DSTs are a largely half-baked feature for now." [2]<p>[1] <a href="https://doc.rust-lang.org/reference/dynamically-sized-types.html#r-dynamic-sized.struct-field" rel="nofollow">https://doc.rust-lang.org/reference/dynamically-sized-types....</a><p>[2] <a href="https://doc.rust-lang.org/nomicon/exotic-sizes.html" rel="nofollow">https://doc.rust-lang.org/nomicon/exotic-sizes.html</a>
> putting the record data right after the CacheEntry members<p>I assumed they couldn't do that because they're using it with some kind of generic HashMap<K, V>. In that situation, can "V" be dynamically sized?<p>A dynamically sized "V" would mean you can't have an array of them, which might preclude some hash map implementations.
> All type parameters have an implicit bound of Sized. The special syntax ?Sized can be used to remove this bound if it’s not appropriate.<p>, which HashMap does not do, i.e. the keys and values have to have a statically known size.
Unfortunately, Rust is not a good choice for this kind of tricks. This is where Zig shines. In Rust, you can’t even use proper arenas, which can help a ton with allocations.<p>Cloudflare started to pick Zig recently, for projects, that have memory constraints.
> In Rust, you can’t even use proper arenas<p>You definitely can and this is done a lot. What you might mean is that you can't use standard library's collections with them (this is getting stabilized soon!) and have to use third-party, but that is a different thing than "can't use arenas".<p>> Rust is not a good choice for this kind of tricks.<p>Rust <i>can</i> do those tricks, but it's true that it is hard than in C or Zig. That said there are often crates to help.
I'd like to know why I can't use arenas in rust? Especially considering that I have used them before in rust.
Rust supports arenas just fine ( <a href="https://crates.io/crates/bumpalo" rel="nofollow">https://crates.io/crates/bumpalo</a> ), and if you mean the support for using custom allocators in the standard library collections, that's as stable as Zig is.
System programming always matters. Things are cheap until they aren't one day.
Depends on how the CacheEntry is stored, it's probably stored in a slice of &[CacheEntry] which precludes storing the record data alongside it as the size of each entry must be fixed.
less ergonomic, but still totally doable
I wish more programming languages implemented record types as seen in databases, where dynamically sized fields are packed into a contiguous area of memory.<p>The CloudFlare manually implemented a clumsy version of this.<p>Wouldn’t it be nice for the compiler to manage this for you in the same way that your database engine does when it saves a “row”?
> dynamically sized fields are packed into a contiguous area of memory<p>Are you able to explain this? Do you mean an N sized array where each entry is either a value or a pointer to a value where the 'pointed-to' values are after the end of the array?<p>I'm trying to underatnd how you'd do this without having to parse M-1 elements to get the Mth entry if you did a [{size0, value0}, ....., {sizeN, valueN}] arrangement
I think they mean the cache entry is a collection of dynamically sized fields. It would be nicer to store that as a single contiguous allocation, rather than a bunch of pointers to individually allocated dynamically sized items. At least in this case, it might.<p>In a row oriented database, you get a contiguous spot for the whole row even when there are multiple variable width fields.
There are various ways of implementing this, someone from a C programming background mentioned on option where the heap-allocated record objects aren't fixed size structs, but instead the allocated space is dynamically sized and the struct is just a <i>prefix.</i><p>So logically you'd have the equivalent of:<p><pre><code> struct FooRecord {
int fixed_sized_field;
char some_other_field;
string first;
string last;
string title;
}
</code></pre>
Physically the compiler would generate something like:<p><pre><code> struct FooRecord {
long __length__;
int fixed_sized_field;
char some_other_field;
char* first;
char* last;
char* title;
}
</code></pre>
Where 'first', 'last', and 'title' are sequentially stored after the struct in the heap memory.<p>There are variants of the above, of course. Instead of pointers the compiler could use lengths, offsets, or a pointer to the <i>end</i> of the variable length field -- this works because the beginning of the first field is at a fixed offset, and then pairs of pointers delimit the rest.<p>You can rely on the heap allocator to track the "__length__" instead, or you can encode it into the record explicitly to make "dynamic sized copies" simple.<p>Windows APIs generally work this way! You create a buffer, put a length in the first field, and then the API call writes a fixed-sized prefix followed by the dynamic-sized fields into the buffer. The 'length' is replaced too, so you know how many bytes to copy out without having to understand the structure.<p>Database engines go one step further and pack multiple "records" into a single "row". They typically store the fields "packed" at the start of the row with 16-bit length or offset markers at the end for the various dynamic sizes.<p>Something like:<p><pre><code> fixed_sized_field // Row #0
some_other_field
first
last
title
fixed_sized_field // Row #1
some_other_field
first
last
title
... empty space ...
next_offset // always populated
row#1_title_offset
row#1_last_offset
row#1_first_offset
row#1_offset
row#0_title_offset
row#0_last_offset
row#0_first_offset
row#0_offset // typically the constant zero
</code></pre>
The idea here is that every length is the difference between pairs of sequential offsets. I.e. row#1_title has length (next_offset-row#1_title_offset).
With my own MaraDNS, I aggressively optimized the memory usage of blacklist entries by having a single really big malloc() to allocate the memory for the entries, then traversing that memory block for potentially blacklisted entries.<p>When I was using one malloc() per entry, a large blacklist took up 237 megabytes of memory. The same blacklist, once optimized to be loaded with a single malloc() call, only took up 9.5 megabytes of memory.<p><a href="https://samboy.github.io/blog/entries/MaraDNS.html#BlogEntry-2022-12-28" rel="nofollow">https://samboy.github.io/blog/entries/MaraDNS.html#BlogEntry...</a>
Might be of interest:<p>- <i>TigerBeetle: A database without dynamic memory allocation</i>, <a href="https://news.ycombinator.com/item?id=33192288">https://news.ycombinator.com/item?id=33192288</a> (2022).<p>- <i>Succinct Data Structures: Cramming 80,000 words into a Javascript file</i>, <a href="https://news.ycombinator.com/item?id=2348619">https://news.ycombinator.com/item?id=2348619</a> (2011).
Why do I always find interesting new Twitter accounts just as the person is leaving :)
Twitter has become a cesspool, and there’s a lot of reasons why people are leaving it in droves.<p>My personal issue is the misogynists who have created a hateful completely false narrative that 80% of the women sleep with 20% of the men (including the very demeaning and hurtful notion that all women are sexually promiscuous, but only if you’re one of the 20% of supposedly “Alpha” men) [1] Twitter is also full of—let’s call a spade a spade—racists who constantly post some video from years before showing some random Black person doing a criminal act, and then a bunch of racists comment that that’s how all Black people are and it’s the “evil left wing media” suppressing this supposed “truth”.<p>Just as Twitter has become a right-wing cesspool, Reddit has become a leftist cesspool, so I also avoid Reddit, which, like Twitter, is also becoming a closed walled garden—they just this month started clamping down on people reading old.reddit.com anonymously, so now you have to log in to have a usable interface with Reddit. Excuse me, no.<p>[1] This annoyed me to the point I researched the claims to verify it’s a bunch of bullshit. <a href="https://samboy.github.io/blog/80-20-myth.html" rel="nofollow">https://samboy.github.io/blog/80-20-myth.html</a>
This reminds me how you can save a bunch of bytes just by making sure your structs are aligned. In go for example:<p><pre><code> type Wasteful struct {
a int16
b int
c byte
}
type Aligned struct {
b int
a int16
c byte
}
</code></pre>
Will have sizes of 24bytes and 16bytes (on a 64bit system). Same data 8bytes more. If you are storing millions of those objects, then it adds up.
Rust does that automatically unless you switch to the C layout.<p>In langages that don’t there’s a tension between memory use and human readability / consistency of the layout.<p>There are also other domains which can be affected e.g. databases, it’s a concern / issue when using postgres for instance as it uses aligned columns and stores them in schema order.
Why this is not done automatically by the compiler? That seems something quite easy to calculate to me.
There is no way in C to express that you don't care about the orde. When you express a struct in C, you list what you want in the struct and (sometimes without wanting it) exactly in what order you want it.<p>Interestingly, there is also no way to write a loop on i for all the values between 0 and 99 without specifying the order. Luckily, in this case, the compiler is allowed to prove that the order has no impact (because it's local), and to decide that it will scan the values in a different order for optimisation purposes.<p>So the compiler could do it on a structure as well, as soon as it's able to prove that the structure is not exposed in any way to any code that it doesn't control, but that's much more difficult than proving that variable i is not visible outside of a tight loop.
These seem like some fairly standard approaches for reducing memory usage. I can't help to think that the approach of joining several distinct list into a single one in some way undercuts Rust's safety guarantees.<p>If you previous had three distinct Vec objects, then Rust would guarantee that you can't index out of bounds. If you now put all those objects into a single Vec and rely on offsets, then you now open the door to indexing out of range of these sub-slices without any panics.<p>It's a minor point, and it doesn't really invalidate the optimization, but I'm surprised the article didn't mention it.
I think it’s more of a time vs code tradeoff, if done properly.<p>For example in the Vec case, you could theoretically build an alternative which encodes the “three sections” property internally, and ensures correctness at construction time for the pointers. Not as completely safe as a Vec, but you can still get similar benefits for the “business logic”.<p>But I agree, just having a custom structure that does not provide a safe wrapper around this would be sacrificing standard guarantees.
It's the exact thing Rust is made to protect against, on a more local scale. Every memory corruption bug is just an out-of-bounds index that wasn't protected against.
you could always do a .get into the vector and handle the error, it doesn't necessarily need to panic.<p>Thank being said in this case it should be impossible to index out of bounds so maybe a panic is warented.
You can make a wrapper type that abstracts the offset lookup logic with a safe interface. If it's a transparent struct then rust will compile it away into nothing but you still get the abstraction in your code.
> I can't help to think that the approach of joining several distinct list into a single one in some way undercuts Rust's safety guarantees.<p>Not really. You just need to make the underlying fields private and provide methods to get slices to the data you need.
Tools exist to serve us, not the other way around.
Sure, and usually one of the ways Rust serves us is with safety guarantees.<p>Which isn’t to say this optimization is a bad idea, just to say it’s sort of a straw man to imply coding in Rust to take advantage of safety guarantees is “serving Rust”
Not sure what they use to hold the cache key and entry. If a hashmap is used, then a radix tree (adaptive radix tree) would be better in saving memory space. Most of content of the qname field of the CacheKey is hostname, like www.site.com. The reverse version com.site.www fits nicely in navigation path of a radix tree. The common prefixes like "com." are shared and compressed in the parent nodes of the tree.<p>Even a BTree with compressed prefix keys can save space in the qname.
One of my proudest professional moments was when me and three others managed to reduce memory load of the game Wavetale from 20+GiB to under 3GiB so we could port it to Nintendo Switch.<p>The 100 TiB number almost gives me vertigo. Though in this context it was "just" 50%
Funny thing about cloudflare. I have a dns warming script that uses their top 1k or 10k addresses. Then when my master starts up it warms the entire cache. Everything else uses memcache so the cluster is nice and toasty. As far as I can tell no one else releases domain statistics like them.
General theme: A programming language's native in-memory object format is typically optimized for random access, uniformity, and mutability (fields at fixed offsets, etc). Serialization formats for network or disk tend to be designed explicitly to be more compact. But you can design your own in-memory representation too, with the properties you need.
That’s the old school of thought. These days, designers of newer serialization formats realize that designing a more compact format doesn’t really buy much on modern CPUs and modern networks. See for example Cap’n Proto (whose inventor, kentonv, also works at Cloudflare) and flatbuffers.
It's weird that it took so long for these trivial optimizations but it might just be that they were working on optimizing other stuff.
Are they selling that RAM?
We're finally seeing more appreciation for this kind of engineering. Not everything needs to be solved by throwing more hardware at the problem
I've run into issues with using public wifi when I override my MacBook's DNS server to 1.1.1.1 or 8.8.8.8. I believe this is because captive portals require custom resolution of the name captive.apple.com. And external DNS servers will not resolve that correctly to the local gateway's authorization page.
AFAIK (at least it worked like that some 10 years ago) the captive portal just intercepts the HTTP page load and inserts its own content (most often a 302). So it just has to be a http web page. Firefox uses <a href="http://detectportal.firefox.com/canonical.html" rel="nofollow">http://detectportal.firefox.com/canonical.html</a><p>Relevant support page, though light in details: <a href="https://support.mozilla.org/en-US/kb/captive-portal" rel="nofollow">https://support.mozilla.org/en-US/kb/captive-portal</a><p>Edit: ah, yes, DNS can be hijacked too (requires intercepting outgoing traffic on port 53 therefore incompatible with DoH), that may require fewer computing resources. Still need http otherwise the server cannot use the correct cert chain.<p>Edit 2: Wikipedia says both methods are used: <a href="https://en.wikipedia.org/wiki/Captive_portal" rel="nofollow">https://en.wikipedia.org/wiki/Captive_portal</a> and also mentions RFC 8910. I suspected something like that existed, hence my initial disclaimer.<p>My point was: that domain is not treated any differently from other domains.
I've had reliable success by using <a href="http://neverssl.com" rel="nofollow">http://neverssl.com</a> to force a basic HTTP connection for kickstarting a public WiFi portal login, although I have to disable NextDNS (iOS) too.
Can we take a minute to appreciate how utterly broken this state of affairs is? The dogged over centralization of DNS is an endless source of problems.
That’s a Mac bug if so—it should be always using dumb udp/53 for captive detection, not some fancy DoH thing.
Dumb captive portals, which do still exist in some places, usually do MitM attacks on the connection, so you need some http(no-s) site that you can abuse as "yeah, this can get attacked by the WiFi" to then answer the portal.<p>The <i>right</i> way is that there's DHCP option for the network to signal "I have a captive portal", that's been standardized for over a decade.<p>… or … IDK … just stop shoving ads down people's throats just because they want WiFi.
> 56% A records, 25% AAAA, and 19% TXT<p>And they say nobody uses IPV6.
The most interesting result to me is that the richer parsed representation was not necessarily the faster one. If the hot path is mostly “read from cache and serialize back to DNS,” parsing everything upfront only to serialize it again can become unnecessary work and hurt locality....
So they optimized from Vec to Box, but they're still using Box all over and spending 16 bytes on it? The things they're boxing need 2 bytes for length, and their memory use is low enough that they could cram the pointers into 4 bytes. Trying to pack that into 6 bytes is probably too much fuss for the benefit, but I see no reason to use more than 8 bytes.
Why do people seem to think that optimization is something you only have to deal with once the software scales so much that 100s of TB of memory or disk space (or thousands of hours of processing time) are being wasted.<p>It is almost like nobody even thought during the design phase about what might happen down the road.<p>This is why so much software is bloated and often buggy. Just gets something that half-way works out the door ASAP and worry about the rest later (too often, never).
It can be quite hard to predict where particular usage patterns will take a piece of software under extreme load, especially with things that have lots of internal state. Obviously when you get to spend 100 T or more the pay off of an optimization is much larger than what it is in the case of 1T or less, and your typical developer is not going to have that kind of memory even in aggregate to play with. I tend to be forgiving when it comes to watching software bloat that I did not cause myself (and yet, I'm frustrated that Ubuntu's start-up greeting message takes a whopping 500 M).<p>In the case of internet infrastructure I don't think there was anybody even up to the year 2000 who had any idea of how bit this was going to be. And even now we have IPV4 and lots of legacy to deal with. Cloudflare is not my favorite company, let's put it like that, but in this case they show how the sausage is made and I think that should be applauded. Much better than 'why were down again for X hours'.
I recently heard a great analogy for this exact problem under the premise of "Make it work, make it right, make it fast".<p>Assume a sorting algorithm as a metaphor for your whole program.<p>You make it work by implementing the simplest thing you know how to write: bubble sort. Works.<p>At the end you notice it's way too slow and replace with something much better: quicksort.<p>Now, how much of your program is surviving? Almost nothing, perhaps except for the "greater than" comparison.<p>If you apply that idea to a real world program, we're pretty much talking about a full rewrite.
The intermediate level Rust dogma is to try your hardest to avoid the heap, and to tear your hair out at the throne of monomorphization. While both are broadly true, it's articles like this that show that a single pointer (or call) indirection can sometimes be better.
I'd say that boxing large enum variants is itself an intermediate level Rust topic, and a well-accepted practice. Clippy will even point out places where you might benefit from boxing an enum variant: <a href="https://rust-lang.github.io/rust-clippy/master/index.html?search=variant+large#large_enum_variant" rel="nofollow">https://rust-lang.github.io/rust-clippy/master/index.html?se...</a>
Frankly weird that they were resorting to high level containers for this in the first place. Also, this line struck me as odd<p>> Big Pineapple uses jemalloc, an allocator designed for multithreaded, allocation-heavy workloads.<p>jemalloc multithreaded performance is actually poor(ish) compared to other modern allocators, which makes it a weird choice. But even weirder is why they're even using an allocator in the first place compared to a va MAP_ANON | MAP_NORESERVE arena carveout approach? You can also do punning that way too, which I'm not even certain if Rust supports?
An approach like that would be at constant war with the borrow checker in Rust. Apparently it is possible but there is enough friction that these guys went a different route.
I would also have instinctively reached for a large VM reservation to exploit demand paging. I have used that pattern a lot in C++ but not in Rust, so I don't know how difficult it would be to implement there.
Rust supports punning via pointer casting, but you'll want to use #[repr(C)] on any data types used
Where are their users coming from? Besides the few manually putting 1.1.1.1 in their settings.
The Record struct contains rtype and data where RecordData is a tagged union. Aren’t those two always in sync? Not a DNS expert, just wondering if this is redundant or there is a reason both are there. Doesn’t matter anymore if they store it already serialized but I would be interested why it was this way.
Great article, but I'm surprised they waited until they were using $2 million USD of memory before shaving off all the unused bytes at the end of a vector.
How much is this in euro or do we measure money in ram now?
Currently $15 per GB, he saved Cloudflare $1,500,000 and got exactly $0 bonus. He must really believe in cloudflare's vision (global enshittification). In related news, three times today Cloudflare told me that I'm a bot and shall not pass - not that it needs to check if I'm a bot before it lets me pass.
Obvious question: why wasn’t this done earlier? It looks like all the data was already available. At THAT scale, reducing memory usage is a must-have, not a nice-to-have. Weird.
probably agents going through tech debt or finding wins<p>every dept knows what they could do with more budget, the budget for those things just never comes<p>now agents have utilized budget more effeftively, unbottlenecking many things, including engineering blogs
Cloudflare talks about having datacenters in 300+ cities. Presumably they have at least a few servers per datacenter. They saved 130 servers worth of memory... not even the minimum number of servers they have (seriously though, they probably have a LOT of servers)... a few GBs of memory per server running the service. At that scale this is a nice-to-have.
> we store the records as a single Box<[u8]> containing each record encoded as a 2-byte length prefix followed by its raw bytes.<p>Interestingly this is exactly how netlink works-ish: <a href="https://manpages.ubuntu.com/manpages/focal/man3/netlink.3.html" rel="nofollow">https://manpages.ubuntu.com/manpages/focal/man3/netlink.3.ht...</a><p>You start, get the type & length, and then that is how many bytes you read.<p>Some issues with that when you deserialize, from a raw stream in to `[u8; 4096]` buffer, the alignment is only guaranteed to be on 1 byte, not 4 bytes.<p>In practice it is 4 bytes, but if you run those tests with Miri, you'll get yelled at. So the fix there is to declare the buffer with a type that mandates the alignment of the largest type that you're going to be deserializing.<p>So then you start your buffer as follows: `[u32; 1024]`, and with `slice::from_raw_parts` you get to turn that into `[u8; 4096]` with the expected alignment.<p>As an exercise I wrote a streaming parser for netlink, the current existing package serializes everything, all at once.
This kind of encoding[0] is ubiquitous in networking protocols. It scales down to small silicon well and enables the receiver to estimate resource requirements or skip parts of a serial byte stream without storing it in memory first. These encodings usually aren't aligned by design.<p>[0] <a href="https://en.wikipedia.org/wiki/Type–length–value" rel="nofollow">https://en.wikipedia.org/wiki/Type–length–value</a>
It's called TLV encoding - tag/length/value. It's very common in all sorts of network protocols and serialisation formats. It allows you to skip unidentified tags. Sometimes, like in the PNG file format, there's a fixed bit in the tag that tells you whether it's safe to skip or if you have to reject the whole thing because you don't understand this tag.<p>Hey dang can I get my rate limit turned off pretty please?
my first thought was if is this will impact the memory market prices :)
REWRITE IT IN C!
EdgeDNS was rewritten and is now EtchDNS <a href="https://etchdns.dnscrypt.info" rel="nofollow">https://etchdns.dnscrypt.info</a>
I'll buys some spare RAM you now have. I only need 64GB.
I wonder at their scale, why wouldn’t it make sense to store the entries lightly compressed in memory?
The Art of Production.
this is so interesting, memory and storage is cheap until it isn't and then you optimize.
[dead]
[flagged]
[dead]
[dead]
[flagged]
[dead]
[dead]
[dead]
[dead]
[dead]
[dead]
Now put the 100 terabytes of memory back to the market. Stop hoarding RAM.
<i>sees cloudflare</i><p><i>leaves</i>
One question the article doesn't answer is: why are they cacheing at all? If your cache is that big it isn't a cache. How much bigger is the dataset in question? There are 250 billion entries. Assuming 80/20, that implies 1.25 trillion records?<p>What's the speed of service/response time relative to the data source?<p>At that point it might be enough to replace your multiple caches with fewer in-RAM databases?<p>It's an interesting problem.
Maybe I'm misunderstanding, but this powers 1.1.1.1, it doesn't front an internal dataset. A cache miss hits a nameserver. Which is to say, the dataset is "every DNS record in the world"
You have to cache, cloudflare doesn't know all the records ahead of time, they have to do recursive lookups to the authoritative servers that own the records and that is only good for the period of the TTL of the record. There is no "global" DNS record database or something like that.
>that is only good for the period of the TTL of the record.<p>Not really, TTLs are often short, but IPs might not change for years.<p>You can probably generate your own TTL, at scale, and avoid many DNS requests.
Why would anyone want to use a DNS resolver that tampered with records on a large scale? The TTL is intentionally set by the originator of the record.<p>Or alternatively, if you don't tamper why would I want to use a service that serves stale data?
In DNS, the owner of each record has full control over its TTL. Intermediary DNS servers are required to honor them and are not permitted to replace TTLs with their own.
DNS servers do in fact do that but it would not be a good look for the world's largest DNS provider.
Actually that is not true. The IETF has expanded the definition of “TTL” and explicitly permits resolvers to serve “stale” RRs beyond their expiration time.<p><a href="https://www.rfc-editor.org/info/rfc8767/" rel="nofollow">https://www.rfc-editor.org/info/rfc8767/</a><p>As a corollary, there is obviously no floor on refetching unexpired RRs, of course, except for efficiency concerns.
You are obliged to pass on the TTL, you're not obliged to cache according to it.<p>At least in my country (UK) I know of no law relating to DNS caching.<p>Why throwaway perfectly good data every few minutes that is only modified every couple of years, just so someone can move their domain quickly when they eventually wish to? It is my contention that a [caching] DNS service can do far better. Trusting user (domain owner) input blindly is not for me.
It's not some sort of public law with public enforcement, but it is in the RFCs that govern the protocol.<p>I should be a bit clearer here; the TTL is an <i>upper bound</i> on how long it can be cached. Caches are free to consult more frequently but not less frequently. That said, out of respect for upstream cache operators and authoritative servers, most DNS caches honor TTLs as best they can.
then they would be breaking DNS at scale.
It's a recursive resolver. The global DNS dataset is not something you could collect to serve directly vs caching from observations.<p>The data source is authoritative name servers operated by third parties, some of which are slow on their own, some of which are behind slow or lossy networks. Origin response times vary between probably 1 ms and 2 seconds +/- origins that never respond.
They’re adding the cache consumed across all of their servers. It’s not one giant deep cache.
The simple answer is that if you didn't cache, DNS traffic would skyrocket, and the load would pile up on the authoritative servers, which were intended to be small, and during the early days of the Internet, were frequently on bandwidth-constrained links.<p>DNS is designed to distribute query load to the edge as much as possible, and that's enabled by caching. It just so happens that "the edge" is now becoming concentrated among a small set of providers because they wanted to make a business out of it.[1] They knew that this would be expensive going in, though.<p>[1] Nobody <i>has</i> to use 8.8.8.8 or 1.1.1.1. Most people can use their ISP's cache or a local cache instead without any noticeable difference in behavior.
The problem is there is a noticable difference in behavior because the ISP cache is overloaded so queries take longer. Sure, that's not everyone's experience, but there's a reason people chose to use alternate servers.
> If your cache is that big it isn't a cache.<p>This is an incorrect statement. Caches do not have a requirement of being smaller than their source data set. CDN is an example of a cache that generally matches the size of the source data.
> Once we store a DNS response in the cache, however, we never modify it again. The capacity field serves no purpose, but still costs 8 bytes per Vec<p>Were there no design discussions/reviews when the system was setup to catch trivial things like this?
Rob Pikes 5 Rules of Programming:<p>Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.<p>Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.<p>Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. (Even if n does get big, use Rule 2 first.)<p>Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures.<p>Rule 5. Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.<p><a href="https://web.archive.org/web/20260314210910/https://users.ece.utexas.edu/~adnan/pike.html" rel="nofollow">https://web.archive.org/web/20260314210910/https://users.ece...</a>
> Data structures, not algorithms, are central to programming<p>So you agree that they should've designed the system to use the appropriate data structure from the beginning?
Notice rules are ordered. You don't optimize until you know you need it. They started with a data structure they though would be fine. Clearly it was fine since it worked and they decided it was later worth optimizing.
Rule 5 is superseded by Rules 1 & 2. Without the measurements to back it up, you're chasing phantoms.
The existence of 1.1.1.1 speaks to a much larger design problem. If you want to talk about what should have been done, you need to step much, much further back.
> Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.<p>Genuine question, is software performance really linear like that, that one can and should only fight the tightest bottleneck, one workload at a time? Never really sounded right.<p>It also sounds like the typical sleight of hand where the difficult bit is simply laundered a layer up, in this case the choice of what workload one investigates.
It can be. Sometimes you take a profile and there's a big smoking gun and nothing else matters.<p>Sometimes it's a lot of small things everywhere and you can pick up significant performance after a lot of small value fixes. In this case, caching wire data instead of structured data is almost one of these, because the contribution to response time for serving a cache hit is small... otoh it happens so often than a small improvement matters; but this is a pretty focused use case, you usually hit the many smalln improvement issue in a less focused application where there are many code paths.<p>Sometimes the whole code structure / data structures are so wrong, but it works and perf is bad and profiling will never tell you. This article is <i>not</i> that case; these data structures only needed refinement.
In many cases, yes. A software pipeline can only achieve as much throughput as its slowest stage, and much of the software we write can be modeled as a sequence of processing stages.
It is often not worth optimising in the early days. You don't know how popular it will become, you might not know how many DNS records you will hold, it was possibly written in an earlier language and ported as-is.<p>At the point someone queries the 100TB of RAM, then maybe it is worth revisiting but even that has risks. You have to design the migration path, have fallback mechanisms etc.
Premature optimization argument fits right in. Now that memory is up to 10x more expensive it is worth considering optimizing programs with large memory footprint.
Discussing trivial optimizations is a waste of valuable design time. You're never going to "forget" an optimization. The running system will remind you when the optimization is actually needed.
Boxed slice isn't really the most well known type/optimization,
There usually aren't that many vec's that it makes a big difference.
it was working so no one thought to check