I wonder how we could handle that in a simpler way with durable workflows (e.g. Temporal, Restante, DBOS) – which are similar to Erlang processes but with persistent disk storage. This could avoid the need to maintain the 1000 row inventory.<p>Perhaps each shopping cart would have its own workflow, and the inventory item would have one as well. Then, whenever a customer put an item in their cart, their cart workflow would send a signal to the inventory item workflow and wait for the response. The inventory item workflow would maintain a ledger controlling to which cart each unit goes, and it could batch the writes to this table. This way, even if 100k customers try to purchase the same item in the same second, it should handle the load.<p>After the batch is written to the ledger, the inventory item workflow would reply signals to each cart workflow confirming that the reservation was completed. The end-to-end latency from the consumer point of view would be a fraction of a second, without needing the 1000-row hot-inventory heuristic.
Why even have a blog when you can't be arsed to write the posts. This is so obviously LLM-written. I have a positive view of Shopify engineers, but this kind of made a dent in that confidence.
What exactly makes it "obvious" that this is written by AI? I could totally believe that AI was used to generate parts of it, but I really don't get the sense that the whole thing was written that way. I've seen way worse examples on this site.<p>As software engineers we are constantly told that we need to heavily use these tools for our daily work. So is it surprising that software engineers use the same tools as writing aids? Using AI does not mean no human effort was involved.
Subheading and dot point spam, low density writing (the opposite of standard technical english), including useless detail (like enumerating stats on Shopify's scale), using contrastive parallelism, and other llm-isms. Even if it's not AI it's bad writing done by someone who has picked up AI's worst ticks.<p>For example this subheading:<p>> "The real bottleneck: connections, not CPU"<p>That's two AI smells. AI likes to say vacuous punchy statements like "the final takeaway" or "here's the rub". Then contrastive parallelism "connections, not CPU". Contrastive parallelism should scarcely exist in technical writing, regardless of whether it's AI generated.
The good news:<p>Your worst complaint has nothing to do with the overall content or accuracy of the post.<p>Just style bashing.
I really wish that slop writing was disincentivized in whatever RLHF they do. I don’t want to read the weird LinkedIn pop-sci tone for the rest of my life in such amounts.<p>I wonder if the average reader is also getting annoyed like this or whether they just don’t care - especially seeing what seems to get upvoted on your run of the mill social media sites. They probably collectively shape things more than I do.
Taken verbatim "But the hardest lesson wasn't about database design. It was discovering that the real bottleneck wasn’t what we were observing and measuring. "
Because the post is just rambling without a clear intent or direction. Why do you need "oversell protection" if you have "transactions". Isn't the whole point of a "transaction" that it handles concurrency and disk failures?
People see an — dash and kneejerk. Writers are deciding to stop using this valid punctuation because of this exact reaction, it's ridiculous.
I agree with you, it doesn't take long to identify the itemized style that LLMs use and it's always incredibly annoying.<p>I would rather read the original thoughts of the engineer, grammar mistakes and stylistic imperfections included, than prompt generated AI slop.
unfortunately its new normal now
I just wanted to drop a note that I almost never go into an article or blog wondering if its AI or not. I am trying to be a more picky reader and try to actually do something occasionally but I do browse database articles and this is a good one. So I wonder, am I becoming insensitive about AI writing? Am I being hypnotized into taking whatever color pill thats had a color representing a surrender to the AI "hive mind" whatever that is?<p>So I'm sort of curious, did you not like the AI writing style or just rejecting AI in general? I myself am very conflicted, I think AI in its current form is the wrong tech at the wrong time yet I don't mind reading AI text and sort of mooch off of googles free tier.<p>Also I'm starting to notice lots of AI smearing, I read folks online describing someone elses contribution as "obviously AI" and I'm suspecting in some cases these could be false accusations.<p>I'm thinking about starting a blog, so when my AI gf writes posts, do I ask her to try to "not read like an AI"? Anybody try that? I'm gonna try that. Hehe for all you know I already did hehe
Mostly unrelated but shopify is incredibly annoying. They introduced this delivery tracking app called "shop" and it has become unavoidable when buying electronics from china. Recently looked at it with mitmproxy and it ships home more than gets shipped to me.
> Instead of one row per item with a quantity column, we use one row per sellable unit. An item with 10 units has 10 rows.<p>> But one row per unit for all inventory would break down at scale—an item with 50,000 units across 10 locations would mean 500,000 rows, and the reserve query would slow as it scans through them. Instead, we maintain a bounded pool of available rows, capped at 1,000 per item/location combination. Reservations consume rows from this pool; a replenishment process refills it from the inventory ledger.<p>Shouldn't I feel uncomfortable with such approach? It seems to create a backoff (pool) for lowering the chance of having a synchronization issue.
I agree, it does seem awfully complicated and there are quite a few pieces missing for this to be a complete solution.<p>I'm a bit surprised about the scalability case against a simpler solution. This is not about Shopify's scale. We're talking about contention for a specific SKU of a specific seller at a specific warehouse location.<p>How many shopping carts are competing for a single SKU at the payment stage at peak hours? Can this really be too much lock contention for a single database row?<p>I realise Shopify engineers are neither stupid nor inexperienced. Hence my surprise. I would have liked to hear more about that specific problem.
We observe 200+ (can’t say closer number) purchases per second for single SKU with good marketing and price.<p>The other thing that bothers me - why not real stable, but maybe „too old, medieval” solution with Redis as the main source of through - without any sync with SQL at all in terms of stock… it worked in my previous job with much higher traffic (1000s/s). Yup, we ended up with app-side sharding, but it was stupid-simple.
Flash sales are a huge scaling issue for Shopify. There are celebrities who want to sell thousands of items in a few minutes window at the end of an advertised countdown.<p>Basically this is an incredibly rare case but a feature that they want to support.
I would call this one-row-per-contract-type, and this is the most general model for the problem (e.g. the model cannot be further broken down into finer level), thus, the most scalable model given storage is dirt cheap.
Comes down to type of items, when you have physical inventory the number is limited so more manageable and interestingly enough the problem only applies to physical inventory.<p>You are just spending some more disk space to avoid synchronization issues. Denormalization for performance is a really common pattern, just that people do not start with it in the first place itself
> an item with 50,000 units across 10 locations would mean 500,000 rows<p>I don't get it, Wouldn't that still only be 50,000 rows, just divided across locations?
they're saying they started with the dumb thing,<p><pre><code> item1_location1
item1_location2
...
item2_location1
item2_location2
</code></pre>
every (item, location) combo gets its own row, and then they moved to the smarter thing.
you should, their design is not the best. There is middle ground between "one row per SKU" and "1000 rows per SKU".<p>Its called one row per shopping cart*SKU combo.<p>if two people order 100 and 500 items of the same SKU, respectively, the table should have only two rows: for order1 and order2. Not 600 rows.
The problem is that in this case you have to do splits/merges. And while there are products that are sold by 100 units at a time, I think in most cases people by 1-2 items so the hassle might be not worth it.<p>Also you might not understand the original problem. Imagine if 100 customers want to buy product A. One thread starts a transaction, searches for amount of product A and UPDATE's it and goes searching for other products. The database locks the row until the end of transaction and other 99 treads cannot continue until first transaction commits (they can read but cannot update the rows).<p>This is why they made a row per item. In this case, transaction 1 hopefully locks only several rows with items of product A. Transaction 2 instead of waiting for lock release skips them (due to SKIP LOCK) and locks several next rows. And so on.<p>Obviously you do not need to make a row per item - if the available amount is really large (10 000 items), you could have for example 100 rows having 100 items each. In this case each transaction locks the whole row (100 items) even if it wants to reserve just one item. The problem though is that now every row might have different amount of available items and you have to do more work to reserve the amount you want.
Can you explain how that works? With the row-per-item I can see how you’d use locking primitives etc easily to deal with multiple concurrent shopping carts claiming available inventory.. but how does your solution solve contention? There’d need to be some “number of items in inventory” row, wouldn’t there be contention on that?<p>The point of one row per item is that thousands of concurrent shoppers don’t need to block each other as they can each claim as many free rows as they need for themselves?
One other advantage is item serial numbers. Or something else that makes an item that seems the same but actually be unique (perhaps the warehouse it’s in?)
explained below in <a href="https://news.ycombinator.com/item?id=49228432">https://news.ycombinator.com/item?id=49228432</a>
Maybe the example numbers are just bad - but now you expect your system to fall down if you scale from 10 to 100 locations?
"Number of locations" is an input so if the system has been designed to handle up to 10, and not 100, then yes I would absolutely expect it to fail with the higher value.<p>Developers (and everyone else really) need to think about systems, with the system taking inputs like "number of locations", and producing outputs like "available inventory", and when the input parameters change outside of the designed scope, without the system itself changing, then you <i>should</i> expect things to break.
I guess it depends on how the replenishment process works. Unless you're ordering over 1000 of an item, I doubt it would be a problem.
Depends on the scale. Most companies don't approach the scale where this matters.
I'm familiar with the reserved row approach (I use SELECT FOR UPDATE SKIP LOCKED) and yeah this replenishing idea terrifies me.
It seems there could be a simpler solution.<p>1. Deduct the reservation from the inventory when the user starts to order, but in the same txn also maintain a separate row for the in progress order flow.
2. If the order flow is aborted or times out have a background process that returns these to the inventory.<p>That seems simpler than this approach and involves no locking. Though their presented approach is also reasonable, there must be some reason not to choose a simpler flow. It is not that difficult to have a gc service that scales, but may be they didn't want to separate that.
My understanding is: your proposal is not very different from what Shopify is doing except they are tracking 'reserved units' (one per row) and you are proposing tracking 'orders' as the temporary state to then reconcile back with inventory quantities.
Yes, at a high level. It doesn't rely on skip locked, which is not cheap at DB level. DB has to still typically run query and keep going until it finds an unlocked item. Deducting and checking inventory counts are simpler ops inside the DB.
The moment you added a background process you just replaced the complexity.<p>1. Backgrounds process can back up<p>2. They need context of the user and need to switch context per user<p>3. What if they fail, you create some DLQ or another process to handle the failure<p>4. Who looks on those failure and how do they act<p>TLDR; there is always a cost
Can you clarify why this involves no locking? There can still be 2 actors fighting for the same row.
now you have two problems. what happens when your reservation system backs up?
I was investigating Durable Objects (DO) and had Fable walk me through where in my app they might be appropriate. One place had a dependency with billing (where I use a transaction now) and the proposed re-work to allow for concurrent editing with DO looked very much like this, reservations with idempotency keys. And if you add hierarchical allotments then it scales pretty well.<p>I disagree with the other posters about the bg process, if you have any bg processing already you should be able to handle the few edge cases without too much trouble.
Why is it so hard for many people to accept, that this is a solution for a specific problem of shopify? They did not say that Redis is bad and MySql is good. They only a solve their problem.
I never spent much time with the whole NoSQL movement, it always seemed something out of people that don't get how to optimise SQL queries, or suffer from SQL allergy, only to reinvent it badly in custom languages.
Pretty interesting read. One thing I’m curious about is the DB size trade off. Going from a quantity in Redis to one row per reservable unit seems like it could create a lot more rows, even with the 1,000 row cap per item/location.
Could not they shard the inventory table by shop_id? As I understand, the order includes only items from one store, so there is no need to keep all the stores in a single table.<p>Also, I wonder why they could not have a row status (available/reserved) and UPDATE it instead of deleting the rows.
They never said they don’t shard it, however this doesn’t solve the problem they were facing. Even if they have a single store (therefore a single shard), the burst demand may be high for the item in that shop, which creates contention for “remaining item quantity” resource. Their solution spreads this contention across several rows.<p>> Also, I wonder why they could not have a row status (available/reserved) and UPDATE it instead of deleting the rows.<p>This requires a row per item unit, doesn’t it? If you have 50k units you’ll have to track status of every item, meaning 50k rows. They also mention this as a rationale to use at most 1k rows, and treat it as a buffer.
not the best design to have 1000 rows for each shop*SKU combination. If a candidate proposed this solution during Shopify's System Design interview, i doubt he would be vetted for Senior+ position.<p>Instead of having 1000 rows per shop*SKU, why not just have one row per shopping cart*SKU?<p>That way a single row would represent a single cart, and will hold info of multiple items of the same SKU.<p>No need a cludge with 1000 rows limit and replenishment process. Instead of dealing with N rows, you always deal with a single row.
> not the best design [...]<p>So those engineers at Shopify worked hard for months on a more performant system, but they missed the obvious structure? They chose a complex denormalization for no good reason?<p>It may be true, but I think it's presumptuous to belittle their work when we have only partial information. My guess is that they had good reasons to think that the more obvious ways would not scale.<p>And from reading your comments in this thread, I believe your structure would fail at their scale. A SQL query that uses 2 sub-queries with "group by" is probably too heavy. From the post, at peaks there would be millions of active shopping carts.<p>BTW, I suspect most orders are just for 1 or 2 of each item, so the denormalization is not as heavy as it seems.
i also work in big tech and know that a lot of bullshit design creeps into system design and prod, because everyone is overworked, overstressed, wants to just get things done for the quarterly performance review as to not get shitcanned with severance<p>re concurrency, it is not a big issue at all. stock exchanges deal with HFT traders and can easily deal with concurrency of orders. Same can be implemented with shopify, but I doubt they face the same level of concurrency as stock exchange anywhere near
Famously, stock is settled on a delay (and generally doesnt involve physical products that are not fungible). Im sure theres a lot to glean from how they handle concurrency but Im not sure they are solving the same problems.
> re concurrency, it is not a big issue at all<p>I would really appreciate it if you could write this up as an article. It would be an extremely interesting and valuable read
You might not have noticed that essentially the entire blog post was AI written.<p>There's even this bit where they discover a remarkable trick:<p>> Each round trip to the database has a cost. For carts with multiple line items, we batch reservation queries using UNION ALL so we fetch all needed units in one round trip<p>Insights like that really don't read like senior level output, and of course, it's LLM output. I'm not sure it's presumptuous to question it.
I have never worked anywhere where describing how their system actually works would pass the company's own system design interview
Others are almost never as dumb as you hoped, and you’re rarely ever as smart as you think.
Your mental model here is mapping too close to an actual cart in a retail, at a in person, setting.<p>The assumption that a SKU maps 1 to 1 to a cart item is flawed.<p>If the first item in the cart is a bundle of SKU-A and SKU-B, the second item is a bundle of SKU-A and SKU-C and the third item is 5xSKU-B where do you do you keep the re-agregation of the SKU-X's to track them?<p>This is without accounting for item location in the reservation - and rules that may apply around that.<p>You haven't even gotten to the part where different customers will have different rules around shipping from different locations - because that can eat into margins.<p>You're also making a bunch of other assumptions around transaction flow and where carts are actually stored (and how they get converted to an invoice, with payment attached) that likely do not hold true.<p>Could you do it more like what you're sugesting -- maybe -- but only in a single tenant system.
> Instead of having 1000 rows per shop<i>SKU, why not just have one row per shopping cart</i>SKU?<p>At what point that row is inserted?
per my reading of the article, the protection is only needed for a few seconds, while payment is being processed by the payment system.<p>so the row is inserted when Payment is initiated, and row is deleted when Payment succeeds<p><pre><code> What is oversell protection?
Reserve: When payment starts, we mark items as reserved (a short hold, e.g. several minutes).
Claim: When payment succeeds, we permanently deduct quantity from the inventory ledger (source of truth).
</code></pre>
but that system could be easily improved to reserve item when user Adds item to a cart, to prevent scenario when user adds item to a cart, goes through checkout, and after initiating payment gets "soldout error":<p><pre><code> 1. Let user add item to a cart by default (happy path)
2. Initiate async check in the background for SKU and quantity
2a. The check sums up rows for all SKUs and compares to Inventory table (very cheap check since its done to only active shopping carts)
3. After few seconds the check comes back, and we let user know that item is soldout, before/the moment user goes to Checkout.</code></pre>
Ok, but before inserting you must ensure that inventory is not depleted, which means you need to know the count and you need to lock the row. So you still have contention on that item. Them having a 1k buffer allows not to take a lock on a single row every time, and only do it when buffer is empty
there is no need to lock the row, since you a dealing with a shopping cart, not individual item piece. when you run aggregate functions, lock is no needed, it is actually better to run it with SET TRANSACTION ISOLATION LEVEL READ UNCOMMITTED; for aggregation<p>the check for oversold items is extremely cheap:<p><pre><code> with current_order as (
select $SKU1, $q2 as quantity
union
select $SKU2, $q2 as quantity
),
with carts as (
select sku, sum(quantity) as reserved
from active_carts
group by sku
),
with warehouse as (
select sku, available_units
from inventory
group by sku
)
select * from current_order
inner join carts using (sku)
inner join warehouse using (sku)
where warehouse.available_units - carts.reserved < current_order.quantity
</code></pre>
assuming there are indexes on sku field in both, results in efficient index seek and agg over 2 tables
The item is reserved when the user decides to place an order, but before paying for it. Not when a product is added to the cart because the user can keep it there for a month and end up not buying.<p>You reserve the product by creating an "active_cart" entry. Your solution has a problem, that when you run the check, it might say the product is available, but before you create an "active_cart" to reserve it from thread A, another thread B reserves it and you end up reserving a product that is not available anymore. You end up with SUM(active_cart.quantity) > inventory.available_units.<p>That is exactly why the database has locks - to prevent this situation. With locks, thread A decrements inventory.available_units and that row is locked until the end of transaction. Other threads (if they do SELECT FOR UPDATE instead of SELECT) cannot see the old, invalid value until thread A either commits and the value is updated or rollbacks. However, locks cause performance issues and that is why shopify uses the architecture from the article - instead of 100 users fighting for the lock on the same row with available amount, each user locks only rows with units they plan to buy.<p>Interestingly, MySQL docs has the documentation page with a similar case: <a href="https://dev.mysql.com/blog-archive/mysql-8-0-1-using-skip-locked-and-nowait-to-handle-hot-rows/" rel="nofollow">https://dev.mysql.com/blog-archive/mysql-8-0-1-using-skip-lo...</a>
I don’t understand how this should prevent oversold. You have a check that reports empty or oversold inventory. But how does that check prevent 2 concurrent actors fighting for the last item from inserting 2 rows?
how does current design resolve concurrent actors fighting for the last item ?<p>there is ultimately needs to be some global mechanism resolving this conflict. Currently it is an order in which db engine processes transactions by locking rows for a transaction, whoever got the first lock, wins the last remaining items.<p>my design is the same, except it does not need this dance with moving rows between tables, locking them, and the cludge with replenishment process.<p>in the simplest form, run the sum() over active non-finished orders and compare to inventory. you get the same result: whoever got the first to run sum() and get positive answer will get the last remaining items.<p>but the problem as formulated, imho, is not even correctly defined.<p>Shopify incorrectly formulated the very problem they are trying to solve.<p>Trying to solve it at the payment time is too late, its better to resolve it earlier, before the checkout.<p>the "PAY" button should only do one thing: deduct money from cc and that's it. Resolving inventory availability must be solved way earlier, the moment user clicks Checkout, not when user clicks Pay.<p>So ideally, the error for oversold items should be shown to a user when he clicks Checkout, not when he click PAY
> Shopify incorrectly formulated the very problem they are trying to solve.<p>That’s a bold overconfident statement. Cart abandonment is real. People never clear their carts they just walk away<p>Shopify purposefully chooses to do it at payment time because doing it earlier results in lost sales as people “reserve” items and then walk away causing other to see out of stock and then also walk away<p>Whoever puts up the money first gets the item<p>That’s the design constraint they chose you can’t just say “their solution is wrong because they solved the wrong problem”. Each design is a different user experience and I think it’s safe to say they chose which experience they want consciously.
that's why I mentioned active carts in my post, there are ways to define active cart to get rid of abandoned carts ( ignore carts where last user action was > N seconds ago).<p>Ok, let's accept the design goal that whoever paid first wins. You can use the same metric (how many milliseconds ago did user click PAY) and impose a global monotonic non-decreasing counter to distribute the scarce inventory. This is how order matching engines work at stock exchanges with HFT orders (FIFO logic).<p>the goal is to know with 100% certainty, before sending payment request to payment processor, who will have item and who won't, and you dont need to move mountains of rows for that.<p>the payment processor should be just a binary answer: payment succeeded or not, but currently it combines Inventory availability check & payment processing, which is the root cause of confusion. For clarity it is better to make that stage of order processing an explicit separage stage, instead of coupling it with payment stage.<p>some stores split payment into two stages: Payment and Final order confirmation. at the Payment stage you can pre-authorize money at cc and do inventory availability, and at final confirmation you capture $$
Clearly this is for high concurrency cases where there are many people racing to get all the available items. It's not clear that it's in shopifys or the sellers interest to let items get sequestered in people's shopping carts, which is a spot where there isn't a strong commitment to complete the purchase. At payment time, you can be more assured that the item will actually be purchased.<p>Still I think their solution is a bit weird. I'd want to commit the reservation transaction with inventory decrement along with a payment key and then use a different transaction to drop the reservation when the transaction completes. If the transaction does not complete in a timely manner you probably need to query external systems anyway to resolve whether the payment actually occurred or not.<p>They talk about lock contention in this case, but I also wonder about latch contention since these rows are adjacent. If it's a small transaction that's not interactive, does mysql resolve it with just the latches on the needed tables?
> how does current design resolve concurrent actors fighting for the last item ?<p>It resolves with skip locked. Assuming we have only 1 item left. First query scans the buffer table, locks as many rows as needed (1 in our case), and moves rows to another table. Second query scans the table, finds no rows (even if first one hasn’t finished yet, the row is locked and ignored), checks if it can increase buffer, finds out that it’s fully sold and aborts. Db guarantees that you can’t oversold.<p>> my design is the same, except it does not need this dance with moving rows between tables, locking them, and the cludge with replenishment process.<p>I can’t evaluate whether it’s the same or not, because you still haven’t clarified when exactly you’re going to insert the row. In the article they’re inserting in the same transaction. Would you also do it in the transaction? Because if you’ll introduce a separate global mechanism to resolve conflicts, on a high level it would be the same as their approach with redis (you need to have 2 systems)<p>EDIT: wording
think about for a moment what that skip locked actually means, all these 1000 rows per SKU are logically equivalent to a Inventory table with a single row where available_units=1000 per SKU.<p>now let's think again, do we need to lock 900 rows to place order on 900 items? or can we insert a single row where order_quantity=900 ?<p>shopify's design relies on DB to lock rows for transaction as a way to "decrement the counter" of available units. What I am suggesting, is you can just decrement counter by updating a single row, no need to lock 900 rows. Shopify moved from one extreme (single global variable in redis) to another extreme (1000 rows in db) and forgot about the middle ground.<p>The dance with moving rows per each item between tables is completely unnecessary, it's like counting numbers one by one in a for loop, when you can just substract number directly.<p>if I were to solve the problem, I would have solved it differently, at the Checkout state, before user clicks PAY. This removes the race condition at the user UI level, before any request lands in backend/db:<p><pre><code> 1. Have a table with active shopping carts (cart_id, cart_status, sku, quantity)
2. when cart_status changes to 'Checkout' run inventory availability check
3. If inventory availability check fails, show error to user (before he clicks Pay) and suggest replacement items.
4. If inventory availability succeeds, proceed to charge cc
</code></pre>
availability check is the SQL above: inventory-sum(active_carts.quantity)-current_order must be > 0
> if I were to solve the problem, I would have solved it differently, at the Checkout state, before user clicks PAY. This removes the race condition at the user UI level, before any request lands in backend/db:<p>In order to avoid races you need to insert reservation and decrement availability atomically. Your proposed approach is not atomic. For it to be atomic you will need to lock whole range, to make sure no new rows appeared between the points “check for availability” and “record reservation”. Actors will be effectively competing for the single aggregate row. This is the same as having a single inventory row with quantity field, which they rejected in the beginning of the article<p>> now let's think again, do we need to lock 900 rows to place order on 900 items? or can we insert a single row where order_quantity=900 ?<p>In the proposed schema nobody is waiting for these locks, they’re skipped by concurrent queries. In your schema actors would have to wait before they can insert without breaking invariants.
assuming their "reserve item" function is just "update the table set N rows to reserved=true where reserved==false"<p>more transactions can commit at the same time, but with one counter they would conflict (as it did in the Redis case)<p>they should use CRDT (and trying to model that with this 1000 row workspace, no?)<p>still, eventually at some point they need to do the math
[dead]
Thanks! I don't uSe 'with' enough
And that's why these interviews can be stupid, you can mention the real solution and interviewers might reject because it's not the textbook solution<p>But the real world is different
I had a client and they weren't to bothered if they sold the last item twice, they would call the customer, apologise, and offer a discount on an alternative and keep the sale.
"But the hardest lesson wasn't about database design. It was discovering that the real bottleneck wasn’t what we were observing and measuring."
Tobi Lütke also made some rather controversial statements lately, too, agreeing with a retired TD Bank CEO that more votes should be given to the rich. On the surface, this sounds awful, but what Eric Thor actually said was that the number of votes should be tied to the amount of income tax a person pays. Considering that (from what I've heard) billionaires pay no income taxes, I'd say it's not a bad idea. No tax: no vote.
This is absolutely fascinating. I enjoy real life stories like this. I went to a Node meetup in 2013 when Target had just switched to Node from PHP and it was a similar experience to see their metrics and hear their strategy.
Makes sense... if you are counting something in MySQL and now your counter is in Redis that's already strange<p>But I guess the point is that even in the MySQL scenario the 'reserved_quantities' is almost like a temporary table so either way is not the 'Real' inventory
It’s fascinating that in order to do this, they had to remove 50% of reads and 33% of transactions from the main DB.
> 3. Consistent lock ordering: avoiding deadlocks<p>This section is badly written. For example, it refers to different table names than those previously introduced.<p>The slop shows. While I appreciate the post, I wonder why they didn't bother using an LLM in a way that would at least ensure internal consistency.
At that revenue, why not make your own filesystem, database and index structure? There is no way mysql is the best possible software for this use case. Why stop innovation and hand everything over to ops?
Most likely? Time.<p>Using off the shelf software means you mostly design how to plumb things together and how to make them correct , safe and scalable.<p>The things you mention, on the other hand, carry the same requirements but are also much complex to develop AND to maintain.
The social network VK internally uses highly specialized database engines per business domain. They don't use stock DBs. They have a DB engine for posts, a DB engine for likes, etc. They have a team of DB engineers. Their DB load was around 250 mln RPS 3 years ago. Stock DBs were harder to scale for them. I guess if you have immense highload, having a team of DB engineers can be cheaper because you can save a lot on servers. I reviewed their code. A DB engine's source code is pretty compact and simple (relatively speaking) because they deal with very specific domain entities, so they don't have to account for all the possible user query combinations that a general-purpose DB would have to support. It was mostly shards+binlog+snapshots+views in RAM. Considering that Telegram was founded by former VK engineers, I suspect they have something similar.
Is it really the right choice to drop Redis and go back to a disk based relational database just to wrap transactions into a single unit?<p>Redis handles tens of thousands of concurrent connections in a single event loop, while MySQL uses one thread per connection. No matter how I look at it, that seems like a step backward.<p>Of course, performance isn't everything. And if performance isn't a problem, having everything in one place does make it easier to reason about. But I'm worried that under spike traffic, this approach might actually cause more problems.<p>I think putting a scheduling layer in front of the DB would be a better approach. The application server could handle concurrent connections and only write to MySQL when correctness is actually needed. That seems like a cheaper way to do it. but is it different for large-scale enterprise distributed systems?
Redis doesn't have transactions and persistence.<p>No persistence means the data gets lost if machine shuts down or process crashes. Furthermore, after restart you will need to regenerate the data which can take time. That's why Redis is a cache and not a database. You can fix the persistence issue (Redis can write WAL log, don't remember if it does fsync or not), but then Redis won't be able to handle those thousands of concurrent connections.<p>Redis (and other NoSQL storages) don't have some magic architecture that gives them advantages over SQL databases. They just cut corners on ACID guarantees and skip fsync. Once you start doing fsync, your transaction throughput will drop to SQL database level.<p>Redis also doesn't have transactions which means every app error damages the data. You will spend engineer hours investigating and fixing the problems. Transactions save so much time and worries.
Everything you said is incorrect. Redis does have transactions, as well as data persistence. All cloud providers provide managed redis instances with automatic backups as well.
I mentioned that Redis can write changelog (called AOF in the docs [1]). However, if you tell it to do fsync on every update (like SQL databases do), it stops being that fast and spends time waiting for the filesystem.<p>Furthermore, the RDB snapshot mechanism (when Redis forks and forked process writes the snapshot) can double memory consumption and cause thousands of page faults in Redis process if there are many writes happening.<p>The docs contains corresponding warnings. "Cloud backups" are marketing terms and not ACID guarantees.<p>As one more disadvantage, Redis has no SQL and you cannot easily view the data.<p>As for transactions, indeed it seems to have them, but their execution is serialized, i.e. when MySQL can prepare 100 transactions in parallel, Redis will execute them sequentially.<p>[1] <a href="https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/" rel="nofollow">https://redis.io/docs/latest/operate/oss_and_stack/managemen...</a>
Does Redis become that slow when you enable both AOF and RDB? Sure, there's a write cost, but it doesn't lose its ability to maintain tens of thousands of connections. Redis supports AOF and lets you choose the fsync policy.<p>But I think using only MySQL is unnecessarily expensive, just to get single transaction tracking for bug tracing. So the article's argument seems to be:<p>'Use only MySQL as a solution to the distributed transaction consistency problem between two different storage systems, Redis and MySQL!'<p>But I think using Redis is much more elegant. It's easier to scale. I'd even argue that something like Saga would be a better approach. Of course, we might just have different opinions. But in my experience, reducing layers always ends up making things more complicated in the long run.<p>p.s. We have different views, but I do think some of your points are valid, so I upvoted your comment
The fsync policy equivalent to SQL database would be "fsync on each change before reporting successful update to the app". Redis (as many NoSQL databases) also doesn't have SQL and is a pain to view the data, you need to write extra tools when investigating the problems.<p>RDB snapshots can cause multiple page faults due to use of fork() and CoW.<p>> It's easier to scale<p>The company in question manages online stores and they could easily scale by allocating a separate database for each store (sharding).<p>> But I think using Redis is much more elegant.<p>I cannot agree because I think using a single database for all the data is more elegant, than multiple different databases and there are less problems to deal with. I dislike microservice-style architecture strongly and believe it is mostly good for wasting company's money.<p>> 'Use only MySQL as a solution to the distributed transaction consistency problem between two different storage systems, Redis and MySQL!'<p>I read it as "do not create unnecessary work by using a single database".
It's interesting that our views differ. I think it's because of our different experiences. I believe MSA is the right approach. But this doesn't seem like a debate that can be resolved through discussion. It's something that needs to be implemented and tested.<p>Still, I respect your perspective and your experience. We clearly have different values, but I think you have a mature engineering mindset. Ultimately, I think only real measurements can settle this. Have a great day.
Shopify’s founder and their coo both fund far-right extremism, and its founder thinks only rich people should be able to vote. But anyway, they switched databases.<p><a href="https://www.techwontsave.us/episode/340_shopifys_leaders_are_pushing_right_wing_politics_in_canada_w_rachel_gilmore" rel="nofollow">https://www.techwontsave.us/episode/340_shopifys_leaders_are...</a>
The link you shared is just a podcast and does not contain even contain “far right”. Can you provide specific concerns, otherwise I don’t see anything wrong with a CEO of the most successful tech company should not be concerned about a horribly performing country from a GDP perspective.
I included a link to a podcast because that's a good high level overview of the topic. If you want to dig into it further, the podcast episode page I linked to has links to more reporting and plenty of keywords you can type into google.com if that's not enough.<p>But here's another link if you need something that includes the phrase "far right": <a href="https://pressprogress.ca/shopify-executives-right-wing-media-website-rails-against-immigrants-while-defending-a-legally-designated-terrorist-group/" rel="nofollow">https://pressprogress.ca/shopify-executives-right-wing-media...</a><p>Anyway, if you think a country having a low gdp per capita is how you measure if it should suspend voting rights for disabled people and stay at home parents, then I suspect you're not actually reading any of this.
You should be concerned what they spot as the problem and what they propose as the solution<p><a href="https://www.ctvnews.ca/business/article/shopify-ceo-draws-criticism-for-apparent-support-of-giving-wealthy-more-voting-power/" rel="nofollow">https://www.ctvnews.ca/business/article/shopify-ceo-draws-cr...</a>
[dead]
Yeah it's an awful place to work unless you're a far-right bro. My old director used to use slurs and vape in the office. The founder hires pro gamers with no technical expertise because he thinks they're cool.
They are hiring with AI slop:<p><a href="https://www.shopify.com/careers/disciplines/engineering-data" rel="nofollow">https://www.shopify.com/careers/disciplines/engineering-data</a><p>Pair programming and forced AI, that sounds like absolute hell. Glorification of Lütke who didn't do that much in open source and now props up his ego by thinking "AI can do it so it wasn't all that difficult all along."<p>I don't think he ever worked on complex parts of Ruby. The people he now oppresses did.<p>Ruby should note that this company is actively repelling people from using the language. I really want to switch, but then I see Claude contributions in Ruby core, the influence of this slop company, and think it isn't worth it.<p>Oh, and they bought DHH in 2024 for his 180° turnaround on AI. He is now an AI booster, so Rails is out of the question as well.
They were really so proud of that AI image that they just had to tack it on at the end? Did nothing but make the blog post feel like cheap mass produced slop
This is Shopify, the leadership is full steam ahead on AI in a big way and they review employee performance based on AI usage.
The blog probably was.shopify was pretty early and publicly all in on using AI for everything
outside the slop, i liked this post that was linked on innodb locking: <a href="https://jahfer.com/posts/innodb-locks/" rel="nofollow">https://jahfer.com/posts/innodb-locks/</a>
[flagged]
[dead]
[dead]
[flagged]
so this is interesting to me, im in retail i work closely with platforms ive used shopify ive used magento ive used smaller players ive helped implement various pieces of all of them.<p>and i was excited to get some insight, then i realized that this whole thing was written by AI and im going to guess the idea and implementation were probably very AI driven.<p>> The solution: SKIP LOCKED
> Core idea: one row per unit, bounded by design<p>cool, thanks claude.<p>Now I'm wondering what the engineering culture is even like at shopify.<p>Here's the thing. I like databases, I think there's a lot of shit in this space that went and smoked a shit ton their own good stuff to come up with these pure event driven designs that lock you into event workflows with no isolation and remove the ability to do broader bulk-functions.. and then do something even stupider and say "all you need for the interface is graphql" and such service/platform doesn't give you any other way to reconcile or do reporting for your org you have to warehouse from graphql.. this is crap. So seeing a headline where shopify says they want to kinda get behind a unified database strat behind the scenes even if it's not necessarily customer facing, like that's good imo. SQL is many decades of relational algebra that makes insane computations acrossed vast sets of data pure magic and one of the best query dml interfaces of all time.<p>..however i dont even agree with the claim their making here that redis isnt the tech for a reservation system. redis when used correctly feels like an insanely awesome way to do a reservation system, i lurv redis for stuff like that.<p>I'm just gonna go forward with the assumption that current and future shopify updates are pure vibeslop. I already hate their data interfaces, but compared to other saas offerings i appreciate that they do have bulk-features.
I found Shopify’s post very easy to read, and learned about some features of MySQL. On the other hand, I didn’t get any value from reading your comment. You seem to have a bunch of opinions about how things should be done, but haven’t given any details about how you came to these conclusions.
I've done multiple large scale implementations with shopify paired with many flavors of order managmeent systems as well as competing offerings in the space. The only one i haven't touched that i'd like to get my feet wet with is commerce tools.<p>I really just disagreed with the assessment that redis is not good enough for the job for a reservation system. I use sql database all the time, I prefer them. But I'm seeing a claude written article here that seems to heel turn on a proven technology, it would at most be insightful if there was human content in here from actual engineers at shopify who want to vouch for and explain the challenges they were up against with redis rather than just expect me to take claudes word for it. Anyone who's been dabbling with AI knows damn well that you can convince claude to write up a dissertation on any hill you want to die on.
I found it really hard to read, the llm-isms are just too distracting. Does no one proof blog posts anymore?
You should be able to do these increments/decrements in a database at the rate you can write WAL to the disk. But the problem is in a lot of these databases the transaction will hold locks until the WAL hits the disk which causes a massive serialisation problem when you have lots of writes to the same row.<p>For example if it takes 20ms to write a batch to the WAL then if you do 5 updates to the same row then that is a minimum of 100ms. But without waiting on locks if you can batch all the WAL writes together then this could be just 20ms.<p>I don’t think holding locks while waiting for WAL is strictly necessary. There is definitely some anomalies that can happen if you don’t wait for WAL to be durable because transactions that don’t write WAL can observe non-durable writes in some situations. So for example conditional updates that don’t perform work. But I assume this can be fixed by making these wait on the commit for dependent transactions to become durable if they are empty. There is also the problem of failing writes that reveal information about non-durable writes which is more tricky. For example you try to insert into a unique index and it fails, but the duplicate was due to a non-durable write that is lost.<p>Pure reads should be fine when using MVCC because you just show the latest durable version of the DB. I know some other replication systems will run all transactions including reads through the WAL/replicated log in order to not have anomalies.
Example of their culture: <a href="https://x.com/tobi/status/1909251946235437514" rel="nofollow">https://x.com/tobi/status/1909251946235437514</a>
They do heavily use AI, but you haven’t refuted their point that if inventory is in SQL, storing reservation in a second storage system increases complexity.
I mean I'm all for everything collapsing into sql. SQL all the things. Not really against it, I'd just rather not-AI write the challenges they were up against. It seems like these are all very behind-the-scenes scaling issues they faced, so it'd be cool to hear from them. Redis has great qualities, I don't use it often but I've also for years now understood why Redis was put in front of these use cases to handle them. Complexity be damned generally you're trying to enforce a first come first served or some level of idempotent behavior, so if sqls doing that now then hell yeah. It's just super off-putting to try and upend an important design pattern with an AI written article.
The lengths companies will go to avoid running different pieces of software...
Most companies would be best served picking MySQL or PG and only adding something else if absolutely necessary. Every piece of software added increases complexity.
It can be easier and cheaper to solve problems via technology changes than operations and people<p>Now you only need MySQL expertise and maintenance rather than Redis <i>and</i> MySQL
The default should be that every additional piece needs to be justified