This topic always surprises me. I do not understand the sequence of logic that leads people to build synchronisation mechanisms based <i>only</i> on webhooks.<p>Webhooks aren't at-least-once, nor at-most-once, nor are they guaranteed in-order. Some people build systems to make them more reliable, but if you really care about the data you need to think of a webhook delivery as best-effort, a bit like UDP.<p>That's before you get into all the extra complexities around these systems being owned by different people. For example, either or both system might have to roll back their database. Or either side might have a long-term bug in how they process webhooks, and now you have months of broken data.<p>My view is that the only reasonable thing is to start with the process that gets things back into sync if everything is broken. That almost certainly involves a poll or query of at least the upstream side, and maybe both sides.<p>I find that if you put a decent bit of engineering effort into that "disaster recovery" synchronisation, it can often act as the main or only synchronisation process for quite a lot of systems.<p>Stepping up from that, it's often useful to introduce webhooks as notifications only; that is, to provide a signal that some or all of the data is stale. You have to do a bit of consolidation, but this approach is usually enough to get completely reasonable latency for the kind of applications the author is describing.<p>Only if that wasn't enough for speed/scale reasons would I reach for a truly "push-driven" fast path. But you always have to be able to disaster recovery assuming the stream is wildly out of sync.<p>Some bits of the author's idea seem reasonable: certainly, I would love for there to be a standard protocol to request new data since some cursor or since some timestamp, ideally with some webhook notifications to give hints on when to poll.<p>The problem I have with the author's idea is that it is very strongly event-based, but the desired outcome <i>isn't</i> event-based. The desired outcome is almost always "the <i>state</i> over here looks like the <i>state</i> over there". Relying too strongly events ends up at the same kind of problem another level down: the "disaster recovery" script ends up wanting to compare the states anyway to figure out whether the events are broken.<p>Going fully event-sourced can work (although, I think, less often than advertised), but it really relies on everyone collectively agreeing on the same event stream being the source of truth. Once you start doing work across multiple organisations then that coordination is relatively rare.<p>What really surprises me is the variation in maturity on this topic. There seem to be people at all experience levels who are both doing this well and doing it badly. I have worked with people with decades of experience whose whole design just collapses if you ask "but what if X?" for some really banal values of X like "we have an outage for more than five minutes" or "we have to restore the DB to yesterday" or "someone, one time, accidentally merges a bug into master".<p>As an aside, I do find the obvious LLM-ness of the blog post and the proposal a bit disheartening. These are problems that require diligence and precision of thought. LLMs may be able to achieve those things, but that level of quality just isn't expressible in "Claudish".
The proposed “feed” solution is functionally indistinguishable from incremental reconciliation.<p>The article presents a good framing and is well-written, but doesn’t really propose anything new.
With webhooks, consumers get to asynchronously respond to updates from a provider. If no data has changed, a provider will not send any updates.<p>With SCROLL, consumers are responsible for choosing when to ask a provider for updates. Without a mechanism for knowing when data has changed, consumers will be forced to be pessimistic and poll providers for new data on some cadence.<p>I see two issues with the proposal: (1) SCROLL will lead to an increase in unnecessary network traffic for both the consumer and provider, and (2) because a consumer cannot know when data has changed, the lag between a consumer's local model and the provider's data model will be larger when with Webhooks.
Assuming you're not using the proposed streaming option, I suppose you could always send a webhook for that fact alone? In other words, an empty notification, with semantics of "something has probably changed, better poll the SCROLL if you aren't already".
If your API is just wrapping Kafka, it can long-poll
The end here reminds me of "The Log: Real-time data's unifying abstraction" [0], which has unfortunately had a bit of link-rot since 2013.<p>One complication in this approach involves access-windows: What if my system is only supposed to be seeing <i>stuff</i> that happened during two separate weeks in the year, because those are the spans when it was subscribed or authorized?<p>So the data-host would need to maintain a concept of "connection history" for other services, and also use that to filter/modify its real event stream, inserting artificial "initial state" roll-ups of events that happened in dark periods.<p>[0] <a href="https://news.ycombinator.com/item?id=6916557">https://news.ycombinator.com/item?id=6916557</a>
Webhooks are a painful problem. To clarify, Stripe's events API definitely ships a cursor and polling it has been the method preferred by large consumers for a long time.
Dude is not wrong ... but good luck convincing the Internet to switch to a sane system, when everyone already thinks web hooks are a "solved problem".