<p><pre><code> > I’m going to blame Open Telemetry. Yes, it’s powering the modern observability stack and yet I blame it for the mass confusion
</code></pre>
Of course. It’s the classic Internet dev thing of “make a blog post about how using tool X well was a boon to what you were trying to do” and a mass of people walk away with “tool X magically solves things in its default configuration without me learning anything”.<p>The nice bit is this post will then lead to a bunch of companies’ log storage prices spiking and then a post about how you don’t need wide events trending.
Very ignorant take. If all of us had infinite infrastructure there would indeed be no reason to separate the three at generation, but for those that have to actually count the costs, paying for servers to store that amount of duplication is real issue, and even more (for some reason, by order of magnitude or more) if you're using one of big observability providers.<p>It is divided mostly because database to store metrics, and database to store <i>essentially</i> freeform documents are very different products and need different features from query language, and having "just metrics" allows to store them with compression reaching 1byte per metric data point or less.<p>Logs and traces I admit, split here is unnecessary, and it should be just presentation difference. But metrics is often just unrelated to any single event summary of the state of the system. Sure you can just log every single log entry with request metadata and get request rate, percentiles etc. by calculating the events, but that is just too wasteful for most.
wide events feel like structured logs, great for high cardinality data but not so much for time series
(2024)<p>prev. submission - <a href="https://news.ycombinator.com/item?id=39529775">https://news.ycombinator.com/item?id=39529775</a>
Potential PII nightmare. You're now potentially circumventing access controls from sensitive data.
Tbh, it doesn't seem much different than having to watch out for not stuffing PII in logs.<p>Besides, with wide events, it seems possible to apply data lineage ideas and mark certain fields as sensitive and restrict access to them.
To me this seems easier than the alternative model (logs, metrics, traces), in that you only have one abstraction to contemplate handling PII in?<p>In what way is this worse than the alternative?
Why is 77rushi77's comment flagged + dead? Seemed like a good question.<p>Edit: Looks like it's been restored.
The Scuba example is convincing for debugging. What the post doesn’t cover is cost. If you keep raw wide events instead of pre-aggregating, how do you handle storage and query cost at scale? Is sampling the main lever, or do teams still keep cheap metrics for alerting and use events only for investigation?
The whole wide events thing is Honeycomb’s marketing pitch. It’s essentially offering developer velocity, ie you don’t have to think about what should be a metric in advance. They have an open source project called Refinery which is a smart sampling proxy that samples successful requests but always forwards errors, slow queries, etc.<p>You can do anything you like with any approach but if they are mostly about ad hoc queries, dumping everything into BigQuery as a wide event can be the most economical. However if you want to run alerting, SLOs, etc with frequent queries over aggregated metrics, the best bet is using OTel Collector and metrics store of your choice like Prometheus or Mimir.<p>Most companies could save a bundle on observability if they were to move to a hybrid that includes both of those approaches instead of cloud provider logging, log-based metrics, and alerting.
[flagged]