I'm thinking quite a bit about this at the moment in the context of foundational models and their inherent (?) regression to the mean.<p>Recently there has been a big push into geospatial foundation models (e.g. Google AlphaEarth, IBM Terramind, Clay).<p>These take in vast amounts of satellite data and with the usual Autoencoder architecture try and build embedding spaces which contain meaningful semantic features.<p>The issue at the moment is that in the benchmark suites (<a href="https://github.com/VMarsocci/pangaea-bench" rel="nofollow">https://github.com/VMarsocci/pangaea-bench</a>), only a few of these foundation models have recently started to surpass the basic U-Net in <i>some</i> of the tasks.<p>There's also an observation by one of the authors of the Major-TOM model, which also provides satellite input data to train models, that the scale rule does not seem to hold for geospatial foundation models, in that more data does not seem to result in better models.<p>My (completely unsupported) theory on why that is, is that unlike writing or coding, in satellite data you are often looking for the needle in the haystack. You do not want what has been done thousands of times before and was proven to work. Segmenting out forests and water? Sure, easy. These models have seen millions of examples of forests and water. But most often we are interested in things that are much, much rarer. Flooding, Wildfire, Earthquakes, Landslides, Destroyed buildings, new Airstrips in the Amazon, etc. etc.. But as I see it, the currently used frameworks do not support that very well.<p>But I'd be curious how others see this, who might be more knowledgeable in the area.