When Kepler published his models:<p>* <a href="https://en.wikipedia.org/wiki/Kepler%27s_laws_of_planetary_motion" rel="nofollow">https://en.wikipedia.org/wiki/Kepler%27s_laws_of_planetary_m...</a><p>People did not understand how or why they worked, but they switched over to them from the Ptolemaic (Tychonic, <i>etc</i>†) system anyway because the math was easier. It was only later (Newton, Einstein) that we understood the "why" and "how".<p>† <a href="https://tofspot.blogspot.com/2013/08/the-great-ptolemaic-smackdown-down-for.html" rel="nofollow">https://tofspot.blogspot.com/2013/08/the-great-ptolemaic-sma...</a>
Kepler did understand the data he had, where it came from, what his laws were claiming, how to verify that the data fits the laws, how the data dis-proves heliocentricity etc, which is really what the article is talking about.
Kepler's models showed <i>that</i> the planets travelling in elliptical orbits matched observations, but Ptolemy (and Brahe, <i>etc</i>) also produced models that (tried to) match observation.<p>If Kepler's models explained the <i>how</i> and <i>why</i>, why were Newton and Einstein necessary?
Newton and Einstein answered "how" questions with more generality and more precision.<p>Kepler's model predicted the behavior of the planets, and so does Newton's model, but Newton's model is more general, applying not only to planets, but also to apples, cannonballs, trains, and other objects with mass.<p>Einstein's model is more accurate and precise than Newton's model, especially about the orbit of Mercury. Einstein's model also predicts gravitational lensing (e.g. during eclipses), which Newton's model doesn't.<p>"Why" questions are very hard for physicists. Generally, when we ask "why" questions, we're asking what prior Event X caused Event Y. But for universal laws, there is no prior event that caused the universal law to be true. The best we get is more generality. Planets orbit the sun for the same reason apples fall to the earth, but we don't know why gravity happens, only how. Magnets resist each other for the same reason the floor is hard, but we don't know why magnetism happens, only how.
Weren't Kepler's models worse at prediction, at first? We adopted them because they were more parsimonious.
Newtonian mechanics doesn't explain the why of itself, it explains why Kepler's laws are true, an explanation that won't make sense if you don't know what Kepler's laws are saying, and which won't hold any significance to you if you don't know why Kepler's laws are thought to be true.
> how to verify that the data fits the laws<p>What? He worked meticulously to match the observations.<p>I don't understand your claim here at all.
This is just going to devolve into more tribal elders pontificating to be possessed of knowledge the average person won't be able or allowed to falsify; the expert can clearly see how their interpretation is correct and it's disrespectful to challenge elders nonsense<p>Tao and such are merely biological beings serving biological imperative to survive. Educated as they are they are well aware social society expects them to engage in knowledge role-play to barter for food<p>The recursive non-fractal nature of reality where "function exits" when a human dies naturally fosters signal attenuation and the formation of new spoken traditions among the future generations<p>There is no propagating our social ways 100 years into the future. We can only propagate our material outputs. The semantics we define to describe something are lost to time.<p>The provenance to future people was past people; that's all that matters in the future not how we communicated together now
When the article brought up the cancer drug analogy, I thought roughly the same thing. We still don't know <i>why</i> certain drugs work (take gabapentin as a prime example), but we're still comfortable using them on a regular basis.
Yeah. Medical practice is often based on purely statistical statements like "in case of A associated with conditions B if you do C you'll help D% of the patients". Sometimes looking up a drug's mechanism of action yields nothing, literature just says it's unclear and gives us some educated guesses if we're lucky. Makes me a little sad that medicine has been reduced to this instead of the romantic "understand how the body works, how it fails and how to fix it" ideal.
I'd be careful with the inference you're making here. Tbh we could easily conclude that neither Newton nor Einstein explained "why". One could argue we still don't know and won't even after we have a TOE.<p>I'd take Asimov's "Relativity of Wrong" here. And I also think there's motivation for why scientists care so much about elegance.
> And I also think there's motivation for why scientists care so much about elegance.<p>It's an intuition you build pretty early on in science courses in undergrad, somewhat related to Occam's Razor. It's why the optimal packing of 11 squares[1] is so unsatisfying. It flies in the face of all the intuition we've built that simple (and symmetric) solutions are more likely to be true. Machine learning practitioners use this intuition when performing model selection as well: the simpler model (i.e. with fewer free parameters) is less likely to be overfit to the data.<p>[1] <a href="https://en.wikipedia.org/wiki/Square_packing#/media/File:Packing_11_unit_squares_in_a_square_with_side_length_3.87708359....svg" rel="nofollow">https://en.wikipedia.org/wiki/Square_packing#/media/File:Pac...</a>
<p><pre><code> > It's an intuition you build pretty early on in science courses in undergrad
</code></pre>
I think this intuition is much harder to reach than many think, which causes the frequent over confidence [0]<p><pre><code> > It's why the optimal packing of 11 squares is so unsatisfying.
</code></pre>
Idk, maybe one of those "beauty is in the eye of the beholder" kinda things?[1]<p>[0] <a href="https://m.youtube.com/watch?v=hV41QEKiMlM" rel="nofollow">https://m.youtube.com/watch?v=hV41QEKiMlM</a><p>[1] <a href="https://pingyou.com/papers/eleven-squares.pdf" rel="nofollow">https://pingyou.com/papers/eleven-squares.pdf</a>
> <i>I'd be careful with the inference you're making here. Tbh we could easily conclude that neither Newton nor Einstein explained "why". One could argue we still don't know and won't even after we have a TOE.</i><p>Why be careful? What would be wrong with that conclusion (if it's true)? I'm comfortable with arriving at ambiguity:<p>* <a href="https://en.wikipedia.org/wiki/Principle_of_sufficient_reason" rel="nofollow">https://en.wikipedia.org/wiki/Principle_of_sufficient_reason</a><p>* <a href="https://edwardfeser.blogspot.com/2023/04/what-is-law-of-nature.html" rel="nofollow">https://edwardfeser.blogspot.com/2023/04/what-is-law-of-natu...</a>
We all know Terry Tao, do you know Peter Scholz?<p>Fields medalist and as Tao listed as one of the most important mathematicians of our age.<p>Scholz is known to find
„precisely the correct and cleanest way to incorporate all the previously done work and find an elegant formulation for that — and then, because he found really the correct framework, go way beyond the known results“<p>In a way that man is the complement of an AI, he generates understanding rather than just proofs.<p>Maybe he is the right template for the mathematicians of the future.<p><a href="https://www.quantamagazine.org/peter-scholze-and-the-future-of-arithmetic-geometry-20160628/" rel="nofollow">https://www.quantamagazine.org/peter-scholze-and-the-future-...</a>
Even Peter Scholz is not always sure of correctness of extremely complicated mathematical statements and welcomes computer assistance in verification of his results.<p>See the Liquid tensor experiment:<p><a href="https://xenaproject.wordpress.com/2020/12/05/liquid-tensor-experiment/" rel="nofollow">https://xenaproject.wordpress.com/2020/12/05/liquid-tensor-e...</a><p><a href="https://www.ma.imperial.ac.uk/~buzzard/xena/pdfs/liquid_tensor_experiment.pdf" rel="nofollow">https://www.ma.imperial.ac.uk/~buzzard/xena/pdfs/liquid_tens...</a><p>"
I spent much of 2019 obsessed with the proof of this theorem, almost getting crazy over it. In the end, we were able to get an argument pinned down on paper, but I think nobody else has dared to look at the details of this, and so I still have some small lingering doubts.
"<p>In a sense, mathematics is not very different from software engineering.<p>More precisely: Curry–Howard correspondence<p><a href="https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspondence" rel="nofollow">https://en.wikipedia.org/wiki/Curry%E2%80%93Howard_correspon...</a>
He is also a member of the association of human mathematicians.<p>“
1. Members commit to not working with commercial AI companies by providing technical labor, mathematical knowledge, consultation, or publicity.<p>2. Members commit to not publishing mathematical texts (including papers, reviews, referee reports, lecture notes, teaching materials, and books) generated by AI models. Members only publish their own writing, as opposed to outputs of AI tools. We affirm that mathematical writing should have human authorship.
“<p><a href="https://www.ahmath.org/members" rel="nofollow">https://www.ahmath.org/members</a>
Just like small numbers appear in almost all places in mathematics. Small numbers are limited. Sometimes these may appear as coincidences. Similarly small elegant explanations are limited in numbers. As mathematical universe expands, only way to explain these additional results will be to have complex explanations.
<a href="https://en.wikipedia.org/wiki/Peter_Scholze" rel="nofollow">https://en.wikipedia.org/wiki/Peter_Scholze</a><p>> Scholze was a student of Rapoport, who was a student of Deligne, who was a student of Grothendieck, who was a student of Schwartz; in this chain, Scholze, Deligne, Grothendieck, and Schwartz are all Fields medallists.
Ramanujan is another one that did much in his head and lacked a lot of formality in his methods. Many intuitive mathematicians are this way (Shinichi Mochizuki is another although his work may just be so hard for others to undertsand that it appears to lack rigor).
I did not; thank you.<p>One bit stands out to me: to better understand, Scholz will work the problem backwards. Sometimes that strategy is just so spot on.<p>--<p>To my eternal shame, philosophy (like this OC) makes my head hurt. Pretzels my brain, ouroboros like (aka "head up my own butt").<p>Caveats covered, here's my cave man cold take on two aspects of this topic.<p>#1 As we're all trying to paraphrase here: answers are great; understanding is even better. I like the word "legibility" (h/t James Scott). The current models are not very legible.<p>Which directly leads to:<p>#2 Understanding enables predictions.<p>If LLMs are stochastic parrots, then pi.dev (GPT + reasoning + ralph loop) is a drunken sailors walk (thru a solution space). A kind of messy heuristic for optimization. With some kind of gate for declaring "done!" Pretty useful.<p>However, I don't expect Pi.dev to make predictions. That'd require something else. Maybe that something is being worked on; am noob, so I wouldn't know. (I'm still unclear about whatever Jeff Hawkins is doing, in practical terms.)<p>--<p>Wrapping up: Automated prediction would be cool. That's very different from finding solutions to problems. And the current cohort of AI can't even explain why, much less make predictions.<p>Assuming predictions requires understanding. Maybe there are other strategies to generate predictions. Like a meta drunken sailor's walk, with drunken sailors all the way down.<p>(Someone please correct me as needed. TIA.)<p>[Aside: That article has an ad for their podcast They Joy of ₍₎. Perfect. I have no idea if it lives up to its promise.]
> One bit stands out to me: to better understand, Scholz will work the problem backwards. Sometimes that strategy is just so spot on.<p>In David Deutsch's book The Beginning of Infinity, he argues, that the purpose of science is not to merely predict or to model the world, but to explain it. And he gives in my opinion convincing arguments, why a better explanation is needed to advance science. Therefore, he deduces that pure empiricism has failed. Because no matter how much data you have gathered, an explanation does not follow from it. The explanation is always an insight the scientist has to contribute. Then it is open to scrutiny by criticism or experiments.<p>I guess he would argue the same is true for mathematicians. Merely constructing proofs to arrive at theorem, is of no intrinsic value, if it doesn't spark an idea of a more elegant formalism that corresponds to a simpler and even more far reaching explanation.<p>I cannot recommend this book highly enough to anyone who is intellectually curious, not only about epistemology.
Many high stakes problems are blocked by a specific insight, and once known, the people who are highly invested in those problems won't put sufficient care into whether we've built a supporting framework of ergonomic theory for humans.<p>That's not to say there isn't value in building mathematical insight for humans, just that there is a goldmine and a lot of people know what to do with gold, even if they don't understand the ecosystem surrounding the gold.
Ahhh, epistemology... Does any academic researcher even care about these things anymore?<p>If you're in academia, you've probably noticed a major shift recently. Research production at an industrial scale through AI.<p>Grants have "always" been awarded to professors based on quantity multiplied by quality. Now, some professors are churning out 1–3 papers a month with the same group of co-authors. All the grant money goes to them. Grant givers are just only counting the numbers of papers being published.<p>The Industrial Revolution has hit academia, and some academics are busy arguing for the artisanal value of research while some are creating factories for research.
Yes, true scientists do care about epistemology.<p>The problem is the systemic disincentivation of true science, the pursuit of truth and knowledge unencumbered by <i>greed</i>.<p>When you speak favorably of "factories for research", you adopt the position of a mindless <i>consumer</i>, relishing the prospect of "abundant" <i>products of science</i>.<p>You fail to notice, livestock lives in abundance as well.
Nobody will speak favourably about the factorization of anything.<p>The "publish or perish" culture is going through what is essentially the same transformation brought by the invention of the assembly line. Academia, "as an industry", is experiencing what the automobile industry went through a century ago.<p>Grant approvals are public. Just go through them, and you will see a pattern emerge.<p>The argument I keep hearing is, "AI is good in the hands of a good researcher."<p>5–7 papers based on the same dataset? 5–7 papers on the same core topic, published within a six-month period?<p>They tell me that this is their core specialization and that I simply don't understand how science works.<p>Science is a self-governed, self-serving, and artificially defined concept.
You conflate a host of different issues and ultimately miss the point.<p>Science is an <i>ideal</i>, the pursuit for knowledge for its own sake. Science is a boon for humanity because it provides an unparalleled degree of <i>prediction</i> and <i>control</i> of the natural world.<p>But it doesn't tell you <i>what</i> to do with that.<p>A "capitalist" will seek personal, short-sighted gain from it, socializing repercussions, no matter how devastating. They will gladly poison society and the planet and sell it to you as "your own fault". Their life-span is limited still, so what do they care?<p>Technological capability exceeding human wisdom is the core issue here.<p>You need scientific understanding <i>on your own side</i> in order to know <i>and judge</i> the alternatives beyond what vested interests feel obliged to tell you.
I think the actual truth may be more important than we give it credit for. For instance I've noticed that most propaganda consists of lying about factual information, rather than what to do with the information.<p>The decision about what to do only becomes relevant if the underlying information is true.
Well, what to do and why it will accomplish what you (and everybody else) want and nothing else is the point. The facts being actually true is a <i>necessity</i> to pull that off.<p>Think of AI as a magic wand or a genie that makes your wishes come true.<p>While that sounds great at first, <i>actually thinking it through</i> reverses that mirage completely.<p>Whom would you want to have one? Who should get to decide? How do you control the whole matter and prevent it from going horribly wrong?<p>Being irresponsible doesn't work already when you're a grown-up human with only your bare hands.
As you said, "Science is an ideal." I think you and I have fundamentally different views of science. You are an idealist, whereas I am a practical person who has seen how the sausage of science is actually made.
You're talking about <i>academia</i>, society's attempt to gain <i>economic utility</i> from science. You assume, I didn't know about it based on what?<p>Exploiting science for economic utility alone misses the incurred second order effects, as I posited above. You completely ignore that crucial part, despite of it being the whole (very much practical) point here.
A practical person would be off doing something else. You have an axe to grind.
Still there's something to the notion of people who prefer Academia to Industry. As much as I tend to agree with your broader points that it's a game and people love the prestige game and they now can pull levers to accelerate their winning in that Prestige game.. still they chose a path that is less about money and more about ideals than industry. It may be less and less ideal driven and less and less of a cloistered monastery but it's still a place for people who chose against industry. I say that as an industry guy who can just feel when I speak to academics that they are driven by something different than those of us industry. And to be clear those resident industry are not only driven by money we're also driven by building and changing the world but it's still different
[dead]
True Scotsman fallacy. No meaningful argument was made above due to taking the stance of "true".<p>If "false" scientists end up getting all the money, resources, and social status from spamming LLM-generated papers, and the "true" scientists are shunned as slow idealists with 0 value, the first one is more of a scientist in the eyes of society than the latter.
I've seen a modest increase in productivity in bioinformatics. While the field is computational, it's ultimately about analyzing physical processes. The biggest bottlenecks have always been figuring out what makes sense to do and getting the data you need.<p>You can of course try new ideas and make software tools much faster now, but software is much like meetings. They can be productive if properly motivated, but they can also be a way to pretend you are working without contributing anything of real value.<p>Theoretical computer science and mathematics have changed. I'm not sure if they will remain active fields of research in the future. If they will, I would guess that theorems and proofs will eventually become things you include as supplementary data rather than in the main paper. Outside some rare exceptions, people may not consider them publishable results anymore.
Factories for research have existed since Edison. The scale is the only difference.<p>The disdain for human intelligence is dripping off your comment "some academics are busy arguing for the artisanal value of research "
Which is why there is this disorientating feeling. We are all facing now with an ongoing challenge to epistemological trust, like
Someone suffering a mental disorder.
Yes we care! People are having intense debates about this stuff!
The article's perspective reminds me of math class when I was a kid. The teacher didn't want the right answer alone, she wanted to see how I got there. In an absolute sense, the right answer is the right answer with or without my understanding, but that's not how the class was graded. We're approaching a point where AI is to science as my old calculator is to elementary school math -- it's capable of leapfrogging to answers faster and more accurately than we can.<p>Thing is, though, verification is knowledge itself. And if we can get to a point where the important questions have answers, we can surely have the verifiers explain those to us in ways we can understand them.
> we can surely have the verifiers explain those<p>This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.<p>I think this is an artefact of how we train them, and they could be trained to be better at this if we focused on it, but at the same time, it shows its possible for these things to be able to do things they cant explain to us (granted, they can't explain them <i>well</i> - you can bash your way to understanding with them anyway but they aren't good at it)
> <i>This is not obvious to me at all. LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.</i><p>It's amazing to me that language models are still such poor writers and communicators. The mathematics community wrestled with this recently, when many solutions dropped into their laps but were not explained very well...<p>Thing is, (1) generic LLMs will improve, and (2) their lack of ability in this area has a lot to do with the rhetorical and structural habits they pick up during RLHF/instruction-tuning, which trained them for general-purpose comms. Zero-training base models can presumably be trained specifically to communicate scientific concepts. Interesting times.<p>(Edited to add: With respect to (1), compare Opus 5 to 5.5. 5 was a <i>terrible</i> communicator, even of software/coding concepts, whereas 5.5 is excellent and leaves very little to be desired.)
I am curious. What makes you say generic LLMs will improve? I mean, obviously, they will, but to what extent? How can we know what they will or will not be capable of by the time we reach the limit of feasible improvement? Will they ever be able to explain in a way humans can easily digest?
LLMs are excellent writers in the sense that they get the spelling, grammar, syntax, etc... right. They are certainly way better than I am, and they can compete with the best writers considering how AI-generated prose has winned literary prizes.<p>Many people don't like the style, probably by association with slop and fatigue more than the style being fundamentally wrong.<p>Being a good communicator is an entirely different thing, you can have terrible spelling and grammar and still be good at getting ideas across. It is a common situation when the communicator in question is not using his native language.<p>That LLMs are so good at writing and so bad at communicating is not surprizing. The key to good communication is getting into the other people's mind, LLMs can't do that, because they "think" in a way that is completely alien to us. We use words to express our thoughts, thoughts that are based on a physical world model: touch, movement, sound, light, etc... LLMs have no idea about how the physical world work, everything the see is just words, they have no intuitive idea of, say, objects falling down, just that "falling" and "down" go well together. And for explaining things, they will go "someone asks for an explanation, explanations usually use these words, I will be using the same kind of words".
> 5.5 is excellent and leaves very little to be desired.<p>Hm, that may have to do with (faking) confidence. I've often enough with 5.5 still run into cases where it confidently explained something, at the surface it kinda made sense, but the more I dug and verified, the more it became apparent that it was entirely wrong with the most crucial core concepts. It has gotten to a point where you really need a domain expert though to realize this, which makes it tough for using it in areas where you are not an expert and don't have easy access to one either.
In a world of short videos, shallow reads, and now quick answers from LLM's it is easy to accept truth without verifying before moving on to the next thing. I was running into similar cases where the confidence was there but the answer was wrong, and as the article suggests, math separates correctness and understanding, so I thought it would be a great area to test where hallucinations still exist in the models. I made a test on large prime numbers where the LLM's couldn't know the answer and we can easily check the factors it produces to verify if it faked it. Nearly 75% of GPT-6 Luna were faked with a false factor when given a hard number with no tools.
You point at the exact problem here: people are so used to arguing from ignorance, they don't even notice.<p>Verification <i>isn't</i> knowledge. Far from it. You can verify statements without knowing why they're true. Variants of that concept are e.g. used in cryptology.<p>You need humans to truly understand (to know the <i>context</i>) in order to make correct <i>judgements</i>. Without actually knowing the <i>context</i>, you have no way to <i>compare</i>.
An infallible oracle tells Archimedes that the speed of light in vacuum is ~1.08 billion kilometers per hour. Archimedes, with the iron age tools and methods at his disposal, might be able to verify this and make some use of it himself. Maybe. In any case, what he has gained <i>is</i> knowledge itself.<p>The same goes for our own time. A powerful AI can tell us things at the edge of what we can grasp -- or beyond what we can grasp -- and it would nevertheless be knowledge that we possess. We can work to verify it, we can have it explained to us as though we were schoolchildren, or we can simply accept it as a static fact of the natural world.<p>In truth, we're going to have to learn to coexist with minds that are more powerful than ours. At least in some respects, if not in <i>all</i> respects. The good scenario is that they share information with us, and we receive it however we can. "Head in the sand" and "mutual disdain" are bad outcomes.
No, you're wrong.<p>You might want to look up the actual definition of <i>knowledge</i>. It's a pretty interesting topic in epistemology really, far from your simplistic take.<p>You already live in coexistence with minds more powerful than yours. Society works in a way that you never need to notice that, those minds taking care of things beyond your grasp for you.<p>The downside of that "convenience" is you developing an over-simplified bubble-view of the world, rendering you prone to arbitrarily hazardous misjudgements.<p>Here, you align foolishly with interests that cannot possibly be your own. Namely, exploiting scientific knowledge for personalized monetary gain on an unsustainable scale.<p>When a tiny group of people hogs essentially <i>all</i> scientific knowledge, that renders you what?<p>Your idea of being graciously provided with some "basic income" or whatever is blind copium. You would live as livestock at best, to be discarded as useless whenever your masters see fit.
I don't think the social problems you listed can be solved by redefining knowledge to your convenience - they can play that game better than you can. The definition of knowledge in this context is simply <i>that which describes reality</i>. And the person you're replying to was correct in saying that the thing gained from the oracle was knowledge itself, even though they've not considered the epistemological nuances.
It's not me who's "redefining knowledge to my convenience". You weirdly succumb to "them" doing that and come to wrong conclusions.<p>Merely claimed "descriptions of reality" lack the crucial property of <i>truth</i>. The description has to actually fit reality, or it's mere fantasy. It has to be exhaustive, or it remains a facade, leaving you blind to what lies beneath.<p>When you don't actually know whether it does any of that, you're merely a <i>believer</i>, entirely dependent on your "oracle" and its slave in effect.<p>Not caring about nuance leaves you open to exploitation, among other things.
The oracle is infalliable in the toy scenario. That was asserted as an axiom. To us who are god's eye observers of the toy universe, we already know that whatever the oracle says necessarily fits reality in that universe.<p>But I understand what you mean. Suppose I am Archimedes in there. I don't have access to the axiom, so I don't know if what the oracle told me is true until I can verify it myself. So verify I will. This is what is being done with AI-written proofs IRL. Humans may not be able to internalize the proofs, but we can internalize the programs which verify the proofs. So we do know the proofs are correct, or rather, as correct as the verifier programs.
You could have explained why he was wrong before injecting your opinions into the discussion.<p>He's wrong because the epistemological formula is the <i>justified</i> true belief. The infallible oracle simply gave Archimedes a true belief, the needed justification, sensors that can actually give meaning to the precise number, would take hundreds of years to develop. Archimedes would have to sit there with the oracle and develop modern physics together before the oracle can be said to have given him knowledge.<p>And AI won't have given us any actual knowledge until it can fit its findings into the mathematical establishment. I forget who it was that said that society doesn't know anything until you can teach it to a room-full of undergrads, but that's about right.<p>And you will have to actually show your intellectual work before your arguing here will actually accomplish your goal of sounding smart.
The dictionary definition says that knowledge is information gained by study -- i.e., in the very vast majority of cases, <i>by being told.</i> What's your definition? Come on, humor me.<p>The rest of your post is excessively emotional and off-target.
Perhaps be more thoughtful regarding the "dictionaries" you use? The words you use become the borders of your world.<p>How does random data differ from "information you gained by study"? Let alone from "what you're being told"? How do you possibly notice?<p>Other people cannot force you to gain true knowledge. It has to be earned by putting in the necessary effort. Its value is manifold, starting with not gullibly falling prey to other's interests.
To be fair, most people aren't writers, so the "average" writing the statistical model can produce is mediocre writing.
I don't think this is the issue. Most people are mediocre mathematicians, too, yet the companies have selected well enough for math literacy in training data (presumably). I think its a matter of training - they are often post trained by other agents (which read everything), human reviewers not paid enough to care (easily wowed by 10 paragraph length essay with correct punctuation), and deterministic scripts (checking for presence of a specific point, so agents are incentivised to write as much as possible to have more chances of hitting the unknown thing the script is looking for)<p>As far as I know labs try to train the other way, else we'd have monstrously long and useless outputs, but its hard to overcome these natural biases for length
I don't think that's a suitable mental model of how LLMs work. Their output is not just the statistical average of all the training data. If it was then the output would be quite useless.
An LLM trained on the entire internet incorporates essentially all (human-known) <i>aspects</i> of things.<p>Like a superposition of all kinds of writing styles, all perspectives on every topic and all kinds of ways to do things right and wrong.<p>In post-training (and of course already by sensibly curating the input data in the first place), people try to <i>select</i> useful submanifolds from that.<p>Ideally, you get a combination of all the extant strengths of people combined into one model. Self-improvement then leaves the sphere of human capability for good.
And in many cases it is not really the new solution that is important. It is the new techniques, abstractions, or other sense-making or investigative tools that made the solution possible that matter.<p>And the abstractions do still matter, since in many cases it is an issue of tractability. Perhaps an LLM can force its way to a solution through 3 million mind numbingly boring and convoluted steps, but if the number of steps increases exponentially without the right abstraction that means it can only handle a slightly larger problem than a human with pen and paper.
>LLMs are capable of amazing feats of reverse engineering but are terrible at explaining and abstracting things well.<p>I don't think you use them well if you think this.
I have tried dozens of different writing skills and my own prompt ideas, listened to podcasts with engineers from both major american LLM companies, actually read the prompting guides from anthropic for the models I use, and of course, as if it needs to be said, I use the latest and greatest frontier models, for difficult work.<p>What is your genius setup?
Can you give an example of what you wanted explained, the LLM explanation and why it is bad?<p>Obviously if you don't understand an idea you aren't really in a position to evaluate how good an explanation of it is. I have a background in Physics research and find models like Astra to be excellent at explaining concepts at a level for an advanced practitioner. It also has done very well, from what I can tell, for areas I am less knowledgable in. So I am confused by statements like yours that directly contradict my experience and the subjective experiences of many others.
> Can you give an example of what you wanted explained<p>Something I do dozens of times a day:
"Please explain what this PR does, I haven't read it"<p>it will make me 20 pages of gibberish about caches, correctness checking, race conditions, github action runners, tradeoffs between a bunch of invented terminology internal to the PR without defining the terms- you'd have to have <i>already read</i> the code to know what it was referring to<p>without ya know, explaining it. I ask my colleague and he says something like "oh it deletes records from such-and-such a system when they aren't needed anymore after a merge" and I know exactly what he means. A single well composed sentence was all that was needed.<p>You can ask the agent for a single sentence explanation and it will write something like "linked candidate records are trimmed when canonical record determination window is passed", none of which is terminology any of us use, some of which is defined in the new PR and some of which is made up. I can't emphasise enough, I use these things EVERY DAY. it knows what I work on, it knows how I talk. There is no earthly reason it should be this bad given all the skills and advance and context it has. As mentioned, I constantly work to correct its shortcomings in this area but its like trying to hold water in a sieve, it just cannot do it.<p>"What does this PR do?"<p>LLM: "Oh, it specifies the kernel window over the cache candidate list to classify possible invalidated swap identifiers"<p>Colleague: "It removes duplicates in the target_uuid_cache table in the db"<p>The agent has full context of what systems the code talks to, our terminology sheet etc. It just pathologically cannot simplify things down. If you work with them enough you learn that there are some words it reaches for a lot in certain circumstances, like it loves "canonical" in relation to merging things. But this is more like learning to communicate with a deeply autistic person who struggles to adjust their speaking to be relevant to their audience rather than talking with a "good communicator"
Meaning requires integration. But I assume AI can make meaning of it, now or in the future. I assume the same with people. However I don’t see the point of knowledge without meaning.
My teachers tried to explain to me that they were trying to teach me the method, not the answer. I always thought I didn't need the method, if I could find the answer without it. Nowadays, I think that wasn't quite right.
I still don't see the point in knowing a method that takes longer to arrive at the correct answer. For example long division. It's such a convoluted mess.
The issue I had in school was I didn't see the point of it. Why should I solve this problem? Due to that I didn't mind only knowing the answer and ignoring the methodology.
I hated classes that didn't have correct answers, for that reason. But in math, there was a 100% correct answer I could use to impress the teacher and the other students. Fueling my ego was enough of a reason for me... Nowadays, I wonder why I needed to do that.
The other reason it looking at the approach the strategy the reasoning is so much better than a binary sense of whether something is right or wrong answer is that there's ambiguity in what really constitutes a right answer to begin with. At least in most situations. For example in hard Sciences we may learn that we drop second order details and ignore them for purpose of answering the question but still one could make the argument that in so doing the answers we land on are inherently imperfect. Does this make them less right? Could all be debated. But instead seeing someone who explains I'm going to drop the second order detail cuz I think it's not significant to how I would use this result this is the kind of thing that really is valuable
I sucked at remembering goniometric identities, but I was superfast with testing out a few options on my calculator to make sure I was using the correct one. Without a calculator I was nowhere.
I don't think we're there yet for science. Perhaps for math proofs, but that's a very different domain ultimately since we know the rules of the game in a way we don't with science.
Oh, it's going to move up the old ladder, from bottom to top:<p>> <a href="https://en.wikipedia.org/wiki/Hierarchy_of_the_sciences" rel="nofollow">https://en.wikipedia.org/wiki/Hierarchy_of_the_sciences</a><p>First math and CS, where formal logic dominate, and AI can check/verify its own answers very quickly. Then physics, downstream of math. Then materials science, which in some cases is practically a branch of physics. Then chemistry more generally. Then biology. Then various social sciences.
A big difference is that in mathematics, you can do all your experimentation purely in theory and at arbitrary speed, while for example chemistry needs a lab and plenty of time.<p>Math is really super suitable for quick iterations by a machine.
What makes you believe that it will be such a progression?
Surely the way forwards is to take the results obtained by brute force with AI, for example that certain conjectures are true or false, then to use that knowledge to further human understanding.<p>Knowing that something is true or false will at least stop humans wasting time trying to prove or disprove it, and allow them to find a more elegant proof, or investigate the implications of the true/false result.
I think so.<p>Autonomous or human-directed AI agents submit their work to a public repository.<p>Then the scientific community draws from this pool of results, identifies the important ones, and starts working on adding the results to the greater body of scientific knowledge. This work of interpretation, synthesis and question formation should be recognised and rewarded in peer-reviewed academic journals.<p>I wrote a blog here talking about it: <a href="https://colinocallaghan.com/posts/answers-are-getting-cheaper-good-questions-arent/" rel="nofollow">https://colinocallaghan.com/posts/answers-are-getting-cheape...</a>
<p><pre><code> Knowing that something is true or false
</code></pre>
Assuming you can trust it
Maybe but it might also be useless. Knowing a very large number is the product of two very large primes doesn’t at all help the reader to understand factorization, for example.
From fire and wheel to atomic bomb and now LLMs human evolution follows the same laws: a tool can be both and bad, depending on its application.<p>The P vs NP complexity dichotomy is also relevant: they present different views of a problem solution, knowledge and verification respectively. Both are valid science: the first is desirable but in absence of deep understanding the second is acceptable.
"Is science about humans achieving insight, or the archiving of verifiable solutions?"<p>I wonder if this dichotomy fully captures the value of scientific discoveries. One could argue that their practical impact can be independent of whether humans fully understand how they were reached.
Practical impact has always been linked but ultimately independent of knowledge. You can have very sophisticated technology based on zero actual knowledge or understanding, like roman concrete or borosillicate glass. And you can have a very deep understanding without practical impact, like lasers for decades or galactic algorithms for multiplication.
Indeed, if a machine had told Heisenberg in 1943 (or whatever?) that "the critical mass for an atomic bomb is much less than you think", we could very well have had a different world today, regardless of Heisenberg understanding it or not.
> their practical impact can be independent of whether humans fully understand how they were reached<p>See: most of pharma
The problem is trading off short term and long term progress. A solution alone can be useful in the short term, but if humans don't also build up understanding it will probably prevent long term progress from happening. It's much more difficult to build on top of something you don't understand.
Arriving at truth values has limited value in science, obviously (with full understanding being more useful). For example, knowing a chemical reaction will produce heat is useful (cavemen discovered that one), but knowing _why_ unlocks an entire world of useful tools (the combustion engine, space heaters, etc).<p>However, even getting truth answers will soon be beyond AI. It is of course making impacts in Math and in publishing in "data heavy" subjects where you can just formally verify, program, and/or munge data.<p>But in "real science" where we actually need to build a lab, buy supplies, hire personnel, get approval, form physical test harnesses, install sensors in proper locations, calibrate everything, then finally make one measurement, question the whole thing, then finally make a conclusion, AI is of limited benefit.<p>Yes, it can help speed up coming up with some formulas, spitballing, or checking existing research. It can even help write the jank code for sensor calibration, measurement taking, etc. That's not nothing, but we are _very far_ from meaningful contributions to real experimental hard sciences.<p>As for math, theoretical physics, computer science, and other "logic based" studies, the question becomes one of benefit to cost of true understanding versus simply knowing something is true.<p>I'd argue that it paints a starkly negative picture of most science. Once answers are easy to obtain, it seems most people are concluding that actually understanding why its true isn't worth the effort, which means that the question maybe wasn't worth asking.<p>Its the science equivalent to the recent realization in tech: most ideas suck, your product didn't need more bells and whistles. Actually the engineers pushing back on every shower thought from product was a good thing. The friction that stops you from putting every random-ass feature into your already bloated useless SaaS product was good, and now the sales guy is pushing features to prod for a customer demo. We thought "too many ideas, not enough engineers", we now know "not enough meaningful quality ideas".<p>In the same way we thought "too many questions, not enough scientists", we now know "not enough real study on meaningful topics".
I'm gonna be annoying, but space heaters don't have chemicals
I was referring to those big shop heaters that use forced-air propane, butane, or kerosene to generate massive heat. But yes, there are electric heaters too. It wasn't too annoying, just opinionated and misinformed.
This misses that AI is already designing and carrying out experiments autonomously. This will only explode with the growth of capabilities in robotics.
No I accounted for that. Sure, if you have everything setup for full automation, then yeah, you can write a computer program to execute many experiments in parallel, and that is no doubt useful. But at the point your question doesn't fit into a repeated process, the physical provisioning, sourcing, hiring, contracting, politics, space acquisition, calibration, layout, etc. all applies.<p>For many types of study, especially new ones, having such a setup is just as hard as doing the thing that may or may not require massive automation. For example, testing a new type of high acceleration engine. How would you even setup a lab to have hundreds of test beds for this? Would you need to manufacture hundreds of prototypes of the engine to even do the test? Where would you source that if the whole point is proving whether the engine works and produces more thrust. You probably just have one prototype sitting on a table. Sure when you get to the point of refining a tested and well-studied design, automation helps.<p>I'm sure there are some areas like organic chemistry where you can just toss compounds together and vary the heat, etc. But what are you getting? Don't you need to do specific tests and measurements to figure out if its useful? All of that requires setup with all the caveats above. What if you're doing something else, like metallics. Well then you need a whole lab with temperature and atmosphere controlled furnaces, various cooling surfaces, additives, dopants, etc. Knowing when / where to add, how to cool, rotate, etc. could all play a factor. Automating that right off the bat isn't feasible until you've proven it can actually work, so AI help is of limited value.<p>Now, if we have AGI with full embodiment, that's a different story (since with full embodiment it can handle all these tasks if given authority to do so).
> or many types of study, especially new ones, having such a setup is just as hard as doing the thing that may or may not require massive automation. For example, testing a new type of high acceleration engine. How would you even setup a lab to have hundreds of test beds for this? Would you need to manufacture hundreds of prototypes of the engine to even do the test? Where would you source that if the whole point is proving whether the engine works and produces more thrust. You probably just have one prototype sitting on a table. Sure when you get to the point of refining a tested and well-studied design, automation helps.<p>This is the point where robotics can re-arrange their own labs.
The way I’ve explained this concept to folks for the last twenty years:<p>A broken clock is <i>not</i> correct twice a day.
Too many contradictions exist for this one. Plainly, you have null results. A null result is knowledge.
> “The other side argues… human understanding is paramount, because it drives integration of new findings into the canon […] But the first side, those who emphasize correct outputs, dismisses this as gatekeeping or nostalgia.”<p>Is this description AKA. theory vs praxis? Academia vs Engineering?
This is like saying if I read the oven thermometer to obtain the temperature inside the oven, then that is not real knowledge, because I don't know how my oven thermometer works.<p>Okay, it's not quite the same. I could figure out how the thermometer works, but I cannot figure out how the 1 trillion parameter AI works. But the AI didn't give me an answer directly, it gave me a proof that I can run through a verifier. And I could figure out how the verifier works. That is a partial understanding.<p>The barrier to my understanding is not the verifier. So it's the opaque AI model? But human brains are blackboxes too. I cannot tell what's going on in another person's head when they're writing a proof either. I can only look at their behavior, the language they produce.<p>An AI provides a proof written in the same symbols humans write in. It's really long and complicated, more than human-written proofs. It would be nice if I had an excuse for my inability to internalize the proof in a timely manner. I could form a community to make the proofs we don't fully understand yet = "not knowledge".<p>This is delusional self-soothing, and it's just as damaging to my capacity to understand reality as delegating thought to another entity, regardless of whether that entity is a computer or a human.
AI answers are data points.
Science has been industrialized, it has to change.<p>I propose a two-part approach.<p>First, autonomous or human-directed AI agents submit their work to a public repository.<p>Second, the scientific community draws from this pool of results, identifies the important ones and asks the next set of important questions. This work of interpretation, synthesis and question formation should be recognised and rewarded in peer-reviewed academic journals.
I would rephrase it as: "Verification is not human knowledge", but that doesn't really matter. We are going to be several steps behind AI as it reaches a level of intelligence far beyond human capabilities, and we are not going to be able to catch up with it. It's like trying to beat AI at chess: we are bound to be left behind as AI continues to expand its knowledge in mathematics, physics, computer science, and other fields. Eventually, no human being will be able to reach that level of knowledge within their lifetime.
No need to start worrying about AI reaching a level of intelligence far beyond human capabilities until it actually reaches the level of humans and assuming that ever happens it could be a very very long way off. All of the intelligence "AI" has right now comes from the human data that it was trained on. It's been given enough power to crunch numbers better than we do, as any computer can, and more data to play with than any one human can retain, but none of that equals intelligence. Speculating on a sci-fi future however, if AI is ever able to create new and more advanced forms of AI, which can also create new and and more advanced forms of AI we'll be outpaced quickly.
As long as humans are in charge it's vital for us to understand what is going on - be it in science or coding, once you give up all knowledge (yes knowledge = power) not just work to AI, you can be manipulated/controlled/fooled with ease.
I have doubts about that. You don't need AI (understood as the current LLMs) to make a powerful chess engine, or computer vision or anything that reacts faster and better than a human. We've had those forever.<p>A good word predictor happens to be good with closed systems with very defined rules for which a huge training corpus exists and we are suddendly freaking out about how it's gonna become so much better than us at everything that we are basically obsolete.<p>When in doubt about where AI stands, ask it to create a poem, a joke or a riddle. And see how little actual intelligence it has.
How do we know if/when that happens if it’s beyond us to comprehend. An open ended and broad claim like this can always become true one day, similar to flying cars.
Ive never heard such drivel in my life. Are you a shill? The current "AI" is not intelligent, it can beat humans in certain areas but weve always had that with things like Deep Blue. I dont believe llms will reach human intelligence and certainly not super intelligence.
It's probably fair to say that this problem existed without AI already considering the simple scale of human knowledge and in fact the specialist now only focus on their tiny little pixel and as a result struggle to contextualize the meaning im the bigger picture
I see an important issue missing from these critical discussions: if we can produce valid scientific results faster than before, other researchers can build on them, regardless of whether they choose to use AI themselves. Accelerating one discovery can accelerate many others, with or without AI.<p>"If I have seen further, it is by standing on the shoulders of giants". Including AI.
> Verification Is Not Knowledge<p>A bold stand on the P = NP question.
You are conflating science with philosophy
> <i>You are conflating science with philosophy</i><p>Science needs a metaphysical basis and certain assumptions to even get off the ground:<p>> That there is an objective reality shared by all rational observers; That this objective reality is governed by natural laws; That reality can be discovered by means of systematic observation and experimentation; That Nature has uniformity of laws and most if not all things in nature must have at least a natural cause; <i>etc.</i><p>* <a href="https://en.wikipedia.org/wiki/Naturalism_(philosophy)#Providing_assumptions_required_for_science" rel="nofollow">https://en.wikipedia.org/wiki/Naturalism_(philosophy)#Provid...</a><p>Not all worldviews hold these views / make these assumptions.<p>It would probably be 'too much' to think about them everyday if you're going is 'doing science', but that doesn't mean they don't exist or are not important.
> <i>That Nature has uniformity of laws</i> […]<p>Meta: The scifi author Vernor Vinge played around with this concept with his "Zones of Thought", e.g.:<p>* <a href="https://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep" rel="nofollow">https://en.wikipedia.org/wiki/A_Fire_Upon_the_Deep</a>
There is no science without philosophy. In its modern form, it is based on the dominant philosophy and value system of our time, capitalism. We don't see it as philosophy because it is the default religion, "the way the world is". But it is a construct of the mind.<p>This so-called "science without philosophy" is how the world got to this situation, the wonders and horrors of modern civilization and technology. Much of it serves the dominant power structure - and when it does, it is without morals, ethics, aesthetics, or any deeper philosophy other than extracting the maximum value by exploiting everything it can, at the expense of the world and all living beings.<p>Elsewhere in this thread was the imagery, the Industrial Revolution of Academia and Factories of Knowledge.<p>But what is knowledge without the wisdom to use it for the benefit of all? Otherwise it only serves the interests of power-hungry men with no heart, a blackhole of greed and collective self-destruction. There is a dark philosophy there, based on primitive desires.<p>Science that serves humanity needs to draw on the rich history of the world, its cultures and philosophies.
(Natural) Science is an application/ a part of philosophy.
[dead]
[dead]
There is evidence that humans were smarter thousands of years ago. Fossils found had larger brains than modern humans. Maybe civilization, by making life easier, causes humans to become less intelligent. After AI makes our lives easier, maybe humans will lose their minds.