There's some psychological mechanism by which my brain immediately recognizes AI generated text and just short-circuits to "there is no information here".<p>And when I force myself to read AI-generated text I realize I'm making my brain do creative work to impart meaning to the words. It is exhausting because my brain is literally trying to do a just-in-time rewrite of the text into something valuable.<p>Something is deeply wrong with AI generated output, and I say this as someone who is typically very impressed by AI.
The "something deeply wrong" part about AI, that even most technology enthusiasts evidently do not seem to grasp, is that it is still fundamentally a statistical model — an algorithmic construct — and does not possess any real intelligence or critical thought whatsoever.<p>No matter how much investors and tech companies want you to believe that they are on the verge of super intelligence, nothing I've seen to date can not easily be explained by "correlation engine", including the "novel" math solutions, all of which appear to just be "a composition of solutions humans have developed and documented elsewhere" upon deeper inspection.
Some of it the effect of tells. “It’s not X, it’s Y” is not a bad pattern but it was baked into the instruction following training set just like the other patterns. I catch myself about to use it and use something else because I want to look human. I have, a few times, tried to use AI to write something that I was struggling to find the words and I just didn’t like how it didn’t seem like my voice. If there was just one person doing it would be OK but when it is 100s of blog posts submitted to HN a day it is like wearing a “I’m an NPC” t-shirt.
Someone shared with me this system prompt that at least makes assistant outputs usable<p><pre><code> For information retrieval tasks, I want you to provide links to sources and use exact quotes as much as possible. When using a source, consider if it is primary or secondary information. If secondary sources are found, search again for primary sources. Sources and quotes, if applicable, should be mentioned in the answer first before the rest of the response with links.</code></pre>
I just can't accept that it possesses <i>no</i> intelligence. It is not equivalent to human intelligence, obviously, but how can a system without some semblance of rational thinking solve open math problems? Even composing earlier human work into something novel requires intelligence and understanding on some level.
We couldn't agree on what intelligence means before ChatGPT happened. Now, agreement on the term seems even further away<p>If performing well on an IQ test or performing at a high level on knowledge work is intelligence to you, these models are intelligent. If intelligence requires sentience for you, then ... well, I don't think we really agree what that is either, never mind how to measure it. But LLMs certainly don't have it right now<p>But the consistent trend of the last couple decades (arguably since Turing's time) seems to be that any time a computer reaches our definition of intelligence we decide that that was a flawed definition
I don't think "intelligence" needs to carry all the intrigue and woo of related words like "consciousness" or "creative." If we just use "intelligence" to mean "the ability of a system to solve problems that are new to the system," that pretty much matches the dictionary definition and normal usage of the term. We don't need to touch messy questions like "is there something it's like to be a bat" to conclude that bats exhibit intelligence when they navigate long distances and hunt for food.
It has no semantic depth. The sentences and the paragraphs are a statistically viable derivation of existing human text, but once you try to grasp the whole thing with its temporal and spatial dimensions, you are left with a blurry mess that rots your brain. It's a polished, inoffensive and shallow interpretation as written by an opinionated reputation-seeking user of Quora, circa 2019. Assertive, bold, without typos, clean-cut and bulleted, but without an interesting semantic core.
It's just filled to the brim with relations between things. It's good at searching a very large meaning space and create correlations. What it does is to cover great distances and find related things in that large space which needs a long time and large corpus of knowledge to find the connection.<p>This is not intelligence. It's just a good correlation engine with a very big albeit lossy database of things.
Intelligence is compression, compression requires subtraction, and for some reason LLMs are not good at subtracting. To create a coherent model you kinda have to subtract correlations until only the essential parts are still there.<p>What I don't understand is why LLMs haven't been able to do this yet, if it's the harness or some orchestration layer above the LLM that is needed. Because fundamentally if you can identify correlations then it's just another small step to prioritize and remove lower value or irrelevant correlations.<p>I wonder if what's needed is to introduce subtraction tokens in some sense, and in post-training reward the model on that.
Abstraction is compression, and abstraction is definitely a core component of intelligence.
Intelligence is compression? What do you mean? Intuitively that doesn't seem right.<p>>What I don't understand is why LLMs haven't been able to do this yet<p>LLMs are just trained on what humans have said. Why is it surprising that it's still not possible to reconstruct the intelligence that wrote all that by working backwards? Think of your own work experience. When you look at a piece of code, say, are you always able to discern why the person did what they did, just from the code, with no additional context?
I've heard that expression before, but I don't think it can be presented and stated so matter of factly. Where does that put bzip?
<i>Intelligence is compression</i><p>That’s a controversial statement.
The very fact that it is able to search within a meaning-space demonstrates that it understands semantics, to some extent. Philosophically, that is profound, for something that is just one big matrix multiplication. Drawing connections between things in meaning-space is surely <i>a</i> facet of intelligence.
It’s <i>not</i> intelligence if <i>you</i> are the one who gives the correlations to the model in the pre-training. It’s Word2Vec, applied. Model doesn’t learn anything. You embed these correlations and build it from there. It just searches the space.<p>As my AI professor said in the first lecture: “All AI is advanced search”.
Okay, I guess you're right that its ability to do this is just correlational, which doesn't imply it has any understanding. However, you have to conclude that some tasks which we used to believe required intelligence don't actually require any, which is disconcerting.
No, what I would say is the tasks which are handled in a passable manner by LLMs can be mathematically modeled with some reasonable accuracy.<p>Many things are predicted by models in our planet. From weather to production and material science. Building the model needs intelligence, running the model does not.<p>The person who came up with the formulae for CFD was intelligent. The computer running the model is not. Same for LLMs, chess engines, engine ECUs and financial prediction systems.<p>Again, for the example’s sake; the person who came up with an algorithm is intelligent. The model mixing its training data to emit something similar is not.
This starts to feel like you're defining the word intelligence out of any meaning and out of any way we apply that word.<p>So when LLMs can do all human knowledge work, and do it better than humans, we'll be in the mines listening to you go on about how it's actually just autocomplete or just math, a distinction that apparently means nothing.
> This starts to feel like you're defining the word intelligence out of any meaning and out of any way we apply that word.<p>No.<p>> So when LLMs can do all human knowledge work, and do it better than humans, we'll be in the mines listening to you go on about how it's actually just autocomplete or just math, a distinction that apparently means nothing.<p>With a big "if" attached to it. People were saying "computers will program themselves in the near future" for, checks notes, 24 years now, as far as I'm aware.<p>We're constantly building new knowledge and understanding things better than olden days. These models just compress our knowledge and light the blind corners we can't see well. I don't say they are useless, but I say that these things are overhyped.<p>All they can do is regurgitate human knowledge packed into them and highlight some long-distance correlations between items, which is useful in itself, but it can't jump to somewhere where it's not present its training data, but that's something humans and only humans can do.
I get what you're saying. The thing itself is just math. I'll just say it depends on how you define intelligence. If at some point we're be able to simulate a human brain with 100% accuracy, I would say that it is intelligent, it sounds like you would not. (I don't mean to imply consciousness or personhood or anything else by "intelligent".)
For me intelligence is a fairly clean-cut concept, and is somewhat inseparable from consciousness itself.<p>Briefly, any intelligent creature has internal stochastic processes like sensory inputs and feelings to a certain degree. These stochastic inputs and the creature's own actions change the creature in subtle or profound ways. An LLM has no such processes. You push inputs to the same static model, sans temperature which is just a randomness slider.<p>Considering the model even doesn't see the words and work on matrices of numbers is even more telling. One needs to add "tools" and other "experts" to overcome the shortcomings caused by this modus operandi.<p>I can call the algorithm/model smart as in a smartwatch. It can mimic certain things well while having none of the underlying foundation beneath it, or redirect some of the things to correct tools to get deterministic and accurate results if it can't evaluate the query inside its own network in a sane manner.<p>Coming to your question, "simulating a brain" in a static manner would not make that simulation intelligent, but if you can "wire" it completely and let it evolve by itself, now we're entering a territory I have not spent enough time for thinking it through.<p>Oh, as I said "I don't know", an LLM doesn't know what it doesn't know, and can't self correct itself which are required capabilities for understanding something. It just generates something statistically viable via its network.
I suspect like most you don't appreciate how terrifying statistical relationships become when you have truly vast data sets to train on... and also that we as humans aren't as shockingly unique as we think (compared to other humans I mean).
I don't think statistically driven prediction implies reasoning or intelligence.
watch this and see if you think it has intelligence by the end<p><a href="https://www.youtube.com/watch?v=kYUicaho5k8" rel="nofollow">https://www.youtube.com/watch?v=kYUicaho5k8</a>
I wonder if you went back before we had any idea how the brain worked and talked to the smartest people about how neurons work (without giving away that it's a human brain) then asked them all "would such a system be intelligent?" how many would say yes.<p>The main problem I have with people stating it's not intelligent <i>or</i> conscious is I don't think we even have a good definition of either word that satisfies everyone. Philosophers have been trying (and failing) to elegantly define these things forever and everyone out here proclaiming they've got the definitive answer and this specific thing they're seeing doesn't fit under it.
This looks interesting, but would you mind saying a sentence or two about why before I commit to an hour-long video? It looks like it shows how they work internally, which is sort of a non sequitur. Brains also work mechanistically. I'm claiming that any system which is able to do what AIs do must necessarily have some sort of intelligence.
fair reply to an hour video, Scott is just so good to hear his talk is better than I can explain it...<p>go to 24 minutes and 07 seconds.<p>it's statistically determining what the next word should be based on all the text it's been trained on. It's not intelligence and he shows what probability it puts on each word that it chooses, but also shows a lot of the other words it was thinking of using. In a later part he shows how it uses words that are not the highest probability (and you question why did it go this route, it's not more correct), but the user never sees this, they see what they think is the correct answer always...<p>he also shows how context you feed it has a lot to do with what it returns... to the point he can get it to return the capital of France is Marseille, just by typing Marseille a bunch of times before the question. Human intelligence doesn't get confused like that.<p>And it's not a "hallucination", it's just probability of the next token prediction based on the information it's been trained on and fed, it's not intelligence.
> Human intelligence doesn't get confused like that.<p>We do; this is the premise of many children's riddle-games, like the one that goes:<p>"What is white and rhymes with silk? > Milk. What is cheese made from? > Milk. > What do cows drink?"<p>At which point the riddle-guesser is very likely to answer "milk" even though the correct answer is "water".
Isn't this a case of missing the trees for the forest though? The human brain is not an LLM, and an LLM is not intelligent in the same way as a human brain.<p>However, an LLM is a prediction machine, prediction IS at the very least one (or the most fundamental) element of intelligence. The brain most surely contains at least some kind of simulacrum of a prediction machine. How that prediction machine is used or wrapped is another matter.<p>If I said to you: "Blue blue blue, the color of my car is red", would you have absolute confidence in your prediction that my car is red? Or would the way I phrased that sentence make you slightly uncertain, and wonder if there's some miscommunication going on here?
I also like this: <a href="https://laurentiugabriel.github.io/token-town/" rel="nofollow">https://laurentiugabriel.github.io/token-town/</a><p>It shows internals of an LLM nicely, simplified manner.
LLMs are pattern prediction systems with a large training data set. It is not surprising that they can predict patterns, particularly for a well structured field like mathematics that is also amenable to automated proof checking to help steer it.
Are we sure there is some objective, technical definition of what is intelligence and what is not?<p>Isn't it rather a subjective philosophical concept? What if human intelligence is also a statistical model, trained by evolution to make decisions that lead to offspring?<p>The one major difference I see between AI and people is the ability to learn and memorize. All memory/learning solutions that current AI architectures offer just feel like workarounds and simply don't work anywhere near as a person learning something new and remembering it.
Thank you for helping me keep my sanity.
I mean this in the kindest way possible, but you are wrong that the math solutions are that easily dismissed. And there are many more than are publicized. A specific math problem I wanted solved for 3 years did not get solved by any model until fable and, and I tried it on every model and know the literature surrounding it well.
When I read AI-generated prose that is aimed at the general public, I have the exact same feeling.<p>But when I ask Codex a technical question about coding, I don't get it at all. Codex replies to me in a very direct, technical manner, similar to the way I speak.<p>When I ask ChatGPT to be concise and technical, I get the same effect.<p>I think it's because prose aimed at the general public has to be very attention-baity --like the textual equivalent of a Mr. Beast video--, not because AI is incapable of writing like a human.
I use Claude and I find that it speaks in a very obfuscated manner when explaining things. It seems to make up jargon as it goes on top of spending a lot of tokens dancing around a point. I often find myself having to ask it to rephrase things, or speak directly about mechanism or consequence, in order to understand the point.
Using Claude for any kind of technical writing makes me feel like it was trained on snarky Huffington Post articles written by a 23 year old mixed media arts graduate and then was told to intentionally obfuscate the most important elements of any text by extensively rambling about what was not done and for what reason.
GPT is less bad for this, which is why I've mostly shifted to using it.
Completely agree. AI is very impressive in many ways but there is something deeply wrong that is hard to put into words. The output is probable but never true, if that makes sense.<p>I think this is also the mechanism behind why AI generated videos and images are so captivating at first. I remember when Midjourney first launched and it was hours and hours of a brain-melting "Wooooooow". But once you get used to it and start to identify the patterns the brain quickly labels most AI-generated content as blank space.<p>If the image or text wasn't created by a human, then there was no intent behind the content, there is no message or novel information conveyed, and it reads as noise.
Yeah AI generated content hints that there is a whole world behind it, the way that an image pre-AI was a clue that there was a rich 3D space that corresponded to the image.<p>It seems our brains are adapting to that and recognizing "actually the signal behind this message is quite sparse" even when presented with rich imagery.
You are re-compressing information that is in-effect meaningless because it's all decompression artifacts.<p>The AI had a nugget of data and decompressed that into a flood of text.<p>The exhausting thing is that we're then trying to re-compress that or derive the original intent and meaning from noisy decompression.<p>It's like un-zipping a zip file into a probability space of what could have been in the zip -- and then having to find the actual files worth reading.
Yeah, I have the same problem. There's a good quote example of this:<p>> <i>There’s a growing scissor between people who are happy to read AI and those who violently bounce off from it.</i><p>> <i>People adapt in different ways — and some people absolutely cannot look at it. That cognitive split creates a surprisingly powerful opportunity: you can write something that, technically, sits right there on the page, yet an entire sub-population will be incapable of staying with it long enough to actually read it. You can hide entire sub-structures in plain sight. It’s not avoidance — it’s adaptive obfuscation.</i><p>> <i>The paragraph before this one was the only thing generated in this essay and if you just skipped over it I highly recommend reading and really understanding what it’s saying.</i><p>It's quite effective. I think this kind of text functions like the chumboxes you see at the bottom. Taboola and so on. Just mental ad-block takes over.
I kind of wonder if our ability to skim has been stymied.<p><i>blah blah blah<p>- blah blah nugget blah blah<p>- blah blah blah wrong blah blah nonsense<p>- blah blah blah obvious blah blah<p>- blah blah blah off-base<p>blah blah blah</i><p>It is that we HAVE to skim because the text is so cheap, and it wears us out.
> just short-circuits to "there is no information here"<p>That is my experience with the way the models write by default, often even when instructed not to do that. With enough effort you can get even them to slightly unslop the writing so it doesn't read like some LinkedIn/Buzzfeed brainrot, but the problem is that it's not trivial to do and most people won't do it, so the default is indeed horrible.
For some research I looked up some very old Reddit threads a couple of days ago.<p>And, Oh my god, you can actually see how this style of writing influenced AI writing today, I constantly had to remind myself: "this was posted before ChatGPT released".<p>The reddit influence is especially true for "storytelling" writing.
Do you have much exposure to pre-AI corporate memos, mission statements, marketing plans, or white papers? Because they were mostly written in that style. Full of buzzwords, cliche similes, platitudes, jargon and stock phrases.
The thing is, people writing them had a style. Every company has its own style, or feeling for these kinds of texts. Also for the initiated, these buzzword-filled blocks of text provided some between the lines information; sometimes big, sometimes small.<p>AI generated text doesn't have this. Every model has its bias towards a certain style, an overly agreeable tone, some exaggeration to make the user important and smart, but the text has none of the information crumb these pre-AI texts contained.<p>Even when you use tools like Grammarly and allow it to "Impact-MAXX" your text, the resulting text is a bland wall of letters, carrying none of your voice or style, less elegant than a corporate text and emptier than space.<p>It's beyond bland. It's tasteless.
AI tries to make the prose "interesting". I don't want to read interesting prose. I want to read interesting ideas.
The prose is not only interesting, also glorious. Gloriously grandiose, monumentally empty at the same time.<p>It's like a hook of a pop song. Interesting to listen, but entirely empty.
People should notice that it is constantly inventing plausible jargon, some of which may or may not have been used in some specific context.
Exactly, AI-generated text reads so smoothly, that the same short-circuit shifts my attention away from deep focus and onto scanning of the text, looking ahead to get the gist of it. Forcing myself to read the text fully feels almost painful. It's like reading a terms-of-service or any boilerplate document.
we are working on it, the thousands of gig workers tuning frontier models
The junior engineers at my job have a terrible problem of writing AI "proposals" to problems. The proposals are all extremely detailed and verbose to a thought-terminating extent. It takes a lot of effort and self-control to parse out the actual "ideas".<p>I think of the Dwight Eisenhower quote: "Plans are useless. Planning is indispensable."<p>The process of thinking through a system and communicating your design to other humans is a core part of software engineering. You want to build the right abstractions and communicate the right level of detail. Delegating all that thought to an LLM means your proposal isn't clear to the target audience, and it's not helping the author to understand the problem.
It's like if on any website you went to you saw a lot of posts written by the same guy over and over again. Even if he used different names, you'd start to recognize him eventually because of his style. Seeing as he doesn't say a lot of valuable stuff, you'd also learn to skip whatever he says.<p>I do worry that it's just survivorship bias and we're also consuming higher-quality AI output that's indistinguishable from human writing, but we focus on the raw, unedited, low-effort AI slop and think that we're good at recognizing AI text. Even if we really are at the moment, it might not be long until AI companies figure it out. I'm not sure why they haven't yet, given how many books they've burned for this already. Maybe it's just more efficient for the model to stick to a single way of writing, I don't know.
But when that point comes, we'll be back to the usual way of reading and interpreting text because there would be no way to tell what produced it.
Are you sure you are not doing the same thing with other texts?<p>I started to skim a lot more text due to me having read a lot. Like in news article, i stoped reading the first paragraph because it repeats just what it was already written in the short subtext. Then there is the second paragarph which is used to have some historical view or whatever it is.
I am very good at skimming over text. Human-written text I can usually glean the gist from very quickly, and get to choose how much I want to glean from it: The closer I look, the more I find.<p>With AI-written text, it's almost the opposite: the closer I look, the less I find. It is so information-sparse.
I started skimming reports im required to produce quarterly snd annually. I designed them to provide novel information at start and end so I can update them easily.<p>The problem I encounter is both my memory is degrading, but since these reports are largely duplicative, knowing which version im remembering is technically impossible since theres so much overlap. The overlap is tge same problem as context poisoning.<p>Id been doing this for over a decade when i started working with a new engineer with a few years of experience and younger. I tried to explain how i set these docs up so they can be skimmed and you can update the specific facts needed. They exclaimed they would never skim and rewrite it all. There was zero way to explain how exhausting that will become as they age.<p>So theres certain a tension about how people and AI will generate documents.
yes, but now I’m also experiencing that for human-written text
> There's some psychological mechanism by which my brain immediately recognizes AI generated text and just short-circuits to "there is no information here".<p>I think you need to self-correct here, because otherwise you'll be ineffective in an information setting, where I expect AI-generated resources will not only be the norm, they will absolutely swamp the environment.
> my brain immediately recognizes AI generated text<p>I bet it does. I bet it also recognizes some human text as AI text, and doesn't detect other AI text.
I am not claiming to have a perfect AI classifier. That is an unnecessary claim that distracts from the broader point.
Show me AI text that manages to climb out of the uncanny valley, and I'll show you AI text that's been edited by a human.
Yep. It's like it's painful to read for me. It's because the next-token predictor is just mashing (mostly) grammatically-correct and plausible sentences together, without any real intention or meaning. So everything sounds plausible, but almost entirely void of meaning.
> Something is deeply wrong with AI generated output<p>It works just fine for me.
Once you see past the illusion I think there’s no going back. AI writing style is just dogshit. This hype wave is based on the belief that we’re inching closer to AGI but seems to me we just increasingly struggle to define intelligence. LLMs seem smart because they can pump out thousands of LOC quickly, and enthral you with fancy words and bullet points. I don’t fall for the intelligence illusion anymore.
I've got a 3 step instruction to compress Ai text into useful info.<p>1. Ask it to write according to the Google Developer Documentation guidelines. Gets rid of fluff, less emotional statements, no it's not x it's why.<p>2. Tell it you have extreme ADHD and need everything condensed as much as possible. You can always ask for expansion on an answer later.<p>3. Bullet points whenever possible.