Stop for one second and ask yourself a simple question. Where do your words come from?
When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? I think we all do. The word is never the start. The word is just the skin. The idea, the consciousness, is the thing sitting under it.
pigeonwarz · 2026-06-30 14:21:29 UTC
My favourite way of thinking about this is by going back to the End Poem at the end of Minecraft. The fact that words are an "interface", "very flexible" and "less terrifying than staring at the reality beyond the screen".
Maybe that reality is the true understanding of the neurobiology that defines our thoughts. To be reminded that the magic of experience can be reduced to neuronic hallucination is, frankly, horrifying. Maybe we dislike LLMs because we see their vectors and numbers as crude reminders of our own intellectual banality.
Or maybe I'm just feeling hungry and should really go eat dinner.
HelloUsername · 2026-06-30 14:23:26 UTC
There's plenty of people who speak before they think
yieldcrv · 2026-06-30 14:24:27 UTC
We’re stochastic parrots most of the time, with a varying ability to reason but opting for pattern conformity usually
danielmarkbruce · 2026-06-30 14:35:26 UTC
This is getting voted down because people just hate the idea that it's true.
bluefirebrand · 2026-06-30 14:39:21 UTC
No it's down voted because we didn't stochastic parrot ourselves into landing on the moon and building the internet and modern highways and skyscrapers
Dismissing humanity that was is absurd
hallway_monitor · 2026-06-30 14:38:07 UTC
What is reasoning except applying another pattern on top of existing thought? Personally it seems like I and everyone else is simply pattern matching, albeit at a higher level than current LLMs. There is no difference in the process as far as I can tell, just different inputs.
enaaem · 2026-06-30 15:19:27 UTC
In some meditation practices, the task is to observe the stochastic parrot in your head. You will soon realise that you cannot control this parrot. If you cannot control your thoughts, could it be that you are not your thoughts? And if you are not your thoughts--the word stream in your head--what are you then?
next_xibalba · 2026-06-30 14:26:52 UTC
> Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it
There are enough neuroscience experiments demonstrating how we create post hoc explanations of our actions that I wouldn't trust this intuition prima facie.
bluefirebrand · 2026-06-30 14:37:44 UTC
The fact that we're even capable of doing a study like that and introspecting is probably a good indication of our cognitive abilities as humans
Consider someone designing a skyscraper. They're thinking about material strengths and such while they draft it, they aren't just rationalizing their choice later.
jcims · 2026-06-30 14:49:50 UTC
I've noticed that more and more of my mental attention while speaking is observing the words that come out of my mouth rather than 'generating' them. I split time between rough pathfinding in my mind to fit a conceptual framework then kind of sit and listen as it comes out to think about how they sound. If I had to guess this happens on a 1-2 sentence chunk at a time.
mattas · 2026-06-30 14:29:34 UTC
Word is just the skin is the load-bearing idea. The real unlock, if you will.
vitamark · 2026-06-30 14:36:05 UTC
"For an LLM, it is exactly the opposite".
Is it really? LLMs don't have words inside, for the most part they operate by applying transformations to a vector that does not contain any words at all. Words that come out of an LLM are just what sampler gives us by looking at the vector that is the result of those transformations.
Does this vector contain a world model? Some form of thoughts or reasoning to arrive at the result? That's an open question really.
Sure, rerunning that whole process for each token might not be the best solution, although that's an open question too. But saying that LLMs operate on words first is too big of an oversimplification
rcxdude · 2026-06-30 14:39:15 UTC
>Does this vector contain a world model? Some form of thoughts or reasoning to arrive at the result? That's an open question really.
It is, but it has been explored in various forms. Anthropic have some interesting papers where they can map out concepts to some extent as they exist inside the vectors of the LLM, and they can see e.g. that vectors relating to some concept can appear there some iterations before the LLM emits the tokens that spell it out. (though also, quite frankly, this shouldn't be all that surprising. If they were just picking the next token with no representation of what might follow it the result would be more like the 'one-word-at-a-time' party game than coherent text. They used poetry with a rhyming scheme as an example because it's something where you need to have some idea of what might rhyme with the line before and aim at it at the start of the next line in order to have any degree of success).
epihelix · 2026-06-30 15:23:26 UTC
I don't know about you, but I think with words all the time when reasoning through complex ideas. Where do you think the phrase "thinking out loud" comes from?
t0lo · 2026-06-30 14:31:10 UTC
Meditative practices support this. Anyone who's spent even a moderate amount of time with meditation knows we are not our thoughts.
If you don't believe me, learn how to meditate, meditate for literally just 10 minutes, and come back to me.
fedeb95 · 2026-06-30 14:33:22 UTC
this doesn't imply that words are a byproduct of consciousness.
Diogenesian · 2026-06-30 14:31:28 UTC
St Augustine had it right 1500 years ago: humans learn words opportunistically according to their desires and problems and existing deep understanding of the world, and (critically) humans really can't learn language without being natively fluent in great ape facial expressions, gestures, grunts, etc.
Did I not, then, as I grew out of infancy, come next to boyhood, or rather did it not come to me and succeed my infancy? My infancy did not go away (for where would it go?). It was simply no longer present; and I was no longer an infant who could not speak, but now a chattering boy. I remember this, and I have since observed how I learned to speak. My elders did not teach me words by rote, as they taught me my letters afterward. But I myself, when I was unable to communicate all I wished to say to whomever I wished by means of whimperings and grunts and various gestures of my limbs (which I used to reinforce my demands), I myself repeated the sounds already stored in my memory by the mind which thou, O my God, hadst given me. When they called some thing by name and pointed it out while they spoke, I saw it and realized that the thing they wished to indicate was called by the name they then uttered. And what they meant was made plain by the gestures of their bodies, by a kind of natural language, common to all nations, which expresses itself through changes of countenance, glances of the eye, gestures and intonations which indicate a disposition and attitude--either to seek or to possess, to reject or to avoid. So it was that by frequently hearing words, in different phrases, I gradually identified the objects which the words stood for and, having formed my mouth to repeat these signs, I was thereby able to express my will. Thus I exchanged with those about me the verbal signs by which we express our wishes and advanced deeper into the stormy fellowship of human life, depending all the while upon the authority of my parents and the behest of my elders.
It is not all obvious that consciousness precedes language. Hellen Keller provides a vivid description of her pre-verbal mental state as something less than reflective conscious.
> Before my teacher came to me, I did not know that I am. I lived in a world that was a no-world. I cannot hope to describe adequately that unconscious, yet conscious time of nothingness. I did not know that I knew aught, or that I lived or acted or desired. I had neither will nor intellect.
I think this is different if you have vision though. I struggle to understand how someone can look at intelligent animals that don’t have language and not intuitively assume that they’re conscious. Yes this is a tricky problem without an answer but IMO the elements of consciousness can exist without language if visual abstractions are present.
sigbottle · 2026-06-30 14:56:27 UTC
You could redefine language to include mental abstractions and maybe it wouldn't be so weird.
Language is a shared medium. Maybe an intelligence is capable of building abstractions in their head without the need for say, a symbol system to hopefully communicate with species similar to itself.
You may argue that the agent needs persistent memory. Well, so what? If the world has stable regularities from the perspective of the agent - if the sun rises every day, and language is associating certain visuals with certain thoughts (it's not when viewed in its original social definition - but conduct the thought experiment with sociality removed), who's to say that you can't form a kind of language around that?
Of course, this is all armchair, and I'm not trying to separate out what "real language" is or whatever, just thought experiments.
But the original poster has a good point though, that's been proven in papers, that the tokens an LLM spits out, themselves, attract them to certain distributions, and at least a decent amount of the tokens in CoT are there solely to bring itself to the right distribution.
devmor · 2026-06-30 14:57:15 UTC
Animals don’t have language the same way we do, but many of them do have some kind of semi-structured communication. Cats for example, when domesticated, develop an entirely parallel method of communication specifically to interact with humans - one that is neotenous in genesis, but nonetheless absent from their feral counterparts.
brookst · 2026-06-30 15:33:09 UTC
While true, the connection between communication and consciousness is thin. Plenty of plants communicate with each other, but it’s seen as signaling rather than consciousness.
I believe that conscious is a gradient not a binary, and I believe that cats have some level of consciousness, but I’m not sure communication is the thing to hang that on.
gf000 · 2026-06-30 15:37:24 UTC
I think one should bring into Chomsky's Universal Grammar the discussion, or at least some version of it.
If for no other reason, I believe it's quite reasonable to believe that cognition must be Turing-complete (okay, linear bounded automaton), and there must be some kind of support for recursive structures, no matter what the "primitives" of such consciousness is (either visual or linguistic).
I believe in the case of animal intelligence, they (we) have a pretty good hard-coded 3d/4d world model that ought to encode some level of recursion intrinsically (in thought, we very very often use spatial and temporal words), e.g. understanding the motion of another animal, e.g. a rodent running through a tree trunk requires imagination and future prediction already over an abstract agent. Humans just probably have more "stacks" available, and can offload them into either limited "brain memory", or very importantly drawings, text, etc, for unbounded memory.
But I'm just thinking out loud, I'm absolutely no expert on any of these topics.
kibwen · 2026-06-30 14:40:35 UTC
Helen Keller was blind and deaf from infancy. What her teacher gave her was a way of systematizing what little sensory stimulation she was capable of receiving. "Language", in the topical sense of "the textual encoding of language that LLMs ingest and emit", is certainly not required for consciousness. Rather, what you need is a way to identify and predict patterns borne out of repeated observations, which is complicated by the lack of ability to observe.
abound · 2026-06-30 14:44:19 UTC
Similarly, I'm reading "Supernormal Stimuli" now, and there's a whole section on infants raised by various animals (chimps, wolves, chickens, leopards, etc, etc). When they are found and brought back to human societies, basically none of them develop full language and integrate fully.
It really does seem like language is a key part of how we develop our human intelligence.
kibwen · 2026-06-30 14:49:33 UTC
That humans who have grown out of their optimal brain plasticity and hence never acquire the capacity for ordinary human language are not somehow "conscious" is not supported. Rather, it means that, by its very definition, we do not have the means to communicate with them in order to even ask the sorts of questions necessary to determine such a thing. The premise is unfalsifiable.
gf000 · 2026-06-30 15:40:02 UTC
Well, we can surely communicate without language. I certainly communicate via body language with other humans, my dog, etc.
palmotea · 2026-06-30 15:03:02 UTC
> Similarly, I'm reading "Supernormal Stimuli" now, and there's a whole section on infants raised by various animals (chimps, wolves, chickens, leopards, etc, etc). When they are found and brought back to human societies, basically none of them develop full language and integrate fully.
Infants raised by chickens? Chickens!? That doesn't sound real.
So not "raised by chickens," but rather abused and isolated in the presence of chickens. That's more plausible.
vitamark · 2026-06-30 14:44:27 UTC
Well, language is not really required for thinking and conscious experiences, a huge chunk of people don't even have an internal monologue, and some (that'd be me) don't have any "byproducts" of thought process.
Though there are recent experiments showing that unconscious brain retains language processing abilities [1], so the two might be as well independent systems
IMO it's far more likely that people don't agree on what it means to "have an internal monologue" than that people actually bifurcate that dramatically on cognition. Surely there are some people with an actual difference from baseline, but I don't think it's the near-equal split that seems to appear when you just ask people about internal monologue
This is in contrast with aphantasia (complete lack of mental imagery) which can be correlated with outwardly detectable responses. E.g. people who can imagine mental images will dilate pupils when imagining a bright light with eyes closed, while people with aphantasia will not. Not sure there's an analog for internal monologue but if anyone.
devmor · 2026-06-30 14:54:07 UTC
This is in the same vein to what I wanted to say here as well - I find it very likely that people with no internal monologue are executing similar mental processes, just without them being so “front and center” in the mind.
vitamark · 2026-06-30 14:57:29 UTC
Well, I can't say for the general population, but for me there's just nothing unless I actively try to think of words, sentences and other things. One moment you don't have an idea and the next you do, like a state switch.
I really feel like humans argue too much about how computer programs think and at the same time too little about how humans think. The topic is criminally underresearched and the existing research is all over the place.
Fraterkes · 2026-06-30 15:08:59 UTC
Do you have a source for the pupil dilation thing? Sounds fascinating!
lightedman · 2026-06-30 15:24:51 UTC
I scored a 68 on the VVIQ and I can watch my pupils dilate in response to me 'seeing' something in my mind. It doesn't have to be a bright light in my mind's eye, I can imagine something that would trigger a fight or flight response and make the same thing happen.
estearum · 2026-06-30 15:32:28 UTC
Yeah this is another test they use: imagine something frightening and "normal people" will get clammy hands and elevated heart rates while aphantasics will not
amalcon · 2026-06-30 15:11:15 UTC
I don't think a bifurcation of cognition is necessary to explain this. You get both people who claim to always think with words (to the point where they don't understand how it's possible to think without one), and people who claim never to outside of linguistic tasks like writing (to the point where they need "internal monologue" explained to them). In my experience, if you get down to it -- most people will have experienced cases of the thought preceding the word (you can skip the word, or not, but either way it demonstrates a thought without a word) and of the word preceding the thought (people who claim not to have an internal monologue will usually still do this sometimes when writing).
It's just that subvocalization is one powerful technique for organizing thoughts, but not the only one. If you had a swimmer who only ever learned the crawl stroke (freestyle), they might wonder how someone can swim without it. The difference, of course, is that the breaststroke is a demonstrable physical act whereas cognitive techniques are not. The monologuers are just more practiced at subvocalization-based reasoning, and the non-monologuers are more practiced at other types.
It's not that hard to exert limited control over which technique you're using -- and I think there are multiple nonverbal techniques. I'd encourage people to explore the space a bit. Subvocalization-first approaches are very good for linear progressions, whereas subvocalization-last (or never) approaches are usually better for big-picture reasoning.
estearum · 2026-06-30 15:34:13 UTC
I still think it's far more likely that people aren't describing the same thing. People are very, very bad at interpreting and experiencing their mind's inner workings. Then getting them to describe and compare those inner workings is, IMO, largely a fool's errand. I'll wait until someone develops an external validation of this to believe it.
I say this as someone who pays a lot of attention to my mind's inner workings (meditation etc, near-complete aphantasic) and the more I pay attention to the inner workings, the less I know the answer to whether or not I have an inner monologue.
justbees · 2026-06-30 15:34:06 UTC
My mom has aphantasia and we were both shocked to discover it even exists.
She saw something about it online and then started asking everyone in the family if they could picture an apple in their mind. We all said - yes of course, what are you talking about?
And she said, "When you say the word apple, I think of the concept of an apple. It's juicy. It's crisp. It's red, etc, but I don't SEE an apple. I just know what it is."
She had no concept that other people literally saw the apple and I had no concept that people couldn't see the apple.
I asked her what she thought the phrase "in your minds eye" meant for her entire life and she said she thought it just meant exactly what she was describing and not literally seeing anything...
estearum · 2026-06-30 15:38:15 UTC
Yeah this was my experience too as a near-complete aphantasic. Doing the apple test around the dinner table was enlightening, and it also made a whole lot of other things make sense through my life as well: I have virtually no concrete memories from a first-person point of view (I remember the fact that events happened, I do not remember subjectively being there)
It's unfortunately very insulting when someone says, "hey remember when we did that very important thing together?" and my answer has always been "uhh... I remember that we did it?"
high_byte · 2026-06-30 14:48:10 UTC
this was absolutely fascinating to read
pigeonwarz · 2026-06-30 14:48:31 UTC
It is likely that language forms a sort of quanta of reasoning. It is far easier to posit a verbal hypothesis pertaining to verbal subjects and test it than to try and do it nonverbally.
Maths is probably a great example of this. Try and describe Pythagoras' Theorem without using maths notation or words. Difficult, right? Reasoning is a House of Cards, and without understanding what the card is that exercise becomes significantly more difficult.
catlifeonmars · 2026-06-30 14:59:41 UTC
a^2 + b^2 = c^2 is perhaps a poor example, since there are several visual proofs that don’t rely on language. I don’t think this negates your overall point fwiw.
vharuck · 2026-06-30 15:34:47 UTC
Math is a terrible example. Euclid's "Elements" was a core book in Western math education for millennia, and it uses drawn geometric proofs. The proofs are then described in Greek, but the rigorous definitions are the pictures. A lot of mathematical fields can be reasoned about and explored by imagining shapes and objects. The end results today are written in notation because this is a standard language that helps clarify ideas.
bunderbunder · 2026-06-30 15:07:37 UTC
There are real questions about the veracity of some of these Helen Keller quotes, and reason to believe that a lot of it was really coming from Anne Sullivan.
There are really three main observations that give pause. First, after she started working with Sullivan, Keller apparently developed a writing style and nuanced opinions reminiscent of a college-educated person virtually overnight. That’s pretty surprising for any child, let alone one who had only recently acquired language. Second, all of that disappeared overnight when Sullivan died - Keller’s writing style changed drastically, becoming much simpler, and all that eloquent insight also disappeared along with Sullivan. Finally, we’ve never since seen another Helen Keller. On the contrary, later observation of deaf-blind people who don’t acquire language until later in life is part of what inspired a variant of the critical period hypothesis that posits that people who don’t learn their first language (doesn’t matter whether it’s spoken or signed) in early childhood are never able to acquire any language to a particularly high level of proficiency.
That said, even if that Keller quote is now considered dubious, it may not be entirely off base. Supposedly people who don’t acquire language in early childhood also tend to show less capacity for abstract reasoning tasks. Which is still a far cry from the “no-consciousness” that Keller-maybe-Sullivan describes, but does still suggest that language and reasoning are mutually supportive.
neptvn · 2026-06-30 15:38:16 UTC
Sure, but that's a non-sequitur. Sight and hearing are the most important human senses that practically inform consciousness in the sense of where we live, and what we can do. The other 3 (touch, smell and taste) are at best secondary and cannot provide the groundedness in our world experience - you are the easiest prey in the animal kingdom. Language is primarily a tool for communication, not thought [0], and "thought" is a slippery concept when you're lacking a guidance that informs you of the world if you can't see or hear it.
That words are a byproduct of consciousness is a bold claim and requires some justification. I'm not implying it's backwards also for humans, but that the statement might not be true, and something else could be. Also, defining consciousness is hard.
For instance, you can dream words.
cwmoore · 2026-06-30 14:33:53 UTC
Eh? What else could communication be there for?
novacrazy · 2026-06-30 14:37:23 UTC
Survival.
t43562 · 2026-06-30 14:45:18 UTC
Have you ever had impressions about something complicated, e.g. someone's personality which you cannot put into words?
I'm thinking of my kid - her personality when she was tiny. I could see what it was but not explain it.
Hence I think ideas are unquestionably first and words are the inadequate and inaccurate description of thoughts that exist without them.
fedeb95 · 2026-06-30 14:46:41 UTC
That may be true without implying that words are a always a byproduct of consciousness. You can dream words.
IAmGraydon · 2026-06-30 15:14:55 UTC
I agree, and I think that's a good example. Sometimes we can hold a concept in our minds of something and cannot find the words to attach to it. The word we eventually give that concept when it's human, as in your example, is a name. This is because the concept of a personality is too complex to describe with other, preexisting words.
Still, even though I think the author is right about consciousness/concepts preceding language, I think he's wrong about the other part. The statistical models that underlie AI very much remind me of what we call a human concept. It's a statistical model of that thing, person, or idea. However, we humans seem to have this ability to create new concepts that are wildly different than our training data (what we have experienced in life), and we call this creativity. It's missing from AI for some reason. That is the part that makes me question if AI will ever match humans.
epihelix · 2026-06-30 15:17:18 UTC
Have you ever reasoned through a complex coding or maths problem? Tell me, how did you do that without words?
You (and TFA) are making a false dichotomy here. Yes, of course we think in images. But we also have an inner monologue that is critically important for much of our higher-level thought. How do you even write a HN comment without thinking through it in words?
DevarshRanpara · 2026-06-30 15:07:22 UTC
how animals perceive world?
They don't have language, but they are conscious for sure.
jacobr1 · 2026-06-30 14:38:42 UTC
LLMs as clearly doing more than just modeling words. In order to predict word placement, they need to build some kind of model, their latent space, of the types of things they are able to predict. Not really full world models yet - but they have decent "blog space" or "github project" models. And you can see this with multimodality, or non-text modality modals such as images and audio. They map from their latent spaces to the outputs. The fact that multi-modal systems can share the interior layers shows some kind of internal representation is created.
adamzwasserman · 2026-06-30 14:47:25 UTC
Or.. the LLMs are just "telescope" sufficiently powerful to detect the aligned internal representation that already exists.
A more powerful telescope does not "create" new galaxies.
jvanderbot · 2026-06-30 15:27:07 UTC
Love the analogy.
I a lot of people had their mind blown by vector space representation of words. The idea that Female + King averaged out around "Queen". Words might foundational to concepts or just really-well-designed ways to transmit them.
sometimelurker · 2026-06-30 15:29:20 UTC
You have the word 'bot' in your name and judging by your comment history love using em-dashes. Are you a machine or not?
jvanderbot · 2026-06-30 15:30:21 UTC
I doubt I used emdashes, but I do us "-". I dont have emdash unicode bound to my keyboard yet. (edit: Hyphenation is not emdash, of course)
you can view my profile details freely and decide for yourself, and a quick look might even explain the "bot" suffix.
sometimelurker · 2026-06-30 15:34:25 UTC
Oh I figured there's a real human being named Josh Vander and youre his openclaw or something. why do you have the `bot` suffix?
kibwen · 2026-06-30 14:54:58 UTC
Except that the Chinese Room shows that the existence of a mapping from input to output, however emergently it might have been devised, is not alone sufficient to demonstrate understanding.
PaulDavisThe1st · 2026-06-30 15:02:29 UTC
It does so only in the claims of its creator. Plenty of other people have pointed out fallacies in the claim. My favorite is the Dennett/Hofstadter observation that while the man in the room may not understand chinese, the room/system certainly does.
Marha01 · 2026-06-30 15:06:35 UTC
The Chinese Room as a whole understands.
dinfinity · 2026-06-30 15:38:26 UTC
The man in the room is comparable to a human hand, and the magic rulebook to a human brain.
This in the sense that we can easily retain human "understanding" by stripping away almost all parts of the human body or replacing them with fairly trivially made replacements, except for the brain.
In the Chinese Room the equivalent is the magic rulebook: We have no idea how to construct/replace it, yet people somehow handwave that away whilst simultaneously confidently asserting it does not understand anything.
swid · 2026-06-30 15:07:06 UTC
If our own brains were insufficient to demonstrate the fallacy of the Chinese Room; the particles, molecules, cells, synapses, and so forth of our bodies cannot be believed to have understanding, but we recognize it in the sum of the part anyway....
... we now have LLMs to even more pointedly show the deficiencies of the argument.
SpicyLemonZest · 2026-06-30 15:13:43 UTC
"Understanding" is your word, it doesn't appear in the source article nor the comment you're responding to. The Chinese Room argument does not attempt to show that a mapping from input to output can't implement a latent space, or that it can't implement complex models of what the language is describing. If a person can express those things explicitly in the output, or if you have to do them in order to correctly respond how a person would, then the room by definition has those capabilities.
(What's the point of the argument if it doesn't tell us anything about the capabilities or internals of an AI? I'm not sure.)
jacobr1 · 2026-06-30 15:31:46 UTC
Right - the point I was refuting is that LLMs are "just statistical models of words." More is going on. Does that imply "understanding?" I don't know, I'm not sure we have a good enough definition to say. But it does mean that the models are more complex that say, markov chain graphs with corpus frequencies. It seems we are encoding data in the latent space with much higher complexity than "just words." There is higher order semantic information being captured - probably not the same has human "thoughts" - but again - also not _just words_.
Fraterkes · 2026-06-30 15:36:26 UTC
An LLM is a big equation that we solve to get textual output. If Ai proponents already believe an equation can contain consciousness, what about the Chinese Room presents a more compelling counterargument?
sometimelurker · 2026-06-30 15:27:38 UTC
And they do have internal latent vectors for their own states, so there's some kind of recursion/introspectiveness dynamic.
dinfinity · 2026-06-30 15:30:55 UTC
Exactly. The translation back to words is the final step, so in a way very similar to what the post describes.
Improvements in model performance have been made exactly by having intermediate steps stay in the form of internal representations rather than words.
Fraterkes · 2026-06-30 15:28:41 UTC
The assumption of this sort of argument, if I understand it correctly, is that consciousness is just an ordinary byproduct that appears (or grows gradually) somewhere on the spectrum of complexity.
If that’s true, mastering language is basically orthogonal to being conscious (as you’d maybe expect, GPT-2 was pretty good at language, but had a relatively tiny amount of “neurons”).
The same is arguably true for world modeling: a math textbook has a very complex and coherent model of a world, and is, probably, unconscious.
Then the question is: what reason do we have to assume that these models have some form of consciousness?
lowbloodsugar · 2026-06-30 15:36:52 UTC
Likewise, what reason do we have to assume that any given human is conscious?
krunck · 2026-06-30 14:38:49 UTC
For LLMs there is no consciousness.
visarga · 2026-06-30 14:51:22 UTC
We might quibble what to call it, but they use language productively.
2snakes · 2026-06-30 14:38:55 UTC
I do not think the written description of a dog is the same as a dog itself…
t43562 · 2026-06-30 14:42:42 UTC
Nothing is the same as the dog but we probably store a lot of sensory information, emotions and images relating to dogs in our heads.
tsss · 2026-06-30 15:33:46 UTC
Maybe it is. If you think about it hard, you will find that you cannot separate the idea of dog from the word dog. The object dog can only be understood as a perception linked to that idea.
throwaway260124 · 2026-06-30 14:41:06 UTC
"In the beginning was the Word, and the Word was with God, and the Word was God."
And Islam the first verse was ‘read’
The first time I heard about emergent behavior in llms I thought about that.
Maybe thought is too much because I don’t have anything coherent in mind but it feels so captivating like the premise of sci-fi story.
micoti · 2026-06-30 14:44:46 UTC
Word in the Bible is not meaning the actual word (like sematic array of characters) but more like greek word logos
thewebguyd · 2026-06-30 14:56:58 UTC
Correct, in the Koine Greek, logos is the word that's actually used. Logos is a bit deeper than words, and is more akin to a universal principle of order that governs the cosmos.
"In the beginning was the logos" is like saying in the beginning were the laws of physics, and God is the laws of physics and the laws of physics are God.[1]
[1] This also entirely depends on which framework you are translating from. This would be accurate from the stoic interpretation of the logos but differs from a more mystical Johannine interpretation, in which the logos is outside the physical universe, not just operating within it. Logos is like the source code of the universe.
Some folks interpret that sentence in a mystical way - the first word was Om, from which all the universe was created.
chermi · 2026-06-30 14:41:41 UTC
Can you please point me to the proof of the first claim?
talkingtab · 2026-06-30 14:42:05 UTC
Words limit our understanding of words because they are intrinsic to a particular kind of understanding. yes? no?
In "Notes on the Synthesis of Form", Christopher Alexander talks about how we can know something is wrong even when we do not know how to make something right. He was talking about this I think
=z==
as opposed to
+|+|+
or
+|+|+|
I happened upon the word "entanglement" and it seems an interesting alternative to the order that is inherent in words.
Our language is all about Actor-verb-object. Entanglement provides a fundamentally different concept. I cannot say "I drove my car on the road" with entanglement. Or at least with entanglement I can say something equally valid like "the road moved my car with me inside."
And entanglement works for the + and | things, at least for me. There is some kind of entanglement (degree) that creates a gestalt.
At least that is the best I can come up with.
cowlby · 2026-06-30 14:42:35 UTC
Isn't it interesting then how much of our intelligence is captured by our words/language? I would've thought to create AI you needed to replicate neurons and synapses and learning. So it's still amazing to me that statistical modelling of our words creates a Claude Opus.
ordersofmag · 2026-06-30 15:03:39 UTC
Perhaps LLM's do replicate the important features of "neurons and synapses and learning" and perhaps a "statistical modelling of our words" is pretty much what our brains are doing? Most of the counter-arguments I've seen boil down to "but humans are special" which I'm not sure I find compelling.
Not saying LLM's are conscious. Just that much of our amazement about their behavior seems to say more about us realizing the things we can do are not magic, rather than them being so amazing.
speak_plainly · 2026-06-30 14:43:11 UTC
The author is making a big mistake by making a normative claim about 'true' intelligence and consciousness.
Is there a correct way to cognize? And although he feels cognizing works in a very specific direction, his brain is basically doing a very similar guessing game on a deep level with training of pathways that started at birth. Basing the argument in a feeling about consciousness is not convincing.
The author thinks they are describing a unique human magic, forgetting Augustine's insight that human thought doesn't precede the word, but is brought into consciousness by it.
tsss · 2026-06-30 14:43:18 UTC
No, I don't think so. Language is intimately intertwined with our understanding of the world. Many people make the mistake to think that languages are about the spelling or the sound or maybe the grammar. But in reality they are about how words are defined only on relation to each other. A symbol is defined by what it is not. This relationship between symbols structures how we think about the world and how we fantasize and desire.
You can perhaps have a consciousness without words, but not without language.
Comments
When you speak, what comes first, the idea or the word? Do you first feel a thought inside you, and only after that go searching for the right word to wrap around it? I think we all do. The word is never the start. The word is just the skin. The idea, the consciousness, is the thing sitting under it.
Maybe that reality is the true understanding of the neurobiology that defines our thoughts. To be reminded that the magic of experience can be reduced to neuronic hallucination is, frankly, horrifying. Maybe we dislike LLMs because we see their vectors and numbers as crude reminders of our own intellectual banality.
Or maybe I'm just feeling hungry and should really go eat dinner.
Dismissing humanity that was is absurd
There are enough neuroscience experiments demonstrating how we create post hoc explanations of our actions that I wouldn't trust this intuition prima facie.
Consider someone designing a skyscraper. They're thinking about material strengths and such while they draft it, they aren't just rationalizing their choice later.
Is it really? LLMs don't have words inside, for the most part they operate by applying transformations to a vector that does not contain any words at all. Words that come out of an LLM are just what sampler gives us by looking at the vector that is the result of those transformations.
Does this vector contain a world model? Some form of thoughts or reasoning to arrive at the result? That's an open question really.
Sure, rerunning that whole process for each token might not be the best solution, although that's an open question too. But saying that LLMs operate on words first is too big of an oversimplification
It is, but it has been explored in various forms. Anthropic have some interesting papers where they can map out concepts to some extent as they exist inside the vectors of the LLM, and they can see e.g. that vectors relating to some concept can appear there some iterations before the LLM emits the tokens that spell it out. (though also, quite frankly, this shouldn't be all that surprising. If they were just picking the next token with no representation of what might follow it the result would be more like the 'one-word-at-a-time' party game than coherent text. They used poetry with a rhyming scheme as an example because it's something where you need to have some idea of what might rhyme with the line before and aim at it at the start of the next line in order to have any degree of success).
If you don't believe me, learn how to meditate, meditate for literally just 10 minutes, and come back to me.
> Before my teacher came to me, I did not know that I am. I lived in a world that was a no-world. I cannot hope to describe adequately that unconscious, yet conscious time of nothingness. I did not know that I knew aught, or that I lived or acted or desired. I had neither will nor intellect.
https://scentofdawn.blogspot.com/2011/07/before-soul-dawn-he...
Language is a shared medium. Maybe an intelligence is capable of building abstractions in their head without the need for say, a symbol system to hopefully communicate with species similar to itself.
You may argue that the agent needs persistent memory. Well, so what? If the world has stable regularities from the perspective of the agent - if the sun rises every day, and language is associating certain visuals with certain thoughts (it's not when viewed in its original social definition - but conduct the thought experiment with sociality removed), who's to say that you can't form a kind of language around that?
Of course, this is all armchair, and I'm not trying to separate out what "real language" is or whatever, just thought experiments.
But the original poster has a good point though, that's been proven in papers, that the tokens an LLM spits out, themselves, attract them to certain distributions, and at least a decent amount of the tokens in CoT are there solely to bring itself to the right distribution.
I believe that conscious is a gradient not a binary, and I believe that cats have some level of consciousness, but I’m not sure communication is the thing to hang that on.
If for no other reason, I believe it's quite reasonable to believe that cognition must be Turing-complete (okay, linear bounded automaton), and there must be some kind of support for recursive structures, no matter what the "primitives" of such consciousness is (either visual or linguistic).
I believe in the case of animal intelligence, they (we) have a pretty good hard-coded 3d/4d world model that ought to encode some level of recursion intrinsically (in thought, we very very often use spatial and temporal words), e.g. understanding the motion of another animal, e.g. a rodent running through a tree trunk requires imagination and future prediction already over an abstract agent. Humans just probably have more "stacks" available, and can offload them into either limited "brain memory", or very importantly drawings, text, etc, for unbounded memory.
But I'm just thinking out loud, I'm absolutely no expert on any of these topics.
There's also the grim history of language deprivation experiments: https://en.wikipedia.org/wiki/Language_deprivation_experimen...
It really does seem like language is a key part of how we develop our human intelligence.
Infants raised by chickens? Chickens!? That doesn't sound real.
Though there are recent experiments showing that unconscious brain retains language processing abilities [1], so the two might be as well independent systems
[1]: https://www.nature.com/articles/s41586-026-10448-0
This is in contrast with aphantasia (complete lack of mental imagery) which can be correlated with outwardly detectable responses. E.g. people who can imagine mental images will dilate pupils when imagining a bright light with eyes closed, while people with aphantasia will not. Not sure there's an analog for internal monologue but if anyone.
I really feel like humans argue too much about how computer programs think and at the same time too little about how humans think. The topic is criminally underresearched and the existing research is all over the place.
It's just that subvocalization is one powerful technique for organizing thoughts, but not the only one. If you had a swimmer who only ever learned the crawl stroke (freestyle), they might wonder how someone can swim without it. The difference, of course, is that the breaststroke is a demonstrable physical act whereas cognitive techniques are not. The monologuers are just more practiced at subvocalization-based reasoning, and the non-monologuers are more practiced at other types.
It's not that hard to exert limited control over which technique you're using -- and I think there are multiple nonverbal techniques. I'd encourage people to explore the space a bit. Subvocalization-first approaches are very good for linear progressions, whereas subvocalization-last (or never) approaches are usually better for big-picture reasoning.
I say this as someone who pays a lot of attention to my mind's inner workings (meditation etc, near-complete aphantasic) and the more I pay attention to the inner workings, the less I know the answer to whether or not I have an inner monologue.
She saw something about it online and then started asking everyone in the family if they could picture an apple in their mind. We all said - yes of course, what are you talking about?
And she said, "When you say the word apple, I think of the concept of an apple. It's juicy. It's crisp. It's red, etc, but I don't SEE an apple. I just know what it is."
She had no concept that other people literally saw the apple and I had no concept that people couldn't see the apple.
I asked her what she thought the phrase "in your minds eye" meant for her entire life and she said she thought it just meant exactly what she was describing and not literally seeing anything...
It's unfortunately very insulting when someone says, "hey remember when we did that very important thing together?" and my answer has always been "uhh... I remember that we did it?"
Maths is probably a great example of this. Try and describe Pythagoras' Theorem without using maths notation or words. Difficult, right? Reasoning is a House of Cards, and without understanding what the card is that exercise becomes significantly more difficult.
There are really three main observations that give pause. First, after she started working with Sullivan, Keller apparently developed a writing style and nuanced opinions reminiscent of a college-educated person virtually overnight. That’s pretty surprising for any child, let alone one who had only recently acquired language. Second, all of that disappeared overnight when Sullivan died - Keller’s writing style changed drastically, becoming much simpler, and all that eloquent insight also disappeared along with Sullivan. Finally, we’ve never since seen another Helen Keller. On the contrary, later observation of deaf-blind people who don’t acquire language until later in life is part of what inspired a variant of the critical period hypothesis that posits that people who don’t learn their first language (doesn’t matter whether it’s spoken or signed) in early childhood are never able to acquire any language to a particularly high level of proficiency.
That said, even if that Keller quote is now considered dubious, it may not be entirely off base. Supposedly people who don’t acquire language in early childhood also tend to show less capacity for abstract reasoning tasks. Which is still a far cry from the “no-consciousness” that Keller-maybe-Sullivan describes, but does still suggest that language and reasoning are mutually supportive.
Super interesting quote and link though!
[0] Fedorenko et al, https://www.nature.com/articles/s41586-024-07522-w
For instance, you can dream words.
I'm thinking of my kid - her personality when she was tiny. I could see what it was but not explain it.
Hence I think ideas are unquestionably first and words are the inadequate and inaccurate description of thoughts that exist without them.
Still, even though I think the author is right about consciousness/concepts preceding language, I think he's wrong about the other part. The statistical models that underlie AI very much remind me of what we call a human concept. It's a statistical model of that thing, person, or idea. However, we humans seem to have this ability to create new concepts that are wildly different than our training data (what we have experienced in life), and we call this creativity. It's missing from AI for some reason. That is the part that makes me question if AI will ever match humans.
You (and TFA) are making a false dichotomy here. Yes, of course we think in images. But we also have an inner monologue that is critically important for much of our higher-level thought. How do you even write a HN comment without thinking through it in words?
A more powerful telescope does not "create" new galaxies.
you can view my profile details freely and decide for yourself, and a quick look might even explain the "bot" suffix.
This in the sense that we can easily retain human "understanding" by stripping away almost all parts of the human body or replacing them with fairly trivially made replacements, except for the brain.
In the Chinese Room the equivalent is the magic rulebook: We have no idea how to construct/replace it, yet people somehow handwave that away whilst simultaneously confidently asserting it does not understand anything.
... we now have LLMs to even more pointedly show the deficiencies of the argument.
(What's the point of the argument if it doesn't tell us anything about the capabilities or internals of an AI? I'm not sure.)
Improvements in model performance have been made exactly by having intermediate steps stay in the form of internal representations rather than words.
If that’s true, mastering language is basically orthogonal to being conscious (as you’d maybe expect, GPT-2 was pretty good at language, but had a relatively tiny amount of “neurons”).
The same is arguably true for world modeling: a math textbook has a very complex and coherent model of a world, and is, probably, unconscious.
Then the question is: what reason do we have to assume that these models have some form of consciousness?
And Islam the first verse was ‘read’
The first time I heard about emergent behavior in llms I thought about that.
Maybe thought is too much because I don’t have anything coherent in mind but it feels so captivating like the premise of sci-fi story.
"In the beginning was the logos" is like saying in the beginning were the laws of physics, and God is the laws of physics and the laws of physics are God.[1]
[1] This also entirely depends on which framework you are translating from. This would be accurate from the stoic interpretation of the logos but differs from a more mystical Johannine interpretation, in which the logos is outside the physical universe, not just operating within it. Logos is like the source code of the universe.
In "Notes on the Synthesis of Form", Christopher Alexander talks about how we can know something is wrong even when we do not know how to make something right. He was talking about this I think
=z==
as opposed to
+|+|+
or
+|+|+|
I happened upon the word "entanglement" and it seems an interesting alternative to the order that is inherent in words.
Our language is all about Actor-verb-object. Entanglement provides a fundamentally different concept. I cannot say "I drove my car on the road" with entanglement. Or at least with entanglement I can say something equally valid like "the road moved my car with me inside."
And entanglement works for the + and | things, at least for me. There is some kind of entanglement (degree) that creates a gestalt.
At least that is the best I can come up with.
Not saying LLM's are conscious. Just that much of our amazement about their behavior seems to say more about us realizing the things we can do are not magic, rather than them being so amazing.
Is there a correct way to cognize? And although he feels cognizing works in a very specific direction, his brain is basically doing a very similar guessing game on a deep level with training of pathways that started at birth. Basing the argument in a feeling about consciousness is not convincing.
The author thinks they are describing a unique human magic, forgetting Augustine's insight that human thought doesn't precede the word, but is brought into consciousness by it.
You can perhaps have a consciousness without words, but not without language.