When Would LLM Say “I Don’t Know”
I finished the Deep Dive into LLMs like ChatGPT by @AndrejKarpathy. It’s an absolute delight to watch a master presenter break down a complex topic in such a clear and easy-to-understand way. I highly recommend it to anyone interested in LLMs or how to give a great technical training.
My main takeaway from the talk was that LLMs hallucinate and have some basic problems that are easy for humans to understand because they don’t know when to say, ‘I don’t know.’
To know what “I don’t know” is hard. To say it aloud, with clarity, humanity and confidence, is even harder. Socrates captured this truth millennia ago when he declared, “I know that I know nothing.” This ancient statement remains profoundly relevant today—not just for human understanding but amazingly also for the operation of Large Language Models (LLMs).
Just as LLMs construct responses based on patterns in training data, we humans construct our experience of reality through the brain’s interpretation of sensory inputs. Our eyes capture wavelengths of light, our ears detect vibrations in the air, and our skin registers pressure and temperature. But what we perceive—a coherent, colorful, textured world filled with meaning—is not reality itself but a model our brain has built to make sense of the data.
This model isn’t static; it evolves as we accumulate new experiences. Neuroscientists often describe the brain as a predictive machine, constantly updating its internal representation of reality based on sensory feedback. In this way, our mind doesn’t passively receive information but actively constructs our reality—a process remarkably similar to how LLMs generate language outputs by predicting the next likely word based on context.
A fascinating question here is: Is the human mental model inherently language-based? How much does language shape the way we think, perceive, and understand the world?
Language undeniably plays a central role in human cognition. It provides the framework through which abstract ideas are formed, communicated, and remembered. Without language, concepts like justice, time, or infinity would be difficult—perhaps impossible—to articulate or even fully grasp. Many cognitive scientists argue that language doesn’t just reflect thought but actively shapes it, influencing how people categorize experiences and frame their understanding of the world.
Yet language is not the entirety of human cognition. We can think in images, sensations, and emotions. A musician may “think” in melodies; an athlete may “think” in movements and kinesthetic patterns. Still, language serves as a powerful organizing tool for conscious reflection, problem-solving, and social interaction. In this sense, human mental models often rely heavily on linguistic constructs, even if they aren’t wholly defined by them.
The similarities between LLMs and human cognition are fascinating. LLMs generate language by predicting sequences based on statistical probabilities derived from vast training data. They don’t truly understand the concepts they discuss; they simulate understanding by leveraging patterns encoded in their neural network parameters. Likewise, the human brain doesn’t access reality directly—it constructs a predictive model based on sensory inputs, memory, and learned frameworks.
Where human and LLMs differ is in grounding. We experience the world through a body, giving our mental model a tether to sensory experiences. LLMs lack that grounding. Their model exists purely in linguistic space, disconnected from any sensory reality. This is part of why LLMs sometimes hallucinate—making up plausible-sounding but false information—because they have no concrete connection to a physical world to verify their outputs.
But LLMs are not alone. Both humans and LLMs face the challenge of acknowledging what they don’t know. For LLMs, this difficulty arises from the nature of their probabilistic training. They weren’t designed to say “I don’t know” unless explicitly trained to do so. For humans, the difficulty stems from cognitive biases, overconfidence, and the fragmented nature of personal knowledge. We often honest don’t know “what we actually don’t know or don’t understand”.
In both cases, admitting ignorance is powerful. It fosters curiosity, continuous learning, and humility. Just as LLMs improve through feedback and tool use, humans grow by questioning assumptions, seeking diverse perspectives, and refreshing our mental models.
If human perception is a model created by the brain, what about identity? Is the self—the “I” that we experience—just another construct? Cognitive scientists suggest that the sense of a coherent, continuous self is indeed a mental fabrication, a narrative our brain tells itself to make sense of our experiences. This narrative is often deeply intertwined with language, as we use words to define our identity, interpret our past, and project your future.
But despite their limitations, both human and artificial models can produce emergent phenomena—unexpected insights or creative problem-solving strategies. Just as human brain can combine disparate ideas to form novel solutions, LLMs can generate analogies or patterns that may spark new ways of thinking. AlphaGo's famous move 37, a play that was unexpected and considered highly improbable by human Go experts, is one of these moments, where the boundary between artificial and human intelligence blurs.
Whether human or machine, acknowledging the limits of knowledge is essential for growth. The constructed nature of human reality and identity is not a flaw but a feature—one that enables imagination, adaptation, and innovation. Like Socrates, to say “I know that I know nothing” is not an admission of defeat but an invitation to explore.
For LLMs, the challenge is to better mimic this humility through improved training and tool use. For humans, the challenge is to confront the fragile, constructed nature of our reality and embrace the uncertainty that drives progress. In doing so, maybe both can unlock new possibilities for understanding and creativity.