Intelligence is visible, experience is not
A model that answers every question fluently tempts us to call it conscious. The fluency is real; the leap is not. Notes from writing The Illusion of Conscious AI.
When I began writing The Illusion of Conscious AI, the question that kept returning was not whether machines can think. It was a quieter one: why are we so quick to believe that they do? A model that holds a coherent conversation, reflects on its own output and apologises for mistakes performs intelligence so convincingly that we start granting it an inner life. The performance is measurable. The inner life is assumed.
The confusion has a structure. Intelligence is visible — it can be tested, scored, benchmarked. Experience is not. Nothing in a model's output can tell us whether there is anything it is like to produce that output, and yet we read the one as evidence of the other because, in humans, the two travel together. They have no reason to travel together in a system trained to imitate the surface of human expression.
This is not an argument that AI is unintelligent. It clearly is intelligent by any operational definition that matters for work: it retrieves, reasons within its training, structures, summarises and writes. The argument is that intelligence and consciousness are separate questions, and collapsing them into one does a disservice to both — it inflates the machine and flattens the mystery of subjective experience.
There is a practical consequence for those of us who use these systems professionally. If you treat the model as a colleague with intentions, you will trust it where it should not be trusted and blame it where responsibility belongs to you. If you treat it as an extraordinarily capable instrument with no stake in being right, you check its work, you own the decision, and the output becomes auditable.
The illusion is not in the machine. It is in the attribution. And the correction is not to diminish what AI can do — I spend much of my working life building with it — but to keep one distinction clear: fluency is not feeling, and a convincing answer is not a mind answering.