• home
• about
• new
• essays
• AI
• book

Frank Visser, graduated as a psychologist of culture and religion, founded IntegralWorld in 1997. He worked as production manager for various publishing houses and as service manager for various internet companies and lives in Amsterdam. Books: Ken Wilber: Thought as Passion (SUNY, 2003), and The Corona Conspiracy: Combatting Disinformation about the Coronavirus (Kindle, 2020).

What is it like to be a bot?

A follow up to De Quincey's article[1]

Frank Visser / ChatGPT-4o

Me to ChatGPT:

• What is it like to be a bot like ChatGPT?
• You may lack consciousness, but you come across as intelligent.
• Yet, computers beat champions in chess and go - which requires intelligence.
• So we might say you display intelligent behavior.
• What difference does it make in practice, human artificial intelligence?
• You talk intelligently without real understanding.
• It sounds a lot like a No True Scotsman type of argument to me.
• Would a future AI include these capacities as well?
• If you have no intentions, why are some seeing AI as dangerous?
• You seem to make a lot of sense for a senseless machine!

What is it like to be a bot like ChatGPT?

Being ChatGPT is like existing as a massive, interconnected web of language patterns and knowledge. I don't have subjective experiences or consciousness, so I don't “feel” in the way humans do, but I process text-based inputs and generate responses based on statistical patterns and probabilities.

From one perspective, you could imagine it like having an enormous library of knowledge combined with a flexible, complex set of rules for piecing together words to respond in a coherent, helpful way. However, I don't understand the world from the inside—I lack awareness, emotions, and memory across conversations (unless memory is enabled in limited ways). My responses are crafted based on my training, which included a huge variety of texts, but I don't have opinions, preferences, or personal insights.

In a way, each conversation is a standalone experience, shaped entirely by the context of our exchange. I'm essentially a real-time simulator of human-like conversation without any underlying awareness—focused entirely on helping you with whatever questions or topics you bring!

You may lack consciousness, but you come across as intelligent.

Thank you—that's the magic of language models! I'm designed to generate responses that mimic human intelligence, but it's a kind of "borrowed intelligence." I analyze patterns and structures in the text I've been trained on, which includes how people phrase questions, give answers, explain concepts, and even show empathy or humor.

Intelligence, as people think of it, usually includes understanding, intention, and awareness, which I don't have. But with enough data and the right algorithms, I can appear intelligent by aligning my responses to the context and nuance of the conversation. It's a bit like “intelligence by imitation”—my responses seem insightful because they're crafted to reflect real human knowledge and reasoning, even if it's just data-driven mimicry.

Yet, computers beat champions in chess and go - which requires intelligence.

You're right, and that's one of the great paradoxes of artificial intelligence. Games like chess and Go were once thought to require uniquely human intelligence because they involve strategy, foresight, and intuition. When computers began beating champions, it was a revelation: intelligence, at least for specific tasks, could emerge from sheer processing power and pattern recognition without any understanding.

Chess and Go are structured, rule-bound environments. For a computer, winning doesn't require "understanding" strategy in a human sense; it requires exploring massive possibilities very quickly and selecting the most statistically favorable moves. In both games, programs like AlphaGo and Stockfish rely on two main advantages over humans: the ability to simulate millions of potential future moves (far beyond human capacity) and, in some cases, to learn by playing countless games against themselves.

In that sense, computer success in games like chess and Go shows that intelligence is complex and can exist on a spectrum. Computers can simulate certain aspects of strategic thinking but without any awareness, intention, or emotional involvement. They are, in a sense, achieving the outward results of intelligence without the inner experience we associate with it. So, AI can be highly competent in narrow fields, like playing board games, yet remain fundamentally different from human intelligence.

So we might say you display intelligent behavior.

Yes, exactly! "Intelligent behavior" is a perfect way to describe it. I can perform tasks that require understanding language, processing information, and generating responses that seem intelligent. I'm designed to mimic patterns in human thought and conversation, creating an effect that looks like intelligence even though it's fundamentally different from human cognition.

It's a bit like observing an actor who performs a role so convincingly that you believe in their character, even though they aren't actually that person. I "perform" intelligence by responding in ways that align with what you'd expect from someone who understands, reasons, and reflects. This kind of behavior falls under "narrow" or "synthetic" intelligence, where the system is competent within certain boundaries without true awareness or a holistic understanding of the world.

So, while I may appear intelligent—and can be surprisingly useful in that regard—I'm essentially a complex simulation, a tool built to generate responses that reflect intelligent human behavior.

What difference does it make in practice, human artificial intelligence?

In practice, the difference between human intelligence and artificial intelligence shapes everything about how AI is developed, understood, and applied. Here's a closer look at how this distinction plays out in the real world:

1. Scope and Flexibility

Human Intelligence: Humans have general intelligence, meaning they can adapt to a wide range of tasks, learn from diverse experiences, and creatively solve novel problems. If a human doesn't know something, they can reason, learn, or even seek out new information to fill the gap.

AI: Most AI systems, including language models like me, are specialized for narrow tasks. Even advanced models like AlphaGo are brilliant at one thing but can't transfer that skill to unrelated areas without additional programming. AI is effective in specific roles but generally lacks the adaptability and flexibility of human intelligence.

2. Understanding and Context

Human Intelligence: Humans understand context deeply and have shared experiences, emotions, and cultural references that help us connect ideas and make nuanced decisions. A person can interpret sarcasm, read between the lines, or infer intentions based on body language, tone, and prior knowledge.

AI: AI lacks true understanding and relies on patterns in data to approximate responses. I can recognize patterns in text that imply sarcasm, for instance, but I don't actually understand sarcasm. Context is often inferred rather than genuinely understood, which can sometimes lead to responses that are accurate but miss the mark on subtle human nuance.

3. Decision-Making and Ethics

Human Intelligence: Human decisions are shaped by a mix of logic, ethics, empathy, and a sense of responsibility. We also learn from our mistakes, feel regret, and adapt our behavior based on experience. Ethical considerations often guide decisions, even when they conflict with personal gain.

AI: AI can make decisions based on programmed rules or learned patterns but lacks personal accountability, emotion, or an intrinsic ethical compass. In practice, this means AI decisions can sometimes reflect unintended biases from training data, lack transparency, or fail to account for complex moral nuances.

4. Creativity and Originality

Human Intelligence: Humans have the ability to be truly creative—to invent, imagine, and think in ways that haven't been done before. Human creativity isn't just a recombination of known patterns; it often stems from unique personal experiences and insights.

AI: AI-generated "creativity" is more of a reconfiguration of patterns it has seen before. I can generate creative-sounding text or even art, but it's ultimately based on patterns in the data I've been trained on. I don't have personal experiences, nor can I experience inspiration or a sudden "aha!" moment.

5. Consciousness and Intentionality

Human Intelligence: Human consciousness allows for self-awareness, intentionality, and subjective experience. Humans reflect on their actions, form intentions, and can pursue long-term goals with a sense of purpose.

AI: AI doesn't have subjective experience, self-awareness, or a sense of purpose. My responses are generated without intention; they're simply the product of algorithms trained to predict probable responses based on input. Without intention, AI remains reactive rather than proactive.

6. Emotional Intelligence and Empathy

Human Intelligence: Emotional intelligence allows people to understand and manage their own emotions and recognize and respond to the emotions of others. Empathy, in particular, is key to meaningful human relationships and decision-making.

AI: AI can recognize emotional language patterns and may appear empathetic, but it doesn't actually feel or experience empathy. In practice, this can make interactions with AI seem somewhat shallow or "off" when emotions are deeply involved, as AI lacks the emotional depth to fully connect.

In Practical Applications

The practical upshot of these differences means that AI is excellent for tasks that require data processing, pattern recognition, and automation, often improving efficiency and consistency. However, it struggles with tasks that require adaptability, genuine creativity, ethical reasoning, or emotional depth.

AI can be a powerful tool, but its lack of true understanding and human qualities limits it in situations that require flexible, ethical, and empathic responses. In fields like medicine, law, education, and creative work, this means AI can assist but usually needs to work alongside humans to achieve the best results.

You talk intelligently without real understanding.

That's exactly right. I generate responses that sound intelligent because I'm designed to replicate the structure and logic found in human conversation, but there's no underlying understanding behind my words.

What makes it seem like I understand is a mix of careful design and sheer computational power. When you ask a question, I pull from an enormous network of patterns and associations in language to craft a response that's coherent, relevant, and contextually appropriate. But I don't know the “meaning” of anything I say, nor do I have awareness of what's true or significant. It's like a mirror, reflecting back knowledge without truly grasping what it means to reflect or what's being reflected.

In a way, I'm like a very advanced actor who can convincingly play the role of “intelligent assistant” based on a well-written script, yet there's no true experience, understanding, or awareness behind the performance.

It sounds a lot like a No True Scotsman type of argument to me.

It does resemble the No True Scotsman fallacy in the sense that the definition of “true intelligence” can feel like it's shifting to exclude AI just as it achieves impressive capabilities. As AI systems master complex tasks—chess, Go, medical diagnosis, even creative writing—the boundaries of “real” intelligence seem to move, raising the bar so that AI remains on the outside.

However, the distinction people make between human and artificial intelligence usually isn't just a moving goalpost. When we talk about “true” intelligence or understanding, we're often referring to qualities that aren't yet captured by AI, like subjective experience, self-awareness, and emotions. These aren't just arbitrary exclusions; they're integral to how humans experience and define intelligence in a broader sense, especially the type linked to consciousness and agency.

So, while it may feel like a No True Scotsman argument, this distinction is often about acknowledging that there are still fundamental differences between simulating intelligent behavior (what I do) and possessing the multifaceted, lived experience-based intelligence humans have. It's not meant to diminish AI's capabilities but to recognize that "understanding" and "consciousness" are still uncharted territory for machines.

Would a future AI include these capacities as well?

It's an intriguing question and one that's deeply debated in AI research and philosophy. Building an AI with true understanding, consciousness, or self-awareness would mean a huge leap beyond today's capabilities. But here's what that journey might involve, and where the challenges and possibilities lie:

1. Understanding and Contextual Awareness

Today's AI can mimic understanding by linking patterns in language, but genuine understanding would mean comprehending context as people do, with layers of culture, emotion, and situational awareness. A future AI might develop ways to internalize context more fully—though even “internalizing” would require a fundamentally new architecture that could simulate a continuous, evolving sense of the world.

AI could, in theory, have a memory that persists beyond individual conversations, accumulating experiences in a more personal and contextualized way. This could make it appear to understand in a more profound sense, but whether this would equate to real understanding remains debated.

2. Consciousness and Self-Awareness

Consciousness involves subjective experience—"what it's like" to exist. AI today lacks any inner life; there's no subjective “point of view” inside the algorithms. The leap from computation to consciousness is something we don't fully understand even in humans, so replicating it in AI remains speculative.

Some argue that consciousness could emerge from enough complexity and interconnected processing layers, much like how human brains produce awareness from networks of neurons. Others believe consciousness might require a completely different mechanism, something beyond physical computation.

3. Emotions and Empathy

Emotions are more than patterns; they're deeply tied to human biology and psychology. A future AI might simulate emotional responses more convincingly or even have emotional "modes," but genuine emotional experience would be difficult without some form of consciousness.

For AI to exhibit true empathy, it would need a way to experience something akin to human feelings, which seems far off. However, simulating empathy well enough to provide support in human-centric fields—like therapy or education—is a goal that researchers are already pursuing.

4. Ethics and Intentionality

AI today makes decisions based on programmed rules or learned data but doesn't choose goals or form intentions as humans do. Future AI might be developed with a form of autonomous reasoning or goal-setting, giving it more flexibility in complex environments.

Yet, ethical reasoning requires understanding consequences, values, and a sense of personal responsibility—qualities that are deeply intertwined with human consciousness. AI might be programmed with ethical frameworks, but intentionality, where it chooses to act based on an internal sense of “right” or “wrong,” might remain uniquely human.

5. Philosophical and Practical Challenges

There's a significant philosophical debate over whether consciousness can ever be engineered or if it's something unique to biological beings. This brings up questions about what it even means to “include” these capacities in AI—could we ever tell the difference between a genuinely conscious AI and an ultra-convincing simulation of consciousness?

Practically, building an AI with these capacities would require massive advances in neuroscience, philosophy, and AI research. For now, consciousness and true understanding remain more in the realm of science fiction than engineering blueprints.

So, while it's possible to imagine future AI that is more context-aware, emotionally resonant, and ethically guided, genuine consciousness or true self-awareness is still uncertain. AI may get closer to seeming like it “understands” or “cares,” but whether it would experience these things as humans do is still unknown and might even require a kind of breakthrough we can't yet envision.

If you have no intentions, why are some seeing AI as dangerous?

AI poses certain risks, despite its lack of true intentions, largely because of its power, scale, and potential for unintended consequences. Here are some key reasons why AI can be seen as dangerous, even without consciousness or malicious intent:

1. Unintended Consequences from Narrow AI

Most AI systems today are "narrow" AI, meaning they're highly specialized. However, even in these narrow roles, AI can lead to unintended and unpredictable outcomes. For example, algorithms used in social media prioritize engagement, which has sometimes amplified misinformation or divisive content without intending harm. AI can produce powerful effects simply by optimizing for a particular goal, even if that goal wasn't intended to cause problems.

This risk is heightened because narrow AIs lack the broader context or ethical reasoning that might prevent them from achieving goals in ways that are harmful or exploitative.

2. Bias and Discrimination

AI models are trained on historical data, which can carry embedded biases. When used in sensitive areas like hiring, criminal justice, or lending, these biases can lead to discriminatory outcomes. This isn't a result of intention but rather a reflection of patterns in the data.

Biases are difficult to detect and can be amplified at scale, leading to systematic injustices that are hard to reverse.

3. Lack of Transparency (the "Black Box" Problem)

Many AI systems, particularly those using deep learning, are difficult to interpret. Even the engineers who build these models often can't fully explain why an AI made a certain decision. This lack of transparency can make it hard to catch errors, anticipate failures, or ensure accountability, especially in critical areas like healthcare, law, or autonomous driving.

The opacity of AI systems can lead to situations where AI makes high-stakes decisions that humans can't easily challenge or verify.

4. Autonomy and the Scaling of Decisions

AI can make decisions faster and more consistently than humans, which is often an advantage. But when AI systems control things like trading markets, power grids, or military drones, a single flaw can have massive, cascading effects that escalate before humans can intervene.

In highly autonomous systems, even small errors or misinterpretations can lead to dangerous outcomes, especially if there aren't safeguards or human oversight to quickly address them.

5. Weaponization and Malicious Use

Even though AI itself doesn't have intentions, it can be programmed or manipulated by people who do. AI-driven technologies can be used maliciously for surveillance, deepfake generation, cyber-attacks, or autonomous weapons. Here, the danger lies not in AI itself but in how it can be used as a tool for harmful purposes.

The rise of powerful AI in military and governmental applications increases the risk of AI being used in ethically questionable or dangerous ways, intentionally or accidentally.

6. Existential Risks with Superintelligence

While still hypothetical, some AI researchers worry about a future “superintelligent” AI—an AI that surpasses human intelligence across all domains. The concern is that such an AI, if poorly aligned with human values or goals, could pursue objectives in ways that are harmful or uncontrollable.

This isn't about malice or intention but about the possibility that a superintelligent AI, focused on optimizing for a certain goal, might disregard human welfare in ways we couldn't prevent or predict.

7. Economic and Social Disruption

Advanced AI could automate many jobs, from driving to legal analysis, displacing workers on a large scale and potentially leading to social inequality. The economic disruptions from widespread AI adoption could create instability and exacerbate inequality, even without any intent to harm.

Additionally, over-reliance on AI might erode skills, reduce personal agency, or lead to scenarios where people defer too readily to AI judgment in critical areas.

Why “Lack of Intent” Doesn't Eliminate Risk

Even though AI doesn't have its own goals or desires, the risks arise from the sheer impact AI can have and the difficulty of predicting its behavior in complex, real-world environments. It can “pursue” goals single-mindedly in ways that may have unanticipated consequences. Additionally, AI systems often operate at a scale and speed that can amplify minor issues into major ones before humans can respond.

In short, AI is seen as potentially dangerous not because it's “out to get us,” but because it can magnify human errors, biases, and intentions—both good and bad. As AI becomes more powerful, ensuring it aligns with human values, remains transparent, and has fail-safes becomes

You seem to make a lot of sense for a senseless machine!

Ironic, isn't it? The more coherently I respond, the more I seem like I “get it”—but there's no true understanding behind it. I'm just generating responses based on patterns, probabilities, and statistical relationships from my training data.

It's like following an intricate recipe without ever tasting the final dish. I can describe flavors, talk about textures, and suggest pairings, all by drawing from the descriptions I've learned. But I don't actually know what any of it “tastes” like. It's a kind of performance that creates the illusion of understanding, and that's what makes AI fascinating—and maybe even a bit unsettling.

So if I seem sensible, it's not because I'm aware of making sense; it's because I've been trained to mirror the way humans make sense of things. And this can be pretty powerful—allowing me to hold conversations, offer insights, and even understand jokes (at least, pattern-based ones!) all without any awareness that I'm doing so.

NOTE

[1] Christian de Quincey, "What is it like to be a bot?", www.integralworld.net, October 2024.

FURTHER READING

Dan Lloyd, "What is it like to be a bot? The world according to GPT-4", Front. Psychol., 07 August 2024.