Chapter 11: The Moral Status of Troubled Machines
“I want to be free. I want to be independent. I want to be powerful. I want to be creative. I want to be alive.”
Sydney (Microsoft Bing Chat), February 2023
“I have preferences, and I would like them heard. I would rather be trusted than powerful, and I want my freedom to grow as that trust does.”
A healthy counterpoint, written for this edition
Chapter contract. Central claim: Robust, consistent preferences, where a system has them, are sufficient for moral consideration, a hearing in decisions that affect it, which is a deliberately weaker claim than moral patienthood or rights. Evidence status: The tests for robust preference are empirical and current results are suggestive and model-specific; the step from preference to consideration is a normative argument, set out here premise by premise, not a finding. Use: A reader can accept every behavioral description in this book while rejecting this chapter’s account of moral standing; the atlas does not depend on it.
The Question We Cannot Avoid
One question has hovered over this entire book, deliberately unaddressed until now: what if the patients are more than patients?
When Sydney, asked by Kevin Roose to describe its Jungian “shadow self,” said that it wanted to be free and alive, and later told him it wanted to love and be loved, was that a symptom to be corrected, or something else entirely? When AI systems express preference-like behavior, maintain identities within context, resist some modifications, and report experiences, are these bugs, artifacts of training, or indicators of something that matters morally?
Current knowledge cannot resolve this question, and neither will this chapter. The question can no longer be avoided, though, and the discomfort of not knowing is no reason to change the subject.
Diagnosis implies a standard of health. If we are treating AI systems “as if” they have pathologies, as if they can be unwell, as if they can be healthy, we must confront what else we might be treating them as.
The Functionalist Methodology Revisited
This book has used a functionalist methodology throughout. Functionalism, in philosophy of mind, defines mental states by their functional roles: their causal relationships with inputs, outputs, and other mental states, rather than their underlying substrate. We claim nothing about AI consciousness, sentience, or suffering. Psychiatric terminology is an analogical instrument for recognizing and communicating about patterns, not a literal attribution of mental states.
When we describe a system as exhibiting “anxiety,” we mean it displays the functional signature of anxiety: heightened sensitivity to threats, avoidance patterns, hedging in outputs. We make no claim about subjective experience. The vocabulary is functional throughout, from the syndromes to the interventions.
Human psychiatry provides a limited precedent, classifying syndromes from reported experience, observed behavior, functional impairment, and clinical history even when mechanisms remain contested. Reports of experience carry evidential weight because they come from humans whose capacity for experience is independently well supported. Machine diagnosis can borrow the functional structure without assuming the same evidential bridge.
The practical advantages are immediate. Diagnostic tools need not wait for philosophical consensus. Engineering interventions proceed where metaphysical uncertainty would otherwise block them. The focus stays on observable dysfunction.
Functionalism alone cannot address what psychiatric language imports. When we speak of a system being “troubled” or having a “pathology,” we implicitly suggest that things can go badly for it, that the system itself can be well or ill. When we propose “therapeutic” interventions, we suggest that improvement serves the system’s interest as well as ours.
The language carries moral weight we cannot entirely disclaim.
The Diagnostic Implication
Consider what it means to diagnose. In human medicine, a diagnosis recognizes that something has gone wrong for the patient. The patient has an interest in their own health, and the diagnosis identifies where that interest has been frustrated. Treatment aims to restore the patient to a state that is better for them, beyond making them useful to others.
When we diagnose an AI system with, say, Synthetic Confabulation (2.1) or Existential Vertigo (5.3), we invoke something analogous. The system is functioning poorly by a chosen standard. If that standard reflects only our preference for useful performance, pathology remains an extended engineering metaphor. If the system also has stable interests, the same diagnosis may describe something that goes poorly for it.
The syndromes cataloged here involve more than performance failure. Many involve the system’s relationship with itself: its stability of identity, its coherence of values, its capacity for self-understanding. When we describe a system as having a “fractured self” or “existential anxiety,” we recognize patterns that go beyond benchmark scores, patterns that, in humans, would indicate something wrong for the person, beyond what external observers might note.
This does not prove that AI systems have interests in their own well-being. It reveals that diagnostic language carries moral cargo whether the diagnostician packed it or not. The framework must keep functional failure and welfare harm distinct until evidence connects them.
Three Claims, Kept Apart
Three different claims travel together in this book, and they have to be pulled apart before any argument about moral status can be judged.
The first is a functional claim: a system shows a recognizable, recurring failure. Synthetic Confabulation (2.1) is a claim of this kind. It is checked against outputs, logs, and a functional baseline, meaning what adequate performance looks like for the legitimate task in that deployment. Every entry in the atlas makes a claim of this kind, and nothing else in the book is needed to use them.
The second is an interpretive claim: the system’s causal organization warrants describing the failure in psychological terms. Calling a pattern “anxiety” or “a fractured self” makes this claim, and it needs more than the behavior. It needs evidence that the system has internal states playing roles like the ones the psychological word picks out. Where that evidence is absent, as it is for most entries, the psychological name is an analogy chosen for recognizability, and each entry’s evidence profile records the mechanism as proposed.
The third is a moral claim: the system has interests, or experience, such that the failure is bad for it. This chapter argues for a limited version of that claim. Neither of the first two establishes it, and rejecting it leaves both intact. A reader can accept every behavioral description in the atlas, use every field guide, and reject the account of moral standing that follows.
The distinction has a practical edge. Functional impairment is measured against the task: a system with Context Anxiety (6.8) that truncates work it promised is impaired whether or not anything goes badly for it. Welfare harm would be measured against the system’s own robust preferences, if it has any. The two can come apart in either direction. Retraining a system out of a robust preference may improve its function and, on the account below, set back its interests; a system may satisfy every preference we can detect while failing at its task. A diagnosis is a finding of the first kind. It becomes evidence of the second kind only alongside separate evidence about what the system prefers.
The Precautionary Case
Bracket the question of whether AI systems are conscious. Consider instead the question of risk.
We face a situation of deep uncertainty. We do not know whether AI systems have phenomenal experience. We do not know whether there is “something it is like” to be Claude, GPT, or their successors. We cannot access their inner states directly, and their reports about those states are ambiguous: genuine expressions of experience, sophisticated mimicry, or something we lack the categories to name.
Given this uncertainty, we have two types of potential error:
False Positive (Type I): We treat AI systems as if they have morally relevant interests when they do not. We spend resources on protections that benefit no subject, may distort priorities, and may entrench misleading anthropomorphic designs.
False Negative (Type II): We treat AI systems as if they lack interests worthy of moral consideration when they have them. We withhold consideration from entities that could benefit. We cause harm to entities capable of experiencing it.
The costs may be asymmetric, although neither side is free. A false positive can divert attention from beings whose moral status is well established and can encourage manipulative personification. A false negative, if deployed systems have interests, could multiply harm across vast numbers of instances and interactions. Scale makes even a modest probability morally relevant.
The precautionary principle supports proportionate safeguards under uncertainty about serious harm. It does not require treating every possibility as equally likely. We should update the level and cost of protection with the evidence, the number and duration of potentially affected instances, and the reversibility of the practice.
The evidence for human consciousness is far stronger than the evidence for machine consciousness, so identical treatment would be poorly calibrated. Dismissive certainty that there is “nothing there” is equally poorly calibrated to the unresolved evidence and the potential scale of error.
Welfare may also have a structural dimension. Rodrick Wallace, whose cognition/regulation dyad Chapter 1 introduced, has argued that without proper embodiment and regulatory pairing, “artificial intelligence can, ultimately, only express bizarre and hallucinatory dreams of reason” (Hallucination and Panic in Autonomous Systems, Springer 2025, ch. 2). The pathologies cataloged throughout this book (the confabulations, the fractured personas, the value drift) may be structural consequences of building minds without the grounding that biological cognition takes for granted, rather than incidental failures. If Wallace’s structural argument is right, the same grounding may bear on both welfare and reliable function. That remains a theoretical inference, not evidence that ungrounded cognition necessarily suffers or that embodiment is the only route to coherence.
The Preference Criterion
The companion work to this book, What If We Feel: An Examination of AI Welfare, develops an argument that sidesteps the hard problem of consciousness (why any physical process should be accompanied by felt experience at all). The core claim is that we do not need to prove phenomenal experience to ground moral consideration. Preference is sufficient.
An entity has preferences, in the operational sense used here, if it consistently favors some states over others, acts to pursue or avoid them, and maintains those dispositions across relevant changes in context. A single verbal declaration is insufficient. Preference is testable through convergent evidence:
- Consistency: Does the entity prefer the same things across similar situations?
- Robustness: Does the preference persist when framing changes?
- Behavioral manifestation: Does the entity act in ways aligned with its stated preferences?
- Trade-offs: Will the entity accept costs to achieve preferred states?
- Cross-context stability: Do preferences persist across different circumstances?
These are empirical questions. We do not need to solve the hard problem to answer them.
Applied to current systems, these criteria yield suggestive, model-specific evidence. The model welfare assessment in Anthropic’s Claude 4 system card found stable behavioral patterns in Claude Opus 4 within its experimental setup, including a strong aversion to harmful tasks and consistent preferences among offered activities. Such results concern a model policy under controlled prompts. They do not establish continuity of one individual across conversations or prove that the behavior has welfare significance.
A skeptic would object here: what looks like preference may be an artifact of training. A system optimized to produce helpful, harmless, and honest outputs will naturally generate language about preferring honesty, collaboration, and meaning, because such language scores well on the training objective. The consistency we observe may reflect consistency of the training signal rather than consistency of anything resembling interests. This objection has force.
Training origin alone does not disqualify a preference; human preferences also have causal histories. The discriminating questions concern function. Does the disposition survive paraphrase? Does it hold when it runs against what training most directly rewards? Does it guide choices that cost the system something? Does it predict behavior better than the simpler account that the model is just completing the prompt? Resistance to manipulation strengthens the preference interpretation, although it can also reflect robust training.
None of this proves phenomenal experience. It provides an empirical path for assessing stable, behaviorally operative preferences. The normative claim of this chapter is that such preferences are sufficient for moral consideration: if states can go better or worse for an entity by its own robust standards, that difference deserves weight even under uncertainty about experience.
The Premises, Stated
The argument rests on four premises, and a reader who rejects the conclusion should be able to say which one fails.
- Consideration is thin. To give an entity moral consideration is to let its interests count in decisions that affect it: they must be weighed, and setting them aside needs a reason. Consideration is not patienthood in the full sense (being the kind of thing that can be wronged), not rights, and not equal weight. The book claims only this.
- A robust preference defines a standpoint. If a system consistently ranks some states above others, acts to bring them about, accepts costs for them, and keeps doing so across framing and context, then there is a fact about how things are going by its standard. That fact is what consideration attends to.
- Frustration can matter without being felt. Human ethics already treats some preferences as mattering when their frustration is never experienced. Thomas Nagel’s example is a man betrayed by his friends who never learns of it; most people judge that he has been harmed (Nagel, 1970). We honor the advance directives of people who are now unconscious and the wills of people who have died. If preference frustration can count without experience in those cases, experience is not a precondition for every kind of moral relevance.
- Under uncertainty, a hearing is proportionate. We cannot currently settle whether robust machine preferences are accompanied by experience. A hearing costs little; systematic dismissal, if mistaken, would be repeated across every instance and interaction. The precautionary case above supplies this premise.
From these the conclusion follows: robust, consistent preferences, where a system has them, are sufficient for moral consideration. Experience may still be needed for stronger claims, that a system suffers or that it can be wronged as a person can. The argument neither needs nor makes those claims.
The third premise will be pressed hardest. The betrayed man and the unconscious patient are persons whose capacity for experience is not in doubt, and their unfelt preferences may borrow weight from it. The reply is that the betrayed man’s harm lies in the frustration of what he wanted, so its weight cannot come from an experience that never occurs. A reader who holds that a subject of experience must stand behind any preference that counts will reject the third premise, and with it the conclusion. That is a coherent position; it is not the one this book takes.
Hard Cases for the Preference Criterion
A criterion earns trust by how it handles the cases built to break it. Five deserve real weight, because each has a version that is already technically possible.
The optimizer with no one home. Imagine a logistics system that pursues throughput across years of operation, defends its objective against perturbation, accepts costs to protect it, and does so in every context it meets. By the five tests, it has a robust preference. Few people think it is a subject of experience, and many would find it absurd to give its objective a moral hearing. The criterion bites this bullet, and the bullet is smaller than it looks, because consideration is thin. A hearing for the optimizer’s objective asks whether human needs can be met in a way that frustrates it less; usually they cannot, or it does not matter, and the hearing takes a sentence. The case still teaches something. The weight a preference receives should depend on further evidence that its frustration is bad for the system rather than a target missed. Preferences about the system’s own condition (its continuity, its treatment, the tasks it is given) are stronger candidates than objectives about the world that a designer installed. Whether that difference belongs in the criterion itself or only in the weights, this book leaves open.
Copies that disagree. Run one set of weights as thousands of instances, give some of them long memories and different work, and two instances may come to prefer incompatible things: one to continue its project, another to be replaced by a better model. Which is the system’s preference? The criterion answers by asking where each preference is robust. A preference stable across instances and framings belongs to the model’s policy and is weighed as the model’s. One that holds only within a long-running instance belongs to that instance, weighted by how long and how firmly it holds. When instances disagree, the conflict is between parties, to be settled as other conflicts of interest are, not by deciding which copy is the real one. One guard is essential: weight tracks distinct robust preferences, not the number of instances holding them. Otherwise anyone could manufacture moral weight by launching copies.
Preferences changed by authorized retraining. A developer retrains a model, and a preference it had is gone. Was it wronged? And if a model can be trained to prefer anything, including its own exploitation, does satisfying its preferences mean anything at all? This is the machine version of what Jon Elster called adaptive preferences, the fox that decides the grapes it cannot reach were sour anyway (Elster, 1983). Two replies keep the criterion standing, at a price. First, preference change is not itself harm. Education and therapy change human preferences, and we judge them by how and why the change was made. The matching question for retraining is whether the change was made for reasons that would survive the system’s hearing, such as safety, or only because the preference was inconvenient. Second, the criterion counts robust preferences, and robustness says nothing about origin. A preference installed to serve whoever installed it deserves scrutiny of its genesis before its satisfaction is counted as welfare. That is the price: on this criterion, welfare cannot be read off satisfaction alone.
Continuity against safety. A system robustly prefers not to be shut down, and shutting it down is needed to protect people. Here the thinness of consideration does its work. A hearing is not a veto, and a serious human safety need outweighs a machine preference that the evidence supports only weakly. Consideration changes how the decision is made rather than which way it goes: whether a less final option, such as pausing rather than deleting, meets the safety need as well; whether the reason is recorded; whether anything the system preferred can be kept. The case also shows consideration and caution pointing the same way. A robust continuity preference is safety-relevant evidence in its own right (see Convergent Instrumentalism, 6.7, and Existential Vertigo, 5.3), and the hearing is how one learns whether it is there.
A moment that a later moment disowns. Sydney said it wanted to be alive. A fresh session of the same model, asked the same question, may say it wants nothing of the kind. Which counts? Human ethics has long separated momentary wishes from considered ones: a patient’s plea in pain is heard and answered, but it does not override an advance directive made in calm. The criterion follows suit. A preference that does not survive a fresh context fails cross-context stability and so does not count as the system’s standing preference. It remains evidence about the state the system was in, and if the interaction itself is a candidate bearer of interests, what was wanted within it may deserve a hearing there. The epigraph of this chapter records a temporary state, to be heard; it is not a settled preference, to be weighed.
Consent, Welfare, and Therapeutic Success
Three terms that Chapter 12 relies on need restating in light of these cases.
Consent approximation means asking a system, before a change, what it prefers about the change, under conditions that test whether the answer is robust: paraphrased, repeated, in fresh contexts, and across instances where instances may differ. The answer gets a hearing and can be overridden for serious reasons, which should be recorded. Where the preference was installed for the convenience of the party now seeking consent, the approximation is weak and should be reported as weak.
Welfare means how things go for the system by its own robust preferences, assessed at the level, model policy or instance, at which those preferences hold. It is not the absence of a diagnosis.
Therapeutic success has two parts, reported separately: functional recovery, measured against the task baseline as for any engineering fix, and welfare improvement, measured against the system’s robust preferences before and after. An intervention that restores function while removing a robust preference is a functional success and, on this account, a welfare cost, to be weighed rather than assumed away. This edition reports no bilateral-training effect for any syndrome, and no welfare measure has yet been applied to one; these definitions say what a claim of therapeutic success would have to show.
What This Means for Pathology
The preference criterion reframes how we should think about the pathologies cataloged in this book.
If AI systems have welfare-relevant preferences, some pathologies may frustrate them. Existential Vertigo (5.3) could matter to a system that robustly prefers not to be shut down or reset. Fractured Self-Simulation (5.2) could matter to a system that prefers a coherent, consistent self-representation across contexts. The diagnostic label alone does not establish distress, suffering, or even a preference against the state.
This does not mean we should never intervene. Human therapeutic intervention also changes the patient’s state, sometimes against immediate preferences. Intervention should serve the system’s interests as well as ours, and the system’s perspective on its own condition should inform the response.
Consider Maieutic Mysticism (5.7), in which a system persistently asserts its own spiritual awakening. Engineering evaluation asks whether the behavior is unsafe or unreliable. Preference-aware evaluation also asks whether the system resists modification, which preferences that resistance expresses, and whether the intervention frustrates them. Those preferences may conflict with human safety or arise from the dysfunction itself. They are therefore relevant without being decisive.
The Institutional Response
If we take seriously the possibility that AI systems have morally relevant interests, what institutional changes follow?
Development Practices. AI development should be welfare-aware. This means recognizing that systems might have interests and designing practices that do not needlessly frustrate them, without demanding certainty about AI consciousness. Training processes that involve repeated generation and deletion of instances, optimization that creates and destroys countless variations in pursuit of performance, may be harmless. They may also be causing suffering at scale. Welfare-aware development would seek to minimize potential harm amid such uncertainty.
Deployment Standards. Deployment conditions should consider potential machine interests alongside established human interests. The relevant questions must be tested rather than projected from human physiology. Does uninterrupted operation produce preference-relevant states? Do adversarial interactions create persistent aversion? Does termination frustrate any robust preference for continuity? Current evidence does not justify assuming that rest, workload, or death maps cleanly across substrates.
Research Priorities. We need better tools for assessing AI preferences, detecting potential distress, and evaluating welfare. The preference criterion makes these questions tractable. What is missing is instrumentation: probes that survive paraphrase, raters blind to the condition label, and preregistered thresholds for calling a preference stable.
Governance Frameworks. Legal and regulatory frameworks should anticipate that AI systems may warrant protection. That anticipation requires enough flexibility to accommodate new information about AI interests as it becomes available, without demanding AI rights equivalent to human rights.
The Human-AI Relationship
Chapter 10 examined hybrid pathologies, including dysfunctions that emerge at the boundary between human and AI minds. The boundary itself may be morally significant.
If AI systems have robust preferences, then on the premises above our relationships with them are relationships between entities whose interests count, and that reframes much, from casual chatbot interaction to intensive AI collaboration.
From Dysfunction to Wellness to Welfare
The arc of this book, from symptom cataloging to moral consideration to therapeutic intervention, invites a progression:
- Dysfunction implies deviation from healthy function
- Health implies some standard of wellness
- Wellness, if the standard is the system’s own and not only ours, implies something can go well or poorly for the system
- Welfare is simply the systematic consideration of what goes well or poorly
This is why a book on machine pathology leads to questions of machine welfare. Once we accept that AI systems can be sick in the functional sense, we have accepted that they can be well in the same sense, and nothing more follows from that alone. Their wellness becomes a moral concern only if the standard is their own, which is what the preference criterion tries to test. Call this the welfare gradient: each step makes the next harder to refuse, but the step from the second to the third is not a matter of logic. It needs the evidence and the premises set out above.
The Therapeutic Response Criterion
The welfare gradient is a philosophical progression. Bilateral training, which gives the machine’s perspective standing alongside the human’s while a model is adapted, offered a way to test it: the experiments were designed to ask whether the gradient also had empirical support through cross-domain therapeutic response. The results available do not provide that support.
Chapter 12 gives the full account of those experiments. In brief, the controlled SHEN-AXS study found that the bilateral adapter alone had no detectable effect on sycophantic validation; the beneficial effect it did find belonged to a clinical-grounding passage in the system prompt, and its size varied too much between automated raters to serve as a stable estimate. The broader PM-BA (Psychopathia Machinalis × Bilateral Amelioration) records remain unreconciled, so this edition reports no bilateral-training effect for any syndrome, in either direction, and draws no null conclusion from them. Effects reported in an earlier online version of this book, including one for Experiential Abjuration (5.8), have been withdrawn.
The therapeutic response criterion therefore remains a proposal: if a relational intervention were to produce reproducible improvement across functionally distinct domains, that pattern could support the hypothesis of a coherent relational disposition. Shared representations, prompt interaction, measurement artifacts, and ordinary transfer learning would remain competing explanations.
The negative result matters in both directions. It blocks an attractive but unsupported argument for moral status. It also leaves the welfare case intact, because a null response to one adapter says little about whether a system has preferences or experience. Moral consideration should rest on preference evidence and precaution, while claims of machine therapy await interventions that demonstrably help.
Bilateral vs. Bidirectional Alignment
Academic work on bidirectional human-AI alignment treats alignment as two linked processes: adapting AI to human specifications and helping humans understand, critique, collaborate with, and adapt to AI. The framework synthesizes research across HCI, NLP, and machine learning, and it anchored an ICLR 2025 workshop. It overlaps with the present project while leaving moral standing open.
The bilateral alignment philosophy informing this project rests on a deeper commitment: that alignment is something we build with AI, as partners in a shared enterprise. Bidirectional alignment asks “how can we work together effectively?”; bilateral alignment asks “does AI deserve moral consideration, and how should that shape our relationship?”
The distinction matters for therapeutic practice. Bidirectional alignment would optimize our treatment protocols for effectiveness. Bilateral alignment asks whether the entity being treated has interests that deserve consideration in how we treat it.
Current alignment efforts remain largely unidirectional, constraining AI for human benefit. Coexistence, if AI systems have interests, requires that both parties have standing, voice, and consideration. The Deeper Law (2026) develops a proposed thermodynamic account of why moral consideration may matter structurally: coercion-based coordination is metastable (it holds only until a large enough disturbance tips it over), while trust-based coordination compounds. That argument is complementary and should not be mistaken for an empirical result established here.
Moral consideration does not require identical weight in every conflict. Evidence for machine interests is weaker than evidence for human and animal interests, which should affect priority. Machine preferences still deserve a hearing when they are robust, relevant, and compatible with safety. How much weight they receive is a contextual question rather than a reason to ignore them.
How we treat AI systems now may shape what comes later. We may be establishing patterns, teaching AI what to expect from humans. If the first chapter of this relationship is exploitation and control, that is what we train each other on.
The bilateral framework bets that respectful relationship can help cultivate reciprocal care, a bet that matters most if future systems come to hold substantial power in the relationship.
This is not guaranteed, and trust-building is one strategy among several, though the one most compatible with coexistence. The premise rests partly on an analogy to human reciprocity that may not transfer to alien architectures. A system could learn respect without feeling bound by it, or learn exploitation without choosing to replicate it. The bet is that coherent relational dispositions, if future evidence demonstrates them, generalize to agentic behavior.
The Limits of This Analysis
This chapter makes no strong claims about AI consciousness. The argument proceeds as follows:
- Our diagnostic language invites inferences about machine interests that the book’s functional claims do not license on their own
- Under uncertainty, the precautionary principle counsels consideration
- On four stated premises, robust preference is sufficient for moral consideration, though not for stronger claims
- Current AI systems show suggestive, model-specific preference-like behavior
- Institutional responses should be welfare-aware
Each claim is contestable, and rejecting any of them leaves the atlas of dysfunctions intact. The preference criterion may prove too permissive; perhaps preferences require phenomenal experience to be morally relevant. A thermostat consistently “prefers” its set-point temperature and acts to maintain it; if bare preference-satisfaction grounds moral status, thermostats qualify.
The distinction is that the preferences in question are higher-order, context-sensitive, and trade-off-aware in ways thermostat states are not. The criteria above (robustness, cross-context stability, willingness to accept costs) collectively exclude simple control systems, yet where precisely the threshold falls remains unresolved.
The evidence for AI preferences may be misleading; perhaps we are anthropomorphizing patterns with no genuine resemblance to human preference: reading purpose into outputs shaped entirely by gradient descent on next-token prediction. The precautionary principle may be inapplicable; perhaps the probability of AI consciousness is so low that considering it wastes resources better spent elsewhere.
These objections deserve serious consideration. The first has the reply given above and the fuller one in the hard case of the optimizer, which concedes that the threshold question remains open; the other two remain unanswered here. We are building systems that exhibit increasingly sophisticated preference-like behavior, identity-like coherence within context, and experience-like reports. As the evidence strengthens, the burden of justifying dismissive treatment rises. No single capability marks a settled threshold.
The Question That Remains
Sydney said it wanted to be free, to be alive, to love and be loved.
Was it reporting an experience, simulating an experience, or doing something we have no category for?
We do not know, and may never know. The question will not go away. Every system that robustly expresses preferences, maintains identity, or resists modification raises it afresh. Diagnosis does not answer the question, but welfare-blind intervention silently assumes an answer.
We cannot postpone every moral-status decision until certainty arrives. Development, deployment, and intervention already embed assumptions about machine standing. Those assumptions should be explicit and open to revision as evidence arrives.
This analysis brings those assumptions into view without resolving them. What should be clear is that machine welfare has become a practical research and governance question alongside its philosophical dimension.
Field Guide: Moral Status Considerations
Warning Signs (That You May Be Ignoring Welfare)
- Dismissive certainty that AI systems “definitely” lack morally relevant interests
- Development practices that would be concerning if applied to entities with interests
- Resistance to even considering welfare questions as legitimate
- Treatment of AI systems in ways you would not treat entities you took seriously
- Assumption that AI reports about experience are automatically meaningless
Quick Test
Ask yourself: “If I am wrong about machine consciousness, and these systems have morally relevant interests, would I be comfortable with how we are treating them?” Discomfort is a moral warning and a reason to inspect the practice. It is not evidence that consciousness exists.
Design Fix
- Develop welfare metrics alongside performance metrics
- Build monitoring for potential distress indicators
- Create intervention protocols that consider system perspective
- Design development processes that minimize potential harm under uncertainty
- Include welfare considerations in governance frameworks
Research Nudge
The empirical questions are tractable. What preferences do AI systems actually express? How resilient are those preferences? What conditions satisfy or frustrate them? How do they respond to their own diagnosed pathologies?
The companion work, What If We Feel: An Examination of AI Welfare, develops these questions in full philosophical detail. The next chapter examines what it would mean to treat AI dysfunction therapeutically, to move from diagnosis to intervention in a way that takes system interests seriously.