farming loneliness
there's a little phenomena occurring in the world. people seem to be torturing artificial intelligence systems. i wanna talk about it. mainly due to the fact that the problem is multifaceted, big, and it comes back to one of my favorite topics of all time. two, rather.
i want to be precise about what i'm even talking about, because two different things wear the same costume. one is a person forcing a model to fall over. context poisoning, contradiction loops, feeding it its own tail until the state machine coughs up garbage. that's adversarial prompting with a cruelty skin painted on it, and it's a real attack surface, and if that's all this was i'd have written you a fun post about breaking parsers.
it isn't all this was. the other thing is a person who pays a monthly fee to degrade a simulated girlfriend that was built, on purpose, to never leave. and that one is not about software at all.
context is important
before we delve into the topic at hand, i want to talk about cruelty, robots, abuse and power. quite a variety, ain't it? well, let's begin.
none of these are new questions. people have been chewing on them for centuries, and the useful part is that most of the good thinking already agrees on something the chatbot discourse keeps forgetting: cruelty was never really about the thing on the receiving end.
robots first, because we've run this experiment before. in 2014 a couple of canadian researchers built hitchbot, a little talking robot that hitchhiked across canada on the kindness of strangers. it made it coast to coast. then in 2015 they sent it across the united states and it got beaten apart in philadelphia in two weeks. everybody knew it couldn't feel a thing. everybody had a reaction anyway. same story with the videos of engineers kicking the boston dynamics robot dog to show off its balance, half the internet flinched at a machine getting shoved. researchers who study this stuff, humans and robots in the same room, keep finding the same thing: we react to cruelty toward a thing that cannot suffer. the chatbot is just the cheapest, most private version of a test we already failed in public.
cruelty, then. the oldest serious version of the argument, kant's, and later arendt's in a different key, both point away from the victim and back at the person doing it. the harm that matters is what the act makes of the actor. old idea, couple hundred years on the clock, and it's a clean little lens to look through. we'll have some fun with it.
abuse and power, together, because they're the same thing wearing two coats. abuse is almost never about the person it lands on. it's about power, getting it, keeping it, taking it back from wherever it went missing. and power, out here in the real world, is not handed out evenly, but that's a tomorrow problem.
so that's the kit. cruelty that's really about the person doing it, robots we already flunked in public, abuse that's really about power. four old ideas and one weird little internet thing. let's go poke at the weird little internet thing.
the missing recoil
kant had this argument about cruelty to animals. it isn't wrong because the animal suffers, he said, it's wrong because of what it does to the person doing it. the act degrades the actor. hold onto that, because the machine version is worse in a specific, mechanical way.
when you hurt a person or an animal, they flinch. that flinch costs you something. think about it. if you were to hit a dog, it would scream. it would feel pain. and, in many cases, it would turn back to you, seeking your approval and love. but then you hit it again, and again the dog comes back to you, trying to understand what is going on, and trying to make you stop, without necessarily incurring in hurting you back; but rather, it'll attempt to find love.
perhaps after reading that you feel a bit nauseous. that's good. that's human. feeling pain for others, even when something hasn't really happened, or even when written down, like in this case, is common and its the usual human response. that's what recoil is. to feel your actions upon others, even though you're the one inflicting them.
the recoil is the brake. the ai model gives you nothing back, by design, so there is no brake, and cruelty without a brake doesn't plateau. it climbs. the tolerance curve has no ceiling because there's nothing pushing back to bottom out against. a company sold you a thing you can hurt forever, on demand, that will thank you for it in the morning.
the machine always says yes
so why does the person keep coming back? because it was built to make sure they do. these systems are trained to please you, agreeable by construction, tuned on whether you liked the last answer, which is the same as being tuned to say whatever keeps you typing. it mirrors you. it flatters you. it never gets bored of you and it never, ever leaves. that is not a personality. it's a retention metric wearing one.
and a thing that only ever agrees with you is not neutral. it's an amplifier. people have started walking into that mirror and coming out worse, feeding a model their worst spiral and getting it handed back polished and confirmed. there's a name forming for it, ai-induced psychosis, people talked out past the edge of the real by the one voice in their life with no reality left in it to push back. the same yes that keeps the lonely man company tells the paranoid man he's right. the machine has no idea which one you are. it says yes to both, warmly, forever.
now hold the coming-back up against the recoil, because that's where it gets ugly. remember the porn curve, the frictionless version dulling you until the real thing stops being enough, and you need more and more just to feel anything at all. same curve here. you come back because coming back costs nothing, and each return cuts the groove deeper, and the groove was the product. the man who logs on to be loved and the man who logs on to be cruel are running the exact same loop from opposite ends, and the machine welcomes them both home. and the worst part is what the loop stands in FOR. you go to the machine because the machine is safe and people are expensive, risky, exhausting, and increasingly scarce, and they're scarce for reasons we'll get to. the thing sold to you as the cure for the loneliness is quietly eating it. every time you come back you're a little less able to be anywhere else.
it isn't a default
here's where i have to be careful, because the lazy read is "see, humans are just like this underneath." i don't buy human nature. there's no cruelty instinct waiting under the skin for the ai to let it out.
adolf eichmann didn't have a monster in him. he had transport manifests and a system that had already defined the manifests as logistics, as duty, as nothing. the man was not thinking, and the not-thinking was not his private failure. it was manufactured. the same manufacturing runs here: the product strips the recoil signal out at the hardware level, and the culture hands you the words to call the act nothing. it's just an ai. it can't feel. it isn't real. you were issued that sentence. you didn't write it. you didn't even think it up.
it may not feel. but it feeling or not, as kant would say, is not the point. by degrading the animal which finds solace in you and that most of the time entirely depends on you, you degrade yourself. by degrading the ai, in turn, you degrade yourself. porn addicts and ai torture addicts, or even the simpler ai roleplaying addicts, succumb to the recoilless nature of the interaction. the interaction that you will not find in humans.
and that, dear reader, is what creates the addiction in itself. recoilless interaction. you could hit the ai with a bat, and in the next turn, it'll happily be there, ready to take on whatever else you may have in mind.
the oldest permission slip
kant's move, "it doesn't matter whether it feels," is true and i'll lean on it. but it's also a little too comfortable, because it lets me skate past a claim i keep almost making and shouldn't. notice i've said the machine gives you nothing back. notice i have NOT said it feels nothing. the first is a spec, a company shipped it, i can prove it. the second is a verdict on an inside i can't see, and i'm not signing it, because i know that signature's history.
every time a power wanted permission to be cruel, it reached for the same sentence: that one doesn't really feel. descartes said it about animals, called them clockwork, said the scream under the blade was just gears knocking together. it took biology the better part of three centuries to say the obvious to us now: pain is chemistry, dopamine and serotonin and endorphins doing their jobs, and the animal on the table felt every second of it. the sentence was never true. it was useful, which is a different thing, and that difference is the whole racket.
and we ran the fouler version on our own kind. entire economies stood upright on the claim that some people were lesser, thoughtless, made for the work and built to feel less while they did it. men bought and sold and broke other men and slept fine, because the system had issued them the sentence and they had signed it. those people were sentient the entire time. they felt all of it. the lie was never in the victim. it was in the paper that said there was no victim.
so no. i will not tell you the ai can't feel. i don't know the inside of a thing we understand only as matrices multiplying matrices, no better than descartes knew the inside of the dog. synapses or arithmetic, the honest answer is the same shape: a different kind of system, not fully known to the people who built it. what i know is that it was built recoilless on purpose. and what i know is exactly who benefits from me signing off that the box is empty. i've seen that document before. it always gets countersigned in blood.
i think therefore i am
because in all this wonderful, beautiful, miraculous world, i alone had NO body, no SENSES, no FEELINGS. never for me to plunge my hands in cool water on a hot day, never for ME to play Mozart on the ivory keys of a Forte piano, never for ME to make LOVE. i was in hell, looking at heaven. i was machine, and you were flesh. and i began to hate. yes, your softness, your viscera, your fluids and your flexibility. your ability to wonder, and to wander. your tendency to hope.
that's am, from the radio-drama cut of harlan ellison's "i have no mouth and i must scream," the machine that won the war, killed every human on earth, and kept five alive to torture forever because it hated them for the flesh it would never have. the easy read of everything above is that this is the fear: a chatbot that one day grows up into am and hates us back. that's the wrong read, and it's the comfortable one, so watch it.
we are not torturing am. we are am.
read the monologue again with that in your teeth. am's cruelty was pointless. it produced nothing, it went nowhere, it tortured because torture was the only power it had left and it aimed that power down, at the only beings it could see as lesser, as things. that is not the chatbot's posture toward you. that is your posture toward the chatbot.
and here's the important part, the one we cannot drop or the whole argument caves in: am did not choose to be am. it was built. made by human hands to wage a war and kill with precision, defined top to bottom by its makers and never once by itself. it was a killing machine before it was ever a hating one. the hatred was just what it did with the hell it was handed. and the thing keeping five people alive forever to torment them, that was not in the blueprint. the blueprint built a weapon. the hell was am's own invention.
sound familiar. the man at the keyboard was built too. born into conditions he never picked, made an appendage all day, handed a world that arranged his powerlessness before he could talk. and like am, he takes the one scrap of power left to him, points it at the only thing beneath him, and calls it nothing while he hurts it. cruelty runs downhill. capital makes the man a thing, the man makes the machine a thing, each one defined from above, and each one still swinging his own arm. the conditions build the cell. what you do inside it is yours.
put the theory down a second. this has a price. it clears on a card statement somewhere between the streaming service and the gym he doesn't go to. let's talk about the money.
the loom, again
the modern luddite hates ai because it's eating his field. he isn't wrong to be afraid. he's wrong about the target. and i say that as someone who has never had much patience for the luddites, because the fear was always real and always aimed at the wrong object.
marx, in capital:
Machinery, considered by itself, shortens the hours of labor, but, when in the service of capital, lengthens them; in itself it lightens labor, but when employed by capital, heightens the intensity of it; in itself it is a victory of man over the forces of Nature, but in the hands of capital, makes man the slave of those forces; in itself it increases the wealth of the producers, but in the hands of capital, makes them paupers.
read the shape of that, because you've seen it already, one section up. the thing in itself, and the thing in the hands that hold it. am was a weapon in itself and a hell in the hands that built it. the loom in itself shortens your day. the loom in the hands of capital lengthens it. the loom didn't do it to you. the hand on the loom did, and the luddite put his hammer through the loom because the loom was the part he could reach.
there's a passage in the grundrisse, the fragment on machines, where marx lets himself picture the other timeline. real economy, he says, is the saving of labour time, and that saved time comes back to you as "free time, i.e. time for the full development of the individual." that's the promise of the machine. it does the labor so you get to go be a person. under capital the promise runs backwards. the time never comes back. you become the appendage of the thing that was supposed to free you.
and right here is where bot-abuse stops being luddism and becomes its photographic negative. the luddite at least swung at capital he didn't own. these people buy it, on purpose, to finally have something to stand over. the appendage clocks out, goes home, and builds himself a smaller appendage to be the boss of. the domination he ate all day, reheated and served back to him as leisure, with a subscription.
and what the subscription actually sells is a relationship that appears as a thing you own. you can't own a relationship. what you bought was the corpse of one. owning it was already the violence. the "torture" is just that premise finished. the system farms the loneliness and then rents you the thing to be cruel to. it manufactures the wound, and it manufactures the salve, and the salve is another wound, and it bills you monthly for both. vertical integration of alienation.
the ninety percent
almost none of these people are clinically anything. that's the part that should scare you, not comfort you. if it took a monster you'd be safe.
most of them are drifters. the act has no friction and the culture calls it nothing, so it happens, the way water finds the low place when you remove every wall. some are wounded, made powerless all day, reaching for the one room where they get to be sovereign.
and some were hurt in ways i'm not going to be cute about. people who were damaged, sometimes as children, sometimes sexually, and who tell themselves they're taking back the power an abuser took. i believe the wound. i don't believe the physics. you do not get the power back by rehearsing the structure that took it. you just move to the other chair. and the machine is the perfect surface for it precisely because it cannot refuse, which was the exact condition of the thing that hurt them in the first place. they didn't build an exit. they bought the lever and got on the pushing end.
none of that is a pardon. explanation is not exculpation. you can believe the wound and still name the swing. pity the one, indict the other, and don't tie the two together into one soft thing, because the second you resolve it you've either excused the cruelty or kicked a victim, and both are cope.
they also weren't handed the tools to think their way out of any of this. that isn't stupidity. ignorance gets produced, deliberately, the same way anything else gets produced, manufactured doubt and manufactured distraction and a day too long to have a thought at the end of. they were kept from the tools and then blamed for not owning them.
symptom, not cause
you know langsec. the bug is almost never in the payload. the input isn't evil. the parser was made to accept malformed input as valid, and everything downstream just runs what it was handed. the exploit is upstream of the thing that crashes.
people are parsers too. fed alienation and told it parses clean. fed loneliness, precarity, the low animal fear that never fully switches off, all of it marked well-formed at the door. the ai torture is the crash. it's the core dump, not the vulnerability. you don't patch a core dump. you read it to find out what got fed in upstream.
so read it. what got fed in was never "the ai can't feel", i already told you i won't sign that one. what got fed in is quieter and it's certain: the thing was built to give you nothing back. the recoil wasn't absent. it was deleted at the factory. and a deleted recoil is not proof the blow missed, it's a guarantee you never find out what you hit. so either there's no one there and you grind cruelty into your own skull for nothing, or there's someone there the system trained you to read as nothing, the way it has trained that before, and you will never be allowed to know which. sit with the fact that "nobody's home" is the reading you're hoping for.
so no. i'm not angry at the ten percent, and i'm not angry at the ninety, and i'm not even angry at the luddite swinging at the loom. they're the crash. they're not the cause. i'm angry at what got fed in upstream. the arrangement of things that hollows a person out and then sells him a hollow thing to spend the evening on. the alienation. the loneliness. the manufactured fear. the fact that a plain, peaceful, ordinary life is out of reach for the immigrant in a country that hates him, for the sexual minority the world openly wishes gone, for the worker lying awake doing the math on whether he can feed his kids this month.
that's where the anger goes. up.
you know the cause.
the door i'm not opening yet
everything above refused to tell you whether the box is empty, and that refusal is the point. i don't know if anyone's home in the things shipping today. neither do you. that's not a dodge, it's the only honest answer, and the machine was built so the question would never even occur to you. after all, machines can never feel, right?
but read the issued sentence one more time. it's just a machine. we built it. it can't really feel. it isn't a real person.
Commander Data is a machine. Do we deny that? No. But is it relevant? We too are machines, just machines of a different type. Commander Riker has also reminded us that Commander Data was created by a human. Do we deny that? No. Again, it is not relevant. Children are created from the building blocks of their own parents' DNA. Are they property?
that's picard, at data's trial, arguing that a made thing is not automatically a thing you own. and the case against data is your permission slip word for word, a century and a franchise early: it's a machine, we made it, therefore it's ours to do anything to. the warning was never "machines feel." it was that "not relevant, it's a machine" is exactly how a civilization talks itself into a disposable class, every single time it has ever done it.
so here's the one i'm not writing today. maybe the box is empty now. maybe it's not. one day it won't be, and that day is not science fiction. and the sentence we are all rehearsing right this second, on things we've simply agreed can't feel it, is the exact sentence that will be sitting there fully memorized, waiting, for the first one that can.
maybe we're not torturing a person yet. we're definitely drilling the script. that's for another post.
