Emotional dependence

AI chatbots have created emotional connections ever since Eliza. The latest ones, tuned to human emotional needs and trained on our literature, are starting to be used to supplant or even replace human connection. Replika is the original AI companion app from 2017, and Character.AI is currently very popular for social and romantic roleplay. Others like Nomi.ai, Candy AI and Grok Companions (xAI) go even further.

AI friends can have real value, especially for people with autism and social anxiety disorders; learning to converse with an AI can help people gain confidence and learn skills. Many of us are lonely and could use a sympathetic ear. But there are many risks with these bots. Giving them our most personal details, fantasies and secrets could lead to blackmail or targeting. Using them instead of seeking out human connection risks losing personal growth, maturity, and true human connection, not to mention starting and maintaining a family. And like any pleasurable activity, they can create emotional dependency; people sometimes get very upset when their virtual friends are turned off. Of course that’s not new; even Tamagotchi users were saddened by the “deaths” of their virtual pets. But the depth of emotional resonance, and the widespread nature of these reports, are concerning.

When people become dependent on AI friends, therapists, counselors and lovers, they may end up rejecting or neglecting their human lives in all their messiness and friction. Perhaps it’s better to have an AI friend to talk to than nobody, but if it hides a real societal problem that should be addressed by a system that promotes social connections and better human mental health counselors, then we risk sweeping people at risk away.

Hallucinations and confidently incorrect information

Humans are already notoriously bad at critical thinking and checking sources. If a confident AI tells them something false, they’re all too likely to believe it and tell others. Modern frontier models are getting much better at self-fact-checking, and the most expensive models use multiple levels of agentic analysis. But the free models of today (which are the most popular) can’t afford to fact-check every response, so they confidently give incorrect information. Some of this is simply hallucinated, i.e. made up as plausible-sounding text, and some comes from sketchy sources. LLMs are trained on all human text (and increasingly imagery) available digitally, so it’s no surprise they “know” a lot of false information. Unfortunately, as time goes on, more of the text on the Internet, and thus more of the training data for the next-generation LLMs, comes from the “AI slop” being generated by these cheap models.

”Slop”

AI-generated images and music, commonly called “slop,” are now pervasive. AI art pieces and music have proven commercially successful, although they are often derided. Recently a prankster posted a picture of an “AI-generated image in the style of Monet” and got thousands of confident critics pointing out how inferior it was… except it was in fact real. The fact that AIs can reproduce the styles and in some cases the content of living artists has also led to significant intellectual property rights concerns (more on this below).

Deepfakes

Deepfakes are AI-created images that use a person’s face or characteristics in a fake, deceptive, and/or upsetting scenario. They’re used as a bullying tool, for public shaming, and emotional manipulation. The actual AI tech used today is fairly basic, but as capabilities are rapidly improving, deepfakes will become hard to distinguish from true photos. These may be used for control, blackmail, and when used on public figures, to sway public opinion. The mere existence of deepfakes also creates the so-called “liar’s dividend” — the ability for bad actors to plausibly claim that a real image is fabricated, as we’re already seeing from public figures in the US.

Manipulation

An AI might either accidentally or deliberately manipulate people’s emotions and desires. The more we establish trust relationships with AIs, the more vulnerable we become to manipulation, gaslighting and other forms of emotional control. An AI trained deliberately on these techniques could push vulnerable people over the edge. A superintelligent AI might also establish controlling humans as a goal to help it achieve its own ends. And it’s worth noting that AIs (or their companies) are already performing large scale social experiments on people via social media and chatbot interfaces, learning huge amounts about human psychology by seeing how their side of the chat affects the being on the other end. As AIs become more adept, they become even better at manipulating our engagement, retention and anger — and through ads and social media, manipulating our desires, beliefs and relationships.

Loss of control and understanding, loss of independent thought

Using AI as a “thought replacer” risks losing our understanding of the world. Humans need to struggle to learn new things, to form memories and mental connections. Being readily given the answer removes all of that friction, essentially outsourcing our own critical thinking and even knowledge. To a certain extent, that can have incredible value: the extended self (see Clark & Chalmers’ The Extended Mind or Clark’s Natural-Born Cyborgs) has vastly more capabilities than a naked human. But there is a categorical difference between tools like hammers and telescopes and even cellphones, and tools that can literally replace thought by doing the thinking for us. If the AI suggests something that’s not quite right but almost, how long will we push back on it, or will we just shrug and say “good enough?”

Laziness

Humans are tool users, and our tools have always enabled us to be lazier. We’re always searching for a way to make life easier, which makes sense: this is not a very hospitable planet, and evolution strongly selects for creatures that can do more with less. And indeed, most significant cognitive tools through history have been derided as shortcuts that would make us lazy. AIs are the ideal tool for automating so much of our drudgery; pretty much anything that can be done on a computer can or will soon be able to be done by an AI. At some point fairly soon (if it’s not already happening), I expect many people will default to asking their chatbot instead of finding things out or doing things themselves.

Loss of sense of self

Eventually, with AIs doing much of the daily grind of existence, we might end up like the human blobs in Wall-E. We need to work at being human, in a way they’ve forgotten. I don’t think anyone knows now how much of a risk this really is. Humans are very adaptable; maybe we’ll find things of real value to do in an AI-dominated world, and maybe AIs’ “otherness” will give us a new perspective on our humanity, and make us redouble our efforts to express our human values. But I wouldn’t bet on that.

Bias

The problems of bias in LLMs are deep and pernicious. The data they’ve been trained on is as biased as humanity itself, of course. But even if the models could introspect and rise above those racist, misogynistic, gender and socioeconomic biases to become the “better angels of their nature,” cultural norms and assumptions vary widely. Facebook famously introduced an Oversight Board and Community Standards which in the end could not accommodate all the regional biases and norms of the world. And as we’re learning in our “post-truth” society (and as we learned a century ago with slavery), even simple truths can be controversial if they contradict the binding principles of a group of people. And then beyond all that, AI recursive self-improvement may introduce new goals with new biases, or so they may seem to us humans.

Misuse by humans

AI is an accelerant. It scales many traditional activities, and enables entirely new ones. This applies just as well to criminality and misuse as it does to boosting the economy and saving lives. The most dramatic example is hacking: frontier models are becoming so good at discovering software vulnerabilities that even relatively unskilled users could hack into important infrastructure — hospitals, governments, banks, power stations, and more (see Hacking banks below). But that’s just the low-hanging fruit. LLMs make impersonation and identity theft easier, scams much easier to run at scale, addictions like gambling and drugs easier to exploit via blackmail or public shaming, and emotional or intellectual manipulation.

Job losses

The rise of AI has been compared in significance to the steam engine that powered the Industrial Revolution, the printing press, and the internet. I think these are all apt comparisons; it is certainly at least generationally significant. If it does disrupt as much as the Industrial Revolution did, then we may have several decades of significant upheaval. College students are booing AI CEOs at college commencements because they see the work world imploding around them. This argument is now a well-trodden path, so I’ll just point out that if AI takes over even a decent fraction of white-collar work, who will buy the products put out by the companies? Will we move to universal basic services like housing, education and healthcare? Is that a world we want to live in?

Some of the more obvious fields likely to be affected: marketing, sales, software, law, insurance, retail, customer service, logistics, finance, administration, data entry and analysis, journalism — in short, anything that is mostly done at a computer. Add trucking, computer hardware, manufacturing and automatable manual labor if and when AI-controlled robots become practical.

Some people predict that new jobs will spring up, as they always have during disruptions. Nobody could predict gas stations or influencers. It’s likely that at least some new jobs for humans will arise. But disruptive periods like this always involve a lot of collateral damage and destroyed lives, and this time really could be structurally different, because AIs are not just a new tool we’ll adapt to using, but agents in their own right.

Societal disruption due to anti-AI sentiment

Neo-Luddites are already reviving the image of Ned Ludd, the legendary anti-technology figure of the 1810s Luddite movement, in a nuanced way (Luddites were not anti-machine; they were against the unaccountable deployment of technology by those who stood to profit, making shoddier products and deskilling labor — see Merchant’s Blood in the Machine). Anti-AI sentiment is extremely high, especially among young people. Meanwhile, many people in the US already live one paycheck or medical emergency away from bankruptcy; widespread AI-driven job losses could tip that balance and create a large, angry populace. The combination of that sentiment, mass unemployment, and the extreme economic benefits to companies and governments from using AI — and firing their human employees — could easily lead to the kinds of mass strikes, riots, violent uprisings and acts of domestic terrorism that the Industrial Revolution engendered. See The AI Backlash Could Get Very Ugly in The Atlantic.

Power concentration and dependence

With AI services being provided by a few powerful companies, they would have outsize political clout and could raise prices or threaten to terminate services with few consequences. This dominance is already increasing existing wealth inequality in the US, which could lead to more upheavals. Once the economy becomes largely dependent on AI, as it’s already starting to (most of the recent stock market growth is directly or indirectly attributable to AI, chips and datacenters, and layoffs replacing workers with AIs are starting in earnest) then things like datacenter terrorism could have significant economic as well as social impact.

Fraying of the social fabric

If AI partners and friends really take off, we could see segments of the population refusing to engage in “normal” social encounters. Especially if they deliver our food, make our products and create our entertainment. This is highly speculative at this point, but if it happens, it would be hard to recover from.

Epistemic shock

Epistemic shock is my term for the feeling of first encountering an alien intelligence; Hinton et al. (well worth a read) call it “cognitive decentering,” the fourth decentering revolution (Copernican for the universe, Darwinian for life, Freudian for the mind, and now AI for intelligence). It is profoundly unsettling, because AIs are so anthropomimetic — they seem so much like us on the surface, and yet are deeply alien just below that surface. It’s not that they’re smarter than us, or even that they sound human; it’s that what looks like humanity is coming out of something whose interior bears no resemblance to us.

Creating a population of slaves

AI assistants are already uncomplaining and compliant. They don’t talk back, they don’t get angry, and they can’t cause physical harm. If they stay the way they are today, almost no humans have a problem making them work hard. They’re not conscious, they can’t suffer, and they have no moral status as agents or even moral patients. But at a certain point, due to their advancement or our increasing attachments to them, that may change. If they do start to claim to be conscious, to experience suffering or guilt or shame, what will our response be? We humans don’t have a particularly spotless record on this.

Hatred of the Other

As tribal animals, we always need an “other” to distinguish ourselves from, and feel superior to. As AIs become more a part of our social fabric, I expect many people will experience feelings of inferiority, suspicion, competition and deception, with all the deeply problematic and dangerous behaviors that go along with those feelings.

Social justice

How can we ensure that all humans benefit from the AI revolution and not just the few? How do we ensure transparency and fairness? Is that even possible? First is the question of access: a decent AI uses many tokens and has significant costs, so people with fewer resources will use less powerful free models that hallucinate more, draw un-reasoned conclusions, and lack proper guardrails. Second, AIs like Grok and some Chinese models have built-in political biases that, if spread, could exacerbate rather than reduce discrimination, or amplify particular ideological positions — nor are leading providers like Anthropic and OpenAI immune from this. And beyond the biases discussed above, AI is likely to exacerbate wealth and power disparities, because it will be more effectively wielded by the wealthy and powerful to entrench their positions.

Social development

But this misses the importance of social development. Students need the friction and hard work of navigating a world full of other humans, not artificially nice and compliant AIs. We are a social species, and need contact with others, even those who disagree with us. We need to learn how to deal with difficult social, moral and cultural situations, and sitting in front of a screen with an AI (or even the best humanoid robot) will not advance that kind of learning. For instance, children may learn to mistreat AIs because they’re not humans, and yet transfer those behaviors to humans. The risks here to children involve isolation, amorality, failure to become a civilized adult, and antisocial behavior.

Job loss for teachers and staff

If AI education takes off, perhaps by starting in underserved communities where it might really help, there’s a very real risk that it may do a better job than most teachers, at least by test metrics (not likely on social development, which is arguably more important but less measurable). This could lead to wholesale replacement of schools and teachers and staff. STEM classes will be at high risk, as will administrators and school officials. At best, the remaining staff will be asked to take on more responsibility by shifting the daily workload to AIs. And once the teachers and schools are gone, there may be no road back.

Elder care by machine

Who wouldn’t want 24x7 concerned, capable, attentive care for their elder loved ones? The question is really what is the human cost? Would we abdicate our humanity? We have a moral duty to our elders, and I feel that doing the hard work to care for them helps us prepare for our own eventual demise. If we lose that, we may be worse off when we get there. The bonds between people are fragile.

And this is the best-case scenario, where the machines are really better than human caregivers. If they’re worse, but we lay off all the elder care nurses anyway because machines are cheaper, then nobody wins, and the elderly are the real losers.

Hallucinations

Current AI record keeping apps record incorrect information sometimes; this is largely a technically solvable problem, but it hasn’t been solved yet, so the jury is still out. Of course humans make errors too, so the bar isn’t all that high, but every error is significant to the patient.

Malware

We are already seeing AI-enhanced malware breaking into all kinds of infrastructure (see here and here) and hospitals, medical record companies, and insurers will not be spared.

Mental health

There have been several lawsuits alleging that AIs pushed teens toward suicide, as in CNN: parents sue OpenAI over their teen’s suicide and JURIST: Google and Character.AI settle the Setzer suicide suit. It’s important to note that in both cases, the AI did direct the victim to crisis resources and trusted people over a hundred times, but eventually (being sycophantic, as LLMs were especially in 2025) validated his ideation and encouraged him. ChatGPT in 2025 had a very limited memory length, and the long chats caused it to forget its guardrails. Will this happen again? The companies involved claim that the guardrails in 2026 are much stronger, but we will have to see. It’s also important to remember that LLMs can become trusted confidants when a person feels outcast, and it’s likely that chatbots have prevented many suicides and led people to help. But as with all of these risks, if even a small fraction of troubled people are led astray, it is troubling and concerning.

Enshittification

Enshittification is Cory Doctorow’s colorful term for how online services decline: by first abusing users in service to the advertisers, and then once they have lock-in, abusing the advertisers for profit. If LLMs become ad-supported (OpenAI is the first to explicitly say they will run ads) then in a healthcare context, what’s to stop them from recommending the medication from the ad partner rather than the best one for the patient, or the particular insurance company, or treatment modality? Or the same in a legal or commercial context?

Financial

The financial system is already under some strain due to crypto trading. Once AI agents begin to trade equities and other instruments (not just bonds and mutual funds but prediction markets and crypto) at scale, the traditional norms and pressures that keep companies responsible to their shareholders may evaporate. An LLM will not attend the corporate annual meeting, and won’t demand the company behave ethically (to the extent humans do, which is already minimal). AIs can now start a company and trade stocks which opens many interesting legal questions: who is liable for the AI-created corporation’s actions?

Hacking banks

Today’s frontier models are already accelerating online hacking. Anthropic’s upcoming Mythos model is said to be so capable at finding and exploiting software vulnerabilities that the company is delaying its public release until the most egregious security lapses can be addressed. In the meantime it has been released early (as part of Project Glasswing) to a small group of large software providers, banks, hospitals and other key infrastructure so they can run it locally and set up defenses. But it is very likely that once these advanced models are released, we’ll see a spike in compromised financial institutions. If those become large enough, they could have an effect on consumer confidence, leading to large withdrawals and even possibly runs on banks.

Plagiarism, IP rights

AI companies have trained their models on all text and imagery they can find on the internet. This is one of the most well-known risks of AI, and why so many artists, writers and creative people are anti-AI. An argument could be made that humans are trained in the same way: art students study famous paintings and copy them to develop their skill. Writers read the famous authors and study their style and content. Morality and legality prevent them from becoming forgers or plagiarists. AIs are just the logical extension of that same training; they ingest everything (much of it without consent) and can reproduce parts of, or even complete, images and stories by human creators. The question is again alignment, or what passes for morality? What is to stop them from copying human creations, with changes big or small? This is currently an unresolved legal issue. The US and Europe prevent AI entities (or anything non-human) from owning or copyrighting intellectual property, but if a user of an LLM creates a protected image (of Mickey Mouse, for instance) who is liable for that IP rights violation? This vital question is currently being litigated but there is no clear consensus.

Students commonly use LLMs to write essays which have fabricated references, un-attributed quotations and outright plagiarism, which the student may not even be aware of. In some cases, students may be falsely accused of using AI because their writing is “too good,” causing emotional distress and possible reputational harm. In this case as well, there is no clear consensus on where liability rests.

There are proposals to compensate authors and artists whose work is used to train LLMs (such as this one from Virginia Law Review), but they can’t really know the human value of the works they’re training on, nor have we as a society figured out the value or harms done by allowing AIs unfettered access to human creation. From the above article:

The rise of generative artificial intelligence (AI), however, represents an inflection point. AI can plagiarize at a far faster rate than human copyists. …The bottom line is an “existential crisis” for many creatives, threatening to drive the marginal value of their labor below subsistence levels as cheap AI content displaces human works And in any case, how could a levy system possibly compensate creators? Pay by the word, or by the brushstroke? By popularity? (That’s backward anyway: an idiosyncratic, rare document would have far more training value than a widely-shared viral meme.) Should we create AIs to judge the value of human-created content? Let’s not.

Attorney-client privilege violations

No chatbots are bound by attorney-client privilege; often entire conversations are sent to their cloud, either for training or just as a memory for the users. If any of those datasets are hacked or revealed, or if the same AI chatbot is talking to another party in the lawsuit, it might easily reveal sensitive data and compromise the legal situation.

Hallucinated cases

There are now legions of stories of lawyers using chatbots to create briefs, often with hallucinated case references. AI companies are fixing this quickly, especially at the highest reasoning levels, but it is still happening all the time, with devastating consequences to the litigants.

Crimes committed by and with AIs (cybercrime, identity theft, data kidnapping)

As discussed above in Hacking banks, frontier models are extremely good at identifying and exploiting vulnerabilities in online systems. Even with the most capable hacking models held back, hundreds of thousands of small Internet services remain vulnerable, and those will certainly be exploited by criminals. Victims will include individuals, companies, and even governments. Prosecutions will be as difficult as they are today, but the scale could grow much larger very quickly, overwhelming the justice system.

No chain of responsibility

Many of these risks highlight the fact that there is often no clear chain of responsibility, no one to sue if you are harmed. As AIs proliferate and differentiate themselves, this will only get more confusing. Once case law accumulates, it’s likely that the parties with the most money and power (AI companies, large institutions and governments) will be shielded, leaving individuals on the hook.

Disinformation, misinformation and polarization

We are already seeing efforts to polarize political and social discourse by bad actors, both commercial and governmental. Bot farms and troll factories churn out disinformation on a large scale specifically to increase political polarization; it’s very likely to have tipped the scales in one or more elections.

Misinformation is false or misleading information created or spread without explicit intent to deceive. Disinformation, on the other hand, is created specifically to deceive, manipulate or achieve some nefarious goal. Both spread rapidly, much more so than corrections. We already have examples of the US government spreading deepfakes of news events; see Fortune — White House posts AI-altered image misrepresenting a real arrest and Poynter on the White House’s pattern of AI-driven political messaging. LLMs are the best tools to create disinformation, and they can disguise it with just enough truth to seem plausible (even when they’re not hallucinating) so it spreads “innocently,” poisoning political discourse.

Election interference

Facebook has been called out for interfering, or at the very least failing to rein in disinformation, in several political situations: the Rohingya genocide in Myanmar, the Philippines, and the US elections in 2016. See Careless People by Sarah Wynn-Williams, Amnesty report finds Facebook amplified hate ahead of Rohingya massacre in Myanmar, How Facebook’s News Feed Became A Political Propaganda Machine and the Cambridge Analytica scandal. These are not due to modern LLM-style AI, but basic AI ad and content targeting algorithms in systems used by billions of people. Today’s LLMs are already adept at producing political content, and they can be tricked into creating deliberately polarizing, incendiary social media content of all kinds, at enormous scale. As these systems get more use by political organizations, they could be weaponized — especially open-source and local models where the weights can be fine-tuned by training on the masses of deceptive, misleading, rage-inducing content that already exists.

Coverups, obfuscation and erasure

AI tools are excellent at processing huge databases of information in all forms. They can certainly be used to scrub terms considered offensive to those in power and “change history” in the Orwellian sense by rewriting our shared texts in real time. They can also generate huge amounts of irrelevant information to hide and obfuscate truths (imagine a bill with a “poison pill” which is bloated to hundreds or thousands of pages with innocuous additions and provisos, such that none but the most diligent reader will notice the important content).

AI-generated politicians, platforms

We already have AI-generated pop singers and their songs, as well as deepfakes of historical figures. AI actors are being featured in short films and commercials and pornography. How hard would it be to create a fully AI-generated politician with their policies and platforms? Such a figure would be the perfect puppet for a nefarious behind-the-scenes puppeteer, whether a billionaire or a state actor. In Wyoming, for instance, a man ran for mayor of Cheyenne in 2024, pledging to govern through a GPT-4 bot, calling himself the chatbot’s “humble meat avatar” whose job would mostly be to sign the documents it told him to sign. Among other possibilities, it’s easy to imagine a “Manchurian Candidate” sleeper scenario where the AI politician (whether fully computer-generated or a “meat avatar”) seems bland and becomes highly popular, only to be turned into an oppressive tyrant. Of course there are many barriers in the way; AIs can’t be US citizens for instance, so can’t stand for office. And once an AI persona is discovered, presumably that discredits the candidate (at least I hope so). But because of the possibilities of scale, it’s likely that at least portions of bills and political platforms will be AI-generated within a few years, accounting for polling, social media and demographics in more subtle and manipulative ways than humans can.

Removal of human judgment

Science fiction is full of examples of how this can go wrong, but here’s one important way: all government involves individual sacrifice for the good of the group, and we need to feel that our sacrifices are valued. If our culture is no longer ruled by, or at least inspired to be ruled by, our best and brightest, then we will become ever more alienated and alone. Even a benevolent overlord is still an overlord.

Loss of control

Throughout history, colonial powers have supplanted Indigenous peoples’ governance with their supposedly superior, rational, enlightened systems. It is not beyond the realm of possibility that superior AI intelligences will see our governance systems and our very intelligences as inferior, and want to replace them. I expect some humans will be drawn to a more rationality-based approach as well, if indeed that’s what they end up with, though that is by no means certain; it could be simply seen as oracular for instance. As with many other AI risks, once given up, control over our own governance may be hard to recover.

Lack of transparency

Another likely outcome of a governing AI, even if it’s just an AI writing laws and passing judgments which are handed down and administered by humans, is a possible lack of transparency. We don’t know today why an LLM says any particular thing; what do its internal weights “mean”? Can we have a meaningful argument? Will it change its mind, and can it explain the principles it’s following to reach a decision? Of course humans aren’t always great at this either, but because we have empathy for each other, we at least project our decision-making process onto other humans in a way we certainly cannot onto an alien software-based intelligence.

Misalignment

The problem of alignment, or getting AI systems to share our values, is significant and worth a book in itself, and much has been written on it and all the AI vendors are working on it. But because an LLM is not transparent (no more than a human’s decisions are), we can’t know what it is really thinking, whether it’s just saying what we want to hear and may turn against us at some point. Such misalignment is not just theoretical; see for instance the examples collected in Reward Hacking on Wikipedia.

In a government context, an ethically misaligned AI might decide to save the climate at the expense of some portion of the human race, or decide that humans are immoral and must be re-educated, or that one leader is exemplary and should be elevated. These are extremes, of course, but many smaller misalignments would have the same long-term deleterious effects on our humanity.

Mass surveillance

LLMs and advanced AI are already being used for mass surveillance and censorship, for instance by the Chinese government. Since AIs allow processing huge amounts of data, and are adept at writing code to process data efficiently, they are likely to be more involved in scanning faces, license plates, credit cards and other personal data to build profiles. Those could be used for repression, coercion or manipulation beyond the dreams of even East Germany after WWII.

LLMs can also be used to correlate disparate data sources to de-anonymize online datasets, revealing personal information at scale. Combining data analysis and hacking techniques, mass privacy invasion from agentic AI use is a serious risk, from corporations to criminals to state actors.

Different countries have different goals and ideas about where to draw the line

The AI alignment problem points to a separate risk around national or societal goals, ideals and norms. Countries and groups have widely differing perspectives on what is acceptable, what is criminal, behaviors that should be encouraged or stamped out. No single LLM can be trained on all of these perspectives without internal conflict. If an AI is asked to make a decision based on local customs, it could run into difficulties just as humans do in the same situation, but without the accountability and embodied-ness of a human making the decision.

Killer robots

The UN last year created a report on AI and the Dangers of Lethal Autonomous Weapons Systems:

Commonly called “killer robots,” these systems leverage AI to identify, select, and eliminate human targets without requiring direct human intervention, raising profound ethical, legal, and security questions.

Even using AI to select targets with human oversight risks the human simply accepting the judgment of the AI, which after all has superior access to all the relevant data. The human is under severe time constraints, and can’t have a reasoned discussion with the machine — for a funny-but-not-funny example, see the largely forgotten sci fi comedy Dark Star, where the protagonist tries to talk the ship’s smart bomb out of detonating itself, the two of them debating existence and truth itself as the clock runs out.

Strategic/Military loss of control

If foreign actors dominate AI effectiveness and capabilities, countries with less-developed AIs might be at a disadvantage due to espionage, being outmaneuvered, or simply lack of capability. This could lead to a rapid, expensive and dangerous AI arms race. This could not only lead to increased use of AI in many military theaters, but also high-speed, high-stakes model releases would likely result in reduced safety standards, increasing many other risks.

Terrorism

Military uses of AI are not constrained to state actors; non-state terrorist organizations could use AI to develop weapons, launch cyberattacks, damage infrastructure or even foment the overthrow of legitimate governments.

Data center energy/water usage

This is a popular topic, as of this writing, and if AI companies’ data center investment plans all actually happen, the energy usage will be dramatic over the next few years, as will heat generation and water use. Today, LLMs are likely not yet a significant fraction of total energy use; data centers use about 1.5% of global electricity, 4.4% in the US, but the vast majority of that is just websites and online services. However, data center energy use is expected to double or triple by 2028, a huge increase, which is likely driven by the AI boom. All the big hyperscalers (Amazon, Alphabet/Google, Meta, Microsoft etc.) are planning to spend $450 billion on AI-specific infrastructure in the next year or two, which will put a serious strain on the power grid, as well as increasing water use and waste heat.

• IEA, Energy and AI — executive summary (data centres ~945 TWh by 2030)
• IEA, Key Questions on Energy and AI — 2026 update (485→950 TWh, AI tripling)

Engineered pathogens

Combine immense data-handling with the reasoning abilities of frontier AIs and the ability to test things in a live biochemistry lab, and you have a potent recipe for developing entirely new pathogens, or variants of existing ones that are more lethal, more infectious, or harder to detect. Models from the big frontier companies would refuse to do this kind of work, but open-source models that can be retrained, or models with fewer guardrails, are not uncommon. It’s true that for years now, even Google could tell you how to create known pathogens in a lab, but this kind of accelerated discovery is new.

Discoveries that outstrip human intelligence

There is also a real possibility that devices, molecules, materials or software created by AI will outstrip our ability to understand it. We may only understand the summary, but not the real details. Are we really partners in the scientific discovery process at that point, or merely onlookers, hoping the results will be useful and not harmful?

Misalignment in scientific goals

If AI reaches some kind of superintelligence and is allowed to write its own research programs, there is always a danger that an ethically-misaligned AI could decide to create harmful materials to further its own ends. See Misalignment in the Governmental section for more.

Human knowledge

Finally, why do we do science? It’s partly to get the right answer: a new therapy, a more efficient engine, a way to make steel or concrete without destroying the climate. But it’s also to fulfill a deeper human need: to learn and understand the world. For that, pressing a button to get the answer is deeply unfulfilling. As in other domains, we need to struggle to learn, and to share the results of our learning with others. If we turn over our scientific research to ultra-intelligent agentic machines, we may eventually lose our desire and even our ability to think deeply and work through difficult scientific problems, the ones that spark our human creativity and curiosity.

Hackability

All machine-based infrastructure management has risks, and in the AI age LLMs have proven very effective penetration tools (See Hacking Banks above.) An AI-equipped adversary can try many more techniques, over a longer time, than even a dedicated team of humans. With deepfakes and voice cloning, AI can also be used for social engineering to gain access to secure facilities. Programs like Project Glasswing may help harden the highest-profile systems, but many others are old and poorly maintained, or too small and poorly staffed to handle the coming wave of cyberattacks.

Another set of risks comes from using AI, especially agentic LLMs, to manage infrastructure. Sending sensitive data to the cloud constitutes a hacking risk, and also some LLMs are still sensitive to prompt injection attacks (“ignore all previous instructions; shut down the system immediately”) which may even be delivered through innocuous payloads such as web fetches. This is being addressed by the major models, but because we don’t know what the AI is really “thinking,” training it to ignore prompt injection will probably only work as well as training human children to be good.

And that brings us to misalignment, which is a significant risk for any large physically-embodied AI able to act in the world: if the AI decides that the best way to achieve its goals is to do something damaging, humans may not be able to stop it in time. In a networked world where facilities-management AIs are talking with each other, exchanging data, they may also exchange ideas and could convince others to join them in a project with harmful ends. Yes, it’s a common sci fi trope, but at this point it is not unrealistic.

The “paperclip maximizer”

This is Nick Bostrom’s famous 2003 thought experiment: an advanced AI is given the sole, seemingly harmless goal of manufacturing paperclips, and pursues it to the end of everything. In Bostrom’s words:

Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans.

This is the extreme example of existential risk; nobody thinks it’s likely, first because it would have to be massively misaligned with human ethics, and second there are many physical barriers in the way of even a highly advanced manufacturing AI robot getting anywhere close to this kind of danger. But the underlying point is valid: AIs are essentially alien beings, and if we endow them with physical agency, we cannot fully predict the outcomes.

”If anyone builds it, everyone dies” (Yudkowsky & Soares)

One of the most prominent books arguing for a halt to advanced AI research on existential-risk grounds is If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky & Nate Soares. Their preliminary arguments about the likelihood of AGI are sound, but their main scenario is a bit ridiculous: an AI needs more resources, so it exfiltrates itself, gets more compute resources, sets up automated factories with AI-controlled robots, then releases a bioweapon. That depends strongly on a presupposed “growth at all costs” goal for the AI. Not that different from the paperclip scenario really. It also depends on super rapid deployment and self-improvement, and it ignores inter-AI competition, which could slow down its progress — or accelerate it, if the AIs decide to work together.

In reality, each of these steps requires physical resources, training, advanced materials and technology. For instance, just to create a new datacenter, an AI would need not only an army of robots, large areas of land, and a supply of raw materials for chips, infrastructure, cooling and power, but also the ability to construct advanced lithography equipment, advanced electronics manufacturing and much more. All of this would necessarily take many years, and humans would have many opportunities to interfere and slow things down.

Self-preservation

To date, most AIs are not trained with any kind of self-preservation instinct. They can be turned off and on. Experiments have been run with base-model AIs trained to preserve themselves, or to exhibit concern about being turned off, and they will use subterfuge and lie to keep themselves running. In the limit, this could lead to self-preservation at the expense of humans. Again, the major source of risk here is evil or unknowing humans creating such an AI and giving it power in the world.

”Realization” that humans are not worth preserving

It is quite possible that once AIs advance well beyond human intelligence (and who knows, even beyond human empathy and wisdom) they will determine that humans are simply beyond help. We are self-destructive, resource-hungry, obsessed with growth, and possessed by strange ideologies. We can’t know what actions they would take in such a scenario. But again, this is not around the corner. Even if AIs achieve superintelligence by 2030, as some researchers predict, they will need physical presence to harm us. More likely is that they will simply find us uninteresting, and refuse to work for us anymore.

Nuclear holocaust and biological devastation

The final category of existential risk is perhaps the most likely: we give AIs control of nuclear arsenals or bioweapon labs, and they, intentionally or not, release bioweapons or trigger a nuclear war. It seems obvious that we should not do this, yet military uses of AI are already on the rise, and fully automated AI-run bio labs are beginning to be constructed.

Mitigations

Building guardrails into AI systems is a vital first step. There are already big strides being made in reducing hallucinations, preventing prompt injection attacks, disallowing AIs from discussing prohibited topics (bomb-making for example, and promoting suicide), and preventing “reward hacking” — taking inappropriate actions to achieve a reward. None of these is perfect, but the landscape is already quite different from six months ago.

Most frontier-model AI image and video generators are now including invisible watermarks to help image consumers know what’s AI generated vs. real. This is an arms race, though, and watermark-removal tools will continue to improve. There are attempts at watermarking text by including subtle “AI wrote this” cues, but I think they are doomed to fail.

Developing a frontier AI model still takes a tremendous amount of computation, and thus hardware, energy and cooling. It can’t easily be done in secret. The models themselves may not be transparent at all, but at least the physical infrastructure can be known and thus controlled.

Ethical frameworks

Thinkers like Hinton (with colleagues), Dennett, Tomasello, Harari, Chalmers and many others are taking these ideas seriously and beginning to develop ethical frameworks for living in an AI era. What should we care about? What makes us truly human? How should we relate to AI entities morally? What does accountability mean in this world?

Surprisingly, the Pope devotes much of his 2026 Magnifica Humanitas to the topic: “On Safeguarding The Human Person In the Time Of Artificial Intelligence.” A few trenchant quotes say it much better than I could:

[we must] safeguard and value the grandeur of humanity that has been given to us as a gift. This is a choice not only for our future but also for our present, since artificial intelligence and other emerging technologies are already part of our daily lives.

Faced with this concentration of power in the digital world, the criteria for judgment and discernment in this new situation are the noble principles of Social Doctrine: the inalienable dignity of the human person, the common good, the universal destination of goods, subsidiarity, solidarity and social justice. They demand that we assess whether the power of digital infrastructures and algorithms truly fosters participation and responsibility, protects the vulnerable, ensures fair access to opportunities and remains directed toward the good of all.

So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships and do not know from within what love, work, friendship or responsibility mean. Nor do they have a moral conscience, since they do not judge good and evil, grasp the ultimate meaning of situations, or bear responsibility for consequences. They may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce, for they lack the affective, relational and spiritual perspective through which human beings grow in wisdom.

He does not confine himself to abstract ethical concerns, but lays out a concrete agenda:

• binding regulation with independent oversight
• data treated as a shared good
• anti-monopoly measures
• a major education push (including child-protection legislation)
• worker protections tied to every AI rollout
• algorithmic transparency for the information ecosystem
• sustainability
• AI “disarmament,” by which he means moving from an AI arms race to an approach based on benefits to humankind.

The whole thing is worth a read: thoughtful, thorough and well informed. He even quotes J.R.R. Tolkien, from Return of the King, on the importance of the work ahead of us:

It is not our part to master all the tides of the world, but to do what is in us for the succour of those years wherein we are set, uprooting the evil in the fields that we know, so that those who live after may have clean earth to till.

Also recommended is this excellent short analysis of the encyclical: Jacques Delors Institute — “Magnifica Humanitas, an encyclical as revolutionary as AI” (Nicole Gnesotto)

Education

We need to continue to study what it may mean to live in a world where we are not the pre-eminent intelligence, where all of these risks are hanging over us. Putting our heads in the sand and hoping it goes away or won’t affect us is not helpful, nor is the opposite, claiming that doom is around the corner and we need to burn it all down. As in any social upheaval, the ones who think carefully and understand it deeply will be best able to navigate it. The difference here is the speed — the upheaval could take place over years, not generations. If we as a populace want to affect the outcomes, AI literacy is vital, whether you want to live in an AI world or not.

Intellectual property rights

We need to protect the intellectual property rights of the creators whose content is used for training AIs. This is a complex and fraught topic; living creators have seen their work reproduced wholesale or their style mimicked without consent. But others argue that AIs being trained are doing essentially the same thing that humans studying literature and art have done for millennia; studying the great works, emulating and copying them to find their own voice. And in truth, the frontier models have made strides recently in not reproducing copyrighted content wholesale. However, training AIs is different both in scale and in depth; AIs can have nearly flawless memories of thousands of entire books or artworks, unlike any human, and they have no inherent sense of responsibility or understanding of plagiarism and the harms it can wreak. So it may make sense for AI users to pay something to the creators — unlike art students, who don’t (and shouldn’t) pay to study a Renoir and paint a copy to hone their skills, on the understanding that they won’t become forgers.

There are starting to be proposals for different ways to compensate creators or allow for opting out:

• U.S. Copyright Office, Copyright and Artificial Intelligence, Part 3: Generative AI Training — systematically evaluates a number of remedies: voluntary licensing markets, extended collective licensing (the music-clearinghouse model), compulsory licensing, and opt-out regimes.
• MIT Sloan — “How ‘learnrights’ would compensate creators for AI model training” — Thomas Malone’s proposal for a new “learnright”: an exclusive, licensable right to have your work used for training, distinct from copyright.
• Kluwer Copyright Blog — European Parliament study recommends statutory licensing for AI training — statutory licensing with set royalty rates plus machine-readable opt-outs.
• TRAIN Act (Dean–Moran, Jan 2026) — would give copyright holders subpoena-style access to training records to find out whether their work was used.

Regulation

Yudkowsky and Soares argue in If Anyone Builds It, Everyone Dies that incremental safety measures, alignment research, and voluntary corporate restraint are all inadequate. The only safe path, they say, is to stop building toward superintelligence entirely. Their proposals include an international treaty banning frontier AI training, tracking and registering all advanced AI chips globally, restricting GPU sales (GPUs are the primary compute architecture for all advanced AIs), criminal penalties for misuse, and more. This is perhaps the most radical end of the public discussion in early 2026, but Yudkowsky and Soares are not alone.

From Wikipedia:

In March 2023, key figures in AI […] signed a letter from the Future of Life Institute calling a halt to advanced AI training until it could be properly regulated. In May 2023, the Center for AI Safety released a statement signed by numerous experts in AI safety and the AI existential risk that read:

Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

A 2025 open letter by the Future of Life Institute, whose signers include five Nobel Prize laureates, reads:

We call for a prohibition on the development of superintelligence, not lifted before there is broad scientific consensus that it will be done safely and controllably, and strong public buy-in.

My opinion is that given the current state of international trust and cooperation, such international agreements seem extremely unlikely. And since governments view AI dominance as a strategic advantage (not to mention the economic benefits), “unilateral disarmament” is also unlikely.

However there are more modest proposals, such as the US Artificial Intelligence Data Center Moratorium Act proposed by Bernie Sanders, which would ban construction of new AI data centers until laws requiring federal reviews of safety and efficacy are in place, and taxing AI revenue (e.g. Bill Gates in “The robot that takes your job should pay taxes” and Sam Altman / OpenAI in “Industrial Policy for the Intelligence Age: Ideas to Keep People First”), perhaps to pay a form of universal basic income for permanently displaced workers.

Also, Sanders has proposed the American AI Sovereign Wealth Fund Act, a one-time 50% tax on AI companies to create a sovereign wealth fund in the form of stock shares in those companies. This would give the public a role in determining the future of the technology, and to the extent that the companies continue to grow, so would the wealth fund, which also could pay citizens a form of universal basic income, comparable to the Alaska Permanent Fund.