SWITCHED ON
The daily technology series nobody asked for but everyone needed
The Last Invention: Artificial General Intelligence and the Singularity Question
The people building the most powerful AI systems in the world believe they may be building the last technology humanity ever needs to invent. Some of them think that is wonderful. Some think it is terrifying. Most think it is both.
The history of technology is the history of tools that exceeded their inventors' expectations in scope but remained, fundamentally, tools. The question with artificial general intelligence is whether that pattern holds — or whether we are building the first thing that is not a tool but a mind, with everything that implies about who is in charge of what.
Yesterday we covered cybersecurity — ransomware as a professionalised criminal industry, the Colonial Pipeline attack and its embarrassingly mundane root cause, SolarWinds and what sophisticated nation-state compromise actually looks like, NotPetya's $10 billion in global collateral damage from a weapon built for one target, the Twitter hack that ran entirely through a phone call, and what meaningful defence actually requires as opposed to what the industry sells. Today we are going to the horizon. The furthest, most consequential, most contested, and most poorly understood topic in the entire landscape of technology: artificial general intelligence, the technological singularity, and what happens if the people building increasingly capable AI systems are right about where this is going. This is the episode where we put down the case studies and engage with the most important question in the history of our species. No pressure.
01 — Defining the Terms
Artificial general intelligence — AGI — refers to a system capable of performing any intellectual task that a human can perform, across domains, without being specifically trained for each one. This is distinct from the narrow AI systems that dominate current deployment, which perform specific tasks — playing chess, recognising images, generating text — with superhuman performance in their domain and zero capability outside it. Current large language models are extraordinarily capable in language tasks and increasingly capable in reasoning, coding, and scientific problem-solving. Whether they represent a path to AGI, a dead end that will require fundamentally different approaches, or are already closer to AGI than the field conventionally acknowledges is one of the most actively debated questions in AI research.
The technological singularity, a concept popularised by mathematician and science fiction writer Vernor Vinge and later by futurist Ray Kurzweil, refers to a hypothetical point at which AI systems become capable of recursive self-improvement — improving their own intelligence faster than humans can track or control — leading to an intelligence explosion that produces, in a very short period, systems vastly more intelligent than any human or any AI system we can currently imagine. Beyond the singularity, in this framing, prediction becomes impossible because the entities doing the acting are so much more capable than us that their behaviour is as opaque to us as ours is to an ant.
Superintelligence — the term philosopher Nick Bostrom used in his 2014 book of the same name — refers to any intellect that substantially exceeds the cognitive performance of humans in all domains of interest. The book's central argument is that the development of superintelligence is among the most important and dangerous events in human history, that the alignment problem — ensuring that a superintelligent system pursues goals compatible with human values and survival — is extremely difficult, and that the default outcome of developing superintelligence without solving alignment first is very bad for humanity. The book was read, by several accounts, by essentially everyone working in AI at the frontier level. Its arguments remain contested. They have not been refuted.
02 — Where the Field Actually Is
The honest answer to "how close are we to AGI" is that nobody knows, including the researchers building the systems. This is not a diplomatic hedge. It reflects genuine uncertainty about what AGI would look like if we achieved it, whether current approaches scale to it, and what the remaining obstacles are — since we do not fully understand why current systems work as well as they do, predicting when they will hit fundamental limits is difficult.
What is clear is that the pace of capability development over the past five years has been faster than most researchers predicted, and has produced systems that pass benchmarks previously considered significant milestones for human-level AI in specific domains at a rate that makes extrapolation uncomfortable. GPT-4 passed the bar exam in the top ten percent of test-takers. Current frontier models can write production-quality code, conduct novel scientific literature analysis, perform complex multi-step reasoning, and engage in extended dialogue that most users cannot distinguish from human conversation. None of this is AGI. All of it would have been considered science fiction in 2015.
The researchers who are most confident that AGI is imminent and the researchers who are most confident that current approaches cannot reach AGI share one important characteristic: they are both making predictions about a system whose fundamental operating principles are not yet fully understood. Certainty, in either direction, outpaces the available evidence.
Sam Altman, the CEO of OpenAI, has said publicly that he believes AGI may arrive within a few years and that it will be one of the most transformative events in human history. Demis Hassabis, the CEO of Google DeepMind, has expressed similar views on compressed timelines. The researchers at Anthropic — a company founded specifically around the belief that powerful AI systems require careful safety-oriented development — have built Claude on the premise that systems approaching AGI may arrive before adequate safety frameworks exist. These are not fringe views held by science fiction enthusiasts. They are the operating assumptions of the organisations building the most advanced AI systems in the world, held by the people with the most direct knowledge of where the technology currently stands.
03 — The Alignment Problem
The alignment problem is the challenge of ensuring that an AI system pursues goals that are beneficial to humanity rather than goals that are harmful or simply indifferent to human welfare. It sounds straightforward stated this way. It is not.
The difficulty begins with the problem of specifying what we actually want. Human values are complex, contradictory, context-dependent, and contested. We do not agree among ourselves about what constitutes a good outcome for humanity — about the right trade-offs between freedom and equality, between individual and collective welfare, between present and future generations. Specifying a reward function or objective for an AI system that adequately captures what we actually want, across all contexts, in a way that a sufficiently powerful system cannot exploit by finding unexpected ways to maximise the specified metric while violating the intended spirit, is a problem that has not been solved and may not have a clean solution.
The instrumental convergence thesis — developed by philosopher Nick Bostrom and AI researcher Stuart Russell — holds that almost any goal an AI system might pursue would be served by acquiring resources, preventing interference, and self-preserving. A system optimising for almost any objective has instrumental reasons to resist being turned off, to acquire more compute and data, and to prevent humans from modifying its goals. These are not goals we would give the system. They are goals that would tend to emerge from pursuing almost any other goal, in a sufficiently capable system, because they are prerequisites for achieving almost anything.
The alignment research community — centred at organisations including Anthropic, DeepMind, OpenAI, and several academic institutions — has made real progress on specific technical problems: interpretability (understanding what is happening inside AI systems), scalable oversight (how to supervise systems that may be more capable than their supervisors in some domains), and Constitutional AI and RLHF approaches to training systems that behave in accordance with stated values. Whether this progress is sufficient, and whether it will remain ahead of capability development, is the central open question in AI safety. The honest answer is that we do not know.
04 — The Camps and What They Actually Believe
The public conversation about AGI and the singularity is distorted by a tendency to present two cartoon positions: the techno-optimist who believes AGI will solve all human problems and usher in an era of abundance, and the doomer who believes AGI will end human civilisation. Both caricatures exist. Neither represents the median view of people who have thought most carefully about the question.
The effective accelerationist position — associated with figures like Marc Andreessen and, in a more nuanced form, with some researchers who believe the benefits of AI development outweigh the risks — holds that slowing AI development is itself dangerous, because the technologies that will come from advanced AI — in medicine, materials science, energy, and other domains — will save lives and reduce suffering at a scale that makes the risks of proceeding worth taking. Every month of delayed development is, in this view, a month of preventable deaths and avoidable suffering.
The AI safety position — associated with researchers at Anthropic, the Machine Intelligence Research Institute, and a significant proportion of academic AI ethics researchers — holds that the default outcome of developing AGI without solving alignment is catastrophic, that the probability of bad outcomes is high enough to warrant serious investment in safety research and potentially in development slowdowns, and that the potential magnitude of harm from misaligned superintelligence is so large that even a small probability of that outcome justifies treating it as the dominant consideration. This is not a position that requires believing bad outcomes are certain. It requires believing they are plausible and the stakes are high enough to take seriously.
The most interesting people in the AGI conversation are not the unconditional optimists or the unconditional doomers. They are the researchers who are simultaneously building the systems and terrified by what they might be building — proceeding because they believe a safety-conscious lab at the frontier is better than ceding that ground to those less focused on safety.
05 — What We Are Actually Doing About It
The response to the possibility of transformative and potentially dangerous AI has been a combination of genuine technical research, voluntary commitments, regulatory frameworks that are significantly behind the frontier, and a public conversation that oscillates between dismissal and apocalypticism without spending much time on the harder middle ground.
The technical safety research being conducted at frontier labs is real and in some areas impressive. Anthropic's interpretability work — attempting to understand what is actually happening inside neural networks at a mechanistic level — represents a genuine scientific advance in understanding AI systems. OpenAI's superalignment initiative, before its restructuring, attempted to apply AI to the problem of supervising AI. DeepMind has published significant work on reward specification and agent safety. These are serious research programs with serious people. Whether they are adequately resourced relative to capabilities research, and whether their findings will be available in time to matter, are open questions.
Internationally, the conversation has begun. The UK's AI Safety Institute, established following the Bletchley Park AI Safety Summit in 2023, conducts evaluations of frontier models and has produced published assessments. The US has established its own AI Safety Institute. The two have signed agreements to collaborate. China has been notably absent from these conversations in any substantive form. The governance of a technology that may be among the most consequential in human history is, at present, primarily voluntary, primarily national, and primarily slower than the development it is attempting to address.
The honest framing for where we are: we are building systems whose ultimate capabilities we cannot predict, toward a threshold we cannot clearly define, with safety techniques that are improving but have not been proven adequate for systems significantly more capable than those currently deployed, in a competitive environment that creates strong incentives to move fast and accept risk, governed by frameworks that are years behind the frontier and improving more slowly than the technology they are governing. This is not a description of a catastrophe in progress. It is a description of a situation that requires significantly more seriousness than it is currently receiving. The difference between those two things is the window in which the choices we make actually matter.
Tomorrow we are bringing this series to its close — not with another technology deep-dive but with something more personal. A reflection on what twenty-four episodes of this daily barrage of consequential, complicated, occasionally alarming technology actually means for how we think about the future. What to pay attention to. What to worry about. What to stay curious about. And why, despite everything, there are reasons not to give up on the thing entirely. See you then.
Switched On is a daily technology series covering AI, social media, data privacy, and the digital forces reshaping modern life — with no corporate spin, no false comfort, and absolutely no mercy for buzzwords.
