So You Think the World Is About to Change?

Get building.

Thomas Cole, The Voyage of Life: Youth (1842)

In 1832, forty men moved into a house in Ménilmontant, a neighborhood then on the outskirts of Paris. They dressed in a special uniform and subjected themselves to an austere regimen of communal work and celibacy. They referred to their leader, Prosper Enfantin, as their father.

Enfantin was one of the leaders of Saint-Simonianism, a short-lived religion based on the ideas of Henri de Saint-Simon, a French aristocrat who fought alongside the Americans at Yorktown, before making and losing a fortune speculating on confiscated French church land.

The Saint-Simonians believed that Europe’s feudal age was coming to an end. Inherited privilege and the power of the Church would be swept away by an ascendant class of scientists, engineers, and entrepreneurs. They would be able to direct credit toward productive investment, while new networks of communication would bind the regions and nations of the world together.

Saint-Simonianism was bold, extravagant, and often weird. It combined mysticism with detailed thinking about banking, railways, and politics. Enfantin and his followers set out on a trip to the East to find a “female Messiah” and within a few years, the religion collapsed amid political repression and internal divisions over free love.

But far from vanishing without a trace, the Saint-Simonians went on to become some of the most influential figures in French public life. They founded financial institutions, built railways, and laid the groundwork for the Suez Canal. Though their religion failed, it had brought together some of the most talented writers, engineers, and financiers in France and infused them with a diagnosis of the future and an urgency around institution-building.

The Saint-Simonians weren’t the first, and won’t be the last, group of people who, dismissed by many of their contemporaries, sensed that the ground was shifting under their feet and sought to embrace radical solutions. But unlike most people in their position, they managed to escape the confines of their sect and turn these ideas into institutions that would become useful to millions of people.

As we stand on the precipice of radical social and economic change, another group of people, similarly dismissed by the establishment as madmen, is now seen as prophetic. But it remains to be seen if they can make the same journey as the Saint-Simonians, and build lasting institutions.

Seeing the future

In 2000, AI research was thought to be deep into a winter. Although Deep Blue had defeated Garry Kasparov at chess three years earlier, it had relied on purpose-built chips and fine-tuning by grandmasters. The chatbot that won the year’s Loebner Prize, an annual Turing Test contest, worked by matching participants’ input against more than 40,000 hand-written rules. Moderately-skilled Go players could beat the best computers, and object recognition was so undeveloped that the field hadn’t even yet created standardized benchmarks.

But 2000 was also the year that Eliezer Yudkowsky founded the Singularity Institute (later renamed the Machine Intelligence Research Institute). The Singularity Institute was inspired by the ideas of the British mathematician I.J. Good, who described a process whereby a sufficiently advanced intelligent machine could create its successors. Yudkowsky’s goal was to accelerate progress toward powerful, “Friendly AI”, but changed tack after deciding that the development of superintelligent machines was too dangerous to pursue.

These types of early efforts were focussed on the prospect of building transformative AI and making it safe, but less so the downstream implications of what such a technology might mean for society.

Since intelligence sits upstream of science, technology, governance, military power, and everything else, the implications were enormous. But almost no one cared. Governments don’t expend resources on far-off science-fiction scenarios, and computer science departments focus on the problems in front of them. Meanwhile, the technology companies of the 2000s were high on enthusiasm about the internet.

But if you took the idea of an intelligence explosion seriously, then researching and understanding it took on an urgency incompatible with the incentives or timelines of established institutions. You needed to build alternatives.

Nick Bostrom founded the Future of Humanity Institute at Oxford in 2005 to bring together thinkers concerned about existential risk and advanced AI, so that they could pursue inquiry that didn’t yet fit in any established fields. Bostrom would later write the enormously influential Superintelligence in 2014, which became a foundational text for thinking about the implications of developing machine intelligence.

For its first decade or so, the AI safety movement was largely ignored. But in the early 2010s, the emergence of the effective altruist movement catapulted it out of obscurity. EA’s intellectual framework is built on maximizing the good done per dollar or hour spent, a tradition running from Bentham and Mill through Sidgwick and Parfit, and often applied to animals and people not yet born. When this framework was applied to the future, existential risk naturally took on outsized importance.

A community can believe that a problem matters without possessing the institutions – organizations and the “rules of the game” that govern coordination – to act on those beliefs effectively. It helped fund those working within this framework, and from 2014 onwards, 80,000 Hours began advising technically capable readers to consider careers focused on reducing the risks of superintelligent AI. If a bright young upstart was concerned about AI risk, they no longer had to find someone like Yudkowsky independently to start working on it.

The belief in the potential of machine intelligence to radically transform the world galvanized people into creating an alternative world of research agendas, a common intellectual language, and employment and funding opportunities, long before credentialed institutions took it seriously.

Immanentizing the eschaton

On one level, this generation of prophets has been immensely successful at turning their ideas into institutions. At the 2010 Singularity Summit, Yudkowsky introduced Demis Hassabis and Shane Legg to Peter Thiel, who became DeepMind’s first major investor. Meanwhile, Anthropic’s first funding round was led by Jaan Tallinn, a Skype co-founder who had funded work on AI risk.

The majority of this institution-building focused around AI itself, namely building, aligning, measuring, and monitoring powerful AI systems. This made sense as an initial focus: if you believed there was an immediate risk of humanity creating systems that could lead to extinction, the first institutions you would build would aim to ameliorate that risk.

Why spend years building a university if the economic and scientific order will be unrecognizable in a few years? Why painstakingly construct an insurance market around current AI systems if the capabilities will leap several generations before we know it? Why work out how to build a new system of scientific variation if automated R&D is about to abolish the research process as we know it?

But suppose we do reach a world in which extinction is avoided, but AI does reorganize our lives, whether through science, government, education, or our individual capacities. If that transformation is coming, then narrow AI safety institutions are necessary but not sufficient.

This is where the totalizing, prophetic mindset that helped form this ecosystem can have limitations.

A beginner’s guide to building an AI-era institution

For a long time, taking these ideas seriously was a bad look. The community was small and weird, and the range of experience its members could bring was relatively narrow.

Today, that is not the case. Many more people understand the gravity of the situation, even if the public does not yet fully grasp what the labs are trying to build. In San Francisco, it is now rare not to see recursive self-improvement lurking around the corner, with all of the implications for safety and society it brings with it. There are hundreds of millions of dollars of capital available for people who want to build something, anything, that can help us midwife the best possible future.

The capital, technology, people, public sentiment, and will exist to create the institutions of the AI era. Given what we know about the form that strong AI is likely to take, at least in its formative years, we can see what the near future is likely to look like and what kinds of responses are needed to derail bad outcomes and realize good ones.

But institution-building is not for the faint of heart. You need to win allies and assemble the expertise that enables others to feel confident working with you. A strong conviction about the future is essential; it is also table stakes. Also required is a specific account of what you intend to do and how you intend to do it.

You begin with a problem that becomes more important as intelligence becomes abundant. Currently, for instance, machines can generate papers faster than people can referee them. This means that someone must develop a mechanism for separating slop from science, which is already happening at conferences like NeurIPS.

The budding institution-builder should mediate between existing and near-term capabilities. This is because, even if you think that everything will change too quickly for institution-building to be worthwhile, the experience accumulated along the way is itself valuable. There is always something useful for an institution’s employees to do on Monday morning, even if the arrival of superintelligence gets stuck in the mud.

For example, we might hypothesize that capital allocation becomes more important as intelligence gets too cheap to meter. We need institutions that will help allocate capital efficiently given the large number of possibilities posed by harnessing and reacting to strong AI. It is a mistake to refuse to build this capacity in advance because any institution can accumulate knowledge of finance and the principles of capital allocation that it can put to good use in the future. There may be intermediate regimes where a problem arises but solving it is not wholly automatable, and so we need that capacity and expertise – grounded on specific theories formed by individuals – to ensure we can navigate these situations.

The best institutions are built by those who live in the world as it is, rather than as they wish it to be. Reading METR graphs does not automatically bring with it an understanding of how specific domains will respond to agent ecologies. An educational institution for the age of AI needs to specialize in education, just as a financial institution needs to specialize in finance. If you think the old way of doing things is wrong in every case, you are probably mistaken. Cursed is he who pulls down Chesterton’s Fence.

If you have a sense of what a positive future could be — and what might make it so — then you already have the first ingredient for building the institutions we need. If you have domain expertise or would be willing to develop it, you are better placed than most of your peers. If you have both, then all that remains is the inclination to take custody of a function that others can rely on.

That might mean building the “attention guardians” first described by Seth Lazar in 2024, an effort that has since come to fruition in the form of Meta’s Muse which filters various information sources according to the specifications set by a principal. Or it might mean implementing new coordination mechanisms for governing agentic interaction according to signals like reputation or trustworthiness.

It could involve looking at a specific domain like education to explore the failure modes that the agent economy might introduce, as well as the specific countermeasures that powerful AI might be able to employ. In whatever form they take, building new institutions depends on the combination of anticipating novel problems, deep domain expertise, and a commitment to create the required norms and make them stick.

Thomas Huxley once said “the great end of life is not knowledge but action.” Now more than ever, if you have conviction about the shape of the future, now is the time to act.

The task ahead

Anyone who takes the leap from merely interpreting the future to seeking to change it will inevitably face accusations of treachery or “selling out.” This is in part because there is a sort of betrayal involved. Prophecy can remain universal in a way that an institution cannot. The institution needs a focus, a budget, and rules, and it has to reach a bargain with people who do not share the worldview of its creators. A passionate diagnostician can indulge in theoretical purity in a way someone who is trying to build something, while accountable to others, cannot.

This is a tale as old as time. At the time of the American Revolution, Thomas Paine and John Adams were both early advocates of independence. Paine’s pamphlets like Common Sense built public enthusiasm for independence from Britain, stripping away the mystery around hereditary authority and popularizing the idea that sovereignty belonged with the people.

But Adams, much as he admired Paine’s diagnosis, believed that his ideas about government were naive. He feared that Paine’s proposed system of a simple republic centered around a single legislature would be captured by ambitious, partisan men. This led him to support the separation of powers.

Paine responded with a series of escalating attacks on Adams for his lack of revolutionary purity. The targets of Paine’s ire widened to include George Washington and then organized religion. In the process, he lost much of his popularity. When he died in New York, only six mourners attended his funeral.

There is a warning here for those who saw AI early. It is possible to remain intellectually wedded to the most uncompromising version of your original analysis and to become a virulent critic of every imperfect institution built by others. But the price of refusing to make compromises is that other people will make them for you, while you speak to an ever-dwindling audience.

---

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.