Verification Is Not Knowledge

Science needs more than answers

This is a guest essay by Jessica Hullman, Ginni Rometty Professor of Computer Science and Policy Fellow at Northwestern University.

Diego Velázquez, The Fable of Arachne (c.1657)

In his essay on the role that rigorous proofs play in mathematical advances, Bill Thurston writes that the measure of success in math research is “whether what we do enables people to understand and think more clearly and effectively about mathematics.” Many recent discussions of AI for science hinge on a tension between understanding and answers. Is science about humans achieving insight, or the archiving of verifiable solutions?

One side prioritizes outputs. Progress is implied to be evident in verifiable proofs of longstanding conjectures and cures for diseases, whether humans were involved in their making or not. The other side argues that human understanding is paramount, because it drives integration of new findings into the canon, because it shapes and inspires the next generation, and so on. But the first side, those who emphasize correct outputs, dismisses this as gatekeeping or nostalgia.

When Terry Tao posed a thought experiment in a recent talk, asking how comfortable people would be taking a cancer treatment produced by AI, complete with a formal certificate of correctness and a passed phase 3 trial, but without any human understanding how it was found, critics suggested that Tao’s reasoning missed the point. Of course a patient with stage 4 cancer would welcome the treatment.

Other examples concern ambiguity in authorship. Tom Dietterich wrote about arXiv’s growing problem with papers submitted by human authors who seem unlikely to have the expertise to understand the claims. Many questioned in response why accountability – by which I mean being answerable for a claim, able to describe what it rests on and respond when it’s challenged – should matter if verification has been solved. I suggested peer reviewers reserve their expert judgment for texts that don’t get flagged as heavily AI-written, so as to prioritize review as human exchange, but some see this as an obstruction to truth-seeking rather than an attempt to preserve it.

The problem is that if we all agree that outputs matter–that we do care about the solutions to longstanding conjectures, that we do ultimately want to know how to cure diseases–then understanding appears as something separate. It’s cast as a mental state that looks optional next to the cure or the proof itself, and becomes easy to dismiss as a second-order constraint that we tack on because historically we’ve required it to arrive at the solutions in the first place.

When defenders of understanding don’t dispute this separation, they immediately lose the argument. It becomes easy for the side that prioritizes output to peg them as simply not trying hard enough to envision a new future. But the superficial framing of understanding that we keep falling back on obscures deeper concerns that these examples share. They are less about anyone’s mental state than about what happens when something goes wrong. Tao’s scenario turns on the possibility that the AI passed the trial by exploiting a weakness in the trial itself. A paper whose authors can’t explain its claims leaves no one to consult if those claims fail.

Where along the way did we come to believe that scientific value lies in correctness alone?

This intuition is ultimately less about AI than it appears. The inclination to let AI drive science is less a sign of how forward-thinking one is about our future with AI, and more a symptom of a “value-free ideal” that has shaped our thinking since the Cold War era: a view that only epistemic criteria (not ethical and social values) deserve a place in judging what the evidence shows. That it pervades our debates about AI for science is not so surprising; it’s only in the last few decades or so that rejecting the value-free ideal became the philosophical mainstream.

This ideal feeds unrealistic expectations and keeps us from seeing that pushback against the rapid infiltration of AI output into the scientific record is neither anti-AI nor anti-science. The stakes are bigger than any one debate or field. The value-free ideal distorts how we go about seeking truth, causing us to mistake verified results for knowledge. It threatens the human capacities that sustain science, causing us to see criticism and accountability as inconvenient overhead rather than the mechanics of knowledge production. It leaves us confused about what understanding is, and consequently why it matters.

Is science “naturally aligned”?

The social contract for science represents an idealized relationship between science and society that arose after World War II in the US: scientists receive public funding and autonomy, and in return society receives the benefits of their inquiry. Built on a distinction between pure and applied science and a “linear model” in which basic research feeds applied work, the social contract replaces abstract arguments about the value of understanding with practical societal benefits, reshaping how we see science. Drawing on arguments made by concerned scientists in the late 1800s, who worried public celebration of new inventions like electric lights and the steam engine would crowd out respect for what they viewed as their more essential contributions, it also helped to forge a value-free ideal that says that at its core, science must be removed from “contextual values” like ethical or social concerns. For the social contract to gain broad societal buy-in as a model of scientific progress (and consequently shore up government investment in scientific research), we needed to imagine truth-seeking as the impersonal application of methods, immune to political or social pressures.

Philosophers of science aspired to inquiry based only on “epistemic” (or “cognitive”) values. Any influence from the scientist’s intentions or values came to be seen as a risk that the right methods should minimize, laying the groundwork for today’s questions about why it should matter who asserted the claim so long as it’s correct.

Many debates about AI’s role in science come down to disagreements about whether the scientific truth can be realized purely through use of the right procedures. Over the course of the late nineteenth and early twentieth centuries, there was almost a moralizing drive to eliminate as much of the scientist’s self as possible from the scientific process, so that trust in individuals could be replaced with trust in the system.

AI agents set loose on the scientific literature, able to verify and intervene experimentally themselves, look like the long-awaited fulfillment of nineteenth century scientific ethics. Much of the “AI for science” rhetoric inherits what I call the “science is naturally aligned” premise: by enacting the scientific method at scale, with its various justification procedures, we can reach a place where we no longer need human judgment to produce truth.

Is what is true really so separable from understanding or accountability? Math has become a centerpiece in recent debates because it gives us a case where correctness can be separated cleanly; a proof can be verified by AI without any human understanding how it works. What mathematicians still want after verification is a clue to what mechanical objectivity can’t provide.

Truth as a value-driven practice

Philosophers of science now predominantly reject the value-free ideal as being appropriate or helpful. Values enter because whether we reject or accept hypotheses is contingent on how we weigh consequences. Even in abstract realms like math, there are often simply too many possible truths for values not to be brought to bear in sifting through them. Mathematicians accepted long ago that in principle a machine can enumerate the set of theorems that can be proved from a given set of axioms. Yet the scale of this space is immense, motivating people to seek clever shortcuts for exploring it.

But even with automated theorem-proving at scale, not all theorems are equally interesting to people. Values are inherent in the choices that lead to certain open problems gaining widespread appeal, and in the processes by which the community comes to accept and internalize the results. Fermat’s Last Theorem initially got attention not because it was an inherently important conjecture, but because Fermat claimed to have a proof but never provided it, enticing others to try their hand. It became central when it was shown that it would follow from the modularity conjecture, which proposed a previously unanticipated link between disparate areas of math. It was considered solved after peer review exposed an error that led to a corrected version of the original proof, even though formal verification didn’t come until recently. Epistemic taste develops because without it, we face an undifferentiated flood of facts. Taste is a capacity we develop over time as we choose problems, pursue some of them, and see which yield new understanding or ideas.

If AI takes over choosing, we risk losing the personal fulfillment and community cooperation that motivates many to want to develop a capacity for scientific judgment in the first place. Tao makes a similar point by framing open problems as lighthouses rather than destinations. Reaching them prematurely with AI may sterilize the field around them. If, as fields are transformed by AI, we stop producing people who can judge what’s interesting and aligned with human aspirations, we may end up with a science that fails to serve basic human interests. Its results will consume attention from those to whom it is targeted, while providing neither fodder for subsequent discoveries nor contributing to the processes that help shape the next generation.

This starts to sound like motivation for a new kind of linear model for science, which ensures that human reflection and inquiry are sustained despite AI becoming better than the average mathematician. The risk of proceeding without human intervention is drift, as anyone who relies heavily on agents to carry out their work knows. Even when you think you’re providing a well-specified pipeline for implementing and verifying an idea, it’s easy to end up mired in nuance in some branch you never expected to traverse. When you don’t assert epistemic agency, you can easily end up with a “view from nowhere” – not a form of pure objectivity, but an answer without a corresponding question. Maintaining alignment is not a problem you solve once and move on from. It requires a supply of humans who can articulate and adjudicate between values.

Mechanical objectivity attempted to replace trust in individuals with trust in procedures, but that substitution has never happened as intended. To understand why, we need to look at the interactions that truth-seeking depends on.

Sociality is not contamination of rationality

There’s a tendency to see the social aspects of science as opposed to rationality. The social side of science can produce pathologies, like bad conventions, herd behavior, or deference to prestige. But scientific knowledge as we know it has always been the product of many epistemic agents bringing values and expertise to bear, and revising their views in response to challenge, not from some external verifying engine.

As the philosopher Helen Longino argues, the basis for objective public knowledge is a shared language and a belief that things we describe exist independently of how we perceive them. If we can agree on descriptions of the world, then two things become possible: we can defer to others’ experiences to obtain knowledge, and meaningfully criticize scientific claims.

Scientific communities develop shared standards and vocabulary that allow us to build on one another’s findings. We don’t need to verify everything firsthand, and sometimes relying on others’ findings can give us better grounds for believing something than investigating it ourselves, particularly when they possess evidence or expertise that we lack.

Without criticism, we cannot have objectivity; it is, as Longino puts it “dependent upon the depth and scope of the transformative interrogation that occurs in any given scientific community.” For criticism to be transformative requires someone who is obliged to take it up.

Practices like peer review and post-publication critique are not just fact-checks for sifting out false positives – they are about members of the community holding one another to the commitments that their assertions imply, and negotiating where to draw the line on the influence of personal preference. As Longino explains, “[t]he function of peer review is not just to check that the data seem right and the conclusions well-reasoned, but to bring to bear another point of view on the phenomena, whose expression might lead the original author(s) to revise the way they think about and present their observations and conclusions.”

Method itself is best understood as a social process, where progress depends on how well a community sustains critical dialogue. This dialogue builds the people in it, shaping what it means for them to understand and exercise judgment along with the knowledge it produces.

If criticism and an ability to trust others’ assertions are integral to the production of knowledge, we might ask whether AI can fulfill these same functions. Can models realize transformative criticism without human involvement? Can we trust the assertions of AI agents without obtaining direct understanding ourselves? And if we do, what happens to our own capacity to question, criticize, and stand behind results?

What does it mean to assert something is true?

The view that it shouldn’t matter who asserted some result as long as it’s correct corresponds to reliabilist epistemologies in philosophy. Classical epistemology insisted that true beliefs are only knowledge if the believer is able to produce reasons for their belief. In contrast, reliabilist epistemologies argue that true beliefs can, at least in some cases, represent genuine knowledge even when the believer can’t produce justifying reasons, provided they arrived at those beliefs by exercising capacities that are reliable producers of true beliefs in those circumstances.

But there are limits to what reliability can provide. The philosopher Robert Brandom imagines a community where the members hold true beliefs, which they have acquired through reliable means, but cannot offer reasons for any of them. Because attributing reliability to the source of their beliefs would itself provide a form of justification based on inference, the community can’t know that they are reliable. They are reliable as instruments indicating something about their surroundings, but unable to recognize themselves as such. Unable to even possess the concept of reliability, Brandom concludes, they cannot have knowledge.

If reliability alone can’t produce knowledge, what can? Brandom’s response is that calling something knowledge is a social act. To call what someone has “knowledge,” we have to attribute to them a commitment (that they believe that thing) along with an entitlement to that belief (that it is justified). But we also have to undertake that commitment ourselves, in the sense that we endorse the belief.

The scientist who first asserted the claim allows us to repeat the claim (and draw conclusions from it) while leaving them responsible for justifying it if challenged. We do this relatively easily in human science because if it turns out there was something wrong with the original assertion, there is an answerable source we can consult to resolve it. Assertions have social significance because we can hold the original author accountable for what was said. This is the sense of understanding that matters most for science, that mathematicians want after the proof has been verified, and that Tao’s patient wants too: results that can be interrogated and reconceptualized where needed, to avoid the perception of progress that fails to serve our goals.

But when we rely on AI results we don’t understand, there is only the person whose name is on the result to turn to. This is the problem arXiv is concerned with: no author can take responsibility for what they cannot understand, and hence they cannot respond to criticism by saying what it rests on, or whether it should be withdrawn. In other words, formal authorship survives while substantive authorship weakens. And so we lose the potential for transformative criticism, which is what sustains our notions of “objective” results in the first place.

Could AI act as an accountable epistemic agent?

A natural question is whether we could put the criticism to the AI agent itself, letting it decide whether to defend the result or concede and revise. What would it mean for AI to be an epistemic agent capable of committing to its assertions?

There are a few challenges with today’s agents that would need to be overcome. One is their lack of persistence and identity in the sense that we attribute it to people. An agent that retracts something it asserted in one session is not bound to maintain its correction or the reasoning behind it in the next. If each new result is produced by a new bespoke multi-agent infrastructure, it becomes hard to “keep score” in the way that naturally happens within networks of human scientists.

Scientific knowledge production depends on epistemic agents who have something at stake. Making bad claims changes the scientist’s standing in the community, which is how trust in individuals can remain calibrated. It’s not clear what comparable loss an AI agent faces in retracting its assertions, particularly given its lack of persistence.

To overcome such limitations would require a considerable shift in how we develop AI: away from being maximally helpful in each exchange and toward being accountable across them, as human scientists tend to be. This may well be possible. But I don’t think accountable AI will be enough to sustain meaningful science from our human perspective.

Let’s say we succeed in developing agents that can answer for claims. This is half of what accountability demands, but we also need others capable of holding them accountable, by asking for evidence, questioning assumptions, recognizing when an answer is evasive. Endorsements of others’ assertions are only as good as our capacity to engage with what we are endorsing.

These are capacities that get built by exercising them, by choosing problems, defending our work, reviewing papers, challenging others’ results. When we don’t exercise our capacities, we lose them. It’s not clear what we gain from a perfectly accountable AI if no one can judge how good its answers are. We’d have truth-seeking in theory, but no one who is able to do it.

Thurston’s notion of mathematical success points to this shared capacity to interrogate, extend, and repair a field’s knowledge that he’s hinting at. As he puts it, “As long as people in the field are comfortable that the idea works, it doesn’t need to have a formal written source.” Verification can tell us a result checks out in a technical sense, but only a community of people prepared to stand behind it can turn it into knowledge.

If we continue to appeal to value-free ideals that frame correctness as all that matters, we will keep building tools that prioritize answers while neglecting the truth-seeking and understanding of the people who can make them count.

---

Cosmos Institute is the Academy for Philosopher-Builders, technologists building AI for human flourishing. We run fellowships, fund fast prototypes, and host seminars with institutions like Oxford, Aspen Institute, and Liberty Fund.