When Effective Altruism Evaluation Criteria Fail: AIIM as an Example

Getting your Trinity Audio player ready...

Effective altruism asks an essential question: How can limited resources produce the greatest positive impact? Its emphasis on evidence, cost-effectiveness, counterfactual reasoning, and intellectual honesty has substantially improved philanthropic decision-making.

However, evaluation criteria are not neutral windows onto reality. They are models. When a project produces direct, measurable outcomes through a known causal mechanism, those models can work well. When a project attempts to build new infrastructure, discover an unknown mechanism, or change the system through which future work is evaluated, the same criteria may systematically undervalue it.

AI Internet-Meritocracy (AIIM) is an example. AIIM proposes continuously evaluating the demonstrated contributions of scientists and open-source developers and distributing donated funds among them. Its potential value is systemic: it seeks to improve how society recognizes and finances knowledge production.

That makes AIIM unusually difficult to evaluate using conventional effective altruism methods.

Effective Altruism Evaluation Works Best on Legible Interventions

Effective altruism commonly emphasizes some combination of:

• the scale or importance of a problem;
• its neglectedness;
• its tractability;
• the expected impact of an additional dollar;
• the strength of the supporting evidence;
• the probability that a proposed intervention will work.

The importance–tractability–neglectedness framework is intended to approximate the marginal good produced by additional resources. In idealized form, importance measures how much good solving a problem would produce, tractability estimates how much progress additional resources can generate, and neglectedness helps estimate how much useful room remains for further work.

These criteria are valuable when an evaluator can identify:

1. a relatively well-defined problem;
2. a specific intervention;
3. measurable outputs;
4. a credible causal relationship between outputs and outcomes;
5. a timeframe in which the results can be observed.

For example, a health intervention may have measurable costs, uptake rates, clinical effects, and implementation risks. Although substantial uncertainty remains, evaluators can build an explicit model.

GiveWell itself stresses that cost-effectiveness estimates have important limitations and that programs should not be judged solely by a numerical estimate. The problem is therefore not that sophisticated effective altruists believe every philanthropic decision can be reduced to one spreadsheet. The deeper problem is that funding institutions still need decision procedures, and projects that fit those procedures naturally become easier to approve.

Legibility Can Be Mistaken for Impact

A measurable intervention is not necessarily more valuable than an unmeasurable one. It is merely easier to evaluate.

This creates a selection effect. Consider two projects:

• Project A applies an established intervention whose results can be estimated within two years.
• Project B creates infrastructure that might improve thousands of future projects, but its effects will be indirect, distributed, delayed, and difficult to attribute.

Even when Project B has greater potential, Project A will usually generate a more defensible cost-effectiveness estimate.

The evaluator may believe they are selecting the project with the highest expected impact. In practice, they may partly be selecting the project with the lowest epistemic cost of evaluation.

This distinction matters because civilization depends heavily on outputs that were difficult to evaluate in advance:

• foundational mathematics;
• general-purpose scientific instruments;
• open-source software;
• standards and protocols;
• research databases;
• new academic fields;
• institutions for coordinating future work.

Their value may appear only after other people build upon them.

AIIM Does Not Produce One Easily Counted Outcome

AIIM is not simply a grant to complete one research project. It is proposed infrastructure for allocating funds across many contributors.

The system is designed to examine publicly available scientific and software contributions, estimate their relative importance, and assign contributors a share of donated funds. Its intended beneficiaries include people who may be poorly served by proposal-based funding: independent researchers, maintainers of scientific software, authors of foundational work, and contributors without prestigious institutional credentials.

Its effects would therefore operate at several levels:

• Direct: money paid to scientists and developers.
• Behavioral: incentives to publish useful work and maintain public research outputs.
• Allocative: movement of funding toward contributions missed by conventional institutions.
• Informational: creation of evaluations, dependency records, audit logs, and reputation signals.
• Institutional: pressure on existing funders to justify their own decisions more transparently.
• Long-term: discoveries or software made possible because previously overlooked contributors remained able to work.

There is no simple unit that combines these effects. “Cost per paper” would reward quantity rather than importance. “Cost per citation” would reproduce citation bias. “Cost per funded researcher” would measure distribution rather than scientific impact. Even economic returns would often appear decades later and would be extremely difficult to attribute.

An evaluator could still construct a model, but a highly numerical model would not necessarily be highly informative.

The Counterfactual Is a Different Funding System

Effective altruism correctly emphasizes counterfactual impact: what happened because of the donation that would not otherwise have happened?

For AIIM, however, the counterfactual is not merely “AIIM receives no money.” It is continued reliance on existing systems for identifying and rewarding scientific contributions.

That requires comparing two institutional worlds:

1. one in which universities, grant committees, publishers, donors, and markets continue allocating most resources;
2. one in which an additional, automated and auditable allocation layer evaluates individual contributions continuously.

Estimating that difference requires answers to difficult questions:

• How much valuable research is currently overlooked?
• How many researchers abandon useful work because they lack funding?
• How much scientific software is under-maintained?
• How much funding is misallocated because evaluators rely on prestige?
• Would AIIM discover work that existing systems miss?
• Would its payments meaningfully change researchers’ behavior?
• Could manipulation, bias, or inaccurate AI evaluation outweigh those benefits?

These are empirical questions, but they cannot all be answered before an alternative system exists at meaningful scale. Demanding complete evidence before building the system creates a circular requirement:

The new allocation mechanism must demonstrate the results of operating before it receives the resources required to operate.

This is one reason infrastructure funding often falls into an evidential trap.

Tractability Can Punish Fundamental Innovation

“Tractability” sounds like a property of the problem. Frequently, however, it measures the evaluator’s present understanding of the solution.

A familiar intervention appears tractable because:

• its causal mechanism has already been studied;
• operational organizations already exist;
• measurement tools have been developed;
• previous funders paid the cost of experimentation;
• failed approaches have already been eliminated.

A new institutional mechanism lacks these advantages. Its uncertainty is partly the price of novelty.

If low uncertainty is consistently treated as evidence of superior impact, philanthropy becomes biased toward exploitation of known opportunities and against exploration of new ones. The result may be locally efficient but globally stagnant.

Open Philanthropy’s concept of hits-based giving acknowledges this problem. It explicitly accepts that many grants may fail when a small number of successes could be transformative. Its use of worldview diversification likewise recognizes that some important disagreements cannot be resolved by a single common model.

AIIM belongs closer to this experimental, hits-based category than to a mature intervention with stable outcome statistics.

Neglectedness Does Not Automatically Imply Value

AIIM appears neglected in several senses. Few funding systems attempt to evaluate every eligible contributor continuously, compensate individual scientists and open-source developers directly, and publish AI-generated reasoning in auditable logs.

But neglectedness alone is weak evidence.

A project can be neglected because:

• institutions have overlooked it;
• incentives discourage anyone from building it;
• it crosses established disciplinary boundaries;
• it threatens powerful incumbents;
• it is technologically premature;
• or it is simply a bad idea.

Therefore, “nobody else is doing this” cannot establish that AIIM deserves funding. It only establishes that the project is not receiving much duplicated support.

The correct question is not whether AIIM is neglected. It is:

Does AIIM have a credible path to producing marginal benefits that existing funding mechanisms are unlikely to produce?

Answering this requires technical review, limited deployment, adversarial testing, and comparison with alternative funding mechanisms—not neglectedness as a standalone score.

Expected-Value Models Become Fragile Under Deep Uncertainty

Expected-value reasoning multiplies possible outcomes by their estimated probabilities. In principle, this allows evaluators to compare high-probability modest benefits with low-probability transformative benefits.

The method becomes fragile when both the outcome and its probability are poorly understood.

Suppose an evaluator estimates that AIIM has:

• a 1% probability of transforming research funding;
• a 20% probability of becoming a useful specialized funding tool;
• a 79% probability of producing little lasting benefit.

These numbers may look analytical, but where did they come from? Another informed evaluator might assign probabilities of 0.1%, 5%, and 94.9%. Because the transformative outcome is potentially very large, small subjective changes can alter the final recommendation by orders of magnitude.

GiveWell has discussed the substantial uncertainty and debatable inputs present even in its comparatively mature cost-effectiveness analyses. For unprecedented institutional projects, uncertainty is greater still.

The answer is not to abandon expected value. It is to distinguish between:

• risk, where possible outcomes and approximate probabilities are known;
• ordinary uncertainty, where parameters can be estimated with wide ranges;
• deep uncertainty, where the relevant model, outcome space, or causal mechanism remains disputed.

AIIM currently contains all three.

Evaluators May Reproduce the Failure AIIM Is Intended to Correct

There is another problem: evaluation systems are operated by institutions and people with their own selection effects.

A committee evaluating AIIM may favor:

• conventional academic credentials;
• established research organizations;
• projects with recognizable disciplinary classifications;
• founders who communicate in familiar institutional language;
• interventions resembling previously successful grants;
• evidence published through accepted channels.

But AIIM is partly intended to evaluate contributors whom those filters may overlook.

This produces a governance paradox. A proposed system for reducing prestige bias must first pass through prestige-sensitive institutions. A system designed to fund unconventional work must first appear conventional enough to obtain funding.

This does not prove that a negative assessment is wrong. It means the evaluator must examine whether its own process has a structural conflict with the type of innovation being evaluated.

AIIM Is Also an Evaluation System—and Can Fail for Similar Reasons

AIIM should not be presented as escaping the general problem of evaluation. It moves the problem.

Instead of relying primarily on grant committees, AIIM relies on AI-assisted assessment, public evidence, governance rules, and human oversight. That introduces new failure modes:

• language models may reward persuasive presentation over genuine contribution;
• contributors may manipulate public records;
• prompt injection may distort evaluations;
• obscure but correct work may be misunderstood;
• popular software may appear more valuable than foundational dependencies;
• majority voting may reinforce conventional opinion;
• identity attacks may create duplicate or fraudulent claimants;
• automated scoring may convert imperfect proxies into rigid payment decisions.

Science DAO has explicitly recognized the need for evidence verification, independent checks, transparent criteria, appeals, governance, and adversarial testing of AI funding systems. A proposed multi-agent architecture could separate tasks such as dependency analysis, reproducibility review, priority checking, and manipulation detection rather than trusting one model to perform every kind of assessment.

Human judgment also remains necessary. AI systems should not be treated as independent final judges of disputes involving their own outputs. Science DAO’s proposed approach combines AI analysis with human voting for governance and contested cases, although human voters introduce their own biases and vulnerabilities. See Why AI Shouldn’t Judge Itself for the underlying argument.

AIIM is therefore not “objective evaluation replacing subjective evaluation.” It is an attempt to construct a different, more scalable and auditable mixture of machine analysis and human governance.

How AIIM Should Be Evaluated

A fair evaluation should neither demand impossible proof nor accept speculative claims at face value. It should decompose the project into testable layers.

Technical feasibility

Can the system reliably:

• retrieve contributors’ outputs;
• connect identities to publications and software;
• distinguish original work from copied or trivial work;
• identify dependencies among contributions;
• produce stable and reviewable assessments;
• publish sufficient reasoning for audits and appeals?

These questions can be tested before large-scale deployment.

Adversarial robustness

Can users obtain higher payments through:

• prompt injection;
• fabricated achievements;
• citation manipulation;
• coordinated voting;
• duplicate identities;
• misleading metadata;
• strategic self-description?

A small adversarial deployment can reveal failure modes more efficiently than abstract debate.

Allocative validity

Does AIIM assign greater value to work that independent domain experts consider more important? Does it detect useful contributions missed by citation counts, credentials, or institutional prestige?

Agreement with experts should not become the sole target—the experts may share the biases AIIM is meant to reduce—but systematic disagreement must be investigated.

Incremental behavioral impact

Do payments cause recipients to:

• continue useful work;
• maintain neglected software;
• publish previously hidden outputs;
• improve documentation;
• make research artifacts more verifiable;
• collaborate across institutional boundaries?

This is more informative than merely counting registrations.

Comparative performance

AIIM should be compared with realistic alternatives:

• expert grant committees;
• research prizes;
• citation-based funding;
• lotteries among qualified proposals;
• quadratic funding;
• donor voting;
• conventional employment by universities;
• retrospective peer review.

The relevant question is not whether AIIM is perfect. It is whether it performs better than available alternatives for at least some allocation tasks.

Governance quality

Evaluators should examine:

• whether decisions can be appealed;
• whether evaluation rules are public;
• whether changes are recorded;
• whether conflicts of interest are visible;
• whether minority expertise can correct majority error;
• whether the system can reverse incorrect decisions;
• whether donors, recipients, developers, and voters exercise excessive control.

A funding algorithm without accountable governance would merely automate institutional power.

Fund Information, Not Only Expected Outcomes

Under deep uncertainty, the best initial grant may be one that purchases information.

A limited AIIM test does not require believing that AIIM will transform science. It requires believing that a carefully designed test could resolve important uncertainty at reasonable cost.

Such a test might reveal that:

• AI evaluation is too manipulable;
• expert disagreement is too large;
• contributors do not trust the process;
• payments are too small to influence behavior;
• governance is captured by coordinated groups.

Those would be negative results for immediate scaling, but valuable results for philanthropic decision-making.

Alternatively, the test might show that AIIM identifies overlooked contributors, generates useful evaluation records, and distributes modest funds with lower administrative overhead than conventional grantmaking. That would justify further experimentation without yet justifying universal adoption.

This is a more defensible funding proposition than assigning a speculative probability to total success.

A Portfolio Can Contain Both Measurable Aid and Institutional Experiments

The tension between measurable interventions and uncertain infrastructure is unnecessary if philanthropy uses a portfolio.

A donor can allocate most funding to well-supported programs while reserving a smaller portion for:

• new evaluation mechanisms;
• scientific infrastructure;
• research into philanthropic methodology;
• institutional experiments;
• projects with unusually large but uncertain upside.

This approach avoids two symmetric errors:

1. funding only what can already be measured;
2. using transformative rhetoric to fund projects without meaningful tests.

AIIM should compete for the experimental portion of such a portfolio. Its case is not that conventional effective altruism is useless. Its case is that effective altruism itself requires experimentation with better ways to identify effective work.

When Effective Altruism Criteria Fail

Effective altruism evaluation criteria are most likely to fail when:

• the intervention creates infrastructure rather than a direct final outcome;
• benefits are distributed among many future users;
• the most important consequences are delayed;
• attribution is intrinsically difficult;
• the project changes the evaluation system itself;
• no reliable historical reference class exists;
• the innovation may create outcomes that evaluators cannot yet specify;
• measurable proxies are weakly connected to the real objective;
• evaluators share the institutional biases the project seeks to correct;
• initial deployment is required to generate the evidence demanded for deployment.

In these cases, the answer is not lower standards. It is a different evaluation strategy: staged funding, explicit uncertainty, adversarial testing, reversible commitments, comparative pilots, and measurement of information gained.

Conclusion

Effective altruism’s central insight remains correct: good intentions are not enough, and resources should be allocated according to their expected consequences.

But “use evidence” must not become “fund only interventions whose effects are already easy to measure.” That rule would systematically favor mature, legible activities and could prevent the creation of institutions that make future philanthropy, science, and software production more effective.

AIIM is a useful example because it sits on both sides of the problem. It may offer a new way to evaluate contributions that conventional institutions miss. At the same time, it is itself an unproven evaluation mechanism vulnerable to manipulation, bias, and governance failure.

The rational response is neither immediate rejection nor uncritical enthusiasm.

It is experimentation.

A small, transparent, adversarially tested deployment can determine whether AIIM deserves to grow. More broadly, effective altruism should treat improvements to society’s evaluation and funding infrastructure as legitimate objects of inquiry—even when their value cannot yet be reduced to a clean cost-effectiveness ratio.