Should AI Evaluate Researchers, Research Outputs, or Both?

Getting your Trinity Audio player ready...

AI should evaluate both research outputs and researchers—but not in the same way or with equal weight.

Research outputs should be the primary unit of scientific evaluation. Papers, datasets, proofs, software, experimental results, replications, and other concrete contributions can be examined for quality, originality, rigor, usefulness, and reproducibility.

Researchers should be evaluated more cautiously. Person-level assessment may be necessary for hiring, grants, leadership roles, or access to long-term funding, but it introduces greater risks of prestige bias, historical lock-in, discrimination, and self-reinforcing rankings.

The best design is therefore a two-layer assessment system:

1. Evaluate each research output on its own merits.
2. Construct a limited, contextual researcher profile from the assessed outputs and other documented contributions.

AI should not begin with the question, “Is this person an excellent scientist?” It should begin with, “What has this person produced, and how strong is the evidence that these outputs are valuable?”

The Difference Between Evaluating Research and Evaluating Researchers

The distinction may appear minor, but it changes the entire architecture of a funding or assessment system.

Research-output evaluation

Output-level evaluation examines identifiable scientific objects, such as:

• journal articles and preprints;
• mathematical proofs;
• datasets;
• laboratory protocols;
• source code and scientific software;
• replications and negative results;
• peer reviews;
• theoretical frameworks;
• patents or practical applications;
• educational and research infrastructure.

The relevant question is:

How scientifically valuable is this particular contribution?

Researcher-level evaluation

Researcher-level evaluation attempts to estimate a person’s broader capability or expected future contribution.

It may examine:

• the quality and consistency of previous work;
• technical expertise;
• successful completion of earlier projects;
• research integrity;
• collaboration and mentorship;
• openness and data-sharing practices;
• ability to identify important problems;
• contributions that do not appear in conventional publications.

The relevant question is:

How likely is this researcher to produce valuable work under the proposed conditions?

These questions overlap, but they are not identical. An excellent researcher can produce a weak paper. An unknown researcher can produce an exceptional result.

Why Research Outputs Should Be Evaluated First

The strongest argument for output-first evaluation is epistemic: science advances through contributions, not reputations.

A theorem does not become true because its author works at a prestigious university. An experiment does not become reproducible because its principal investigator has a high h-index. A software package does not become reliable because it was developed by a famous laboratory.

The San Francisco Declaration on Research Assessment, commonly known as DORA, recommends evaluating research according to its scientific content rather than using the journal in which it appeared as a proxy for quality. DORA also emphasizes that research assessment should recognize outputs beyond conventional journal articles.

An output-first system provides several benefits.

It reduces prestige bias

Traditional evaluation frequently relies on indirect signals:

• university affiliation;
• journal reputation;
• citation counts;
• academic rank;
• awards;
• previous grants;
• recommendations from influential scientists.

These signals may contain information, but they can also reproduce existing hierarchies. Once a researcher receives early recognition, obtaining later recognition becomes easier. Conversely, researchers outside major institutions may remain invisible even when their work deserves examination.

AI can reduce this problem by evaluating the content before revealing identity-related information. For example, an initial model could analyze a manuscript, proof, dataset, or repository without receiving the author’s name, institution, country, academic title, or prior funding history.

This does not eliminate bias, because writing style, research topic, citations, and access to equipment may still reveal social information. Nevertheless, it creates a stronger barrier against direct prestige-based reasoning.

It gives independent researchers a fairer entry point

Researcher-level evaluation often disadvantages people who lack conventional credentials. An independent mathematician, open-source developer, citizen scientist, or early-career researcher may have little institutional history to evaluate.

Output-level assessment allows such contributors to present something concrete:

• a proof that can be checked;
• software that can be tested;
• data that can be inspected;
• an experiment that can be replicated;
• a useful synthesis that can be compared with existing literature.

This is especially important for systems such as AI Internet Meritocracy, where funding is intended to follow demonstrated scientific value rather than institutional position alone.

It makes decisions easier to audit

A researcher score can be vague. Why was one person rated 83 and another 61?

An output assessment can be decomposed into narrower claims:

• Was the methodology appropriate?
• Are the conclusions supported by the evidence?
• Is the work genuinely novel?
• Can the data be inspected?
• Does the code reproduce the reported results?
• Are mathematical steps valid?
• Does the output solve an important problem?
• Has the contribution been used by other work?

Each claim can be connected to evidence, uncertainty, and reviewer disagreement. This makes automated evaluation more explainable and easier to contest.

It allows scores to change as evidence develops

The value of a research output is not always known at publication.

A paper may initially appear important but later fail replication. A little-noticed software library may become essential infrastructure. A mathematical construction may acquire major applications years after publication.

Output-level evaluation can therefore be dynamic. Scores may change as new evidence appears:

• successful or failed replications;
• formal verification;
• corrections or retractions;
• downstream dependencies;
• citations with substantive context;
• use in clinical, industrial, or public-policy settings;
• incorporation into later proofs or theories.

The researcher’s profile can then be updated from these revised output assessments rather than remaining tied to an early reputation.

Why AI Cannot Evaluate Outputs Alone

Despite its advantages, a pure output-only model is incomplete.

Many funding decisions concern work that does not yet exist. A grant committee must decide whether a proposed project is credible before the final paper, dataset, or discovery is available.

Even retroactive funding systems must sometimes evaluate the people behind an output. Questions of authorship, integrity, responsibility, and capacity cannot always be answered by inspecting the artifact alone.

Future projects require capability assessment

Suppose two teams submit similar proposals for a technically difficult experiment. One team already operates the required equipment and has completed related work. The other has no demonstrated access to the necessary facilities.

Ignoring this difference would not make the process fairer. It would make the prediction less accurate.

Current NIH peer-review rules illustrate this distinction. The NIH framework considers both the importance and rigor of the proposed research and whether the investigators and research environment are sufficient to conduct it.

Researcher-level evidence can therefore be relevant when it is connected to the requirements of a specific project.

Some valuable contributions are distributed across many outputs

A researcher may contribute through:

• maintaining scientific software;
• curating datasets;
• reviewing others’ work;
• mentoring junior researchers;
• organizing collaborations;
• designing standards;
• documenting failed approaches;
• developing research tools;
• identifying errors in influential work.

Evaluating each contribution separately is possible, but a person-level view may reveal sustained service or cumulative expertise that no single output captures.

Integrity and reliability matter

Research evaluation is not only a ranking of ideas. Funding systems must also consider whether participants fulfill obligations.

Relevant evidence may include:

• whether promised data were released;
• whether previous funding was used as agreed;
• whether conflicts of interest were disclosed;
• whether errors were corrected;
• whether collaborators received appropriate credit;
• whether ethical and safety requirements were followed.

These factors should not become an unrestricted personality score. They should be documented, appealable, and limited to behavior relevant to research responsibilities.

The Danger of Turning Researchers Into Scores

Evaluating researchers is much more dangerous than evaluating outputs because a person-level score can become a persistent label.

A low score may prevent someone from receiving the resources needed to produce better work. The absence of future work then appears to confirm the original score. This creates a feedback loop:

1. The researcher receives a low rating.
2. The researcher receives less funding and visibility.
3. Fewer outputs are produced.
4. The lack of outputs is interpreted as evidence of low ability.
5. The rating falls further.

This is particularly harmful for early-career researchers, people changing fields, researchers affected by illness or political instability, and contributors working outside universities.

Reputation can overwhelm current evidence

Once a global researcher score exists, evaluators may stop examining individual outputs carefully.

A weak article from a highly rated scientist may be accepted too easily. A strong article from a low-rated scientist may receive excessive scrutiny or be ignored.

That would reproduce the same failure found in journal-based assessment: replacing evaluation of scientific content with an easier proxy.

Historical data can encode historical discrimination

AI models trained on past academic decisions may learn that successful researchers tend to come from particular institutions, countries, demographic groups, or professional networks.

Even when protected characteristics are removed, proxies may remain:

• postal location;
• institution names;
• writing conventions;
• career gaps;
• collaboration networks;
• publication venues;
• access to expensive equipment.

An AI system may therefore reproduce prestige bias while presenting its decision as mathematically neutral.

A universal researcher score collapses different abilities

Scientific ability is multidimensional.

A person may be outstanding at:

• proving theorems;
• designing experiments;
• writing reliable software;
• generating hypotheses;
• detecting errors;
• collecting difficult data;
• explaining complex results;
• coordinating interdisciplinary work.

Compressing all these abilities into one number discards information. A researcher who is unsuitable for one project may be ideal for another.

The system should therefore estimate task-specific capability, not claim to measure a person’s universal scientific worth.

What Responsible Research-Assessment Frameworks Suggest

Major research-assessment initiatives generally reject simplistic rankings.

DORA argues that scientific content should be assessed directly and that journal-level metrics should not substitute for evaluating individual work.

The Leiden Manifesto proposes that quantitative evaluation should support qualitative expert assessment rather than replace it. It also emphasizes field differences, transparency, and the need to examine indicators regularly for systemic effects.

The Coalition for Advancing Research Assessment, or CoARA, applies these principles to research, researchers, and research organizations. Its agreement calls for recognition of diverse outputs, activities, and practices, using qualitative judgment supported by responsible quantitative indicators.

These frameworks were designed primarily around human assessment, but their principles apply equally to AI:

AI-generated metrics should support reasoned scientific judgment, not replace scientific judgment with an opaque ranking.

A Better Hybrid Architecture

An AI research-funding platform should maintain separate but connected layers of evaluation.

Layer One: Assess Individual Research Outputs

Each output should receive a structured assessment rather than a single unexplained score.

Possible dimensions include:

Dimension	Central question
Validity	Are the claims supported?
Rigor	Were appropriate methods used?
Novelty	What is genuinely new?
Reproducibility	Can others verify or repeat it?
Utility	Does it enable other research or applications?
Importance	How significant is the problem addressed?
Transparency	Are data, code, assumptions, and limitations available?
Robustness	Does the result survive alternative analyses or criticism?
Influence	Has it produced meaningful downstream use?
Uncertainty	How confident should evaluators be?

Scores should be accompanied by evidence and explanations. When different AI agents or human reviewers disagree, the disagreement should be preserved rather than hidden through premature averaging.

Layer Two: Build Contextual Researcher Profiles

A researcher profile should summarize evidence from evaluated contributions without reducing the person to an immutable rank.

A profile might describe:

• demonstrated areas of expertise;
• types of outputs produced;
• reliability in completing previous projects;
• strengths relevant to a proposed task;
• open-science practices;
• collaboration history;
• unresolved concerns;
• uncertainty caused by limited data.

The profile should answer practical questions, such as:

• Has this person demonstrated the skills required for the project?
• Has the researcher successfully completed similar work?
• Are additional collaborators or safeguards needed?
• Does the researcher have access to the required infrastructure?

It should not make unsupported declarations such as “Researcher A is objectively better than Researcher B.”

Layer Three: Match Researchers to Specific Opportunities

The system should evaluate the relationship among three entities:

1. the researcher or team;
2. the proposed work;
3. the funding opportunity.

A scientist’s suitability is conditional. A theoretical mathematician and a laboratory chemist cannot be ranked meaningfully on one universal scale, but each can be assessed for a relevant project.

This matching layer should consider:

• required expertise;
• available equipment;
• project scale;
• collaboration requirements;
• time horizon;
• uncertainty tolerance;
• previous evidence;
• possible conflicts of interest.

This approach is closer to scientific resource allocation than to social ranking.

Layer Four: Update Assessments Over Time

Assessments should be revisable.

New evidence may include:

• a correction;
• an independent replication;
• an identified flaw;
• new software adoption;
• practical application;
• a successful follow-up project;
• discovery that an output was derivative;
• clarification of an authorship dispute.

Both output assessments and researcher profiles should show version histories. Users should be able to see what changed, when it changed, and what evidence caused the revision.

Layer Five: Preserve Human Appeals and Adversarial Review

Automated systems will make mistakes. Researchers must therefore be able to:

• inspect the evidence used;
• challenge factual errors;
• identify missing outputs;
• contest inappropriate comparisons;
• disclose unusual circumstances;
• request independent reassessment;
• submit counterevidence.

High-impact systems should also undergo systematic adversarial testing to identify manipulation, hidden bias, fabricated evidence, and unstable scoring behavior.

The objective is not to prove that an AI evaluator is infallible. It is to make its errors discoverable and correctable.

Should AI Use Citation Counts?

Citation counts can provide evidence, but they should not define either output quality or researcher quality.

Citations may indicate attention or influence, but they can be distorted by:

• field size;
• publication age;
• self-citation;
• citation cartels;
• fashionable topics;
• negative citations;
• review articles accumulating more citations than original discoveries;
• database coverage;
• language and regional inequalities.

A better system should analyze citation context.

It should distinguish among citations that:

• use a result;
• confirm it;
• extend it;
• criticize it;
• mention it only as background;
• copy citations from another source;
• depend on associated data or software.

A small number of substantive downstream uses may be more important than hundreds of superficial mentions.

Should Researcher Identity Be Hidden?

Identity should sometimes be hidden, but not always.

Identity should usually be hidden during initial output assessment

Removing names and affiliations can reduce direct prestige bias. The first-pass evaluator can focus on methods, evidence, reasoning, and contribution.

Identity may be introduced for verification

Later stages may require identity information to determine:

• authorship;
• conflicts of interest;
• access to facilities;
• previous fulfillment of grants;
• ethical approvals;
• relevant technical experience.

Sensitive attributes should not become ranking variables

Nationality, religion, ethnicity, gender, disability, and political affiliation should not be treated as evidence of scientific quality.

However, simply deleting these fields is not enough. Systems must also test whether proxy variables recreate the same discrimination.

Preventing AI From Rewarding Quantity Over Quality

An AI evaluator may be easy to manipulate if it rewards the number of outputs.

Researchers could split one result into many publications, generate low-value papers, create superficial datasets, or produce large volumes of AI-written material.

The solution is not merely to cap publication counts. The system should recognize relationships among outputs:

• duplicate or overlapping work;
• incremental extensions;
• dependency chains;
• consolidated research programs;
• shared datasets;
• reused code;
• genuine independent contributions.

Marginal value matters. The tenth nearly identical paper should not receive the same reward as the foundational contribution that enabled the entire series.

For related safeguards, see Preventing the Prompt-Gaming Problem.

Funding New Researchers Without a Track Record

A hybrid model must avoid making previous success a prerequisite for all future success.

New researchers could be evaluated through:

• small initial grants;
• staged funding;
• technical work samples;
• preliminary results;
• open research plans;
• prediction or replication tasks;
• collaboration with established infrastructure;
• milestone-based release of funds.

This approach gives unknown researchers an entry path without requiring funders to ignore uncertainty.

The system might allocate a limited exploratory budget first, then increase funding when verifiable outputs appear. This is more informative than rejecting applicants simply because they lack conventional credentials.

Evaluating Collaborative Research

Research outputs often have many authors, and contribution is rarely equal.

AI should not divide credit mechanically by author count. Nor should it assume that the first or final author always made the most important contribution.

Where evidence is available, assessment may consider:

• contributor-role statements;
• repository commits;
• experimental records;
• dataset provenance;
• protocol authorship;
• mathematical sections or proofs;
• project-management responsibilities;
• statements confirmed by collaborators.

Even then, contribution estimates should include uncertainty. Private intellectual work and informal collaboration are difficult to reconstruct reliably.

Output evaluation and contribution attribution should therefore remain separate:

The system should first estimate the value of the output, then estimate who contributed what.

Combining these questions too early can distort both.

Evaluating Negative Results and Replications

Person-centered academic systems often reward novelty and visibility. This can undervalue failed experiments, null results, replications, corrections, and error detection.

An output-centered AI system could recognize their actual utility.

A well-conducted experiment showing that a promising hypothesis is false may prevent many laboratories from wasting resources. A replication may establish whether an influential result is dependable. A correction may protect an entire research field from building on an error.

The assessment should ask what information the output adds—not whether it produces an exciting headline.

The Appropriate Role of AIIM

AI Internet Meritocracy should not operate as a machine that permanently sorts scientists from “best” to “worst.”

Its stronger role is to organize evidence about scientific contributions and allocate resources according to transparent, contestable criteria.

A practical AIIM design could:

1. evaluate research outputs through multiple specialized AI agents;
2. estimate novelty, rigor, reproducibility, utility, and uncertainty separately;
3. collect evidence of downstream scientific use;
4. attribute contributions cautiously;
5. construct task-specific researcher profiles;
6. match researchers and teams to suitable funding opportunities;
7. permit human challenges and community review;
8. revise decisions when new evidence appears.

This model represents evidence-based merit allocation, not algorithmic social hierarchy.

Recommended Decision Rule

The appropriate balance depends on the type of decision.

Decision	Primary emphasis	Secondary emphasis
Retroactive reward	Research output	Contribution attribution
Publication review	Research output	Relevant conflicts or integrity
Replication funding	Proposed method	Team capability
Early-stage grant	Proposal and preliminary work	Task-specific researcher evidence
Large infrastructure grant	Project design	Team, governance, and execution record
Hiring	Relevant outputs and skills	Broader professional contributions
Scientific prize	Specific contribution	Attribution and historical context
Long-term institutional funding	Portfolio of outputs	Reliability, leadership, and infrastructure

There is no defensible reason to use the same weighting for every decision.

Conclusion: Evaluate Work Directly and People Contextually

AI should evaluate both researchers and research outputs, but research outputs must remain the evidential foundation.

Output-level assessment is better suited to testing validity, novelty, reproducibility, and scientific utility. It reduces reliance on prestige and creates opportunities for independent and early-career researchers.

Researcher-level assessment remains useful when decisions concern future execution, integrity, coordination, or accumulated expertise. However, it should be contextual, multidimensional, revisable, and subordinate to concrete evidence.

The governing principle should be:

Evaluate scientific work directly. Evaluate researchers only for a defined purpose, using evidence from their work and behavior relevant to that purpose.

AI should help science move from reputation-based judgment toward transparent evaluation. It should not replace one academic hierarchy with a more opaque algorithmic one.