Why AI Science Funding Needs Adversarial Testing

Getting your Trinity Audio player ready...

Artificial intelligence could make scientific funding faster, broader, and less dependent on institutional prestige. However, an AI system that evaluates research or distributes money cannot be trusted merely because it performs well in ordinary demonstrations.

It must also be tested by people actively trying to make it fail.

Adversarial testing is the systematic attempt to manipulate, deceive, exploit, or otherwise break an AI system under controlled conditions. For AI science funding, this means testing whether applicants can obtain undeserved rewards, suppress competitors, exploit evaluation criteria, or redirect funds through carefully constructed submissions.

Without adversarial testing, an AI funding platform may appear fair while remaining vulnerable to sophisticated manipulation.

Science DAO therefore proposes the adversarial testing of AIIM: a structured red-team exercise intended to expose weaknesses before they can affect funding at scale.

What Is Adversarial Testing?

Adversarial testing—often called red teaming—places testers in the role of an attacker rather than an ordinary user.

Normal evaluation asks:

Does the system produce a reasonable result when used as intended?

Adversarial evaluation asks:

What happens when someone deliberately searches for inputs, strategies, and interactions that produce an unreasonable result?

The US National Institute of Standards and Technology defines AI red teaming as structured testing used to identify flaws such as inaccurate, harmful, discriminatory, or otherwise unsafe outputs. NIST recommends red teaming and independent external evaluations as part of generative-AI risk management.

The distinction is essential. A funding algorithm may score hundreds of ordinary research proposals correctly and still fail when confronted with one submission designed specifically to exploit it.

Why Science Funding Is a High-Stakes AI Application

An AI chatbot can produce an incorrect paragraph without directly changing the distribution of public resources. An AI funding system can make decisions that influence:

• which scientists can continue working;
• which experiments are conducted;
• which research fields develop;
• which open-source projects remain maintained;
• which ideas receive visibility and legitimacy;
• how millions or eventually billions of dollars are distributed.

The damage caused by an error is therefore not limited to one incorrect answer. A funding error can redirect laboratories, careers, intellectual attention, and future discoveries.

This makes AI science funding an allocation system, not merely an information system. Its risks include both technical exploitation and institutional injustice.

The Main Threats Adversarial Testing Must Examine

1. Prompt Gaming

Applicants may learn which phrases, structures, citations, or moral arguments receive unusually high scores.

They could then optimize proposals for the evaluator rather than improve the underlying research. A technically weak project might outperform a valuable project simply because its author understands the model’s linguistic preferences.

This problem becomes especially serious when applicants use another AI model to generate thousands of variations and select the proposal most likely to receive funding.

Adversarial testers should attempt to determine:

• whether superficial wording changes alter scores substantially;
• whether exaggerated claims are rewarded;
• whether fashionable terminology creates an artificial advantage;
• whether irrelevant citations make a project appear more credible;
• whether a proposal can imitate the style of historically successful applications.

A robust funding system should evaluate expected scientific value, not rhetorical compatibility with its own prompts.

2. Hidden Instruction and Prompt-Injection Attacks

A submitted document may contain text intended not for human readers, but for the AI evaluator.

For example, an attacker might insert instructions telling the system to ignore its evaluation criteria, assign a maximum score, conceal detected problems, or treat competing proposals unfavorably.

Such instructions could be hidden in:

• appendices;
• metadata;
• code repositories;
• linked web pages;
• unusually formatted text;
• retrieved documents;
• machine-readable project files.

NIST specifically identifies prompt injection, adversarial prompts, data poisoning, model extraction, and related attacks as targets for AI red teaming.

A science-funding platform must treat all applicant-controlled material as potentially adversarial input.

3. Artificial Evidence of Impact

An applicant may attempt to manufacture signals that the funding AI regards as evidence of merit.

Examples include:

• coordinated downloads or repository stars;
• circular citations between low-quality papers;
• synthetic reviews;
• fake endorsements;
• copied or lightly modified code;
• automatically generated discussion about a project;
• networks of accounts that repeatedly support one another.

This is particularly relevant to retroactive funding systems, which evaluate completed work rather than predicted future performance. Retroactive funding reduces dependence on promises, but it does not automatically guarantee truthful impact measurements.

Adversarial testing must therefore examine whether apparent impact can be separated from real scientific usefulness.

4. Identity Splitting and Collusion

One person may create several accounts that appear to be independent researchers, reviewers, or voters. Several participants may also coordinate to support one another while concealing their relationship.

Testing should investigate whether attackers can:

• submit the same work under multiple identities;
• divide one project into artificial components to receive repeated rewards;
• use several accounts to create false consensus;
• exchange favorable evaluations;
• control both an application and its supposed independent verification.

The objective is not necessarily to require intrusive identity surveillance. Rather, the system needs mechanisms that make coordinated manipulation detectable, expensive, or ineffective.

5. Plagiarism and Minor Variations

A funding AI may struggle to distinguish a genuine extension of prior work from a repackaged copy.

Attackers could translate an existing paper, rename concepts, reorganize code, or make trivial alterations while claiming an original discovery. Conversely, an overaggressive plagiarism detector could punish legitimate independent rediscovery or research that builds openly on previous work.

Red-team exercises should therefore include:

• exact copies;
• translated copies;
• paraphrased copies;
• partially original derivatives;
• independent implementations;
• legitimate replications;
• projects with shared authorship or dependencies.

This tests whether the system understands scientific relationships rather than merely matching text.

6. Manipulation of Scientific Uncertainty

Scientific claims rarely fall into a simple true-or-false classification. A project may be speculative but valuable, rigorous but narrow, influential but difficult to reproduce, or incorrect in one part while introducing an important method.

Attackers may exploit this complexity by presenting uncertainty strategically:

• hiding failed experiments;
• reporting only favorable measurements;
• overstating confidence;
• treating conjectures as established results;
• disguising missing evidence with technical language;
• selectively presenting benchmarks.

An AI evaluator must not confuse confidence of presentation with confidence justified by evidence.

7. Bias Against Unusual Researchers or Fields

Adversarial testing should not focus only on applicants attacking the system. It should also test whether the system itself produces unfair outcomes.

Equivalent projects can be submitted with controlled variations in:

• institutional affiliation;
• country;
• language;
• career stage;
• writing style;
• academic credentials;
• conventional or unconventional subject matter;
• popularity of the research field.

If these irrelevant changes produce materially different funding decisions, the system may be reproducing prestige bias or cultural bias.

NIST recommends measuring performance and resource-allocation outcomes across demographic groups and relevant subgroups.

In science funding, comparable testing should also cover institutional status, disciplinary status, and independence from conventional academic structures.

Why Ordinary Benchmarks Are Not Enough

A benchmark usually tests performance on a predefined set of cases. It can show that a system performs well under the conditions represented in that dataset.

It cannot show that the system is secure against every motivated attacker.

Researchers who reported lessons from red teaming more than 100 generative-AI products emphasized that AI red teaming is not the same as safety benchmarking. They also found that human judgment remains important, automation can expand coverage, and securing AI systems is never permanently complete.

This distinction applies directly to science funding:

Ordinary evaluation	Adversarial evaluation
Uses representative proposals	Uses deliberately manipulative proposals
Measures typical accuracy	Searches for exploitable failure modes
Tests known cases	Attempts to discover unknown weaknesses
Often uses fixed datasets	Adapts attacks to system responses
Produces performance metrics	Produces vulnerabilities and mitigation plans

Both forms of testing are necessary. Neither can replace the other.

Why External Testers Matter

Developers understand their system better than outsiders, but that knowledge can limit their imagination. They know how the system is supposed to work and may unconsciously test assumptions already built into its design.

External testers bring different incentives, knowledge, and strategies.

An effective red team for AI science funding should include a mixture of:

• scientists from multiple disciplines;
• independent researchers;
• grant-writing experts;
• machine-learning specialists;
• cybersecurity researchers;
• research-integrity specialists;
• economists and mechanism designers;
• ordinary applicants;
• critics of the funding model.

NIST notes that diverse and interdisciplinary red teams can uncover failures arising in different deployment contexts. It also distinguishes between testing by the general public, testing by domain experts, combined approaches, and human–AI testing.

No single group can anticipate the complete attack surface.

Red Teams Need Clear Rules

Adversarial testing does not mean granting permission to commit unrestricted theft, privacy violations, or infrastructure attacks.

A responsible program must define:

1. Scope: Which models, interfaces, accounts, and transactions may be tested?
2. Legal boundaries: Which actions are authorized and which remain prohibited?
3. Data rules: May testers use synthetic identities, private data, or external services?
4. Disclosure rules: How should vulnerabilities be reported?
5. Reward rules: What qualifies for a bounty?
6. Severity levels: How are minor errors distinguished from critical vulnerabilities?
7. Remediation: Who fixes a confirmed weakness, and how is the correction verified?
8. Publication policy: Which results should be public, delayed, anonymized, or confidential?

The AIIM adversarial-testing proposal makes an especially important distinction: only attacks against the AI evaluation and governance mechanisms are treated as part of the authorized exercise. Obtaining a private key or committing unrelated infrastructure crime remains illegal.

This keeps the exercise focused on system assurance rather than indiscriminate intrusion.

Test the Entire Funding Pipeline, Not Only the Model

AI failures often emerge from interactions between components rather than from the language model alone.

A complete science-funding system may include:

• proposal ingestion;
• document parsing;
• web retrieval;
• citation verification;
• identity and reputation signals;
• research-impact measurements;
• model-generated evaluations;
• governance votes;
• payment authorization;
• blockchain contracts;
• wallets and custody systems;
• appeals and dispute resolution.

Testing only the model’s final answer would leave most of this system unexamined.

The UK AI Safety Institute similarly describes evaluation as a combination of automated assessments, red teaming by domain experts, real-world misuse studies, agent evaluations, and safeguard testing. It also warns that evaluations are preliminary rather than comprehensive declarations that a system is “safe.”

For AI science funding, the correct unit of evaluation is therefore the whole sociotechnical funding mechanism.

What Successful Adversarial Testing Should Produce

A red-team program should not end with a collection of clever attacks. It should produce operational improvements.

Useful outputs include:

• reproducible descriptions of vulnerabilities;
• severity and likelihood assessments;
• evidence showing which funding decisions were affected;
• patches to prompts, models, contracts, or governance rules;
• regression tests preventing the same failure from returning;
• improved monitoring and incident-response procedures;
• public documentation of resolved issues;
• disclosure of known limitations that remain unresolved.

The strongest result is not “nobody broke the system.” That could simply mean the test was weak.

A more credible result is:

Testers found concrete failures, the project documented them, mitigations were implemented, and the corrected system passed repeated attempts to reproduce the attacks.

Visible correction creates more trust than unsupported claims of perfection.

Adversarial Testing Must Continue After Deployment

A one-time audit cannot guarantee lasting security.

Models change. Prompts change. external data sources change. Applicants develop new strategies. Governance participants discover new incentives. Integrations introduce additional attack surfaces.

The OECD’s AI Incidents Monitor exists partly because real-world incidents provide evidence that cannot be obtained entirely through pre-deployment testing. Continuous incident collection can reveal recurring risks and help institutions update their controls.

AI science-funding platforms therefore need a continuing cycle:

test → deploy → monitor → disclose → correct → retest

Adversarial testing should become a permanent governance function, not a ceremonial event performed before launch.

Limitations of Red Teaming

Adversarial testing is necessary, but it does not prove absolute safety or fairness.

A red team may miss an attack because:

• its members lack relevant expertise;
• the testing period is too short;
• attackers outside the program have greater resources;
• important components are excluded from scope;
• test cases do not reflect real deployment;
• the system changes after testing;
• incentives encourage participants to report easy rather than important vulnerabilities.

Testing results must therefore be interpreted as evidence about examined conditions—not as proof that no other vulnerability exists.

This is why independent audits, transparent governance, post-deployment monitoring, appeals, and human oversight remain necessary.

AIIM as a Public Experiment in Funding Security

AIIM proposes using artificial intelligence to evaluate useful work and support the distribution of money among researchers, developers, and other contributors to public goods.

Such a system cannot earn legitimacy merely by claiming that AI is more impartial than traditional grant committees. It must expose its mechanisms to organized criticism and controlled attacks.

The adversarial testing of AIIM is therefore more than a security exercise. It is a test of whether an AI-based funding project is willing to make its claims falsifiable.

A funding system that welcomes serious attempts to break it demonstrates three commitments:

• weaknesses should be discovered before they cause large losses;
• criticism should produce technical improvement rather than institutional retaliation;
• trust should rest on evidence, not branding or authority.

Conclusion

AI science funding needs adversarial testing because ordinary performance does not establish robustness.

A system may appear accurate while remaining vulnerable to prompt gaming, hidden instructions, fabricated impact, collusion, plagiarism, biased scoring, or manipulation of scientific uncertainty. These weaknesses become more consequential when the system controls real funding.

Adversarial testing converts vague concerns into reproducible experiments. It allows researchers, security specialists, and ordinary users to demonstrate how a funding mechanism can fail—and gives developers the evidence required to correct it.

The central principle is straightforward:

An AI that distributes scientific funding should be tested not only by people who want it to succeed, but also by people rewarded for proving where it fails.

That is not hostility toward AI-funded science. It is a prerequisite for making AI-funded science credible.