Measuring the Impact of Research Software and Mathematical Libraries

Getting your Trinity Audio player ready...

Research software and mathematical libraries should be evaluated by the work they enable—not merely by papers that cite them, repository stars, or download counts.

A useful impact assessment combines several forms of evidence:

• direct scholarly citations;
• software and proof dependencies;
• use in published research;
• reuse across disciplines;
• contribution to reproducibility;
• maintenance and security work;
• replacement cost;
• downstream scientific, technological, and economic outcomes.

No single metric captures all these dimensions. The most credible approach is therefore a transparent impact graph showing how a software package, algorithm, theorem, formal definition, or mathematical library supports later results.

What Is Research Software Impact?

Research software impact is the scientifically valuable activity made possible, accelerated, verified, or improved by a software contribution.

Research software includes much more than large scientific applications. It can include:

• simulation software;
• numerical libraries;
• statistical packages;
• data-processing pipelines;
• laboratory-control software;
• visualization tools;
• theorem provers;
• formal mathematical libraries;
• scripts used to reproduce a published result;
• infrastructure used by other research tools.

The FAIR Principles for Research Software, commonly called FAIR4RS, describe research software in terms of findability, accessibility, interoperability, and reusability. These properties do not directly measure scientific impact, but they make impact easier to generate, observe, and verify.

Impact should therefore not be confused with popularity. A package can be popular without being scientifically important, while a small library may be indispensable to a narrow but critical research community.

Why Citation Counts Are Insufficient

Academic evaluation systems are built primarily around publications. Software is often mentioned only in the methods section, placed in a footnote, linked without a formal reference, or omitted entirely.

The FORCE11 Software Citation Principles argue that software should be treated as a legitimate research product and cited on the same basis as papers, books, and datasets. Proper citation should support identification, credit, persistence, accessibility, and specificity about the version used.

Formal software citations are valuable, but they reveal only part of the influence chain.

Consider a numerical library used by another research package. Scientists may cite the package they interact with while never citing its underlying linear-algebra, optimization, parsing, or visualization dependencies. The upstream library contributes to the result, but conventional bibliometrics may record nothing.

The same problem applies to mathematics:

A theorem may depend on dozens of earlier definitions and lemmas, but a paper normally cites only a small selection of books and articles rather than every intellectual dependency.

Consequently, citation counts tend to measure visible acknowledgment, not complete scientific dependence.

This is one reason citation counts cannot fully measure the value of basic mathematics. The scientific record must be supplemented by dependency, reuse, and outcome evidence.

The Difference Between Software Libraries and Mathematical Libraries

The term mathematical library can refer to several related objects.

Numerical and symbolic software libraries

These provide executable implementations of mathematical operations, such as:

• matrix decompositions;
• differential-equation solvers;
• optimization algorithms;
• symbolic integration;
• probability distributions;
• geometric computations.

Their impact can often be traced through package dependencies, imports, publications, workflows, and deployed applications.

Formal mathematical libraries

Formal libraries contain machine-checkable definitions, theorems, proofs, and tactics for proof assistants.

For example, Lean’s mathlib is a community-maintained mathematical library containing both formalized mathematics and supporting programming infrastructure.

Its internal dependency graph can show exactly which definitions and theorems are required by later formal results. This creates a more explicit representation of mathematical dependence than ordinary scholarly citations usually provide.

Informal bodies of reusable mathematics

A mathematical “library” can also mean a reusable collection of definitions, constructions, lemmas, notation, or methods distributed across papers and monographs.

These contributions are harder to measure because conceptual reuse may occur without direct quotation, software imports, or formal proof dependencies. Evaluation may require expert interpretation supported by semantic analysis.

A Multi-Dimensional Framework for Measuring Impact

A defensible system should report several dimensions separately before attempting to calculate an aggregate score.

Scholarly Recognition

Scholarly recognition includes:

• formal software citations;
• citations to associated papers;
• citations in documentation;
• mentions in methods sections;
• references in textbooks, standards, and technical reports.

This evidence establishes that researchers recognize a contribution. It does not necessarily prove that the contribution was essential.

A ceremonial citation, a comparison with a competing method, and an indispensable computational dependency should not receive identical weight.

Direct Software Dependencies

Dependency analysis asks which projects explicitly require a particular package.

Useful signals include:

• the number of direct dependent projects;
• the quality and importance of those projects;
• the number of active rather than abandoned dependents;
• whether the dependency is optional or required;
• the functions or modules actually imported;
• the versions used.

Counting dependencies is generally more informative than counting repository stars because it records incorporation into other software rather than casual attention.

Yet raw dependency totals can also mislead. One project may automatically generate thousands of trivial packages, while another dependency may support a single major scientific instrument or international research facility.

Transitive Dependencies

A transitive dependency is used indirectly.

Suppose application A depends on library B, which depends on library C. The developers of A may never have deliberately selected C, but A may still fail without it.

An impact system should therefore trace complete dependency paths:

Library C → research package B → scientific workflow A → published result

Transitive impact should usually be discounted with distance, but not discarded. Deep infrastructure is often invisible precisely because users interact only with higher-level tools.

Use in Published Research

A strong impact claim connects software to identifiable research outputs.

Evidence may include:

• archived computational environments;
• workflow specifications;
• package lock files;
• notebooks;
• source-code imports;
• supplementary materials;
• container images;
• provenance records;
• statements in reproducibility reports.

This allows evaluators to distinguish a downloaded package from software that actually contributed to a scientific result.

Reproducibility and Verification

Software can have impact by making other research checkable.

Examples include:

• reproducing published figures;
• validating statistical calculations;
• detecting errors;
• independently implementing an algorithm;
• formally verifying a theorem;
• preserving an exact research environment.

Software and data citation guidance increasingly treats proper citation as part of reproducibility, attribution, discoverability, and reuse rather than as a courtesy added after the research is complete.

A tool that prevents false conclusions may be valuable even when it does not produce a large number of new publications.

Mathematical Dependency

For formal mathematics, impact can be measured partly through proof dependencies.

Relevant indicators include:

• the number of later theorems depending on a definition or lemma;
• dependency depth;
• reuse across distinct mathematical domains;
• whether the result removes duplication;
• whether it enables substantial automation;
• whether replacing it would require extensive proof reconstruction.

A frequently imported elementary definition should not automatically outrank a deep theorem. The system must distinguish routine syntactic dependence from substantive mathematical contribution.

For informal mathematics, possible evidence includes:

• explicit citations;
• equivalent definitions appearing in later work;
• adoption of terminology or notation;
• use in algorithms;
• formalization in proof assistants;
• incorporation into reference works;
• expert-confirmed conceptual lineage.

AI can help identify possible relationships, but mathematical experts must be able to inspect and challenge those relationships.

Cross-Disciplinary Reach

Software may begin in one discipline and later become useful in others.

Cross-disciplinary impact can be measured by the diversity of:

• publication fields;
• institutions;
• software ecosystems;
• geographic regions;
• application types;
• research methods.

Reuse across independent communities is stronger evidence than repeated use inside one closely connected laboratory or development team.

However, specialization is not a defect. A highly technical library serving a small research field may still be essential to that field.

Maintenance and Sustainability

Research software creates value after its initial release.

Maintainers may:

• correct defects;
• respond to changes in operating systems or dependencies;
• review contributions;
• improve documentation;
• preserve backward compatibility;
• address security vulnerabilities;
• support users;
• modernize algorithms;
• release reproducible versions.

A metric that rewards only original authors can systematically underpay the people who keep scientific infrastructure operational.

Maintenance impact should therefore be attributed by version, period, component, and activity. Authorship of the initial design and responsibility for long-term sustainability are related but distinct contributions.

Replacement Cost and Structural Criticality

Replacement cost estimates what users would need to spend to reproduce the relevant capability.

It may include:

• developer time;
• mathematical or domain expertise;
• testing;
• validation;
• documentation;
• migration costs;
• compatibility work;
• delays to dependent projects.

Replacement cost is not identical to scientific merit. Inefficient or poorly documented software can be expensive to replace without being excellent. It should be combined with evidence of genuine use and functional necessity.

Structural criticality asks a different question:

How much of the research ecosystem would be disrupted if this component disappeared or became unreliable?

A small package located at a dependency bottleneck may be more structurally important than a large, visible application.

Scientific Time Saved

Libraries accelerate science by allowing researchers to reuse validated work rather than implement everything again.

Potential indicators include:

• reductions in development time;
• elimination of duplicated implementations;
• shorter proof scripts;
• automated verification;
• faster experimental analysis;
• reduced computational cost;
• fewer errors;
• easier training of new researchers.

These quantities are difficult to estimate precisely. Surveys, development histories, controlled comparisons, and expert assessments can provide ranges rather than false exactness.

Real-World Outcomes

Some research software contributes to outcomes outside academic publishing, including:

• medical analysis;
• engineering design;
• public infrastructure;
• climate modelling;
• education;
• industrial research;
• technical standards;
• commercial products.

Attribution becomes weaker as the causal chain grows longer. A foundational library may be necessary for an application without being responsible for every benefit produced by that application.

An impact system should therefore use fractional, uncertainty-aware attribution rather than claiming that an upstream package created the entire value of all downstream products.

Why Downloads, Stars, and Forks Are Weak Metrics

Platform statistics are useful signals, but none should be treated as a direct measure of scientific value.

Metric	What it may indicate	Major limitation
Downloads	Distribution or automated installation	Bots, mirrors, CI systems and repeated installations inflate counts
Repository stars	Attention or approval	A star does not demonstrate use
Forks	Interest in modification	Many forks are inactive or automatically created
Contributors	Community participation	Contribution size and significance vary greatly
Commits	Development activity	Commit counts can be trivially inflated
Open issues	User activity or unresolved problems	High counts may indicate popularity, poor maintenance, or both
Citations	Scholarly acknowledgment	Many actual uses are never formally cited
Dependencies	Incorporation into other software	Dependency presence does not prove execution or scientific importance

These indicators should function as evidence inputs, not as final judgments.

Building a Research Impact Graph

The most promising approach is to construct a graph containing several types of entities:

• researchers and developers;
• software repositories;
• released software versions;
• packages and modules;
• datasets;
• publications;
• mathematical definitions and theorems;
• formal proofs;
• research workflows;
• institutions;
• standards;
• deployed systems;
• documented outcomes.

Edges would represent relationships such as:

• authored;
• contributed to;
• cited;
• imported;
• depended on;
• executed in;
• validated;
• reproduced;
• implemented;
• generalized;
• formalized;
• corrected;
• maintained;
• deployed in.

Each edge should carry provenance, confidence, date, version, and evidence type.

For example:

Researcher → authored algorithm → implemented in library → imported by simulation package → used in published experiment → independently reproduced

This structure makes impact inspectable. Instead of receiving an unexplained score, a developer or mathematician could see which dependency paths and documented outcomes produced the assessment.

Persistent Identification Is Essential

Impact cannot be attributed reliably when software versions are unstable or ambiguous.

Research software should use:

• versioned releases;
• persistent identifiers;
• complete authorship metadata;
• machine-readable citation files;
• archived source code;
• explicit licenses;
• links between software and publications.

GitHub documents how public repositories can be archived through Zenodo and assigned Digital Object Identifiers. Zenodo’s integration can automatically archive new releases, while Software Heritage supports long-term archival and precise references to source-code versions or fragments.

These systems solve part of the identity problem: they help establish exactly which artifact existed and which version was used.

They do not independently measure its importance.

Attribution Among Multiple Contributors

Research software is frequently collective. Credit must be divided among people who perform different forms of work:

• architecture;
• algorithm design;
• mathematical theory;
• implementation;
• testing;
• documentation;
• release engineering;
• code review;
• maintenance;
• community support;
• security work;
• research validation.

Equal division is simple but often inaccurate. Commit counts are easy to calculate but fail to capture design, review, or intellectual contribution.

A stronger attribution model can combine:

1. contributor declarations;
2. version-control evidence;
3. component ownership;
4. peer assessment;
5. documented roles;
6. dependency-level reuse;
7. challenge and appeal procedures.

The goal should not be to force every contribution into one number. It should preserve distinct contribution categories and show how each category affects recognition or funding.

Measuring the Impact of Mathematical Definitions

A mathematical definition can be highly valuable even before it produces many theorems.

A useful definition may:

• unify previously separate theories;
• expose a hidden common structure;
• simplify proofs;
• generate a new research programme;
• make a field formalizable;
• allow existing results to be transferred;
• provide an interface for software or automated reasoning.

Its impact may initially be underestimated because definitions are not always cited separately from the papers or books containing them.

Possible measures include:

• number and importance of results formulated using the definition;
• number of earlier concepts recovered as special cases;
• adoption by independent researchers;
• use in formal libraries;
• proof compression;
• algorithmic implementations;
• cross-disciplinary applications;
• evidence that it changed how problems are represented.

This evaluation must remain open to delayed impact. Foundational mathematics may become useful decades after its introduction.

Preventing Metric Manipulation

Any metric connected to reputation or money will attract attempts to manipulate it.

Likely attacks include:

• artificial package dependencies;
• automated downloads;
• citation rings;
• meaningless repository forks;
• fragmented commits;
• fake users;
• reciprocal endorsements;
• autogenerated papers;
• duplicated software packages;
• false claims of industrial deployment.

Defences should include:

• identity and provenance checks;
• discounting of closely coordinated activity;
• detection of circular dependency structures;
• separation of independent and affiliated reuse;
• weighting evidence by reliability;
• public explanations;
• reproducible calculations;
• random audits;
• adversarial testing;
• appeals and corrections.

Most importantly, a score should never conceal the underlying evidence.

From Impact Measurement to Research Funding

Better measurement matters because current funding systems often reward proposals, institutional prestige, and publications while leaving reusable infrastructure underfunded.

An impact-aware system could reward different outputs separately:

• a mathematical definition;
• a theorem;
• a formal proof;
• a software implementation;
• a dataset;
• documentation;
• maintenance;
• replication;
• error correction.

The AI Internet-Meritocracy model for research funding proposes evaluating verifiable outputs, dependencies, reuse, and downstream usefulness rather than relying solely on predictions made in grant proposals.

Such a system should not allow AI to declare scientific value without review. AI can map possible dependencies, gather evidence, and calculate transparent indicators. Experts, contributors, and affected communities must remain able to inspect claims, contest errors, and revise the evaluation.

A Practical Impact Dashboard

A research-software or mathematical-library dashboard could display:

Recognition

• formal citations;
• software mentions;
• citing disciplines;
• independent citing groups.

Reuse

• direct dependents;
• transitive dependents;
• active installations;
• research workflows using the software;
• formal theorems depending on a component.

Quality and reproducibility

• archived releases;
• test coverage;
• reproduced studies;
• validation reports;
• documented corrections;
• FAIR4RS indicators.

Sustainability

• active maintainers;
• release continuity;
• issue-response time;
• maintenance workload;
• security status;
• bus-factor risk.

Downstream impact

• publications enabled;
• disciplines served;
• standards or products using the output;
• estimated replacement cost;
• structurally critical dependency paths.

Confidence and limitations

• evidence quality;
• attribution uncertainty;
• missing metadata;
• disputed relationships;
• potential manipulation flags.

This dashboard would give evaluators a profile rather than an oversimplified ranking.

Limitations of Research Software Impact Measurement

Several limitations cannot be eliminated completely.

First, private use is difficult to observe. Companies and laboratories may rely on open software without publicly recording that dependence.

Second, conceptual mathematical influence is partly interpretive. Two theories may be closely related even when terminology and notation differ.

Third, impact changes over time. A library can become obsolete, while an old theorem can acquire new importance.

Fourth, necessity is difficult to prove. A dependent project might have used an alternative, although migration could have required substantial work.

Fifth, different fields have different software cultures. Download numbers, citation practices, project sizes, and publication norms cannot be compared without normalization.

For these reasons, impact assessments should report ranges, uncertainty, and disciplinary context.

Conclusion

The impact of research software and mathematical libraries cannot be reduced to citations, downloads, stars, or lines of code.

A credible evaluation must trace how a contribution is reused, depended upon, maintained, verified, and transformed into later scientific work. Formal citations remain important, but they should be integrated with software dependency graphs, proof dependencies, reproducibility evidence, expert assessment, and documented downstream outcomes.

The central principle is straightforward:

Research infrastructure should be valued according to the scientific capabilities it creates and preserves.

When these dependency chains become visible, research institutions and funding systems can recognize not only prominent paper authors, but also the developers, maintainers, formalizers, and mathematicians whose work quietly supports entire fields.