DependencyUpdateMetric

How fast do open-source packages update their dependencies and remediate known vulnerabilities, over time?

Imranur Rahman, Mahzabina Tamanna, Laurie Williams

North Carolina State University

Paper (soon) Code Algorithm Video (soon)

Architecture of DependencyUpdateMetric: inputs, CLI, historical dependency resolver, OSV index, dependency analyzer, and reporting

Architecture of DependencyUpdateMetric. Numbers give the order of the steps. The resolver resolves each version constraint as of a past date, and the dashed slots show where a new ecosystem plugs in.

Overview

Software composition analysis tools tell you whether a project is out of date or vulnerable now. They do not tell you how the project behaved over time. DependencyUpdateMetric fills this gap. It computes two metrics for the direct dependencies of a package in npm, PyPI, or Cargo: Mean Time To Update (MTTU), how long the dependencies stay behind their newest version, and Mean Time To Remediate (MTTR), how long the dependencies keep a known vulnerability after a fix is available.

The tool resolves every version constraint as of a past date, so it never uses data from the future. It can compute the metrics for one time window or at every past release of a package, and it runs in batches over thousands of packages.

Design Goals

Correctness

Every value is right, and a reader can check it. Constraints resolve as of each past date, MTTR counts a vulnerability only after its fix is published, and optional worksheets list every interval.

Scalability

A study can analyze thousands of packages over their full histories. Batches run in parallel workers, a cache stores registry data, the OSV index is a lookup in memory, and --resume continues after a crash.

Extensibility

Researchers can add ecosystems, data sources, and analysis choices. Every resolver implements one PackageResolver protocol, and every analysis choice is a command-line option.

Quick Start

# Install from PyPI (the command is dependency-metrics)
pip install dependency-update-metrics

# One package, with the OSV index built on the first run
dependency-metrics --ecosystem npm --package express \
  --start-date 2020-01-01 --end-date 2023-12-31 --build-osv

# MTTU and MTTR at every release of a package
dependency-metrics --ecosystem pypi --package requests \
  --start-date 2020-01-01 --end-date 2021-01-01 --per-release

# A batch of packages from a CSV file, in parallel
dependency-metrics --input-csv input.csv --workers 8 --output-dir output

Available on PyPI. See the README for all options and for installing from source, and ALGORITHM.md for how the metrics are computed.

Validation

253

offline tests: unit, integration, regression, and metamorphic

72 / 90

case-study packages whose scores match the reference scores exactly

< 1 g CO₂e

for a full run over 90 packages (6,601 release points)

The test types follow research on testing scientific software; TESTING.md maps each type to its tests. CI runs the suite on Ubuntu, macOS, and Windows with Python 3.10–3.12.

Who Can Use It

Researchers who study technical lag, software supply chains, or maintainer behavior can compute MTTU and MTTR for thousands of packages and get a full time series per package.
Developers can compare how promptly candidate libraries keep their own dependencies current and patched.
Security teams can run a batch over their portfolio to find the projects that lag behind available fixes.
Builders of security metrics, such as the OpenSSF Scorecard, can use MTTU and MTTR as signals of maintenance.

Demo Video

Coming soon.

Related Work from the Authors

No Vulnerability Data, No Problem: Towards Predicting Mean Time To Remediate In Open Source Software Dependencies

CHRONO-RESOLUTION: A Dependency Resolution Dataset at Release Points for npm, PyPI, and crates.io Packages

What's in a Package? Getting Visibility Into Dependencies Using Security-Sensitive API Calls