zFiniazFinia
Editorial

Original research · frozen public-repository benchmark

CrossCheck benchmark: package-manager drift in 1,027 AI-instructed repositories

CrossCheck's default rule produced 47 package-manager configuration findings on four untouched holdout sets covering 947 public repositories. The cited repository state for every emission was manually verified at its frozen commit. A post-publication intent audit then found deliberate multi-manager cases, so this benchmark no longer interprets those 47 emissions as 47 confirmed errors.

First published 18 September 2026 · benchmark evidence published 21 September 2026 · interpretation corrected 21 September 2026

1,027

repositories frozen

947

untouched holdouts

5,904

package scopes evaluated

47 / 47

emissions evidence-verified

Finding

The detector surfaced real coexisting package-manager signals. That does not prove every case is an error.

The measured rule emits when multiple package-manager configuration signals coexist inside the same package scope, such as multiple lockfiles or a packageManager declaration beside another manager's lockfile. Across holdouts v2–v5 it emitted 47 findings. Manual review confirmed that the cited files and configuration signals existed at the frozen commits.

This is an evidence-verification result, not a 47/47 actionable-precision claim. It does not establish maintainer intent, prove that every emission requires remediation, measure recall, or establish a GitHub-wide rate of package-manager drift. The sample deliberately targets active JavaScript or TypeScript repositories with AI-agent instruction files, so it is not a census of all public repositories.

Six experimental findings were emitted across all five sets. They remain in the raw results but are excluded from the headline and cannot fail a CrossCheck run. The benchmark therefore does not make the weaker experimental rules look better by mixing them with the default rule.

Method

Freeze first, tune once, then keep four later samples untouched

  1. Step 1

    Define the eligible population

    Active, non-fork public JavaScript or TypeScript repositories with an AI-agent instruction file, plus the date, size and search constraints recorded in each sample.

  2. Step 2

    Freeze repository identity

    Record the owner/name and full 40-character default-branch commit before running the engine.

  3. Step 3

    Separate tuning from evaluation

    Use 80 repositories for rule development. Exclude them from the four later holdouts, which contain 947 repositories.

  4. Step 4

    Separate evidence from intent

    Keep three unavailable scans as skipped. Verify every cited file and signal at the frozen commit, then treat maintainer intent and remediation as a separate question that static repository state may not answer.

Evidence package

Every aggregate on this page resolves to named raw records

Frozen samples and raw outputs

Repository names, exact commits, CrossCheck results and replay scripts.

Machine-readable summary

Claim-sized totals, explicit limits and SHA-256 hashes for every raw file.

47 review records

One evidence-review record per proven holdout emission, keyed to its repository, commit and cited configuration.

git clone https://github.com/zFinia/crosscheck.git
cd crosscheck
npm run benchmark:verify

The verifier rejects duplicate repositories, malformed commits, sample/result mismatches, missing reviews, changed hashes, or totals that differ from the raw evidence. Full replay instructions fetch only relevant configuration files at the frozen commits and never install or execute sampled repository code.

Use the measured rule

Run a free local scan, or add the advisory Action

CrossCheck 0.1.1 is MIT-licensed, has no runtime dependencies, needs no zFinia account, and makes no scan-time network calls. The Action runs in your GitHub runner and is advisory by default: a new finding creates a warning but does not fail the pull request.

npx --yes @zfinia/crosscheck@0.1.1
name: CrossCheck
on: pull_request

jobs:
  crosscheck:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: zFinia/crosscheck@v0
Install the GitHub ActionRun one hosted scan free

Limits and correction policy

  • • Repository selection depends on GitHub code-search results and the recorded eligibility filters; another search date can produce a different population.
  • • Evidence review was performed by zFinia, not by the sampled maintainers. The public records make independent re-checking possible but do not establish maintainer intent or turn the review into third-party validation.
  • • Multiple lockfiles can be deliberate. Compatibility testing, dependency-update tooling, release workflows or migration state can make a mechanically valid emission non-actionable; CrossCheck should not infer intent from lockfile coexistence alone.
  • • Repository deletion or visibility changes can make later replay incomplete. The frozen manifest remains unchanged and unavailable entries must be reported as skipped.
  • • Corrections should preserve the original raw engine output and update interpretation, review metadata and public claims together. The 21 September correction above is part of that record.