Performance & Memory Benchmarks

Copy Markdown View Source

VulnCheck runs on every pull request, so a warm scan has to stay well inside the time budget of a CI step, and the scanner itself must never become a denial-of-service vector against the BEAM it runs on. This page documents how performance is measured, the numbers on a reference machine, and the invariants enforced by the test suite.


1. Methodology

All numbers come from benchmarks/run_benchmarks.exs, which uses nothing but the Elixir and Erlang/OTP standard libraries:

mix run benchmarks/run_benchmarks.exs                                # medium profile, text table
mix run benchmarks/run_benchmarks.exs --profile all --markdown       # every profile, this page's tables
mix run benchmarks/run_benchmarks.exs --profile large --iterations 20 --keep

The two synthetic profiles generate a complete compiled Mix workspace in a temporary directory: mix.exs, mix.lock, one _build/prod/lib/<app>/ebin per package with .app files and .beam files compiled with debug info, and an offline advisory cache. Every package is a chain of modules whose functions pattern-match, build maps, run closures over lists and format strings, then call the next function or the package's dependencies. Half of the advisories name a function on the call chain from Application.start/2 (reachable), and the other half name a function nothing calls (unreachable), so the solver produces both verdicts.

ProfilePackagesModulesFunctions (approx.)Advisories
medium804214,62050
large1801,00113,000120
umbrella1801,00212,987120

The libraries profile measures the .beam summary cache on real code: the ebin directories of the Elixir and Erlang/OTP applications installed with the running VM (kernel, stdlib, compiler, crypto, inets, ssl, public_key, elixir, logger, eex, mix).

Each stage runs once to warm up (loading VulnCheck's own modules and filling the cache), then is timed with :timer.tc/1 for the configured number of iterations. Every sample runs in a fresh process, as a CLI invocation would, so garbage collection never scans results kept alive by earlier stages. Reported values are the median, p95, minimum and maximum. Peak memory is the median (with min–max spread) of 5 fresh-process runs after :erlang.garbage_collect/0, with :erlang.memory(:total) sampled every 2 ms.

StageWhat is timed
Lockfile and configurationDetecting the project, parsing mix.lock and mix.exs, locating build directories
Advisory feeds (offline cache)Reading and parsing the cached OSV archive and CVE records
Cold .beam extraction:beam_lib.chunks/2 on every .beam plus abstract-code analysis, cache disabled
Warm .beam summariesSHA-256 of every .beam and decoding its cached summary
Entrypoints, call graph and solverBuilding the graph, resolving entrypoints, solving every advisory
ReportsRendering the terminal, SARIF, CycloneDX and OpenVEX outputs
Full scanVulnCheck.Scanner.run/2 end to end, with the .beam cache off (cold) or on (warm)

2. Reference results

Environment

ComponentValue
OSunix/darwin 27.2.0 (aarch64-apple-darwin27.0.0)
CPUApple M5 Max
Schedulers18
Elixir1.20.5
Erlang/OTP29 (ERTS 17.1.1)
VulnCheck1.0.0

10 measured iterations per stage after one warm-up run.

Profile medium

80 packages, 421 .beam files, ~4620 functions, 42093 call edges, 50 advisories with programRoutines (25 reachable, 25 unreachable).

StageMedian (ms)p95 (ms)Min (ms)Max (ms)
Lockfile and configuration7.38.27.18.2
Advisory feeds (offline cache)7.07.56.77.5
Cold .beam extraction60.563.758.363.7
Warm .beam summaries (SHA-256 cache)36.538.436.038.4
Entrypoints, call graph and solver89.598.387.198.3
Reports (terminal, sarif, cyclonedx, openvex)60.863.660.463.6
Full scan, cold .beam cache149.6155.6145.6155.6
Full scan, warm cache120.4124.0116.7124.0
MetricValue
Peak BEAM memory, cold scan (median / min–max)179.8 MiB (179.6–180.0)
Peak BEAM memory, warm scan (median / min–max)161.1 MiB (147.3–165.6)
Atom delta over 10 measured runs of every stage0

Profile large

180 packages, 1001 .beam files, ~13000 functions, 120273 call edges, 120 advisories with programRoutines (60 reachable, 60 unreachable).

StageMedian (ms)p95 (ms)Min (ms)Max (ms)
Lockfile and configuration18.820.017.320.0
Advisory feeds (offline cache)15.616.115.016.1
Cold .beam extraction157.4163.2152.9163.2
Warm .beam summaries (SHA-256 cache)94.7101.386.4101.3
Entrypoints, call graph and solver322.2332.4304.2332.4
Reports (terminal, sarif, cyclonedx, openvex)105.9108.8105.2108.8
Full scan, cold .beam cache497.5504.5481.6504.5
Full scan, warm cache407.5424.9385.6424.9
MetricValue
Peak BEAM memory, cold scan (median / min–max)334.7 MiB (333.9–359.5)
Peak BEAM memory, warm scan (median / min–max)309.5 MiB (309.1–343.8)
Atom delta over 10 measured runs of every stage0

Profile umbrella

180 packages, 1002 .beam files, ~12987 functions, 120220 call edges, 120 advisories with programRoutines (60 reachable, 60 unreachable).

StageMedian (ms)p95 (ms)Min (ms)Max (ms)
Lockfile and configuration18.019.117.719.1
Advisory feeds (offline cache)15.916.815.316.8
Cold .beam extraction154.4160.4150.0160.4
Warm .beam summaries (SHA-256 cache)96.2102.587.6102.5
Entrypoints, call graph and solver324.5402.3314.4402.3
Reports (terminal, sarif, cyclonedx, openvex)112.5115.5111.7115.5
Full scan, cold .beam cache506.7588.2488.6588.2
Full scan, warm cache409.0418.2388.2418.2
MetricValue
Peak BEAM memory, cold scan (median / min–max)375.0 MiB (374.8–375.2)
Peak BEAM memory, warm scan (median / min–max)324.8 MiB (324.3–345.7)
Atom delta over 10 measured runs of every stage0

Profile libraries

835 .beam files of 11 installed Elixir and Erlang/OTP applications (0 without debug info); warm cache hit rate 100%.

StageMedian (ms)p95 (ms)Min (ms)Max (ms)
Cold extraction, 1 worker956.9968.3946.3968.3
Warm cache, 1 worker180.9192.9176.5192.9
Cold extraction, 18 workers186.5193.7180.8193.7
Warm cache, 18 workers81.087.874.187.8
MetricValue
Atom delta over 10 measured runs of every stage0

3. Reading the numbers

  • Warm scans of a medium project take ~120 ms, well under the 200 ms budget, and both large and 3-app umbrella profiles (~1,000 .beam files) complete in ~408 ms. CI time is dominated by compiling the project, not by the scan.
  • The summary cache pays off across both small and large modules. On the installed libraries, the warm cache is about 5.3× faster than cold extraction on one worker and 2.3× faster with 18 workers. On the synthetic profiles, warm .beam loading is roughly 1.6× faster than cold extraction.
  • Compact v4 summary encoding keeps warm peak memory below cold extraction. Cached summaries store a per-module string table and integer-indexed MFA references, and the loader unpacks them through a scan-local string pool so repeated module and function names share binaries across the scan. On large, median warm peak memory is 309.5 MiB (down from 823.7 MiB before v4, a 62% reduction) versus 334.7 MiB cold.
  • Reports and solver costs reflect full evidence generation. Rendering four formats (including SARIF codeFlows / relatedLocations and CycloneDX dependsOn dependency graphs) uses iodata JSON serialization (~61 ms on medium, ~106 ms on large), and pre-read feed validation verifies archive integrity on each cache read.
  • Reports are cheap. Rendering four formats costs a fraction of the analysis; writing SBOM and VEX files next to the primary report does not change scan times measurably.

4. Atom-table guarantees

Atoms are never garbage-collected, so a scanner that converted names from advisories or lockfiles into atoms could exhaust the atom table (1,048,576 entries by default) and crash the VM. VulnCheck keeps every external name as a binary:

  • OSV records, CVE records, lockfiles, .vulncheck.exs and .vulncheck.ignore are parsed into strings and literals; nothing is evaluated and String.to_atom/1 is never called on external input.
  • Cached .beam summaries are decoded with :erlang.binary_to_term(bin, [:safe]). Every atom a summary holds is defined by VulnCheck's own modules, which are loaded before the first decode, so the safe decoder accepts every cache entry in a fresh VM while a tampered cache file that names unknown atoms is treated as a cache miss.
  • Analyzing a .beam file cold reads its abstract code with :beam_lib.chunks/2, which interns the atoms of that compiled module (its module, function and literal names). This is bounded by the code that is actually installed in the project and happens once per changed file; warm runs never read chunks again.

The benchmark reports the atom-count delta over all measured iterations of every stage, which is 0 for every profile. The test suite enforces the same guarantees on every run:

InvariantCheck
Parsing 500 advisories whose module and function names exist nowhere creates no atoms:erlang.system_info(:atom_count) delta is exactly 0 across OSV.parse_archive/1 and EefCna.apply_record/2
Repeated warm scans create no atomsAtom-count delta over five full scans is exactly 0
A warm cache never re-reads unchanged .beam files:beam_lib.chunks/2 call count is 0 during a warm scan and every module is a cache hit (100%)
A changed .beam is re-analyzed, unchanged ones are notExactly one miss after rebuilding one module
Cached summaries decode in a fresh VMEvery atom in summaries of the Elixir standard library belongs to the modules loaded before decoding