VulnCheck runs on every pull request, so a warm scan has to stay well inside the time budget of a CI step, and the scanner itself must never become a denial-of-service vector against the BEAM it runs on. This page documents how performance is measured, the numbers on a reference machine, and the invariants enforced by the test suite.
1. Methodology
All numbers come from benchmarks/run_benchmarks.exs,
which uses nothing but the Elixir and Erlang/OTP standard libraries:
mix run benchmarks/run_benchmarks.exs # medium profile, text table
mix run benchmarks/run_benchmarks.exs --profile all --markdown # every profile, this page's tables
mix run benchmarks/run_benchmarks.exs --profile large --iterations 20 --keep
The two synthetic profiles generate a complete compiled Mix workspace in a
temporary directory: mix.exs, mix.lock, one _build/prod/lib/<app>/ebin
per package with .app files and .beam files compiled with debug info, and
an offline advisory cache. Every package is a chain of modules whose
functions pattern-match, build maps, run closures over lists and format
strings, then call the next function or the package's dependencies. Half of
the advisories name a function on the call chain from Application.start/2
(reachable), and the other half name a function nothing calls (unreachable),
so the solver produces both verdicts.
| Profile | Packages | Modules | Functions (approx.) | Advisories |
|---|---|---|---|---|
medium | 80 | 421 | 4,620 | 50 |
large | 180 | 1,001 | 13,000 | 120 |
umbrella | 180 | 1,002 | 12,987 | 120 |
The libraries profile measures the .beam summary cache on real code: the
ebin directories of the Elixir and Erlang/OTP applications installed with the
running VM (kernel, stdlib, compiler, crypto, inets, ssl,
public_key, elixir, logger, eex, mix).
Each stage runs once to warm up (loading VulnCheck's own modules and filling
the cache), then is timed with :timer.tc/1 for the configured number of
iterations. Every sample runs in a fresh process, as a CLI invocation would,
so garbage collection never scans results kept alive by earlier stages.
Reported values are the median, p95, minimum and maximum. Peak memory is the
median (with min–max spread) of 5 fresh-process runs after
:erlang.garbage_collect/0, with :erlang.memory(:total) sampled every 2 ms.
| Stage | What is timed |
|---|---|
| Lockfile and configuration | Detecting the project, parsing mix.lock and mix.exs, locating build directories |
| Advisory feeds (offline cache) | Reading and parsing the cached OSV archive and CVE records |
Cold .beam extraction | :beam_lib.chunks/2 on every .beam plus abstract-code analysis, cache disabled |
Warm .beam summaries | SHA-256 of every .beam and decoding its cached summary |
| Entrypoints, call graph and solver | Building the graph, resolving entrypoints, solving every advisory |
| Reports | Rendering the terminal, SARIF, CycloneDX and OpenVEX outputs |
| Full scan | VulnCheck.Scanner.run/2 end to end, with the .beam cache off (cold) or on (warm) |
2. Reference results
Environment
| Component | Value |
|---|---|
| OS | unix/darwin 27.2.0 (aarch64-apple-darwin27.0.0) |
| CPU | Apple M5 Max |
| Schedulers | 18 |
| Elixir | 1.20.5 |
| Erlang/OTP | 29 (ERTS 17.1.1) |
| VulnCheck | 1.0.0 |
10 measured iterations per stage after one warm-up run.
Profile medium
80 packages, 421 .beam files, ~4620 functions, 42093 call edges,
50 advisories with programRoutines (25 reachable, 25 unreachable).
| Stage | Median (ms) | p95 (ms) | Min (ms) | Max (ms) |
|---|---|---|---|---|
| Lockfile and configuration | 7.3 | 8.2 | 7.1 | 8.2 |
| Advisory feeds (offline cache) | 7.0 | 7.5 | 6.7 | 7.5 |
| Cold .beam extraction | 60.5 | 63.7 | 58.3 | 63.7 |
| Warm .beam summaries (SHA-256 cache) | 36.5 | 38.4 | 36.0 | 38.4 |
| Entrypoints, call graph and solver | 89.5 | 98.3 | 87.1 | 98.3 |
| Reports (terminal, sarif, cyclonedx, openvex) | 60.8 | 63.6 | 60.4 | 63.6 |
| Full scan, cold .beam cache | 149.6 | 155.6 | 145.6 | 155.6 |
| Full scan, warm cache | 120.4 | 124.0 | 116.7 | 124.0 |
| Metric | Value |
|---|---|
| Peak BEAM memory, cold scan (median / min–max) | 179.8 MiB (179.6–180.0) |
| Peak BEAM memory, warm scan (median / min–max) | 161.1 MiB (147.3–165.6) |
| Atom delta over 10 measured runs of every stage | 0 |
Profile large
180 packages, 1001 .beam files, ~13000 functions, 120273 call edges,
120 advisories with programRoutines (60 reachable, 60 unreachable).
| Stage | Median (ms) | p95 (ms) | Min (ms) | Max (ms) |
|---|---|---|---|---|
| Lockfile and configuration | 18.8 | 20.0 | 17.3 | 20.0 |
| Advisory feeds (offline cache) | 15.6 | 16.1 | 15.0 | 16.1 |
| Cold .beam extraction | 157.4 | 163.2 | 152.9 | 163.2 |
| Warm .beam summaries (SHA-256 cache) | 94.7 | 101.3 | 86.4 | 101.3 |
| Entrypoints, call graph and solver | 322.2 | 332.4 | 304.2 | 332.4 |
| Reports (terminal, sarif, cyclonedx, openvex) | 105.9 | 108.8 | 105.2 | 108.8 |
| Full scan, cold .beam cache | 497.5 | 504.5 | 481.6 | 504.5 |
| Full scan, warm cache | 407.5 | 424.9 | 385.6 | 424.9 |
| Metric | Value |
|---|---|
| Peak BEAM memory, cold scan (median / min–max) | 334.7 MiB (333.9–359.5) |
| Peak BEAM memory, warm scan (median / min–max) | 309.5 MiB (309.1–343.8) |
| Atom delta over 10 measured runs of every stage | 0 |
Profile umbrella
180 packages, 1002 .beam files, ~12987 functions, 120220 call edges,
120 advisories with programRoutines (60 reachable, 60 unreachable).
| Stage | Median (ms) | p95 (ms) | Min (ms) | Max (ms) |
|---|---|---|---|---|
| Lockfile and configuration | 18.0 | 19.1 | 17.7 | 19.1 |
| Advisory feeds (offline cache) | 15.9 | 16.8 | 15.3 | 16.8 |
| Cold .beam extraction | 154.4 | 160.4 | 150.0 | 160.4 |
| Warm .beam summaries (SHA-256 cache) | 96.2 | 102.5 | 87.6 | 102.5 |
| Entrypoints, call graph and solver | 324.5 | 402.3 | 314.4 | 402.3 |
| Reports (terminal, sarif, cyclonedx, openvex) | 112.5 | 115.5 | 111.7 | 115.5 |
| Full scan, cold .beam cache | 506.7 | 588.2 | 488.6 | 588.2 |
| Full scan, warm cache | 409.0 | 418.2 | 388.2 | 418.2 |
| Metric | Value |
|---|---|
| Peak BEAM memory, cold scan (median / min–max) | 375.0 MiB (374.8–375.2) |
| Peak BEAM memory, warm scan (median / min–max) | 324.8 MiB (324.3–345.7) |
| Atom delta over 10 measured runs of every stage | 0 |
Profile libraries
835 .beam files of 11 installed Elixir and Erlang/OTP applications
(0 without debug info); warm cache hit rate 100%.
| Stage | Median (ms) | p95 (ms) | Min (ms) | Max (ms) |
|---|---|---|---|---|
| Cold extraction, 1 worker | 956.9 | 968.3 | 946.3 | 968.3 |
| Warm cache, 1 worker | 180.9 | 192.9 | 176.5 | 192.9 |
| Cold extraction, 18 workers | 186.5 | 193.7 | 180.8 | 193.7 |
| Warm cache, 18 workers | 81.0 | 87.8 | 74.1 | 87.8 |
| Metric | Value |
|---|---|
| Atom delta over 10 measured runs of every stage | 0 |
3. Reading the numbers
- Warm scans of a medium project take ~120 ms, well under the 200 ms
budget, and both
largeand 3-appumbrellaprofiles (~1,000.beamfiles) complete in ~408 ms. CI time is dominated by compiling the project, not by the scan. - The summary cache pays off across both small and large modules. On the
installed libraries, the warm cache is about 5.3× faster than cold extraction
on one worker and 2.3× faster with 18 workers. On the synthetic profiles,
warm
.beamloading is roughly 1.6× faster than cold extraction. - Compact v4 summary encoding keeps warm peak memory below cold extraction.
Cached summaries store a per-module string table and integer-indexed
MFAreferences, and the loader unpacks them through a scan-local string pool so repeated module and function names share binaries across the scan. Onlarge, median warm peak memory is 309.5 MiB (down from 823.7 MiB before v4, a 62% reduction) versus 334.7 MiB cold. - Reports and solver costs reflect full evidence generation. Rendering four
formats (including SARIF
codeFlows/relatedLocationsand CycloneDXdependsOndependency graphs) uses iodata JSON serialization (~61 ms onmedium, ~106 ms onlarge), and pre-read feed validation verifies archive integrity on each cache read. - Reports are cheap. Rendering four formats costs a fraction of the analysis; writing SBOM and VEX files next to the primary report does not change scan times measurably.
4. Atom-table guarantees
Atoms are never garbage-collected, so a scanner that converted names from advisories or lockfiles into atoms could exhaust the atom table (1,048,576 entries by default) and crash the VM. VulnCheck keeps every external name as a binary:
- OSV records, CVE records, lockfiles,
.vulncheck.exsand.vulncheck.ignoreare parsed into strings and literals; nothing is evaluated andString.to_atom/1is never called on external input. - Cached
.beamsummaries are decoded with:erlang.binary_to_term(bin, [:safe]). Every atom a summary holds is defined by VulnCheck's own modules, which are loaded before the first decode, so the safe decoder accepts every cache entry in a fresh VM while a tampered cache file that names unknown atoms is treated as a cache miss. - Analyzing a
.beamfile cold reads its abstract code with:beam_lib.chunks/2, which interns the atoms of that compiled module (its module, function and literal names). This is bounded by the code that is actually installed in the project and happens once per changed file; warm runs never read chunks again.
The benchmark reports the atom-count delta over all measured iterations of every stage, which is 0 for every profile. The test suite enforces the same guarantees on every run:
| Invariant | Check |
|---|---|
| Parsing 500 advisories whose module and function names exist nowhere creates no atoms | :erlang.system_info(:atom_count) delta is exactly 0 across OSV.parse_archive/1 and EefCna.apply_record/2 |
| Repeated warm scans create no atoms | Atom-count delta over five full scans is exactly 0 |
A warm cache never re-reads unchanged .beam files | :beam_lib.chunks/2 call count is 0 during a warm scan and every module is a cache hit (100%) |
A changed .beam is re-analyzed, unchanged ones are not | Exactly one miss after rebuilding one module |
| Cached summaries decode in a fresh VM | Every atom in summaries of the Elixir standard library belongs to the modules loaded before decoding |