Documentation
From docs/benchmark.md · e31dc91 · Built 2026-11-17
| Measurement | Value | Measured | Expires |
|---|---|---|---|
| Runtime dependencies | 0 dependencies | 2026-10-25 | 2027-01-25 |
| Passing Node tests, including Python suite checks | 280 tests | 2026-10-25 | 2027-01-25 |
| Failing Node tests at measurement time | 0 tests | 2026-10-25 | 2027-01-25 |
| Blocking guard rules verified by the suite | 7 rules | 2026-10-25 | 2027-01-25 |
| Packed tarball size reported before this proof refresh | 94492 bytes | 2026-10-25 | 2027-01-25 |
| Files in the packed tarball | 82 files | 2026-10-25 | 2027-01-25 |
| Plain session median input-context tokens per browser step | 188487 tokens | 2026-10-25 | |
| Runner median input-context tokens per browser step | 59725 tokens | 2026-10-25 | |
| Plain-to-runner median browser-step token ratio | 3.16 ratio | 2026-10-25 | |
| Plain arm total browser-step tokens | 1506374 tokens | 2026-10-25 | |
| Runner arm total browser-step tokens | 725151 tokens | 2026-10-25 | |
| Plain-to-runner arm total browser-step tokens ratio | 2.08 ratio | 2026-10-25 | |
| Plain arm whole-run input-context tokens | 2657290 tokens | 2026-10-25 | |
| Runner arm whole-run input-context tokens | 2846792 tokens | 2026-10-25 | |
| Plain-to-runner whole-run input-context token ratio | 0.93 ratio | 2026-10-25 | |
| Estimated break-even browser steps (an estimate from two data points; no run longer than 9 browser steps was measured) | 15.54 steps | 2026-10-25 | |
| Small task (one page, 4 browser steps), native Claude in Chrome: whole-run input-context tokens | 1133195 tokens | 2026-10-25 | |
| Small task (one page), with the package, main session plus runner: whole-run input-context tokens | 1241361 tokens | 2026-10-25 | |
| Small task: plain-to-package whole-run input-context token ratio | 0.91 ratio | 2026-10-25 | |
| Small task: plain median browser step to runner median browser step token ratio | 3.26 ratio | 2026-10-25 | |
| Run 1 plain arm dollar cost at list API prices | 0.5975 usd | 2026-10-25 | |
| Run 1 runner arm dollar cost at list API prices | 0.5787 usd | 2026-10-25 | |
| Run 2 plain arm dollar cost at list API prices | 0.5779 usd | 2026-10-25 | |
| Run 2 runner arm dollar cost at list API prices | 0.4319 usd | 2026-10-25 | |
| Best dollar saving in two measured runs on one setup | 25.26 percent | 2026-10-25 | |
| Pooled dollar saving across two measured runs on one setup | 14.02 percent | 2026-10-25 |
Reproduce a paired comparison
- Choose a public-page task and save its exact prompt. Record the model, browser tools and starting session size for both arms.
- Prepare separate scratch folders for the plain arm and the skill-and-runner arm. In the runner scratch folder, install the package into that project:
npx lean-browser-use --project
- Run the task prompt once in a plain Claude Code session. Run the same prompt once in a session using the
lean-browser-useskill and its runner. Keep each arm's transcripts in its own folder, including subagent transcripts. - Measure each transcript folder with the same package version:
npx lean-browser-use measure --dir ./plain-transcripts
npx lean-browser-use measure --dir ./runner-transcripts
Quote folder paths when they contain spaces. Compare the plain main median with the runner median. For the task totals, sum the main and runner group totals within each arm. Save the meter output and conditions beside the transcripts. How the meter counts explains grouping and token fields.
How far to trust it
These are two paired runs of public-page tasks on one setup. Results move with the model, session size and task. The arms exercise different execution routes, so the comparison describes that pair rather than isolating a single cause.
Tokens re-read are not a price list: cached tokens and fresh input tokens can have different prices. Cost varies by model, and this ballpark guarantees no particular saving. Measure your own similar work to see what changes for you.