Measurement tools

hgrbench

Install, commands, flags, plan file keys, harnesses and outputs for the hgrbench load-test runner.

hgrbench runs load generators against an OpenAI-compatible endpoint, one at a time, from containers on your machine, and writes one HTML report. It does not write to the control plane. For a walk-through see Benchmark an endpoint. For how to judge the numbers see Measurement method.

Install

You need Python 3.11 or later and a container runtime, because the load generators run as containers. hgrbench has no Python dependencies. Nothing is published yet, so install from a checkout of the repository.

pip install ./libs/benchmarking     # provides the hgrbench command
python3 -m hgrbench plans           # or, from libs/benchmarking with no install

Tokenizers are downloaded from Hugging Face and cached in --hf-cache, so the first run needs network access to it.

Commands

Command	What it does
`hgrbench run <plan>`	Take the measurements a plan describes, then write and open the report.
`hgrbench report [<run>]`	Re-render a stored run as HTML without touching the endpoint. Latest run by default.
`hgrbench harnesses`	List each harness, its image, how many canonical fields it reads, and whether it can walk a ladder.
`hgrbench plans`	List the plans shipped with the package, with cell counts and request totals.

<plan> is a shipped plan by name (smoke, deep) or a path to your own TOML file.

`run` flags

Flag	Default	Effect
`--results`	`.local/bench`	Directory where runs are kept, relative to the working directory.
`--harness`	all in the plan	Only this harness. Repeatable.
`--model`	all in the plan	Only this model. Repeatable.
`--hard`	off	Include the saturation search. It can degrade a live endpoint.
`--dry-run`	off	Print every command, run none, keep nothing.
`--timeout`	`1800`	Per-measurement ceiling in seconds.
`--hf-cache`	`~/.cache/huggingface`	Tokenizer cache mounted into the containers.
`--no-open`	off	Do not open the report when done. Opening is only automatic on macOS.

`report` flags

Flag	Effect
`<run>`	A run directory or its `suite.json`.
`--results`	Where to look for the latest run. Default `.local/bench`.
`--out`	Where to write the page. Default `report.html` in the run directory.
`--no-open`	Do not open the page.

Shipped plans

Plan	Time	Matrix	Ladders	Saturation search
`smoke`	about 10 minutes	1 repeat, c1 and c4, 1000 input and 128 output tokens	`vllm-sweep` at 1, 4	off
`deep`	about 2 hours	3 repeats, c1 to c16, 2000 input and 256 output tokens	`vllm-sweep` and `guidellm` at 1 to 32	on, needs `--hard`

Both plans use aiperf, vllm-bench and guidellm in the matrix and measure gpt-oss-120b and gpt-oss-20b. Edit a copy for your own models.

Plan file

A plan is TOML. The endpoint, models, load points and SLA come from the file, not from flags, so what was measured is reviewable in a diff.

Key	Meaning
`name`, `description`	Label for the run. `name` defaults to the file stem.
`repeats`	Repeats of each matrix cell. Default 1.
`cooldown_seconds`	Pause between cells. Default 10.
`input_tokens`, `output_tokens`	Prompt and generation length. Defaults 2000 and 256. A model entry can override both.
`harnesses`	Matrix harnesses. Default `["aiperf"]`.
`[endpoint] url`	Base URL as the containers see it. Default `http://host.docker.internal:8080`.
`[endpoint] probe_url`	Same endpoint as the host sees it. Defaults to `url` with `host.docker.internal` replaced by `localhost`. Before the first cell, `run` sends `GET /v1/models` here and stops if it does not answer 200.
`[endpoint] token_env`	Name of the environment variable holding the bearer token. Default `HANGAR_API_KEY`.
`[[models]]`	`name` is the served model name. `tokenizer` is a Hugging Face repository and is required.
`[[load]]`	`concurrency`, plus `requests`, `seconds` or `request_rate`. `request_rate` is only honoured by open-loop harnesses.
`[sweep]`	`enabled`, `harnesses`, `ladder`, `requests_per_rung`, `seconds_per_rung`, `num_runs`.
`[knee]`	Saturation search: `enabled`, `harness`, `field`, `limit`, `error_limit`, `ladder`, `requests_per_rung`, `seconds_per_rung`.

[endpoint] accepts only url, probe_url and token_env. A token cannot be written into a plan.

Harnesses

Harness	Source	Load model
`aiperf`	NVIDIA AIPerf. The pinned image is the one the plane’s benchmarks module uses.	Closed loop, N in flight
`vllm-bench`	vLLM `bench serve`, from a client-only image built locally.	Open or closed loop
`vllm-sweep`	vLLM `bench sweep serve`. Writes upstream’s own `summary.csv` and plots.	Ladder in one process
`guidellm`	GuideLLM.	Ladder in one process

Only vllm-sweep and guidellm can be listed under [sweep]. Run hgrbench harnesses for the pinned digests and versions on your machine.

Stages

Matrix. Every model, load point, harness and repeat, one cell at a time.
Ladders. Harnesses in [sweep] walk a list of concurrencies in one process.
Saturation search. Climbs the [knee] ladder until field exceeds limit or more than error_limit requests fail, then reports the last rung that held. It runs only when the plan enables it and you pass --hard.

Output

Each run is a directory named <UTC timestamp>-<plan name> under --results.

Path	Content
`suite.json`	Plan source, endpoint, harness image digests, every measurement, readings and named misses. Written after every cell.
`report.html`	The rendered report.
One directory per cell	The harness’s own artifacts, verbatim.

suite.json is the only file report reads, so an interrupted run still renders. A correction to a field map is applied with hgrbench report, never by re-running.

Report sections

Section	Shows
What this run says	The headline findings and caveats.
Behaviour under load	Latency and throughput against concurrency, per model.
Do the harnesses agree?	Each harness’s own value at the same load point and the spread between them.
Run to run	Min, median, max and spread across repeats. With `repeats` at 1 the section only says there is no spread to show.
Every measurement	One row per cell. A value marked with a degree sign is derived; its formula is in the footnotes.

A field a harness could not report is listed as a named miss, not shown as zero.

Behaviour to know about

Behaviour	Consequence
The AIPerf amd64 image hangs under emulation on Apple silicon.	Images are pulled by digest for the host architecture.
`vllm bench serve` exits 0 when every request failed.	A cell is graded on `completed` in the export, not the exit code.
The tokenizer is not the served model name.	Every model needs a `tokenizer`; a model without one is refused.
GuideLLM’s backend validation calls `GET /health`, which the gateway answers with 400 (with a token) or 401 (without).	The runner sets `validate_backend=false`. A “cannot connect” error from GuideLLM run by hand is this check, not the network.
GuideLLM’s `max_requests` applies per rung.	A four-rung ladder at 20 requests sends 80.
GuideLLM’s own `sweep` profile saturates by design.	It is not wired to a plan key. Plans use an explicit ladder.
The vLLM sweep echoes its full command, token included.	Logs are redacted before they are written.
Realised input length differs per harness because of chat templates.	Compare the reported input length in the table, not the flag value.
vLLM’s `max_concurrent_requests` and `max_output_tokens_per_s` are bucketed by integer second.	Neither is read.

Not done yet

GuideLLM’s self-calibrating sweep needs its own --hard-style gate before it can be a plan key.
vllm bench sweep serve_workload works in the image but is not driven by hgrbench.
hgrbench does not submit runs to the control plane.