Measurement

Benchmark an endpoint

Run hgrbench against a Wingsmith gateway with a tenant key, read the side-by-side report and compare runs.

This guide measures how a Wingsmith endpoint behaves under load, using four load generators one after another and one report. Flags and plan keys are in the hgrbench reference. Read Measurement method before you quote a number.

Before you start

A running endpoint (see Serve a model) and its public gateway URL.
A tenant API key for it (see Manage tenants).
Python 3.11 or later and a container runtime on the machine you benchmark from.
A checkout of the repository. Nothing is published yet.

Load tests use real GPU time and share a queue with other callers. Run the smoke plan any time. Run deep and --hard only in a window everyone has agreed to.

Install hgrbench

pip install ./libs/benchmarking
hgrbench plans

plans lists smoke and deep with their cell counts.

Export the tenant key

export HANGAR_API_KEY=<tenant API key>

Plans read the key from the variable named by token_env. It is never written into a plan or a result.

Copy a plan and point it at your gateway

Copy the smoke plan from libs/benchmarking/hgrbench/plans/smoke.toml and edit three things.

[endpoint]
url = "https://<gateway-host>"
token_env = "HANGAR_API_KEY"

[[models]]
name = "<endpoint name>"
tokenizer = "<Hugging Face repository for this model>"

url is resolved inside the containers. For a port-forward on your machine, keep http://host.docker.internal:<port>. Remove the models you do not serve.

tokenizer is not the served name. The harnesses count tokens on the client and fail before sending a request if they cannot load one.

Dry run

hgrbench run ./my-smoke.toml --dry-run

Prints every command and sends nothing. Check the model, URL and load points.

Run it

hgrbench run ./my-smoke.toml

The smoke shape takes about ten minutes. The report opens automatically on macOS. Elsewhere, open the path printed at the end.

Results are in .local/bench/<timestamp>-<plan name>/, relative to where you ran the command.

Read the report

Read the sections in this order.

Section	Look for
What this run says	Caveats first. A failed cell or a named miss changes what the numbers mean.
Behaviour under load	Where p90 TTFT starts to climb as concurrency rises. That is the first sign of queueing.
Do the harnesses agree?	Spread per field. Throughput usually agrees closely and TTFT often does not, because the tools build prompts and define the first token differently. A wide spread is about the tools, not the endpoint.
Run to run	The noise floor. Empty when the plan has `repeats` at 1.
Every measurement	Per-cell values, with reported input length. Compare that, not the configured length.

Each harness’s own value is shown. There is no averaged number, because no tool measured one.

Compare two runs

hgrbench has no command that diffs two runs. Put the two reports side by side and compare the same cells.

Keep everything but one thing identical: plan file, model, load points, harnesses, time of day.
Set repeats = 3 or more in the plan. The Run to run section then shows the spread between repeats of one cell.
Treat a difference smaller than that spread as noise. Cell ids are stable between runs of the same plan, so the same cell is the same measurement in both reports.

One run of a shared endpoint is not enough to call a change real. Measurement method covers how to measure the noise floor properly.

Find the saturation point

Copy the deep plan, set the SLA in [knee] to the number your product promises, and run it with --hard.

hgrbench run ./my-deep.toml --hard

The search climbs the ladder until p90 TTFT passes limit or a request fails, then reports the last rung that held. The shipped limit is a placeholder of 2000 ms. Without --hard the plan skips the search and says so.

Re-render without re-running

hgrbench report                          # latest run
hgrbench report .local/bench/<run>       # a specific run

This reads the stored files and never touches the endpoint.

Troubleshooting

Symptom	Cause
Run dies before sending requests	Model has no `tokenizer`, or the tokenizer repository cannot be downloaded.
GuideLLM reports it cannot connect	Its backend check calls `GET /health`. The runner disables it. If you see this running GuideLLM by hand, that is why.
AIPerf produces no output on Apple silicon	An amd64 image under emulation. `hgrbench` pulls the host-architecture image by digest.
A cell is marked failed though the harness exited 0	`vllm bench serve` exits 0 on total failure. The runner grades `completed` in the export.

To measure only the path through the gateway and not an engine, use hgrtransit. To check that responses match a provider’s, see Check conformance.