hgrconform
Install, commands, flags, profile keys, probes and outputs for the hgrconform conformance checker.
hgrconform sends probes to an OpenAI-compatible endpoint and records what it observed. With a second endpoint as a reference, it diffs the two. It knows nothing about the control plane: a Wingsmith gateway is one endpoint it can check, a model provider’s API is another. For a walk-through see Check conformance.
A probe returns facts, never a verdict. With a reference, the verdict is the diff. Without one, you get observations.
Install
Nothing is published. Install from a checkout of the repository, or build the image. It needs Python 3.11 or later.
The profiles and video clips ship inside the package and the image. The image’s entrypoint is hgrconform, and it runs as a non-root user. Runs are written under /work/.local/conformance in the image, so mount that to keep them. Mount any profile of your own and pass its path.
Commands
--results is a top-level flag and goes before the subcommand: hgrconform --results out run gateway.
| Command | What it does |
|---|---|
hgrconform run <profile> | Ask the probes of one endpoint. With --reference, ask the second endpoint too and print the diff. |
hgrconform diff [<run>] | Re-diff a stored run without touching either endpoint. Latest run by default. |
hgrconform probes | List every probe, what it checks and which keys it compares. |
hgrconform show <probe> [<run>] | Print one probe’s observed values and the raw exchanges, for both endpoints. |
Flags
| Flag | On | Effect |
|---|---|---|
--results | top level | Where runs are written. Default .local/conformance. |
--reference <profile> | run | A second profile to diff against. |
--probe <id or group> | run | Only this probe or group. Repeatable. |
--heavy | run | Include eos.suppression, which sends 50 requests by default. |
--verbose | run, diff | Print the difference kind beside each difference. |
A profile is a name from the shipped set (example, gateway, moonshot-kimi-k3) or a path to a TOML file.
Exit code
| Code | Meaning |
|---|---|
| 0 | No reference, or no differences on the compared keys. |
| 1 | At least one difference. |
Every endpoint differs from its reference on something benign, so do not wire this exit code to a schedule or a gate. There is no accepted-baseline mechanism yet.
Profile file
A profile names one endpoint and the vendor-specific facts no probe may hard-code.
| Key | Meaning |
|---|---|
name, base_url, model | Required. |
token_env | Name of the variable holding the bearer token. Default HANGAR_API_KEY. The token is never written to the run directory. |
context_window | Claimed context window, in tokens. |
modalities | Claimed inputs. Default ["text"]. Values used by the shipped profiles: text, image, video. |
protocols | Claimed API surfaces. Default ["chat_completions"]. Shipped profiles also use responses and messages. |
leak_needles | Strings that must not appear in an error body, such as names of your infrastructure. |
timeout | Request timeout in seconds. Default 120. |
extra_body | Fields merged into every request. Use it to pin an upstream behind a broker. |
routes | Where this vendor mounts a canonical path, for example the Anthropic messages route. |
[thinking] | syntax is one of none, kimi, chat_template, reasoning_effort, system_prompt. Also default_on, efforts, prompt_template. |
[paths] | Dotted response paths for cached_tokens, reasoning and reasoning_delta. |
Shipped profiles
| Profile | Endpoint |
|---|---|
example | Annotated template. Copy it per model. |
gateway | A gateway public listener at http://127.0.0.1:8080, model gpt-oss-120b, tenant key in HANGAR_API_KEY, with leak needles for gateway infrastructure names. |
moonshot-kimi-k3 | An example reference profile for a provider’s own API. It reads MOONSHOT_API_KEY. Its claims were read from the provider and can change, so re-check them. |
Probes
| Group | Asks |
|---|---|
auth, basics, errors, protocol | Authentication, response correctness, error envelopes, the three API surfaces. |
streaming, length, sampling | Streaming and usage location, output caps, parameter acceptance. |
structured, tools | response_format modes, tool-call protocol and round trip, tool call id uniqueness. |
thinking, cache, multimodal, tokens | Thinking mode, cached tokens, image, audio and remote URL input, token accounting. |
video | Video input: acceptance, ordering, token cost, limits, streaming. |
eos, limits | EOS suppression rate (heavy), burst policy. |
hgrconform probes is the authoritative list. Some probes declare nothing to compare, such as errors.leakage. In a diff they are printed separately as observed directly, because a provider will never leak your hostnames.
What is compared
| Compared | Not compared |
|---|---|
The shape of an answer, for example streaming.usage_location records usage_position as absent, final-chunk or every-chunk. | Token counts, latencies, generated text, ids, chunk indexes. |
Prompt token counts in tokens.prompt_accounting, basics.multi_turn and video. | The model’s own choices, such as whether it called a tool. Kept as evidence for show. |
| Rates as buckets, over enough samples to be stable. | Single-sample readings. |
A probe sends no sampling parameter unless it tests sampling, so endpoints that pin temperature still answer.
Output
One directory per run: <UTC timestamp>-<target profile name> under --results.
| Path | Content |
|---|---|
run.json | Probe list and every fact, written after each probe. The only file diff and show read. |
target/, reference/ | Every exchange for that endpoint, verbatim. |
Facts
Each probe records observed values and a status for each endpoint. A not-run difference means a probe has a result on one side only. An interrupted run or an erroring reference shows up this way and never reads as agreement. A probe that two profiles both skip counts as agreement.
Diff
The format, with illustrative values:
<target profile> vs <reference profile>
1 difference(s)
streaming.usage_location.usage_position
target "final-chunk"
reference "every-chunk"
The label is <probe id>.<compared key>. With a reference, each endpoint is asked about everything either profile claims, so a capability one profile omits is observed, not skipped.
Reference flapping
A broker that serves one model id from many upstreams can answer differently per request. Pin the upstream with extra_body in the reference profile, and repeat a run before trusting a difference seen once.
Limits
- It does not run accuracy benchmarks. Use an eval harness for those.
- It does not measure performance or capacity. Use hgrbench.
limits.burstobserves the shape of a rate-limit policy and never searches for its ceiling, so a run cannot degrade a live endpoint. - It does not test availability or recovery.
Writing a probe
A probe is a function registered with @probe(id, title, requirement, compare) in a module under hgrconform/probes/. It returns a dict of observed values. Keys named in compare are diffed; the rest are evidence. Raise Skip("reason") when the profile makes the question not apply. See Contributing for the test setup.