Check conformance
Use hgrconform to compare a Wingsmith endpoint's OpenAI-compatible behaviour with a reference provider and read the diff.
An endpoint that serves someone else’s model is judged by whether it behaves like the provider’s own API. hgrconform asks both endpoints the same questions and shows where they differ. Flags, profile keys and probes are in the hgrconform reference.
Before you start
- A tenant API key for the endpoint and its gateway URL (see Manage tenants).
- Access to the reference: the model provider’s own API and a key for it.
- Python 3.11 or later, or a container runtime. Nothing is published yet, so use a checkout of the repository.
Install hgrconform
pip install ./libs/conformance
hgrconform probes
probes lists every probe and what it compares.
Write a profile for your endpoint
Copy the shipped gateway profile from libs/conformance/hgrconform/profiles/gateway.toml. Set the URL, model and key variable.
name = "my-gateway"
base_url = "https://<gateway-host>"
model = "<endpoint name>"
token_env = "HANGAR_API_KEY"
Keep context_window, modalities and protocols honest: they are what the endpoint is checked against. Add the names of your own infrastructure to leak_needles.
Write a profile for the reference
Copy the example profile and fill in the provider’s URL, model and a different token_env. The moonshot-kimi-k3 profile is a worked example.
Set [thinking] syntax to how that provider switches thinking on. If the provider sits behind a broker, pin the upstream in extra_body, or the reference may change between requests.
export HANGAR_API_KEY=<tenant API key>
export PROVIDER_API_KEY=<provider key>
Run both
hgrconform run ./my-gateway.toml --reference ./provider.toml
Each endpoint is sent the same probes. By default it skips the heavy eos.suppression probe, which sends 50 requests. Add --heavy for it. Narrow a run with --probe streaming.
Read the diff
The output has three parts.
| Part | Meaning |
|---|---|
not compared against the reference, observed directly | Probes with nothing to compare, such as errors.leakage. Check these on their own. A leak here is a finding without a reference. |
N difference(s) | Each is <probe>.<key> with the target value and the reference value. |
no differences on the compared keys | Agreement on what was compared. It is not proof of correctness. |
The command exits 1 if there is any difference.
Look at the evidence for a difference
hgrconform show streaming.usage_location
This prints the observed values and the raw requests and responses for both endpoints. Add a run directory to pick a run other than the latest. Decide from the bytes whether the difference matters to your callers.
Decide what a difference means
| Difference | Likely cause |
|---|---|
| Only the target errored | A probe the endpoint cannot serve, or a profile claim the endpoint does not meet. |
| Same endpoint, differs between runs | A flapping reference. Pin the upstream with extra_body and run again. |
| A field present on one side only | A parameter or response field the gateway or engine drops or adds. |
| A prompt token count differs | The two endpoints tokenise or template differently. |
Many differences are benign. Expect some on the first run, and write down the ones you accept. There is no stored baseline yet, so a scheduled check would fail on them every time. Keep hgrconform a manual check.
Re-diff without re-asking
hgrconform diff # latest run
hgrconform diff <run directory>
Runs are stored under .local/conformance/<timestamp>-<profile name>/. diff reads only the stored files.
Check without a reference
hgrconform run ./my-gateway.toml
With no reference you get observations, not verdicts. This is enough for errors.leakage and for reading the raw behaviour of one endpoint.
For response differences caused by the gateway itself, the repository has a gateway-only check, just conformance-gateway. It runs the probes through a real gateway in front of a stub and through the stub directly. See Contributing.
To measure speed instead of behaviour, see Benchmark an endpoint.