Wingsmith
wingsmith.dev
Serving

Serve a Model End to End

Create artifact, workload, endpoint; wait for ready; call via gateway.

Serve a model across one or more clusters, from publishing weights to making inference requests.

Prerequisites

  • At least one cluster registered and connected
  • Model weights in a location the cluster can reach (Hugging Face, S3, or HTTP)
  • The gateway deployed and reachable

Publish an artifact

An artifact is a content-addressed reference to model weights. Create one in the console under Serving → Artifacts, or with hgr:

hgr artifact create llama-3-8b-instruct \
  --digest sha256:abcdef1234567890... \
  --location hf://meta-llama/Llama-3-8B-Instruct \
  --format safetensors
  • Digest: a SHA-256 hash of the weights (or a placeholder like placeholder.v1 for unversioned sources)
  • Location: where the cluster fetches the weights. Supported schemes:
    • hf://org/repo (Hugging Face)
    • s3://bucket/prefix (S3-compatible storage)
    • https://... (HTTP server)
  • Format: safetensors, pytorch, or tensorrt

The artifact is immutable. Updating weights means publishing a new artifact and updating the workload to reference it.

Create a workload

A workload declares what you want to run globally. Wingsmith places it on one or more clusters as deployments.

In the console, go to Serving → Workloads and click Create Workload. Fill in:

  • Name: a unique identifier
  • Artifact: select the artifact created above
  • Engine: vllm, sglang, or tensorrt-llm
  • Engine version: e.g. 0.6.5 for vLLM
  • Quantization: none, fp8, int8, int4, or engine-specific (e.g. awq, gptq)
  • TP / PP / EP: tensor, pipeline, and expert parallelism degrees
  • Context length: maximum context in tokens (e.g. 8192)
  • Scale: number of replicas
  • Project: the project this workload belongs to

Or with hgr:

hgr workload create llama-3-8b \
  --artifact llama-3-8b-instruct \
  --engine vllm \
  --engine-version 0.6.5 \
  --quantization none \
  --tp 1 \
  --pp 1 \
  --context-length 8192 \
  --scale 2 \
  --project default

The combination of artifact, engine, version, quantization, and parallelism is hashed into a serving_digest. Workloads with the same serving digest share prefetched weights.

Wait for placement and admission

After creation, the plane places the workload on clusters that satisfy:

  • Enough devices for TP × PP × EP × replicas
  • Available quota in the workload’s project
  • Placement policies (if any)

Each cluster becomes a deployment. Check deployment phases:

hgr deployment ls --workload llama-3-8b

Phases:

  • pending: waiting for admission
  • admitted: passed quota and policy checks, waiting for placement
  • placed: devices allocated, prefetch and rendering in progress
  • running: replicas are ready

In the console, the workload’s detail page shows deployments and their phases.

Create an endpoint

An endpoint is a stable model name that routes requests to one or more deployments.

In the console, go to Serving → Endpoints and click Create Endpoint. Fill in:

  • Name: the string clients send in the model field (e.g. llama-3-8b)
  • Tenant: the tenant that owns this endpoint
  • Visibility: public (served on both listeners) or private (internal listener only)
  • Routes: select the workload(s) this endpoint routes to, with optional weights

Or with hgr (note: endpoint creation via CLI may require API access; check hgr endpoint --help).

Routing weights

If an endpoint routes to multiple workloads, requests are distributed by weight:

  • Workload A (weight 80): 80% of requests
  • Workload B (weight 20): 20% of requests

A deployment reported as not ready is deprioritized but not excluded, because readiness is observed and can lag.

Check readiness

A deployment is ready when all replicas are running and healthy.

hgr deployment get <deployment-id>

Look for:

  • Phase: running
  • Replicas: observed count matches desired scale
  • Allocated devices: devices listed with UUIDs

In the console, the deployment shows green status and lists allocated devices.

Call the endpoint

Internal listener

If the gateway’s internal listener is reachable from your network (typically via kubectl port-forward or a private ingress):

curl http://gateway.example.com:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama-3-8b",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 50
  }'

The internal listener requires no authentication and reaches every enabled tenant’s models, including private endpoints.

Public listener

If the tenant’s ingress is public and the endpoint’s visibility is public:

curl https://tenant-slug.hangar.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-Api-Key: hgrk1.abcdef..." \
  -d '{
    "model": "llama-3-8b",
    "messages": [{"role": "user", "content": "Hello"}],
    "max_tokens": 50
  }'

The Host header identifies the tenant, and X-Api-Key authenticates if the tenant’s auth mode is api_key.

See Call the OpenAI Gateway for full details.

Check replica allocation

Each replica runs on one or more devices (GPUs). View device allocation:

hgr allocation ls --cluster <cluster-name>

Allocations show which devices each replica holds. A replica requesting 2 devices (e.g. TP=2) appears as one allocation holding 2 device UUIDs.

Scale a workload

To change the number of replicas:

hgr workload scale llama-3-8b --replicas 4

Or in the console, from the workload’s detail page, click Scale. The operator reconciles the new desired scale into updated DynamoGraphDeployment objects.

Pause and resume

Pause scales to 0 without deleting the workload:

hgr workload pause llama-3-8b --reason "maintenance window"

Resume restores the previous scale:

hgr workload resume llama-3-8b

Both are recorded as operations with the reason you provided.

Troubleshooting

Deployment stuck in pending

  • Check project quota: hgr quota ls --project default
  • Check device availability: hgr device ls --cluster <cluster> --allocatable
  • Check admission events: hgr event ls --deployment <deployment-id>

Replica not ready

  • Check operator logs: kubectl logs -n hangar-system deployment/hangar-operator
  • Check Dynamo graph status: kubectl get dynamographdeployment -n dynamo
  • Check prefetch job status: kubectl get jobs -n dynamo

Endpoint returns 503 no_capacity

  • Verify agent is connected: gateway metrics show hangar_gateway_tunnel_streams_active{cluster_id}
  • Check deployment readiness: hgr deployment get <deployment-id>
  • Check gateway routing table age: hangar_gateway_routing_table_age_seconds should be low

Next steps