Available Providers

QueryMT ships with 15 providers split across two types: WASM (API-based, cloud services) and Native (local inference, runs models on your hardware). This page covers how to configure each type, how to pick the right build variant for your hardware, and provides copy-pasteable configuration recipes.

Provider Repository (repo.query.mt)

QueryMT maintains an always-up-to-date provider repository at https://repo.query.mt.

• latest.json — updated on every push to main (default)
• stable.json — updated on every tagged release

If no providers config exists at ~/.qmt/providers.toml (or .json / .yaml), QueryMT automatically fetches latest.json and caches it to ~/.qmt/providers.json on first run.

1
2

# Refresh to the latest provider list
qmt update

Override the source URL:

1

QMT_PROVIDERS_URL=https://repo.query.mt/stable.json qmt update

WASM Providers (API-based)

WASM providers call remote APIs. They are sandboxed WebAssembly modules that run on any platform and require no local hardware beyond an internet connection. All 12 WASM providers are listed in the provider repository, so you don't need to configure them manually unless you want to pin a version or customise settings.

Minimal WASM configuration

You only need to specify a provider if you want to override defaults or pin a version. Most users can skip this entirely.

1
2
3

[[providers]]
name = "anthropic"
path = "oci://ghcr.io/querymt/anthropic:latest"

1
2
3

[[providers]]
name = "openai"
path = "oci://ghcr.io/querymt/openai:latest"

1
2
3

[[providers]]
name = "google"
path = "oci://ghcr.io/querymt/google:latest"

1
2
3

[[providers]]
name = "mistral"
path = "oci://ghcr.io/querymt/mistral:latest"

1
2
3

[[providers]]
name = "groq"
path = "oci://ghcr.io/querymt/groq:latest"

1
2
3

[[providers]]
name = "ollama"
path = "oci://ghcr.io/querymt/ollama:latest"

1
2
3

[[providers]]
name = "openrouter"
path = "oci://ghcr.io/querymt/openrouter:latest"

1
2
3

[[providers]]
name = "alibaba"
path = "oci://ghcr.io/querymt/alibaba:latest"

1
2
3

[[providers]]
name = "moonshot"
path = "oci://ghcr.io/querymt/moonshot:latest"

1
2
3

[[providers]]
name = "kimi-code"
path = "oci://ghcr.io/querymt/kimi-code:latest"

1
2
3

[[providers]]
name = "xai"
path = "oci://ghcr.io/querymt/xai:latest"

1
2
3

[[providers]]
name = "codex"
path = "oci://ghcr.io/querymt/codex:latest"

Native Providers (local inference)

Native providers run LLMs locally on your machine. They are distributed as platform-specific shared libraries (.so / .dylib / .dll) packaged into OCI images. Unlike WASM providers, QueryMT resolves native providers at download time by inspecting the OCI image index for a manifest that matches your OS and architecture. If a match is found, the native library is downloaded; if not, QueryMT falls back to the WASM variant if one exists.

There are three native providers:

Feature tags

Each native provider is published with multiple OCI tags corresponding to different hardware backends. The tag you specify in path controls which binary is downloaded.

Version-pinned variants follow the same pattern with the crate version prepended: 0.1.0-metal, 0.1.0-cuda12.8-sm89, etc.

NVIDIA GPU to SM architecture

Use this table to find the correct cuda12.8-smXX suffix for your GPU.

AMD ROCm compatibility

The llama-cpp latest-rocm7.2 variant is built on Ubuntu 24.04 with ROCm 7.2.1, matching llama.cpp's Linux ROCm Docker build. One binary contains all of the upstream build's GPU targets; no per-GPU tag is needed:

The OCI artifact contains the provider library only, not ROCm or GPU drivers. Use Ubuntu 24.04 or a compatible environment with the ROCm 7.2.1 HIP runtime, hipBLAS, and rocBLAS (including its GPU kernel data) installed. Follow AMD's ROCm 7.2.1 installation instructions and check the GPU/driver requirements for your hardware. The dynamic linker must be able to resolve libamdhip64.so, libhipblas.so, librocblas.so, and their dependencies. Nonstandard installations, including NixOS, require an environment providing those libraries; the downloaded artifact is not self-contained.

The process also needs permission to access the AMD GPU (/dev/kfd and the appropriate /dev/dri render node). GPUs outside the compiled list can use latest-vulkan or a source build with a custom target list. This initial ROCm variant is not published for Windows or Linux arm64.

Choosing a feature tag

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

What OS are you on?
├── macOS Apple Silicon (M1/M2/M3/M4)  → latest-metal
├── macOS Intel
│   ├── izwi / mrs                     → latest-accelerate
│   └── llama-cpp                      → latest-default
├── Linux
│   ├── NVIDIA GPU
│   │   ├── llama-cpp                  → latest-cuda12.8
│   │   └── izwi / mrs                 → latest-cuda12.8-sm{XX}  (see table above)
│   ├── AMD GPU + ROCm 7.2 (x86_64)     → latest-rocm7.2 (llama-cpp; validation pending)
│   ├── Other AMD / Intel GPU          → latest-vulkan  (llama-cpp only)
│   └── CPU only                       → latest  (or latest-default)
└── Windows
    ├── NVIDIA GPU                     → latest-cuda12.8  (llama-cpp only)
    └── CPU only                       → latest

Provider reference

anthropic

Claude models through the Anthropic Messages API. Requests are shaped per model, so the same session settings are valid on every Claude generation.

Model classes and "auto" effort

The model ID (including Bedrock, Vertex and router forms such as us.anthropic.claude-opus-4-7-v1:0, claude-sonnet-4-6@20260101 or anthropic/claude-sonnet-5-5) is parsed into family and version and falls into one class:

Effort and budget fitting

On adaptive models the effort is sent unchanged as output_config.effort. On budget models the budget is reasoning_budget_tokens when configured, otherwise a default per effort (low 1,024, medium 8,000, high 16,000, max 31,999). It is then fitted to the request: at least 1,024 and at most half of max_tokens. When max_tokens is below 2,048 thinking is omitted. Budget models never receive output_config.

Sampling parameters

• On Opus 4.7 or later and on every Claude 5 or later, Fable or Mythos model, temperature, top_p and top_k are never sent.
• On other models, whenever thinking is sent, temperature and top_k are omitted and top_p is sent only if it is between 0.95 and 1.
• Otherwise the configured values are sent.

Each dropped value is logged at debug level.

Forced tool choice

A forced tool_choice is any or a named tool.

• Budget models: thinking is dropped from that request.
• Opus 5.5 or later, Sonnet 5.5 or later, Fable 5.1 or later, Mythos 5.1 or later: the API rejects forced tool choice, so the request fails locally with an invalid-request error.
• Other adaptive models: sent unchanged.

Thinking replay, usage and stop reasons

Each thinking block becomes its own reasoning item with its own signature. Signed blocks are replayed unchanged, even when their text is empty (the default display: "omitted" on Opus 4.7+ and Claude 5). redacted_thinking blocks are kept and replayed with their opaque data. output_tokens_details.thinking_tokens is reported as reasoning_tokens and subtracted from the inclusive output_tokens, so ordinary output and reasoning never overlap. Streaming usage ends with the final message_delta values: the provisional output count in message_start is not emitted as ordinary output, fields a later event omits (input, cache) keep their earlier values, and an explicit zero replaces them. Cache writes keep the returned cache_creation five-minute and one-hour counts as a breakdown of the aggregate cache_creation_input_tokens (never added on top); with no breakdown the tiers stay absent rather than assumed. Stop reasons map as: end_turn/stop_sequence to stop, tool_use to tool calls, max_tokens/model_context_window_exceeded to length, refusal to content filter, pause_turn to other.

adaptive_thinking override

An optional provider config field overrides the class derived from the model ID:

1
2

[providers.anthropic]
adaptive_thinking = true   # force opt-in adaptive thinking

• true sends adaptive thinking explicitly. This is the right choice for a proxy or custom model ID that can't be parsed but fronts an adaptive model.
• false forces budget thinking. The API rejects budget thinking on Opus 4.7+ and every Claude 5 or later model.

The sampling and forced-tool-choice rules always use the parsed model ID and ignore this override.

Strict tool use (strict_tools)

Strict tool use makes the API enforce a tool's input schema during sampling. One optional provider setting controls it:

1
2

[providers.anthropic]
strict_tools = ["edit", "multiedit", "replace_symbol"]

A tool is eligible when its definition sets strict: true or its unprefixed name is in strict_tools (names are matched before the OAuth mcp_ prefix is applied). The provider itself has no default list: when strict_tools is unset or empty, no tool is selected by name, but tools whose definitions explicitly set strict: true stay eligible. The QueryMT agent fills in ["edit", "multiedit", "replace_symbol"] for Anthropic when you have not set strict_tools; set strict_tools = [] to turn that off. Builtin tool definitions never set strict, so OpenAI Responses behaviour is unchanged.

Eligible tools are sent with "strict": true only when every check passes:

• Model. Opus and Sonnet 4.5 or later, Haiku 4.5 or later, and all Fable and Mythos models. Older or unrecognised model IDs send no strict field and the original schemas (logged at debug level).
• Schema. Strict mode accepts a subset of JSON Schema, so the provider converts a copy of each schema: fixed-property objects are closed (additionalProperties: false), and numeric limits, string-length limits, maxItems and minItems above 1 move into the property description, for example (minimum: 0). Required and optional properties are unchanged. The agent still validates calls against the original schema. A tool whose schema cannot be converted safely (external or recursive references, dictionaries or open objects, pattern, unsupported formats, oneOf/allOf/not and other unknown keywords) is sent non-strict with its original schema and a warning naming the tool and the unsupported feature. Other eligible tools stay strict. This applies to explicit and name-selected strictness alike.
• Budgets. Counted over the successfully converted tools only: at most 20 strict tools, 24 optional properties and 16 union properties (anyOf or type arrays). If any limit is exceeded, every tool in that request is sent non-strict with its original schema and a warning names the limit.

Strict tools work the same over API key and OAuth: OAuth strict requests were verified against the live API on claude-sonnet-4-5 and claude-sonnet-5-5 (see scripts/verify-anthropic-strict.sh).

Usage normalization across API providers

Every provider adapter reports usage in the same exclusive categories: uncached input, ordinary output, reasoning, cache reads and cache writes. Each token appears in one category, so totals and cost estimates never count it twice. What each wire format needs to get there:

These adapters only report what the response states. They never infer a cache counter from the requested cache settings or the provider name.

llama-cpp

Wraps llama.cpp. Supports GGUF models, vision/multimodal models, and streaming. Broadest hardware support of the three native providers.

Supported platforms:

*ROCm is published with compiled target coverage; GPU runtime validation is pending. See AMD ROCm compatibility.

Configuration:

1
2
3
4
5
6
7
8

[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest-metal"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 4096
n_gpu_layers = 99

1
2
3
4
5
6
7
8

[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 4096
n_gpu_layers = 0

1
2
3
4
5
6
7
8
9

[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest-cuda12.8"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 4096
n_gpu_layers = 99
flash_attention = "enabled"

1
2
3
4
5
6
7
8

[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest-rocm7.2"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 4096
n_gpu_layers = 99

1
2
3
4
5
6
7
8

[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest-vulkan"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 4096
n_gpu_layers = 99

1
2
3
4
5
6
7
8
9

[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest-cuda12.8"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 4096
n_gpu_layers = 99
flash_attention = "enabled"

Key config options:

For vision model configuration and the full option reference see the llama-cpp provider README.

izwi

Wraps izwi-core. Efficient local inference with Flash Attention support. CUDA builds are compiled per SM architecture for maximum performance.

Supported platforms:

Configuration:

1
2
3
4
5
6

[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest-metal"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6

[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest-accelerate"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6

[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7

# SM 86: RTX 3060, 3070, 3080, 3090, A10, A40
[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest-cuda12.8-sm86"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7

# SM 89: RTX 4060, 4070, 4080, 4090, L4, L40, L40S
[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest-cuda12.8-sm89"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7

# SM 80: A30, A100
[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest-cuda12.8-sm80"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7

# SM 120: RTX 5070, 5080, 5090, B100, B200
[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest-cuda12.8-sm120"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7

# SM 87: Jetson AGX Orin, Orin NX, Orin Nano
[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest-cuda12.8-sm87"

[providers.config]
model = "/path/to/model"

mrs

Wraps mistral.rs. High-performance inference for Mistral and compatible architectures. CUDA x86_64 builds include both Flash Attention and cuDNN.

Supported platforms:

Configuration:

1
2
3
4
5
6

[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest-metal"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6

[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest-accelerate"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6

[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7

# SM 86: RTX 3060, 3070, 3080, 3090, A10, A40
[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest-cuda12.8-sm86"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7

# SM 89: RTX 4060, 4070, 4080, 4090, L4, L40, L40S
[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest-cuda12.8-sm89"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7

# SM 80: A30, A100
[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest-cuda12.8-sm80"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7

# SM 120: RTX 5070, 5080, 5090, B100, B200
[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest-cuda12.8-sm120"

[providers.config]
model = "/path/to/model"

1
2
3
4
5
6
7
8

# SM 87: Jetson AGX Orin, Orin NX, Orin Nano
# Note: cuDNN included; Flash Attention not available on aarch64
[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest-cuda12.8-sm87"

[providers.config]
model = "/path/to/model"

Complete configuration recipes

Drop one of these into ~/.qmt/providers.toml and adjust paths as needed.

Cloud APIs only

No config needed — repo.query.mt handles this automatically on first run. If you want explicit control:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

[[providers]]
name = "openai"
path = "oci://ghcr.io/querymt/openai:latest"

[[providers]]
name = "anthropic"
path = "oci://ghcr.io/querymt/anthropic:latest"

[[providers]]
name = "google"
path = "oci://ghcr.io/querymt/google:latest"

Cloud + local llama-cpp on macOS Apple Silicon

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

[[providers]]
name = "openai"
path = "oci://ghcr.io/querymt/openai:latest"

[[providers]]
name = "anthropic"
path = "oci://ghcr.io/querymt/anthropic:latest"

[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest-metal"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 8192
n_gpu_layers = 99
flash_attention = "auto"

Fully local on Linux with NVIDIA RTX 40-series

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# llama-cpp: single CUDA build, all SM supported at runtime
[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest-cuda12.8"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 8192
n_gpu_layers = 99
flash_attention = "enabled"

# izwi: SM 89 for RTX 4060/4070/4080/4090
[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest-cuda12.8-sm89"

[providers.config]
model = "/path/to/model"

# mrs: SM 89 for RTX 4060/4070/4080/4090
[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest-cuda12.8-sm89"

[providers.config]
model = "/path/to/model"

Fully local on macOS Apple Silicon

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest-metal"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 8192
n_gpu_layers = 99

[[providers]]
name = "izwi"
path = "oci://ghcr.io/querymt/izwi:latest-metal"

[providers.config]
model = "/path/to/model"

[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest-metal"

[providers.config]
model = "/path/to/model"

Fully local on Linux (CPU only)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

[[providers]]
name = "llama-cpp"
path = "oci://ghcr.io/querymt/llama-cpp:latest"

[providers.config]
model = "/path/to/model.gguf"
n_ctx = 4096
n_gpu_layers = 0

[[providers]]
name = "mrs"
path = "oci://ghcr.io/querymt/mrs:latest"

[providers.config]
model = "/path/to/model"

Further reading

• Plugin Configuration — full providers.toml field reference, OCI signature verification, JSON/YAML format
• OCI Plugins — how OCI image resolution and caching works
• llama-cpp provider README — vision models, full config reference, troubleshooting