QueryMT Agent - Mesh Networking
Mesh networking enables QueryMT Agent to collaborate across multiple machines, allowing sessions to be shared, delegates to run remotely, and LLM calls to be routed to specific nodes.
Overview
Mesh networking uses the kameo actor framework with libp2p for peer-to-peer communication. This enables:
- Cross-machine sessions: Share sessions across multiple machines
- Remote agents: Access agents running on other machines
- Distributed computation: Run heavy tasks on specialized hardware
- Load balancing: Distribute work across multiple nodes
Architecture
flowchart LR
subgraph A["Machine A"]
AS[Agent Session]
AG[GPU Worker]
end
subgraph B["Machine B"]
BS[Agent Session]
BL[LLM Provider]
end
AS <-->|"Internet<br/>(libp2p Mesh)"| BS
Quick Start
High-Level API
Mesh::hybrid() is the recommended default for API users. It starts one process-shared runtime with LAN discovery plus iroh transport enabled, so invites can be joined later on the same runtime.
Starting a Mesh Node
Connecting to a Mesh
Configuration
Basic Mesh Configuration
This is the config equivalent of Mesh::hybrid(): LAN stays enabled, and iroh transport is available for ad-hoc invite joins.
Configuration Reference
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Enable mesh networking |
listen |
string | /ip4/0.0.0.0/tcp/0 |
Multiaddr to listen on |
transport |
string | "lan" |
Transport layer: "lan" or "iroh" |
iroh_gso |
bool | true |
Enable UDP Generic Segmentation Offload on the shared iroh actor endpoint |
discovery |
string | "mdns" |
Peer discovery method |
auto_fallback |
bool | false |
Allow mesh provider discovery fallback |
node_name |
string | OS hostname | Human-readable node name advertised to peers |
identity_file |
string | ~/.qmt/mesh_identity.key |
Path to ed25519 identity keypair |
request_timeout_secs |
u64 | 300 |
Timeout for non-streaming requests (seconds) |
stream_reconnect_grace_secs |
u64 | 120 |
Grace period for stream reconnection (seconds) |
invite |
string | - | Invite token to join existing mesh (supports ${VAR} interpolation) |
Transport Modes
QueryMT supports two transport modes optimized for different network environments:
LAN Transport (Default)
Traditional TCP + QUIC transport optimized for local area networks:
Characteristics: - ✅ Fast: Optimized for low-latency LAN connections - ✅ Zero-config discovery: Automatic peer discovery via mDNS - ✅ Low overhead: Direct TCP/QUIC connections - ❌ LAN only: Cannot traverse NAT or connect over internet - ❌ Subnet limitations: mDNS may not work across different subnets
Use cases: - Development environments on local network - Office networks with multiple machines - High-bandwidth, low-latency requirements
Iroh Transport
Internet-capable transport with NAT traversal using the iroh networking library:
GSO batches outgoing UDP packets to reduce CPU overhead and is enabled by default. On systems with problematic UDP offload paths, opt out explicitly:
This is a compatibility workaround, not a universal reliability guarantee. The
setting applies to the shared actor endpoint across all [[mesh.iroh]] scopes,
including mixed LAN+iroh runtimes and invite joins/reconnects. It does not change
LAN transport, encryption, authentication, or separately created payload endpoints.
The endpoint reads this setting at startup; restart to change it. qmtcode also
honors it from an explicit TOML config file or the profile selected by --profile.
Override it for this process without editing a profile:
Precedence: --mesh-iroh-gso > QMT_MESH_IROH_GSO > config/profile
[mesh].iroh_gso > default true. CLI and environment values must be true or
false; invalid values are rejected. An explicit --mesh-iroh-gso=true overrides
an environment value of false. The resolved value is applied before endpoint
bind for hosting, joining, and reconnecting, and stays fixed across profile
switches/reloads until the process restarts. These overrides require a build with
the remote feature; they do not enable mesh networking by themselves.
Rust callers set MeshRuntimeConfig::iroh_gso. Existing exhaustive runtime-config
struct literals must add iroh_gso: true to preserve previous behavior.
Characteristics: - ✅ Internet-capable: Works across the internet - ✅ NAT traversal: Automatic hole punching and relay fallback - ✅ Encrypted: Built-in encryption and authentication - ✅ Relay network: Falls back to relay servers when direct connection fails - ❌ Higher latency: Slightly higher overhead than direct LAN - ❌ Requires invite: Typically uses invite tokens for secure joining
Use cases: - Remote team collaboration - Distributed nodes across different networks - Nodes behind NAT or firewalls - Cloud + on-premises hybrid setups
Iroh DNS Policy
Iroh 1.3 uses system DNS with deferred public fallback by default. The N0 preset also resolves endpoint addresses through HTTPS pkarr alongside DNS, improving endpoint-ID-only dialing without changing reliable actor/provider streams.
Override DNS policy for the shared iroh endpoint, including hybrid runtimes:
Fallback modes are deferred (query fallback after primary failures), eager
(race primary and fallback), if_system_empty (fallback only when no system
nameservers are available), and never. Without system defaults, if_system_empty
behaves like eager fallback, following upstream semantics. Nameserver protocols
are udp, tcp, tls, and https; encrypted protocols optionally accept
tls_server_name. To replace public fallback resolvers, configure
[[mesh.iroh_dns.fallback_nameservers]] using the same nameserver fields.
Omitting iroh_dns preserves upstream defaults. Public fallback can disclose
queries or bypass corporate DNS policy; use explicit policy where required.
Disabling DNS fallback does not disable N0's public pkarr publisher/resolver
or relays and is not a private-network mode.
Settings apply at endpoint startup across all Iroh scopes. qmtcode preserves
the startup config/profile DNS policy for hosting, invite joins, reconnects, and
profile switches; restart to change it. Rust callers set
MeshRuntimeConfig::iroh_dns (None preserves defaults), or use
join_mesh_via_invite_with_dns for ad-hoc invite joins. Exhaustive runtime-config
struct literals must add iroh_dns: None to preserve existing behavior.
Comparison Table
| Feature | LAN Transport | Iroh Transport |
|---|---|---|
| Network | Local network only | Internet-wide |
| Discovery | mDNS (automatic) | Invite tokens |
| NAT Traversal | No | Yes (hole punching + relay) |
| Latency | Very low | Low-medium |
| Setup Complexity | Minimal | Requires invite workflow |
| Security | Network-level | Built-in encryption |
| Best For | Development, LAN | Production, remote teams |
Discovery Methods
mDNS (Default)
Automatic discovery on local network:
- Pros: Zero-config, automatic
- Cons: Local network only, may miss peers on different subnets
Kademlia DHT
Distributed discovery across the internet:
- Pros: Cross-subnet, internet-wide
- Cons: Requires bootstrap nodes, more complex
Manual Peers
Explicit peer connections:
- Pros: Precise control, reliable
- Cons: Manual configuration required
Iroh Relay Discovery
Automatic discovery via iroh relay network (used with iroh transport):
How it works: 1. Host creates invite token with embedded PeerId 2. Joiner connects to inviter via iroh relay 3. Relay network handles NAT traversal 4. Direct connection established when possible (hole punching) 5. Falls back to relay if direct connection fails
Pros: - Works across the internet - Automatic NAT traversal - No manual peer configuration needed - Encrypted by default
Cons: - Requires invite token workflow - Slightly higher latency than LAN - Depends on iroh relay network availability
Discovery Method Selection Guide
| Scenario | Recommended Method | Reason |
|---|---|---|
| Development on local LAN | mdns + lan transport |
Zero-config, automatic |
| Office network, multiple subnets | kademlia + lan transport |
Cross-subnet discovery |
| Remote team, internet nodes | Invite tokens + iroh transport |
NAT traversal, secure |
| Production, fixed infrastructure | none + explicit peers |
Predictable, reliable |
| Mixed environment | Multi-transport (see below) | Best of both worlds |
Multi-Transport Setup
QueryMT supports running multiple transports simultaneously, enabling nodes to participate in both LAN and internet meshes at the same time.
Multi-Transport Architecture
flowchart TB
subgraph Node["Mesh Node"]
direction TB
LAN["LAN Transport<br/>(mDNS discovery)"]
Iroh1["Iroh Scope: 'personal'<br/>(Invite-based)"]
Iroh2["Iroh Scope: 'team'<br/>(Invite-based)"]
end
LAN <--> Peers1["LAN Peers"]
Iroh1 <--> Peers2["Internet Peers<br/>(personal mesh)"]
Iroh2 <--> Peers3["Internet Peers<br/>(team mesh)"]
Basic Multi-Transport Configuration
Configuration Reference
LAN Subtable ([mesh.lan])
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Enable LAN transport |
listen |
string | Inherited from [mesh] listen |
LAN-specific listen address |
discovery |
string | Inherited from [mesh] discovery |
LAN discovery method |
Iroh Scope Array ([[mesh.iroh]])
| Field | Type | Default | Description |
|---|---|---|---|
enabled |
bool | false |
Enable this Iroh scope |
name |
string | - | Scope identifier (becomes mesh_id when no invite) |
invite |
string | - | Invite token to join existing mesh (supports ${VAR}) |
Multi-Transport Use Cases
Use Case 1: LAN + Personal Internet Mesh
Local development with ability to connect from home:
Benefits: - Fast LAN connections for local machines - Access from home via internet mesh - Single node participates in both networks
Use Case 2: Multiple Team Meshes
Participate in multiple internet meshes:
Benefits: - Single node accessible from multiple teams - Team-specific mesh isolation - Centralized node management
Use Case 3: LAN with Internet Fallback
Primary LAN mesh with internet access for remote workers:
Multi-Transport Routing
When a node participates in multiple meshes, routing considers:
- Direct connections: Prefer direct peer connections
- LAN priority: LAN connections preferred over internet (lower latency)
- Scope matching: Route to peers in same scope when possible
- Fallback: Use relay if direct connection fails
Multi-Transport Security
Each transport maintains separate security context:
- LAN: Network-level security (same subnet trust)
- Iroh scopes: Cryptographic isolation between scopes
- Invite tokens: Per-scope access control
- Identity: Same node identity across all transports
flowchart LR
subgraph Security["Security Layers"]
LAN["LAN<br/>(Network trust)"]
Iroh["Iroh Scopes<br/>(Crypto isolation)"]
Token["Invite Tokens<br/>(Access control)"]
end
LAN --> Iroh --> Token
Migrating to Multi-Transport
From LAN-only to Multi-Transport
Before (LAN only):
After (LAN + Internet):
From Single Iroh to Multi-Transport
Before (Single Iroh):
After (Multiple Iroh scopes):
Multi-Transport Best Practices
- Name scopes descriptively: Use meaningful names for Iroh scopes
- Environment variables: Store invites in env vars, not config files
- Scope isolation: Keep different teams/projects in separate scopes
- Monitor connectivity: Check all transports are healthy
- Test failover: Verify behavior when one transport fails
- Document topology: Keep track of which nodes are in which scopes
Multiaddr Format
Mesh addresses use libp2p multiaddr format:
Examples
Invite Token System
QueryMT provides a secure invite token system for dynamically joining meshes, particularly useful for internet-capable iroh transport. This system uses cryptographic signatures to ensure only authorized nodes can join.
Overview
Invite tokens are signed grants that allow nodes to securely join an existing mesh:
- Cryptographically signed: Each invite is signed with the host's ed25519 identity keypair
- Self-verifying: Joiners can verify the signature offline (no network required)
- Time-limited: Tokens expire after configurable duration (default: 24 hours)
- Use-limited: Tokens can be single-use or have a maximum use count
- Role-based: Tokens grant specific permissions (member vs client role)
- Shareable: Compact format fits in QR codes or URLs
Token Format
Invite tokens are encoded in multiple formats:
- Base64url string: Compact string for CLI usage (~470 characters)
- QR code: Scannable QR code (version 14, 73x73 pixels)
- URL format:
qmt://mesh/join/<base64_token> - Direct CLI: Pass token directly to
--mesh-join
Creating Invites
Via CLI
Via TOML Configuration
Joining via Invite Tokens
Via CLI
Via TOML Configuration
Invite Security Model
Ed25519 Identity Keypair
Each node has a persistent identity keypair:
- Location:
~/.qmt/mesh_identity.key(configurable viaidentity_file) - Generated: Automatically on first run
- Persistent: Same PeerId across restarts
- Secure: Private key never leaves the host
Signature Verification
The invite token verification process:
- Host creates invite: Signs invite grant with ed25519 private key
- Token shared: Base64url encoded token shared out-of-band
- Joiner receives token: Decodes and extracts grant + signature
- Offline verification: Verifies signature using inviter's public key (embedded in token)
- Connection attempt: Dials inviter's PeerId via iroh relay network
- Mesh join: Establishes encrypted connection and joins mesh
sequenceDiagram
participant Host as Mesh Host
participant Token as Invite Token
participant Joiner as Joining Node
Host->>Host: Generate invite grant
Host->>Host: Sign with ed25519 private key
Host->>Token: Encode as base64url (~470 chars)
Token->>Joiner: Share via URL/QR/CLI
Joiner->>Joiner: Decode and verify signature (offline)
Joiner->>Host: Dial PeerId via iroh relay
Host->>Joiner: Establish encrypted connection
Joiner->>Host: Request mesh membership
Host->>Joiner: Grant membership (check expiry, use count)
Invite Grant Structure
QR Code Support
Invite tokens fit in QR codes for easy mobile sharing:
- Size: QR version 14 (73x73 pixels)
- Capacity: ~470 characters
- Format:
qmt://mesh/join/<base64_token>
Invite Management
Environment Variable Interpolation
TOML configuration supports ${VAR} syntax for secure token storage:
Token Lifecycle
- Creation: Host generates signed grant with TTL and use limits
- Distribution: Token shared via secure channel (URL, QR, CLI)
- Verification: Joiner verifies signature offline
- Connection: Joiner connects via iroh relay network
- Validation: Host checks expiry and use count
- Membership: Joiner granted mesh access
- Revocation: Token expires or use limit reached
Example Workflows
Workflow 1: Quick Team Setup
Workflow 2: QR Code Sharing
Workflow 3: Environment Variable for CI/CD
Best Practices
- Use short TTLs: Default 24h is good for temporary access
- Limit uses: Set
invite_uses=1for single-use tokens - Secure distribution: Share tokens via encrypted channels
- Environment variables: Store tokens in env vars, not config files
- Rotate tokens: Create new tokens periodically
- Monitor usage: Track invite acceptance in logs
Security Considerations
✅ Strong cryptography: Ed25519 signatures (64 bytes) ✅ Offline verification: No network needed to verify token ✅ Time-limited: Tokens expire automatically ✅ Use-limited: Prevents token reuse ✅ Role-based: Granular permission control ✅ Signed: Tamper-proof (signature invalid if modified) ✅ Auditable: Each invite has unique ID for tracking
⚠️ Token exposure: Treat tokens like passwords - don't commit to git ⚠️ Transport security: Always use iroh (encrypted) for internet meshes ⚠️ Revocation: Tokens cannot be revoked before expiry (use short TTLs)
Ad-Hoc Invite Join
Use ad-hoc joins when the agent is already running and you want to add a new iroh mesh scope later.
The returned MeshJoinOutcome reports:
mesh_id: the derived logical scope idmesh_name: optional invite labelinviter_peer_id: the inviter peer id from the invitealready_joined: whether this join was idempotent
Remote Agents
Define agents that run on remote mesh nodes:
Remote Delegate
Delegates can run on remote nodes:
Behavior: - LLM calls are routed to the remote node - Tool execution happens locally on the planner node - Enables "remote model, local session" pattern
Identity Management
Each mesh node has a persistent cryptographic identity that ensures stable PeerId across restarts.
Identity Keypair
- Algorithm: Ed25519 (elliptic curve)
- Location:
~/.qmt/mesh_identity.key(configurable) - Persistence: Same PeerId across restarts
- Auto-generation: Created automatically on first run
Configuration
Custom Identity Path
Identity Management Best Practices
- Backup identity: Copy
~/.qmt/mesh_identity.keyto preserve PeerId - Secure storage: Protect identity file (chmod 600)
- Don't share: Identity contains private key
- Regeneration: Delete file to generate new identity (changes PeerId)
Node Name
Override the default hostname advertised to peers:
Useful when: - OS hostname is meaningless (e.g., "unknown" on mobile) - Multiple nodes on same machine - Want descriptive names for peers
Session Management
Creating Remote Sessions
Listing Remote Nodes
Use list_remote_nodes() to discover available mesh peers and their capabilities:
Attaching Existing Sessions
Attach to a session running on another node through the embedded ACP dashboard or the attach_remote_session API.
Forking Sessions
Create a copy of a remote session at a specific message point:
Use cases: - Branching conversations: Try different approaches from same starting point - Experimentation: Fork to test risky changes without affecting original - Collaboration: Team members fork to work on different aspects - Rollback: Fork from known good state if current path isn't working
Resuming Sessions
Reconnect to sessions across node restarts or disconnections:
Recovery semantics:
- Durable identity: a remote bookmark is the authoritative local record for a remote session (session ID, node, metadata). A bookmarked session is never replaced by a locally created session, and its cached history stays readable while the host is offline.
- One recovery algorithm: opening, explicit reconnect, operation recovery, extension attach, and startup reattach all share a single connection coordinator with per-reason time budgets (open 5s, reconnect/attach 15s, operation recovery 10s by default) and per-session single-flight, so concurrent load/prompt/runtime queries share one recovery attempt.
- Connected is verified: an attachment is reported connected only after the control path (bounded health check) and the event-delivery path (direct relay capability acknowledged after forwarder installation; a generation-specific DHT relay name remains as a compatibility fallback) both succeed. A failed candidate never replaces a working attachment, and delayed failures from an old attachment generation cannot detach a newer one.
- Missed events are backfilled exactly once: reconnect captures a durable synchronization checkpoint before subscribing. Live delivery cannot advance this checkpoint; each successfully persisted historical page advances it, so an interrupted backfill resumes safely after either process restarts. New events publish to subscribers, and completion refreshes the host-ordered transcript in place. Cursorless sessions fetch authoritative host pages from the beginning. Legacy rows remain stored, but the completed source-identified snapshot supersedes them in the remote transcript without guessing their source sequences.
- Acknowledged submissions: interactive turns submit with a client_input_id; the session actor caches accepted receipts for bounded same-actor retry safety, not cross-restart exactly-once. The draft is cleared only after a receipt or matching durable acceptance event. Failed inputs remain visible; lost acknowledgements are labelled "Delivery status unknown" and offer Refresh / Reconcile rather than automatic resend.
- Explicit connection state: session summaries and session_loaded carry connecting | connected | disconnected. Connection changes are ephemeral signals; conversation events remain durable.
Remote Session Lifecycle
stateDiagram-v2
[*] --> Created: create_remote_session()
Created --> Active: First message
Active --> Active: Messages
Active --> Forked: fork_remote_session()
Active --> Suspended: Node restart/disconnect
Suspended --> Active: resume_remote_session()
Active --> Archived: Explicit close
state Active {
[*] --> Processing
Processing --> Waiting: Tool execution
Waiting --> Processing: Tool complete
Processing --> Streaming: LLM response
Streaming --> Processing: Stream complete
}
Offline History, Ambiguous Delivery, and Compatibility
Offline-first opening. Opening a bookmarked remote session loads the cached audit history from the local event journal first and reports disconnected when the host is unreachable (bounded open-time recovery, ~5s by default). The frontend keeps the URL, cached transcript, and pending draft, shows a reconnect banner, and offers an explicit Reconnect action. Unknown IDs return a structured session_not_found error; a session that is both a local row and a remote bookmark returns a structured session_location_conflict.
Typed failures and ambiguous delivery. Remote transport failures are classified by kind and delivery certainty (not_delivered / unknown) — never by error-message text:
- Idempotent reads and proven pre-delivery failures may reconnect and retry once, including against a replacement actor. Ambiguous keyed submissions may retry only against the same actor, with the same
client_input_id; materializing a different actor never authorizes an ambiguous resend. - Ambiguous failures on non-idempotent operations (legacy
Prompt, unkeyed submissions, undo/redo, mode/model changes) are never replayed; connectivity is restored and a structuredsubmission_outcome_unknownresult is returned instead. - Remote handler/provider errors are delivered results, not transport failures, and are never retried.
- Protocol mismatch and serialization failures are never retried.
Mixed-version compatibility. New protocol fields (direct relay capability, source cursor, connection state, structured error codes) are additive with serde defaults, so older peers ignore them. New clients fall back to the generation-specific DHT relay name when talking to hosts that predate capability handoff, and hosts that predate cursor backfill are reported as potentially stale history rather than connection failure. Legacy LoadSession/AttachRemoteSession remain supported wrappers over the unified open operation, and legacy prompt RPCs are fulfilled with acknowledged submissions.
Session Interactions
Tool permission requests and form questions (elicitation) follow the same lifecycle whether a session runs on the local agent or on a remote mesh node. A client never needs location-specific handling.
- Request. The session registers a pending interaction on its host node and emits a durable event (
PermissionRequestedorElicitationRequested). The event is relayed to every attached client through the normal event path. - Delivery. The transport the client is connected to (WebSocket or stdio) claims the delivery through the session and sends a standard ACP request:
session/request_permissionorelicitation/create. Claims, re-delivery and recovery go through the session actor, so a question from a remote session is discoverable and answerable through the local agent the client is connected to. - Resolution. The client's answer is submitted to the owning session. Only the first valid answer for an interaction is accepted; later answers get a not-pending or stale result. An invalid form answer is rejected, the tool keeps waiting and the question is re-delivered for correction.
- Retirement. Cancelling the run, or the tool exiting, retires the interaction. A late answer is rejected as not pending and a reconnect cannot revive it.
Sessions and prompts carry no per-connection client bridge. session/prompt works the same for remote sessions as for local ones.
Unanswered permission waits. A permission-required tool call that is not decided by a cached "always" choice, tool policy or hook waits for an explicit user answer. It is not granted because no client is attached, delivery failed, the client disconnected, the response timed out or the answer was malformed. The request stays pending and is delivered again to the next authorized client that attaches.
Unattended runs. Scheduled or headless runs that previously relied on implicit auto-grant now stay paused at the first unanswered permission until a user answers or the run is cancelled. Restrict the enabled
toolslist, setmutating_tools/assume_mutatingso only intended tools need permission, or addpermission_requesthooks that allow or deny explicitly for those runs. See Hooks.
"Always" choices. Answering "Always allow" or "Always reject" applies to later calls of the same tool in that session, for local and remote sessions alike.
Editor workspace queries. The language_query tool needs a locally hosted session with an attached editor connection. For a session on a remote node it returns an "unavailable" result that names remote sessions, and the run continues.
All nodes must be upgraded together. This contract adds new session messages to the mesh wire protocol. Mixed-version meshes are not supported, and there is no fallback to bridge-based or auto-granted behavior. If the local agent attaches to or prompts a session on a node that lacks the interaction messages, the operation fails with RemoteNodeUpgradeRequired. Clients receive it as an ACP error (code -32603) with data.category = "remote_upgrade_required" and data.node set to the outdated node. Upgrade that node, and every other node in the mesh, to the same agent version. Rolling back means reverting the agent on all nodes together.
Prompt response ordering. The response to session/prompt is sent only after the requesting connection has delivered every session/update the prompt produced, for remote-hosted sessions as well as local ones. The wait is bounded: if the relay disconnects, the connection lags and cannot fully recover, or the 30 second safety limit passes, the response is sent anyway and a warning is logged. See Event ordering across nodes.
Session Persistence
Remote sessions are persisted on the host node:
Persistence features: - SQLite storage: Sessions stored in SQLite database - History preservation: Full message history with metadata - Tool results: Tool execution results preserved - Context snapshots: Compacted context stored for recovery
Error Handling
Connection Failures
Remote operations may fail due to network issues, peer unavailability, or timeout. Check logs for specific error details and adjust request_timeout_secs or stream_reconnect_grace_secs as needed.
Stream Disconnections
During stream disconnection: 1. Buffer messages: Messages buffered locally 2. Attempt reconnect: Automatic reconnection attempts 3. Grace period: Configurable wait time (default: 120s) 4. Resume or fail: Either reconnects successfully or fails gracefully
Best Practices for Remote Sessions
- Use descriptive session IDs: Makes it easier to resume
- Regular checkpoints: Fork sessions before major changes
- Monitor node health: Check node availability before delegation
- Handle failures gracefully: Implement fallback to local execution
- Set appropriate timeouts: Balance responsiveness vs reliability
- Use session names: Add metadata to identify sessions
Routing
Routing Table
The mesh maintains a routing table that maps agents to nodes:
Routing Snapshot
Use Cases
1. GPU-Accelerated Coding
flowchart LR
subgraph Local["Local Machine (CPU)"]
PA["Planner Agent<br/>(Lightweight)"]
LM[Local Model]
end
subgraph Remote["Remote Machine (GPU)"]
CA["Coder Agent<br/>(GPU-accelerated)"]
GM[GPU Model]
end
PA -->|"Delegates task"| CA
PA -->|"LLM calls<br/>(routed to GPU)"| GM
CA -->|"Fast model inference"| GM
Configuration:
2. Distributed Team Collaboration
flowchart TD
S1["Session 1<br/>(Feature A)"]
S2["Session 2<br/>(Feature B)"]
S3["Session 3<br/>(Feature C)"]
SS[("Shared State")]
S1 & S2 & S3 <--> SS
Benefits: - Share session state across team members - Collaborate on same codebase - Real-time synchronization
3. Load Distribution
flowchart LR
subgraph LB["Load Balancer Node"]
SR["Session Router<br/>- Distribute<br/>- Monitor"]
end
subgraph WN["Worker Nodes"]
W1["Worker 1<br/>(Handle Tasks)"]
W2["Worker 2<br/>(Handle Tasks)"]
end
SR --> W1 & W2
Configuration:
4. Specialized Hardware
flowchart LR
subgraph DM["Development Machine"]
DA[Development Agent]
end
subgraph SN["Specialized Nodes"]
GPU[GPU Worker]
TPU[TPU Worker]
FPGA[FPGA Worker]
end
DA --> GPU & TPU & FPGA
Streaming Stability
QueryMT includes robust streaming stability features for handling network interruptions and maintaining reliable LLM streaming across the mesh.
Stream Reconnection
When a remote streaming connection is interrupted, QueryMT automatically attempts reconnection:
Reconnection behavior:
- Detection: Stream interruption detected via heartbeat timeout
- Buffering: Messages buffered locally during disconnection
- Reconnection attempts: Automatic reconnection with exponential backoff
- Grace period: Configurable wait time before failing (default: 120s)
- Resume or fail: Stream resumes or fails gracefully after timeout
sequenceDiagram
participant Client as Client Node
participant Stream as Stream Handler
participant Remote as Remote Node
Client->>Stream: Start streaming request
Stream->>Remote: Initiate stream
loop Stream Active
Remote->>Stream: Stream chunks
Stream->>Client: Forward chunks
end
Note over Stream,Remote: Network interruption
Stream->>Stream: Detect disconnection
Stream->>Stream: Buffer messages
Stream->>Remote: Attempt reconnection
alt Reconnect Success
Remote->>Stream: Resume stream
Stream->>Client: Continue forwarding
else Reconnect Timeout
Stream->>Client: Fail gracefully
end
Streaming Timeouts
QueryMT distinguishes between different timeout scenarios for better reliability:
Request Timeout
Timeout for non-streaming mesh requests (e.g., compaction, metadata queries):
Use cases: - Agent metadata queries - Session operations (create, fork, list) - Model listing - Node info requests
Streaming Behavior
For streaming LLM responses:
- First-chunk timeout: The mesh waits for the first response chunk after initiating a stream
- Idle detection: Monitors for stalled streams during streaming
- Reconnection: If a stream disconnects, the system attempts reconnection within the
stream_reconnect_grace_secswindow
These behaviors are managed internally by the mesh transport. Configure the grace period for reconnection:
Transient Disconnect Handling
QueryMT gracefully handles temporary network issues:
Symptoms of Transient Disconnects
- Brief network interruptions (WiFi switching, VPN reconnect)
- Packet loss
- Temporary DNS failures
- NAT re-binding
Handling Strategy
Configuration for Unstable Networks
For networks with frequent interruptions:
Mesh Provider Stability
For remote model providers in the mesh:
Fallback Strategies
Configure automatic fallback when remote providers fail:
Fallback chain: 1. Try primary remote provider 2. If unavailable, try other providers in mesh 3. If all remote providers fail, use local provider 4. If local provider fails, return error
Monitoring Stream Health
Log Monitoring
Enable detailed stream logging:
Key log messages:
- Stream reconnected successfully - Good
- Stream reconnection timeout - Check network
- Provider health check failed - Provider issue
- Transient disconnect detected - Network instability
Metrics to Watch
| Metric | Healthy | Warning | Critical |
|---|---|---|---|
| Stream reconnection rate | < 1% | 1-5% | > 5% |
| Average latency | < 100ms | 100-500ms | > 500ms |
| Provider availability | > 99% | 95-99% | < 95% |
| Error rate | < 0.1% | 0.1-1% | > 1% |
Best Practices for Streaming Stability
- Set appropriate timeouts: Balance responsiveness vs reliability
- Fast networks: Lower timeouts (30-60s)
-
Slow/unreliable: Higher timeouts (120-300s)
-
Monitor reconnection rates: High rates indicate network issues
-
Use health checks: Implement periodic provider health checks
-
Implement client-side buffering: Buffer messages during reconnection
-
Provide user feedback: Show reconnection status in UI
-
Test network resilience: Simulate network failures in testing
-
Log stream events: Enable detailed logging for debugging
-
Configure per-network: Adjust settings for different network conditions
-
Use exponential backoff: Prevent thundering herd on reconnection
-
Graceful degradation: Fall back to local execution when mesh fails
Troubleshooting Streaming Issues
Symptom: Frequent Stream Timeouts
Possible causes: - Network latency too high - Remote node overloaded - Timeout values too low
Solutions:
1. Increase stream_reconnect_grace_secs
2. Check remote node health
3. Reduce mesh complexity
4. Use closer geographic nodes
Symptom: Stream Never Reconnects
Possible causes: - Network partition - Remote node crashed - Identity mismatch
Solutions: 1. Check network connectivity 2. Verify remote node is running 3. Check mesh peer configuration 4. Restart mesh on affected nodes
Symptom: High Latency During Streaming
Possible causes: - Network congestion - Too many hops in mesh - Provider overload
Solutions: 1. Check bandwidth utilization 2. Optimize mesh topology 3. Load balance across providers 4. Use LAN transport when possible
Security
Peer Authentication
Mesh nodes authenticate using libp2p's built-in peer ID system:
Firewall Configuration
Required ports for mesh networking:
| Direction | Port | Protocol | Purpose |
|---|---|---|---|
| Inbound | 9000 (default) | TCP | Mesh connections |
| Outbound | Any | TCP/UDP | Peer discovery |
Example firewall rules:
NAT Traversal
For nodes behind NAT:
- Port forwarding: Forward mesh port to internal node
- UPnP: Enable UPnP for automatic port forwarding
- Relay: Use libp2p relay servers
Monitoring
Node Status
Iroh Transport Diagnostics
The querymt/mesh/status response includes an optional iroh snapshot. It is
null for disabled/LAN-only runtimes. For an iroh endpoint it contains:
closed: whether the endpoint has shut down.relays: home relay URLs, connected state, the last human-readable error, and an optionalauth_denied_reasonfrom relay authentication.metrics: optional cumulative relay and address-lookup counters.
Snapshots read the current relay-status watcher on demand and create no observer task. Empty relay status can mean initial startup or disabled relays. Relay failure does not imply that direct peer connections have failed. Relay auth denial is distinct from mesh admission/invite rejection; its reason is for display, not programmatic string matching. Treat these diagnostics as operational information and do not expose them to untrusted clients.
Enable counters with querymt-agent's remote-metrics feature (which includes
remote), or querymt-remote's iroh-metrics feature (which includes kameo-mesh).
The adapter also offers an independent metrics feature. Without the remote
metrics feature, metrics is null, not a misleading set of zero counters.
Counters include successful/failed/closed/rate-limited relay connections, total
and failed lookups, and results/errors for N0's dns and pkarr services.
Counters are decimal strings to preserve 64-bit precision in JavaScript clients.
They reset when the endpoint restarts and are not an atomic cross-counter sample.
Rate-limited connections count affected connections, not individual warnings.
Rust callers can use MeshHandle::iroh_diagnostics() or
MeshRuntimeHandle::iroh_diagnostics(). These diagnostics do not change peer
liveness, authorization, routing, or application reconnect/backoff behavior.
Batched unreliable datagrams are deliberately not used for actor/provider traffic.
Event Logging
Enable mesh logging:
Metrics
Key metrics to monitor:
- Connected peers: Number of active connections
- Latency: Round-trip time to peers
- Bandwidth: Data transfer rates
- Session count: Number of sessions per node
Troubleshooting
Cannot Connect to Peer
Symptoms: Mesh node shows no connected peers
Solutions: 1. Check firewall allows mesh port 2. Verify peer address is correct 3. Ensure peer is running and listening 4. Check NAT/firewall configuration
High Latency
Symptoms: Slow responses from remote agents
Solutions:
1. Check network bandwidth
2. Reduce mesh complexity (fewer peers)
3. Use closer geographic nodes
4. Increase request_timeout_secs
Peer Discovery Issues
Symptoms: Cannot find peers automatically
Solutions: 1. Try explicit peer configuration 2. Check mDNS is enabled on network 3. Verify firewall allows multicast 4. Use Kademlia for cross-subnet discovery
Session Attachment Fails
Symptoms: Cannot attach to remote session
Solutions: 1. Verify session exists on remote node 2. Check peer has correct permissions 3. Ensure mesh is properly configured 4. Review error logs for details
Best Practices
Network Configuration
- Use static IPs for mesh nodes
- Configure port forwarding for NAT environments
- Monitor bandwidth usage
- Use dedicated ports for mesh traffic
Node Organization
- Group by function: Separate planner and worker nodes
- Consider geography: Place nodes close to users
- Plan for redundancy: Multiple nodes for critical tasks
- Document topology: Keep track of node roles
Security
- Use strong peer IDs: Generate unique keys
- Limit peer access: Only allow known peers
- Monitor connections: Watch for unauthorized access
- Encrypt traffic: Use TLS where possible
Examples
Full Mesh Configuration
Command Line Examples
Related Documentation
- Configuration Guide - Mesh configuration options
- Profiles - Mesh-enabled profiles
- Delegation - Remote delegation patterns
- Examples - Mesh networking examples
- API Reference - Mesh API types and functions
Event ordering across nodes
Relayed session events keep the host's per-session session_seq. The relay fetches durable events it missed from the host when it sees a gap, or when the host's forwarder reports that it fell behind (which also covers a loss at the end of a burst). A session/prompt response on an attached node follows the updates relayed for that prompt. All nodes of a mesh must run a version that supports this. See Event Ordering.