Troubleshooting
This is the deep runbook. Start at /help/troubleshooting for the fast triage flow first.
Command ladder
Run in this order:
remoteclaw statusremoteclaw gateway statusremoteclaw logs --followremoteclaw doctorremoteclaw channels status --probeHealthy signals:
remoteclaw gateway statusshowsRuntime: running,Connectivity probe: ok, and aCapability: ...line.remoteclaw doctorreports no blocking config/service issues.remoteclaw channels status --probeshows live per-account transport status and, where supported,worksoraudit ok.
After an update
Use when an update finishes but the Gateway is down, channels are empty, or model calls fail with 401s.
remoteclaw status --allremoteclaw update status --jsonremoteclaw gateway status --deepremoteclaw doctor --fixremoteclaw gateway restartLook for:
Update restartinremoteclaw status/remoteclaw status --all. Pending or failed handoffs include the next command to run.plugin load failed: dependency tree corrupted; run remoteclaw doctor --fixunder Channels: the channel config still exists, but plugin registration failed before the channel could load.- Provider 401s after re-auth:
remoteclaw doctor --fixchecks for stale per-agent OAuth auth shadows and removes old copies so all agents resolve the current shared profile.
Split brain installs and newer config guard
Use when a gateway service unexpectedly stops after an update, or logs show one remoteclaw binary is older than the version that last wrote remoteclaw.json.
RemoteClaw stamps config writes with meta.lastTouchedVersion. Read-only commands can inspect a config written by a newer RemoteClaw, but process and service mutations refuse to run from an older binary. Blocked actions: gateway service start/stop/restart/uninstall, forced service reinstall, service-mode gateway startup, and gateway --force port cleanup.
which remoteclawremoteclaw --versionremoteclaw gateway status --deepremoteclaw config get meta.lastTouchedVersion```bashremoteclaw gateway install --forceremoteclaw gateway restart```Protocol mismatch after rollback
Use when logs keep printing protocol mismatch after a downgrade or rollback. An older Gateway is running, but a newer local client process is still reconnecting with a protocol range the older Gateway cannot speak.
remoteclaw --versionwhich -a remoteclawremoteclaw gateway status --deepremoteclaw doctor --deepremoteclaw logs --followLook for:
protocol mismatch ... client=... v<version> min=<n> max=<n> expected=<n>in Gateway logs.Established clients:inremoteclaw gateway status --deeporGateway clientsinremoteclaw doctor --deep: active TCP clients connected to the Gateway port, with PIDs and command lines when the OS allows it.- A client process whose command line points at the newer RemoteClaw install or wrapper you rolled back from.
Fix:
- Stop or restart the stale RemoteClaw client process shown by
gateway status --deep. - Restart apps or wrappers that embed RemoteClaw: local dashboards, editors, app-server helpers, or long-running
remoteclaw logs --followshells. - Re-run
remoteclaw gateway status --deeporremoteclaw doctor --deepand confirm the stale client PID is gone.
Do not make an older Gateway accept a newer incompatible protocol. Protocol bumps protect the wire contract; rollback recovery is a process/version cleanup problem.
Skill symlink skipped as path escape
Use when logs include:
Skipping escaped skill path outside its configured root: ... reason=symlink-escapeEvery skill root is a containment boundary. A symlink under ~/.agents/skills, <workspace>/.agents/skills, <workspace>/skills, or ~/.remoteclaw/skills is skipped when its real target resolves outside that root, unless the target is explicitly trusted.
Inspect the link:
ls -l ~/.agents/skills/<name>realpath ~/.agents/skills/<name>remoteclaw config get skills.loadIf the target is intentional, configure both the direct skill root and the allowed symlink target:
{ skills: { load: { extraDirs: ["~/Projects/manager/skills"], allowSymlinkTargets: ["~/Projects/manager/skills"], }, },}Then start a new session or wait for the skills watcher to refresh. Restart the gateway if the running process predates the config change.
Do not use broad targets such as ~, /, or a whole synced project folder. Keep allowSymlinkTargets scoped to the real skill root that contains trusted SKILL.md directories.
If Skill Workshop apply should also write through those trusted symlinked workspace skill paths, enable skills.workshop.allowSymlinkTargetWrites. Keep it disabled for read-only shared skill roots.
Related:
Anthropic 429 extra usage required for long context
Use when logs/errors include: HTTP 429: rate_limit_error: Extra usage is required for long context requests.
remoteclaw logs --followremoteclaw models statusremoteclaw config get agents.defaults.modelsLook for:
- Selected Anthropic model is a GA-capable 1M Claude 4.x model (Opus 4.6/4.7/4.8, Sonnet 4.6), or the model config still carries legacy
params.context1m: true. - Current Anthropic credential is not eligible for long-context usage.
- Requests fail only on long sessions/model runs that need the 1M context path.
Fix options:
Related:
Upstream 403 blocked responses
Use when an upstream LLM provider returns a generic 403 such as Your request was blocked.
Do not assume this is always an RemoteClaw configuration issue. The response can come from an upstream security layer such as a CDN, WAF, bot-management rule, or reverse proxy in front of an OpenAI-compatible endpoint.
remoteclaw statusremoteclaw gateway statusremoteclaw logs --followLook for:
- Multiple models under the same provider failing the same way.
- HTML or generic security text instead of a normal provider API error.
- Provider-side security events for the same request time.
- A tiny direct
curlprobe succeeding while normal SDK-shaped requests fail.
Fix the provider-side filtering first when evidence points to a WAF/CDN block. Prefer a narrowly scoped allow or skip rule for the API path RemoteClaw uses, and avoid disabling protection for the whole site.
Related:
Local OpenAI-compatible backend passes direct probes but agent runs fail
Use when:
curl ... /v1/modelsworks.- Tiny direct
/v1/chat/completionscalls work. - RemoteClaw model runs fail only on normal agent turns.
curl http://127.0.0.1:1234/v1/modelscurl http://127.0.0.1:1234/v1/chat/completions \ -H 'content-type: application/json' \ -d '{"model":"<id>","messages":[{"role":"user","content":"hi"}],"stream":false}'remoteclaw infer model run --model <provider/model> --prompt "hi" --jsonremoteclaw logs --followLook for:
- Direct tiny calls succeed, but RemoteClaw runs fail only on larger prompts.
model_not_foundor 404 errors even though direct/v1/chat/completionsworks with the same bare model id.- Backend errors about
messages[].contentexpecting a string. - Intermittent
incomplete turn detected ... stopReason=stop payloads=0warnings with an OpenAI-compatible local backend. - Backend crashes that appear only with larger prompt-token counts or full agent runtime prompts.
Related:
No replies
If channels are up but nothing answers, check routing and policy before reconnecting anything.
remoteclaw statusremoteclaw channels status --proberemoteclaw pairing list --channel <channel> [--account <id>]remoteclaw config get channelsremoteclaw logs --followLook for:
- Pairing pending for DM senders.
- Group mention gating (
requireMention,mentionPatterns). - Channel/group allowlist mismatches.
Common signatures:
drop guild message (mention required→ group message ignored until mention.pairing request→ sender needs approval.blocked/allowlist→ sender/channel was filtered by policy.
Related:
Dashboard control UI connectivity
When the dashboard/control UI will not connect, validate URL, auth mode, and secure context assumptions.
remoteclaw gateway statusremoteclaw statusremoteclaw logs --followremoteclaw doctorremoteclaw gateway status --jsonLook for:
- Correct probe URL and dashboard URL.
- Auth mode/token mismatch between client and gateway.
- HTTP usage where device identity is required.
If a local browser cannot connect to 127.0.0.1:18789 after an update, first recover the local Gateway service and confirm it is serving the dashboard:
remoteclaw gateway restartlsof -i :18789curl http://127.0.0.1:18789If curl returns RemoteClaw HTML, the Gateway is working and the remaining issue is likely browser cache, an old deep link, or stale tab state. Open http://127.0.0.1:18789 directly and navigate from the dashboard. If restart does not leave the service running, run remoteclaw gateway start and recheck remoteclaw gateway status.
Auth detail codes quick map
Use error.details.code from the failed connect response to pick the next action:
| Detail code | Meaning | Recommended action |
|---|---|---|
AUTH_TOKEN_MISSING | Client did not send a required shared token. | Paste/set token in the client and retry. For dashboard paths: remoteclaw config get gateway.auth.token then paste into Control UI settings. |
AUTH_TOKEN_MISMATCH | Shared token did not match gateway auth token. | If canRetryWithDeviceToken=true, allow one trusted retry. Cached-token retries reuse stored approved scopes; explicit deviceToken / scopes callers keep requested scopes. If still failing, run the token drift recovery checklist. |
AUTH_DEVICE_TOKEN_MISMATCH | Cached per-device token is stale or revoked. | Rotate/re-approve device token using devices CLI, then reconnect. |
AUTH_SCOPE_MISMATCH | Device token is valid, but its approved role/scopes do not cover this connect request. | Re-pair the device or approve the requested scope contract; do not treat this as shared-token drift. |
PAIRING_REQUIRED | Device identity needs approval. Check error.details.reason for not-paired, scope-upgrade, role-upgrade, or metadata-upgrade, and use requestId / remediationHint when present. | Approve pending request: remoteclaw devices list then remoteclaw devices approve <requestId>. Scope/role upgrades use the same flow after you review the requested access. |
Device auth v2 migration check:
remoteclaw --versionremoteclaw doctorremoteclaw gateway statusIf logs show nonce/signature errors, update the connecting client and verify it:
If remoteclaw devices rotate / revoke / remove is denied unexpectedly:
- Paired-device token sessions can manage only their own device unless the caller also has
operator.admin. remoteclaw devices rotate --scope ...can only request operator scopes that the caller session already holds.
Related:
- Configuration (gateway auth modes)
- Control UI
- Devices
- Remote access
- Trusted proxy auth
Gateway service not running
Use when the service is installed but the process does not stay up.
remoteclaw gateway statusremoteclaw statusremoteclaw logs --followremoteclaw doctorremoteclaw gateway status --deep # also scan system-level servicesLook for:
Runtime: stoppedwith exit hints.- Service config mismatch (
Config (cli)vsConfig (service)). - Port/listener conflicts.
- Extra launchd/systemd/schtasks installs when
--deepis used. Other gateway-like services detected (best effort)cleanup hints.
Related:
macOS gateway silently stops responding, then resumes when you touch the dashboard
Use when channels (Telegram, WhatsApp, etc.) on a macOS host go quiet for minutes to hours at a time, and the gateway appears to come back the moment you open the Control UI, SSH in, or otherwise interact with the host. There is usually no obvious symptom in remoteclaw status because by the time you look the gateway is alive again.
ls ~/.remoteclaw/logs/stability/ | tail -5remoteclaw gateway stability --bundle latestpmset -g log | grep -iE "sleep|wake|maintenance" | tail -50launchctl print gui/$UID/org.remoteclaw.gateway | grep -E "state|last exit|runs"Look for:
- One or more
*-uncaught_exception.jsonbundles in~/.remoteclaw/logs/stability/witherror.codeset to a transient network code such asENETDOWN,ENETUNREACH,EHOSTUNREACH, orECONNREFUSED. pmset -g loglines likeEntering Sleep state due to 'Maintenance Sleep'oren0 driver is slow (msg: WillChangeState to 0)aligned with the crash timestamps. Power Nap / Maintenance Sleep briefly puts the Wi-Fi driver into state 0; any outboundconnect()that lands in that window can fail withENETDOWNeven on a host that otherwise has full network connectivity.launchctl printoutput showingstate = not runningwith multiple recentrunsand an exit code, especially when the gap between crash and the next launch is on the order of an hour rather than seconds. macOS launchd applies an undocumented respawn-protection gate after a crash burst that can stop honoringKeepAlive=trueuntil an external trigger such as interactive login, dashboard connection, orlaunchctl kickstartre-arms it.
Common signatures:
- A stability bundle whose
error.codeisENETDOWNor a sibling code, with the call stack pointing into NodenetlookupAndConnect/Socket.connect. RemoteClaw2026.5.26and newer classify these as benign transient network errors so they no longer propagate to the top-level uncaught handler; if you are on an older release, upgrade first. - Long quiet periods that end the instant you connect to the Control UI or SSH into the host: the user-visible activity is what re-arms launchd’s respawn gate, not anything the dashboard does to the gateway.
runscount incrementing across the day with no correspondingreceived SIG*; shutting downline in~/Library/Logs/remoteclaw/gateway.log: clean shutdowns log a signal; transient crashes do not.
What to do:
-
Upgrade the gateway if you are running a release before
2026.5.26. After upgrading, futureENETDOWNerrors are logged as warnings instead of terminating the process. -
Reduce maintenance sleep activity on Mac mini / desktop hosts meant to run as always-on servers:
Terminal window sudo pmset -a sleep 0 disksleep 0 standby 0 powernap 0This significantly reduces, but does not entirely eliminate, the underlying driver flap. The system can still perform some maintenance sleeps for TCP keepalive and mDNS upkeep regardless of these flags.
-
Add a liveness watchdog so a future crash burst that gets parked by launchd is caught quickly:
Terminal window # Example launchd-aware liveness check, suitable for a 5-minute cron or LaunchAgentstate=$(launchctl print gui/$UID/org.remoteclaw.gateway 2>/dev/null | awk -F'= ' '/state =/ {print $2; exit}')if [ "$state" != "running" ]; thenlaunchctl kickstart -k gui/$UID/org.remoteclaw.gatewayfiThe point is to externally re-arm the respawn gate;
KeepAlive=truealone is not sufficient on macOS after a crash burst.
Related:
Gateway exits during high memory use
Use when the Gateway disappears under load, the supervisor reports an OOM-style restart, or logs mention critical memory pressure bundle written.
remoteclaw gateway status --deepremoteclaw logs --followremoteclaw gateway stability --bundle latestremoteclaw gateway diagnostics exportLook for:
Reason: diagnostic.memory.pressure.criticalin the latest stability bundle.Memory pressure:withcritical/rss_threshold,critical/heap_threshold, orcritical/rss_growth.V8 heap:values near the heap limit.Largest session files:entries such asagents/<agent>/sessions/<session>.jsonlorsessions/<session>.jsonl.- Linux cgroup memory counters when the gateway runs inside a container or memory-limited service.
Common signatures:
critical memory pressure bundle writtenappears shortly before restart → RemoteClaw captured a pre-OOM stability bundle. Inspect it withremoteclaw gateway stability --bundle latest.memory pressure: level=critical ... memoryPressureSnapshot=disabledappears in gateway logs → RemoteClaw detected critical memory pressure, but the pre-OOM stability snapshot is off.Largest session files:points at a very large redacted transcript path → reduce retained session history, inspect session growth, or move old transcripts out of the active store before restarting.V8 heap:used bytes are close to the heap limit → lower prompt/session pressure, reduce concurrent work, or raise the Node heap limit only after confirming the workload is expected.Memory pressure: critical/rss_growth→ memory grew quickly inside one sampling window. Check the latest logs for a large import, runaway tool output, repeated retries, or a batch of queued agent work.- Critical memory pressure appears in logs but no bundle exists → this is the default. Set
diagnostics.memoryPressureSnapshot: trueto capture the pre-OOM stability bundle on future critical memory pressure events.
The stability bundle is payload-free. It includes operational memory evidence and redacted relative file paths, not message text, webhook bodies, credentials, tokens, cookies, or raw session ids. Attach the diagnostics export to bug reports instead of copying raw logs.
Related:
Gateway rejected invalid config
Use when Gateway startup fails with Invalid config or hot reload logs say it skipped an invalid edit.
remoteclaw logs --followremoteclaw config fileremoteclaw config validateremoteclaw doctorLook for:
Invalid config at ...config reload skipped (invalid config): ...Config write rejected: ...- A timestamped
remoteclaw.json.rejected.*file beside the active config. - A timestamped
remoteclaw.json.clobbered.*file ifdoctor --fixrepaired a broken direct edit. - RemoteClaw keeps the latest 32
.clobbered.*files for each config path and rotates older ones.
Related:
Gateway probe warnings
Use when remoteclaw gateway probe reaches something, but still prints a warning block.
remoteclaw gateway proberemoteclaw gateway probe --jsonremoteclaw gateway probe --ssh user@gateway-hostLook for:
warnings[].codeandprimaryTargetIdin JSON output.- Whether the warning is about SSH fallback, multiple gateways, missing scopes, or unresolved auth refs.
Common signatures:
SSH tunnel failed to start; falling back to direct probes.→ SSH setup failed, but the command still tried direct configured/loopback targets.multiple reachable gateway identities detected→ distinct gateways answered, or RemoteClaw could not prove reachable targets are the same gateway. An SSH tunnel, proxy URL, or configured remote URL to the same gateway is treated as one gateway with multiple transports, even when transport ports differ.Read-probe diagnostics are limited by gateway scopes (missing operator.read)→ connect worked, but detail RPC is scope-limited; pair device identity or use credentials withoperator.read.Gateway accepted the WebSocket connection, but follow-up read diagnostics failed→ connect worked, but the full diagnostic RPC set timed out or failed. Treat this as a reachable Gateway with degraded diagnostics; compareconnect.okandconnect.rpcOkin--jsonoutput.Capability: pairing-pendingorgateway closed (1008): pairing required→ the gateway answered, but this client still needs pairing/approval before normal operator access.- Unresolved
gateway.auth.*/gateway.remote.*SecretRef warning text → auth material was unavailable in this command path for the failed target.
Related:
Channel connected, messages not flowing
If channel state is connected but message flow is dead, focus on policy, permissions, and channel specific delivery rules.
remoteclaw channels status --proberemoteclaw pairing list --channel <channel> [--account <id>]remoteclaw status --deepremoteclaw logs --followremoteclaw config get channelsLook for:
- DM policy (
pairing,allowlist,open,disabled). - Group allowlist and mention requirements.
- Missing channel API permissions/scopes.
Common signatures:
mention required→ message ignored by group mention policy.pairing/ pending approval traces → sender is not approved.missing_scope,not_in_channel,Forbidden,401/403→ channel auth/permissions issue.
Related:
Cron and heartbeat delivery
If cron or heartbeat did not run or did not deliver, verify scheduler state first, then delivery target.
remoteclaw cron statusremoteclaw cron listremoteclaw cron runs --id <jobId> --limit 20remoteclaw system heartbeat lastremoteclaw logs --followLook for:
- Cron enabled and next wake present.
- Job run history status (
ok,skipped,error). - Heartbeat skip reasons (
quiet-hours,requests-in-flight,cron-in-progress,lanes-busy,alerts-disabled,empty-heartbeat-file,no-tasks-due).
Related:
Node paired, tool fails
If a node is paired but tools fail, isolate foreground, permission, and approval state.
remoteclaw nodes statusremoteclaw nodes describe --node <idOrNameOrIp>remoteclaw approvals get --node <idOrNameOrIp>remoteclaw logs --followremoteclaw statusLook for:
- Node online with expected capabilities.
- OS permission grants for camera/mic/location/screen.
- Exec approvals and allowlist state.
Common signatures:
NODE_BACKGROUND_UNAVAILABLE→ node app must be in foreground.*_PERMISSION_REQUIRED/LOCATION_PERMISSION_REQUIRED→ missing OS permission.SYSTEM_RUN_DENIED: approval required→ exec approval pending.SYSTEM_RUN_DENIED: allowlist miss→ command blocked by allowlist.
Related:
Browser tool fails
Use when browser tool actions fail even though the gateway itself is healthy.
remoteclaw browser statusremoteclaw browser start --browser-profile remoteclawremoteclaw browser profilesremoteclaw logs --followremoteclaw doctorLook for:
- Whether
plugins.allowis set and includesbrowser. - Valid browser executable path.
- CDP profile reachability.
- Local Chrome availability for
existing-session/userprofiles.
Related:
If you upgraded and something suddenly broke
Most post-upgrade breakage is config drift or stricter defaults now being enforced.
What to check:
- If `gateway.mode=remote`, CLI calls may be targeting remote while your local service is fine.- Explicit `--url` calls do not fall back to stored credentials.
Common signatures:
- `gateway connect failed:` → wrong URL target.- `unauthorized` → endpoint reachable but wrong auth.What to check:
- Non-loopback binds (`lan`, `tailnet`, `custom`) need a valid gateway auth path: shared token/password auth, or a correctly configured non-loopback `trusted-proxy` deployment.- Old keys like `gateway.token` do not replace `gateway.auth.token`.
Common signatures:
- `refusing to bind gateway ... without auth` → non-loopback bind without a valid gateway auth path.- `Connectivity probe: failed` while runtime is running → gateway alive but inaccessible with current auth/url.What to check:
- Pending device approvals for dashboard/nodes.- Pending DM pairing approvals after policy or identity changes.
Common signatures:
- `device identity required` → device auth not satisfied.- `pairing required` → sender/device must be approved.If the service config and runtime still disagree after checks, reinstall service metadata from the same profile/state directory:
remoteclaw gateway install --forceremoteclaw gateway restartRelated: