# SSH - Debugging Config, Agents, and Forwards SSH failures follow a diagnostic ladder. `ssh -G` shows the config that ssh will really use, agent error strings tell the agent states apart, and an overlay ping separates network failures from target-host failures. Apparent forward "stalls" often come from multiplexing settings that kill a backgrounded forward. ## Key points - `ssh -G <host>` prints the fully parsed config: the effective resolved values. Run this check before you assume that ssh will behave as the intended config suggests. - `ssh-add -l` returns "Operation not permitted" when the agent socket exists but the agent process refuses the connection. The error does not mean "agent not running". - "Last login: … from localhost" does not always mean a proxy tunnel. The message also appears when the ssh client and the ssh server sit on the same VPN segment and resolve to local addresses. Check the host config for an absent `ProxyCommand` before you chase a tunnel. A tunnel changes which fix path applies. - `launchctl setenv SSH_AUTH_SOCK` at runtime does not reach GUI apps that launched earlier. Set the environment at agent startup through a LaunchAgent plist. The value then persists across reboots and reaches GUI-spawned processes. `launchctl getenv SSH_AUTH_SOCK` shows what the GUI environment sees. This value differs from the shell environment. - `ControlMaster auto` plus a short `ControlPersist` kills a backgrounded `-N` forward one second after detach. The forward forks to the background as a control master and exits when the persist timer expires. Disable multiplexing for long-lived forwards. Every apparent forward "stall" had this cause. - A password-manager SSH agent that serves only GUI sessions refuses to sign for a non-GUI launchd process. A headless forward therefore cannot use the interactive key. The compliant pattern for headless use: a passphrase-protected dedicated key, the passphrase stored only in the login keychain, and a dedicated launchd ssh-agent that reloads the key with `ssh-add --apple-load-keychain`. This pattern also passes device-management policies. - `tailscale ping <node>` traverses the overlay network, through a relay or direct, without sshd and host firewalls in the path. Run this test to separate a network problem from a target-host problem before you touch the SSH config. - `ssh -t` PTY allocation does not work in fully non-interactive heredoc pipelines. Write config files locally and `scp` them instead. This avoids heredoc quoting issues with complex JSON. ## How to apply Work the ladder for a failed SSH connection from the top. 1. Run `tailscale ping <node>`. A reply means the overlay reaches the node, so the failure sits in sshd, a firewall, or the SSH config. No reply means a network problem. 2. Run `ssh -G <host>` and read the resolved values. Confirm user, port, identity, proxy, and multiplexing settings for that host. 3. Run `ssh-add -l` and read the exact error string. "no identities" means the agent runs without keys. "Operation not permitted" means the agent process refuses the connection. 4. Run `launchctl getenv SSH_AUTH_SOCK` and compare the value with the shell environment. GUI apps see the launchctl value, not the shell value. 5. When a backgrounded forward dies, check `ControlMaster` and `ControlPersist` for that host before you debug the network. Disable multiplexing on the forward connection. 6. For a headless process that must sign, do not use the GUI password-manager agent. Create a passphrase-protected dedicated key, store the passphrase in the login keychain, and run a dedicated launchd ssh-agent that reloads with `ssh-add --apple-load-keychain`. 7. Do not pass `ssh -t` inside a fully non-interactive heredoc pipeline. Write the file locally, then `scp` it. ## Observations - [tooling] `ssh -G <host>` shows the effective resolved config before you trust an assumption about SSH behavior. #ssh (source: agent sessions, 2026-06) - [debugging] `ssh-add -l` returns "Operation not permitted" when the agent refuses the connection. The string "no identities" means the agent runs without keys. #ssh (source: agent sessions, 2026-05) - [networking] "Last login: … from localhost" can indicate a proxy tunnel or a shared VPN segment. An absent `ProxyCommand` in the host config rules out the tunnel. #ssh (source: agent sessions, 2026-05) - [macos] `launchctl setenv` at runtime does not reach GUI apps that launched earlier. Set `SSH_AUTH_SOCK` at agent startup through a LaunchAgent plist. #macos (source: agent sessions, 2026-06) - [port-forwarding] `ControlMaster auto` with a short `ControlPersist` kills a backgrounded `-N` forward after detach. Disable multiplexing for long-lived forwards. #ssh (source: agent sessions, 2026-10) - [security] A password-manager agent that serves only GUI sessions refuses to sign for non-GUI launchd processes. Use a dedicated key in the login keychain with a dedicated launchd agent. #ssh (source: agent sessions, 2026-10) - [networking] `tailscale ping <node>` separates overlay reachability from sshd and host firewall problems on the target. #ssh (source: agent sessions, 2026-06) - [automation] `ssh -t` does not allocate a PTY inside a fully non-interactive heredoc pipeline. Write files locally and `scp` them. #ssh (source: agent sessions, 2026-02) ## Relations - relates_to [[resources/documentation/Git - Commit Signing with SSH Agents and gitsign]] (agent reachability and the `ssh-add` state decide whether commit signing works) - relates_to [[resources/documentation/macOS - Developer Environment Pitfalls]] (launchctl and LaunchAgent environment handling for `SSH_AUTH_SOCK`) - relates_to [[resources/documentation/Agents/Agents - Desktop and Remote]] (long-lived forwards and remote sessions depend on multiplexing settings) ## Sources Personal agent-session notes, February 2026 to October 2026