Servers

Open in ChatGPT Open in Claude

A server is a Linux machine that runs the TunnelHQ agent. Every 30 seconds the agent reports the server’s CPU, memory, disk, network, and load. On a VPN server it also reports the users per protocol and per country, and the state of the VPN services it watches. Web servers, databases, and other services are reported too.

Servers have their own section, Servers in the sidebar, and are free during the beta. TunnelHQ links each server to the tunnels and Uptime checks that test it, and alerts you when a server stops reporting, a service stops, or a limit you set is crossed.

Adding a server

Go to Servers and click Add server. You can give the server a name and a group now, or later on its page. Keep the agent up to date automatically is ticked by default.

Copy the install command and run it on the server as root. It looks like this:

curl -fsSL https://dev.tunnelhq.com/agent/install.sh | sh -s -- --key thqa_… --name 'web-1' --group 'Production' --auto-update

The server shows up in TunnelHQ within a minute.

The installer needs:

Linux that boots with systemd. It doesn’t run in a container or a chroot.
An x86-64 or ARM (arm64) processor.
Root. If you’re signed in as another user, run sudo -i first, then the same command.
curl or wget, and sha256sum.

It downloads the agent, checks it against its SHA-256 checksum, and registers the server with TunnelHQ. If any step fails, nothing is installed.

Option	What it does
`--name`	A name for the server, up to 150 characters.
`--group`	A group for the server, up to 100 characters.
`--logs`	Lets TunnelHQ ask for the services’ error and state log lines, cleaned on the server first. Without it, the agent sends no log lines at all.
`--auto-update`	TunnelHQ updates the agent by itself, a day after each release. The copied command has it when Keep the agent up to date automatically is ticked.
`--no-updates`	Leaves the updater out. The agent is then updated only by running the install command again.

A name or group keeps letters, digits, spaces, and . _ - ( ), and is used only when the server has none in TunnelHQ yet.

The install key

The command holds your project’s install key, which starts with thqa_. It can only add servers to the project, and the same command works on every server. Each server then gets its own token, which can only report for that server.

A new install key stops the command people already copied. Servers already installed keep reporting. Make one from the link in the Add a server dialog, with thq hosts install --rotate, or through the API. It needs a role that can edit the project.

Updating the agent

A server’s System tab shows its agent’s version. When a newer release is out, Update agent appears beside it. The Servers list says how many agents are older, with Update all.

Updates, on the same tab, chooses how the server’s agent updates: Automatically, a day after each release, or By hand, from this page. New servers follow the project’s default, Update agents automatically under Integrations, Server agent, which is on.

How an update stays safe:

The agent never writes its own program file. It leaves a request, and a separate updater, started by systemd, does the work.
The updater downloads the release and checks TunnelHQ’s signature against the key built into the agent already on the server. It takes only a newer release.
It tries the new agent once, swaps it in while keeping the old one, and puts the old one back if the new agent doesn’t stay up.
Automatic updates start a day after a release.

A server installed before agent updates came has no updater yet, and its Updates row says so. Run the install command on it once more, and it updates from TunnelHQ after that. A server installed with --no-updates is updated by running the install command again. Running the install command again always keeps the server’s place in TunnelHQ, with its name, group, links, and history.

Updating an agent, or switching a server’s automatic updates, needs a role that can edit monitors. Changing the project’s default needs a role that can edit the project.

Removing a server

To remove a server from TunnelHQ, open it and choose Remove server. Its agent is cut off at its next report, and its history goes. Removing needs a role that can delete monitors.

The agent stays installed on the machine until you remove it there. The command is on the server’s System tab:

systemctl disable --now tunnelhq-agent; systemctl disable --now tunnelhq-agent-update.path 2>/dev/null; rm -rf /usr/local/bin/tunnelhq-agent /usr/local/bin/tunnelhq-agent.prev /usr/local/bin/tunnelhq-agent.new /etc/tunnelhq-agent /var/lib/tunnelhq-agent /etc/systemd/system/tunnelhq-agent.service /etc/systemd/system/tunnelhq-agent-update.path /etc/systemd/system/tunnelhq-agent-update.service; systemctl daemon-reload

What the agent reads and sends

The agent only reads. Its code never changes your VPN, firewall, kernel, or swap. TunnelHQ can ask it to run named checks that read, such as the services that failed, the ports the server listens on, the firewall’s rules, or the processes using the most memory. AI SRE uses them when it looks into an outage, and when something needs fixing, it gives you the command to run.

Before anything leaves the server, the agent removes:

IP addresses, except the server’s own and networks that cover more than one address.
Email addresses and web addresses.
User names, and WireGuard and AmneziaWG public keys.
Keys, passwords, tokens, and private key blocks.
A country’s user count, when the country has fewer than 11 users.

The installer also locks the agent’s service down with systemd. The agent runs with /etc, /usr, and /var read-only except its own state folder, the kernel’s settings read-only, no module loading, no new privileges, and no setuid. It keeps CAP_NET_ADMIN, which reading WireGuard, conntrack, and iptables needs; the agent’s code never writes a firewall rule.

Log lines about a single connection are counted, and only their common shapes are sent, with every digit masked. Report fields TunnelHQ doesn’t keep are dropped when the report arrives, and check results are kept for 30 days.

The full list of what the agent sends, and what TunnelHQ keeps, comes from the agent build itself. It’s at app.tunnelhq.com/agent/what-leaves, with no sign-in.

Server status

Status	Means
Online	The last report is at most 60 seconds old.
Stale	The last report is 60 to 90 seconds old.
Offline	No report for more than 90 seconds (three missed reports), or the server never reported.

A server that never reported doesn’t alert: it has no normal state to compare with.

A server’s page

Open a server to see it in full. It has four tabs:

Overview: its readings now, with charts.
Events: its incidents.
Alerts: its alert settings.
System: the operating system, kernel, processor, memory, uptime, addresses, agent version, the uninstall command, and the certificate its OpenConnect service serves.

Readings are kept every 5 minutes for 7 days. Press J and K to move between servers in the list’s order.

The certificate

When a server’s OpenConnect service serves a certificate, the System tab has a Certificate group: what serves it, when it expires, the names it covers, and whether the file on disk matches. The agent reads it every hour. View details opens the full certificate:

Its common name, who issued it, when it’s valid from and until, its serial number, key, and TLS version, and when TunnelHQ first saw it.
The names it covers, and its chain of trust.
Seven checks, each passing, failing, or not known:

Check	Passes when
Within its validity period	Today is between its valid from and valid until dates.
Covers the tunnel’s address	Every linked tunnel or check that uses a name is covered by one of the certificate’s names.
Issued by a trusted authority	The chain leads to an authority the server trusts.
Complete chain	The server sends every certificate between its own and the authority.
Modern TLS version	It’s served over TLS 1.2 or 1.3.
Strong signature and key	An RSA key of 2048 bits or more, an ECDSA key of P-224 or more, or an Ed25519 key, and no SHA-1 or MD5 signature.
File on disk	The certificate file on the server is the one OpenConnect serves, not a renewed one waiting for a reload.

The agent fetches nothing from the internet, so there’s no revocation check.

You can rename a server, put it in a group, and give it up to 10 tags. Each tag is 1 to 32 letters, digits, spaces, dots, dashes, or underscores. A server with VPN services gets the vpn tag by itself.

Tunnels and checks on a server

A tunnel or check links to a server by itself when it points at one of the server’s public addresses, or when its host name is the server’s hostname. Where TunnelHQ looks:

Monitor	What it points at
Tunnel	Its server address.
Website and keyword checks	The host in the URL.
Ping and port checks	The host.
DNS checks	The DNS server it asks.
Heartbeats	Nothing, so link one by hand.

A host name is looked up, so a link follows an address change. Private addresses are ignored, because many servers share them, so a monitor aimed at one is linked by hand. A monitor can link to several servers.

Link a monitor by hand when TunnelHQ can’t tell from the address, for example behind a load balancer or a CDN. Unlinking keeps a monitor away from a server even when its address matches.

Alerts

Alert	Default	Setting	Opens when	Closes when
Stopped reporting	On, after 90 seconds	90 to 3,600 seconds	No report for that long, from a server that reported before	It reports again for 5 minutes
Service stopped	On, after 60 seconds	30 to 3,600 seconds	A watched VPN service went from active to inactive, and stayed inactive that long	It’s active again for 5 minutes
CPU, memory, disk	Off until you add one	Above a percent (1 to 100) for 60 to 3,600 seconds, 300 by default	Every report for that long is above the level	It’s at or below the level for 5 minutes
Users from a country	Off	On or off	The users from one country vanish while the service runs and users from other countries stay	That country’s users are back in two 5-minute readings in a row
Certificate	On, 14 days	7, 14, 21, or 30 days, or off	The certificate the server’s OpenConnect service serves has fewer days left. Needs agent 1.7.0.	A renewed certificate with enough days left is served for 5 minutes

Changing a server’s alerts needs a role that can edit monitors. On the Alerts tab, the certificate alert is Certificate expires soon, shown for a server whose OpenConnect service serves a certificate.

One message for servers that fail together

Servers that fail for the same reason at the same time share one message, and one Resolved message goes when every server it named is back:

Stopped reporting groups the project’s servers. The message waits until their linked tunnels and checks have run, between 1 and 5 minutes, so it can say what they show: “Its 6 tunnels still pass, so the server is up and its agent stopped”, or “Its 6 tunnels fail too, so the server or its network is down.”
Service stopped waits a minute and groups the servers where the same service stopped.
Certificate groups every server serving the same certificate, with one message and one Resolved for all of them. The message gives the expiry date and the names the certificate covers. When the file on disk was renewed but OpenConnect still serves the old one, it says to run systemctl reload ocserv.
CPU, memory, disk and users from a country are one alert per server.

A server that fails later opens its own message, so the times a message gives are true of every server it names.

Users from a country

This alert is for a VPN server blocked in one country. It opens only when the drop is unusual for that server:

The country averaged at least 30 users over the hour before, and is now gone.
Every watched service is still running, and users from other countries are steady.
It’s not a quiet hour: when the country had fewer than 30 users at the same time last week, there’s no alert.
It’s not country-wide: when half or more of your other servers that had 30 users from the country lost them too, such as in a shutdown, there’s no alert.
With neither last week’s numbers nor another server to compare with, there’s no alert.

Where messages go

To the alert channels of the server’s linked tunnels and checks, or to every active channel in the project when none are linked. Server alerts never go to webhook channels. TunnelHQ restarting pages nobody, and when almost every server stops reporting at once, TunnelHQ treats it as its own problem and waits.

From the API, the CLI, or an AI assistant

Everything the Servers pages do also works without them:

REST API: the /hosts routes. Servers are hosts in the API, with IDs like hst_7, because /servers already means monitors.
CLI: thq hosts. See Manage servers.
MCP server: 11 tools, from list_hosts to update_host_agent. See Tools. A signed-in assistant can’t get the install command or remove a server; those need an API key.