---
title: Hosts (server monitoring)
description: Install watchfor-agent on a Linux server and see CPU, memory, disk, network and processes in WatchFor — pushed as often as every 15 seconds, no inbound ports.
canonical: https://watchfor.io/docs/hosts
---

# Hosts (server monitoring)

Install watchfor-agent on a Linux server and see CPU, memory, disk, network and processes in WatchFor — pushed as often as every 15 seconds, no inbound ports.

Monitors look at your systems from the outside. A **host** is the inside view:
a small open-source agent runs on the server, reads `/proc` and `/sys` every
few seconds, and pushes the numbers to WatchFor over HTTPS. Nothing listens on
the server, nothing to open in a firewall, and the agent never receives
commands — the traffic is one way, out. What you get for it is summed up on
[Server monitoring](/server-monitoring); the agent has
[a page of its own](/agent).

The agent is [watchfor-io/agent](https://github.com/watchfor-io/agent) on
GitHub: Go, a single static binary for amd64 and arm64, Apache-2.0, a handful
of dependencies (YAML, x/crypto and the Charm terminal libraries). Read it
before you run it — `watchfor-agent check` prints exactly what it would send,
without sending anything.

## Add a host

1. **Hosts → New host**, give it a name (the server's hostname is a fine
   default).
2. Copy the install command. It checks that `curl`, `tar`, `sha256sum`
   and `useradd` are present — and, on a terminal, offers to install what
   is missing with your distribution's package manager — downloads the
   release for your architecture, verifies the sha256 checksum and the
   release signature, creates an unprivileged
   `watchfor-agent` user, writes the token to `/etc/watchfor-agent/token`
   (mode 0600), enables the systemd unit and asks whether to turn on daily
   auto-update. It needs no extra packages: the checksums it checks the
   download against are served by watchfor.io only after their signature
   verified there, and the agent then verifies the release signature
   itself with the key built into it. Run the same command again later to
   upgrade, reinstall or change the auto-update setting. Add `--token -`
   to type the token on the terminal instead of putting it on the command
   line:

   ```sh
   curl --proto '=https' --tlsv1.2 -fsSL https://watchfor.io/agent/install.sh | sudo sh -s -- --token wfh_…
   ```

3. Within a minute the host shows **Online** and the first charts fill in.

The token is shown once. If it is lost, **Rotate token** on the host page
issues a new one and invalidates the old one within a minute. Put the new
one on the server by running the install command again (it writes the
token and restarts the agent), or replace only the token file:

```sh
echo 'wfh_…' | sudo install -m 0600 -o watchfor-agent -g watchfor-agent /dev/stdin /etc/watchfor-agent/token
```

Agent 0.9 and newer pick up a changed token file within 30 seconds; older
agents need `sudo systemctl restart watchfor-agent` after it.

> **Info**
>
> Prefer not to pipe a script into a shell? Follow [Install by
> hand](https://github.com/watchfor-io/agent#install-by-hand) in the README for
> the binary, the user and the unit — or `watchfor-agent once` from cron on hosts
> that must not run a daemon. The token goes in its own 0600 file,
> `/etc/watchfor-agent/token`, never in `agent.yml` (which is world-readable);
> the install dialog shows the command for that file and a complete `agent.yml`.

## The agent's screen

`watchfor-agent` with no command opens a screen on the server — the same
binary, no extra install, and it needs no token to look around. Run it as
root (`sudo watchfor-agent`) and the maintenance rows work too. The menu is
three groups:

- **Overview** — *System* (machine, agent, service, where it sends), *Live*
  (CPU, memory, load, disks, network and the busiest processes, sampled
  every two seconds from the real collectors, not from the server) and
  *The batch* (the exact samples the next push will carry).
- **Settings** — the server and token, the host name and tags, the sending
  interval and the spool, disks, interfaces, processes, logging, updates,
  facts (the cloud lookup) and the file itself, each editable in place:
  Enter starts editing the pane in front of you, Space changes a value, Esc
  leaves it. Processes to watch are ticked from the biggest ones running —
  shown by their full names even where the kernel cuts them at 15
  characters — or typed in by hand for a daemon the list does not offer or
  one that is down at the moment. Changes are written the moment you
  confirm them, and a running agent picks them up within half a minute.
- **Maintenance** — *Update*, *Health check*, *Service & log* and
  *Reinstall & reset*. Update is the same upgrade described
  [below](#upgrading-the-agent) and Health check the same checks as
  [`watchfor-agent health`](#health-check). *Reinstall the binary*
  installs this version again over a damaged one (signature and checksum
  verified first, service restarted); *Reset agent.yml to defaults* writes
  a fresh config with this machine's mounts and interfaces, keeping the
  server address and token file and leaving the old file as
  `agent.yml.bak`. Both ask first, with the safe answer under the cursor.

There is nothing else to learn: arrow keys move, Enter opens what the cursor
is on, Esc goes back, `q` quits; inside a section Tab steps between the
things it holds. Long lists — mounts, interfaces, processes — filter as you
type after `/`. A row that cannot run says why on the row itself
(`needs root — run: sudo systemctl start watchfor-agent`), and anything that
replaces a file or stops the agent asks first.

## What is collected

| Module | Metrics |
| --- | --- |
| `system` | CPU usage (user, system, iowait, steal; per core optional), load 1/5/15, memory and swap, uptime, context switches |
| `disk` | Usage and inodes per filesystem; read/write throughput, ops, utilisation and await per disk |
| `network` | Bytes, packets, errors and drops per interface; TCP/UDP socket counts |
| `processes` | Process-state counts, top consumers of CPU and memory, and `watch` entries — "is `nginx` running, how much does it use" |

Host facts — OS, kernel, CPU model and count, memory, hardware (the server
model on metal, the instance size on a cloud), virtualisation, boot time
and the address on each interface — travel with the first batch and then
once an hour.

On a cloud instance the public IPv4 the provider assigned never appears on
an interface (the provider maps it), so since agent 0.8.0 the facts also
carry the **public addresses** the cloud assigned from the outside, IPv4
and IPv6, asked of the cloud's own metadata service: AWS, Google Compute
Engine, Azure, Alibaba Cloud, Tencent Cloud, Scaleway, DigitalOcean,
Hetzner, Vultr, Linode and OpenStack (Oracle Cloud does not publish them).
Only real public addresses get through — a private, link-local or malformed
answer is dropped — and the lookup is link-local, once an hour, with a
sub-second timeout. The host page shows them in the system card under the
host's address, marked **public · cloud**; `facts.cloud_metadata: false`
switches the lookup off (see [Configuration](#configuration)).

Rates need two samples, so the first batch after a start carries gauges
only. Everything is described in `agent.yml`; unknown keys are an error,
and secrets only ever come from a 0600 file or an environment variable.

## Configuration

Everything lives in `/etc/watchfor-agent/agent.yml`. The installer writes it
with every key explained in a comment and the mounts, disks and interfaces
it found on the machine listed inside, so choosing what to watch is editing
a list. A box with forty mounts and six interfaces where one of each
matters:

```yaml
modules:
  system: { per_core: true }
  disk:
    mounts: [/, /data]              # only these mount points; default: every real filesystem
    devices: [nvme0n1]              # I/O rates for these whole disks
    ignore_fs: [nfs]
  network:
    interfaces: [eth0]              # default: all but lo, veth*, docker*, br-*, virbr*
  processes:
    top: 10
    watch:
      - name: nginx
      - cmdline: postgres -D
        user: postgres
```

Change one value without an editor — the file is validated first, so a
value the agent would not start with (an interval below 5 s, an invalid
host name) is refused rather than written:

```sh
sudo watchfor-agent config set interval 30s
sudo watchfor-agent config set host.tags.env prod
sudo watchfor-agent config set log.level debug
watchfor-agent config get interval
watchfor-agent config keys           # every key `config set` can change
watchfor-agent config detect         # what the collectors see: CPU, memory, hardware, mounts, disks, interfaces
```

The keys most people touch:

| Key | What it does |
| --- | --- |
| `interval` | How often a batch is pushed (default `60s`). Your plan sets the floor and the server adjusts a running agent; the agent's own floor is 5 seconds. |
| `host.name`, `host.tags` | The name and tags shown in WatchFor. |
| `spool.dir`, `spool.max_mb` | Where batches wait while WatchFor is unreachable (`/var/lib/watchfor-agent`) and how much to keep (64 MB, oldest dropped first). |
| `log.level` | `info` by default; `debug` puts every push, spooled batch, fact and cloud lookup in the journal. |
| `facts.cloud_metadata` | `false` stops the agent from ever asking a cloud metadata service for the instance size and the public addresses. |
| `modules.disk.mounts`, `modules.disk.devices`, `modules.disk.ignore_fs` | Which mount points to report, which whole disks to measure I/O on, and filesystem types to skip. |
| `modules.network.interfaces` | Which interfaces to report (default: all but `lo`, `veth*`, `docker*`, `br-*`, `virbr*`). |
| `modules.processes.top`, `modules.processes.watch` | How many top consumers to send, and named processes to watch (`name:` for the kernel name, `cmdline:` for a substring of the command line, optionally `user:`). |
| `updates.auto` | Written by `watchfor-agent auto-update on` and `off` — see [Upgrading the agent](#upgrading-the-agent). |

A running agent watches the file and restarts itself on a change that
validates, so an editor, the screen, `config set` or Ansible take effect
within half a minute; a change it would not accept is logged and ignored,
and the running config stays. Every key with its default is in the agent's
[README](https://github.com/watchfor-io/agent#configuration) and in
[`agent.example.yml`](https://github.com/watchfor-io/agent/blob/main/packaging/agent.example.yml).

## Health check

`watchfor-agent health` answers, without the screen, what you would
otherwise check by hand — one line per check:

- `agent.yml` is there and parses;
- the binary is not writable by others, the service user exists, the token
  file is private and the spool is writable;
- the service is running and enabled at boot, and the update timer matches
  what `updates.auto` says;
- the host can reach WatchFor over TLS;
- since agent 0.9: whether WatchFor accepted the last push (or rejected the
  token, or paused the host for the plan), and whether any other process or
  account uses the agent's uid — `-fix` then moves the account to a free uid
  (see [the agent's own processes](#the-agents-own-processes)).

```sh
watchfor-agent health              # one line per check; exit 1 when something failed
sudo watchfor-agent health -fix    # apply every safe repair, then check again
watchfor-agent health -deep -q     # also compare the binary with the signed release; print only problems
```

Repairs (chmod, chown, mkdir, `systemctl enable --now`) run as root, so
they are deliberately narrow: they touch only the installer's own paths
(`/etc/watchfor-agent/agent.yml` and `token`, `/var/lib/watchfor-agent`,
`/usr/local/bin/watchfor-agent`), work on open file descriptors rather than
paths, refuse symlinks and refuse a directory anyone but root can write. A
token file or spool that `agent.yml` points at somewhere else still works,
but the check only reports it. The same checks, with the same repairs, are
the **Health check** row on the screen, where *Compare with the signed
release* is what `-deep` does.

If the host page says the agent stopped reporting, or no batch arrived
after installing, start with these two on the server (the page offers both
to copy):

```sh
sudo watchfor-agent health
sudo journalctl -u watchfor-agent -n 50
```

Exit codes worth knowing in a script: `health` exits 1 when at least one
check failed; `upgrade -check` exits 10 when a newer release is available;
the daemon exits 3 when the server has rejected the token repeatedly for
over an hour (host removed or token rotated — the unit does not restart on
it; a new token in `/etc/watchfor-agent/token` starts it again, see
above) and 75 when `agent.yml` changed and validated (the unit starts it
again at once on the new file). Before 0.9 a host paused by the plan also
ended in exit 3 — see [Plans and the push interval](#plans-and-the-push-interval). The full table is in
the [README](https://github.com/watchfor-io/agent#exit-codes).

## The agent's own processes

The agent is one process and starts nothing while it collects; only
`health`, the screen and `upgrade`, when you run them, call `systemctl` or
`journalctl`. `systemctl status watchfor-agent` lists every process in its
cgroup, and the unit caps it at 64 tasks, threads included (`TasksMax=64`;
`GOMAXPROCS=2` only limits how many of them run Go code at once). Since
agent 0.9 every batch carries what the agent itself uses, and the host page
shows it next to the version: *agent 7 MB · 0.1 % CPU*.

**Why does `ps` show `watchfor-agent` next to processes I don't know?** `ps`
names a process's owner by looking up its numeric uid in this server's
`/etc/passwd`. A container without a user namespace runs its own uid 999 as
host uid 999, and `useradd --system` hands out 999 first on many
distributions — so on a host running, say, mailcow, SOGo's `sogod` workers
show up as `watchfor-agent`. Check with:

```sh
ps -o pid,uid,cgroup,comm -u watchfor-agent   # a docker-… or libpod-… cgroup = a container's process
```

Since 0.9 the agent reports such processes by their numeric uid (*uid 999*)
with a **container** badge in Top processes, and marks its own process
**agent**; new installs take a uid from 60578–61183 that nothing uses. On an
existing install `sudo watchfor-agent health` names every process sharing
the agent's uid, and `sudo watchfor-agent health -fix` moves the account
and its group to a free uid and hands the agent's files over. It is worth
doing: to the kernel, a shared uid is one user.

## Security

A server running the agent should be no more exposed than one without it:

- **Outbound only.** While collecting, the agent opens HTTPS connections to
  `ingest.watchfor.io` (or the `server.url` you set) and nothing else.
  watchfor.io and GitHub's release hosts are contacted only by `upgrade`
  (by hand or from the daily timer), and the cloud's metadata service
  (link-local, sub-second timeout) only when the firmware says the machine
  is on that cloud — `facts.cloud_metadata: false` switches that off.
  Nothing listens; there is no inbound port to open.
- **No remote commands.** The server's answer to a batch carries two
  things: the push interval your plan allows and the newest agent version.
  Nothing in it is executed, and nothing in `agent.yml` is executed either.
- **Unprivileged and sandboxed.** The service runs as the `watchfor-agent`
  system user under a hardened unit — `ProtectSystem=strict`, an empty
  capability set, `NoNewPrivileges`, a system-call filter, a 256 MB memory
  cap, and write access to its spool directory only. The token file is
  mode 0600 and owned by the service user; a world-readable token is a
  start-up error. The unit with every directive is
  [`packaging/watchfor-agent.service`](https://github.com/watchfor-io/agent/blob/main/packaging/watchfor-agent.service)
  in the repository.
- **Signed, reproducible releases.** Every release ships a `checksums.txt`
  signed with the WatchFor release key (minisign). Binaries are static,
  built from the tag with `CGO_ENABLED=0 -trimpath`; a second CI job
  rebuilds them on a fresh runner and fails if a byte differs, and every
  asset carries a GitHub build-provenance attestation
  (`gh attestation verify <tarball> --repo watchfor-io/agent`). The
  installer always verifies the signature — there is no switch to skip
  it — and the agent verifies releases itself with the key built into it.
- **Readable before you run it.** Apache-2.0, a handful of dependencies,
  and `watchfor-agent check` prints the exact batch without sending it.

To verify a release by hand, with `checksums.txt`, `checksums.txt.minisig`
and the tarball from the release page:

```sh
tar -xzf watchfor-agent_<version>_linux_amd64.tar.gz watchfor-agent
./watchfor-agent verify -checksums checksums.txt watchfor-agent_<version>_linux_amd64.tar.gz
sudo watchfor-agent health -deep     # the binary already installed, against the signed release
```

## The hosts list

Every host with its status, the latest CPU, memory and disk readings, its
address, system and tags. **Address** is the host's own address — the one
it uses to reach WatchFor, as the agent sees it (agent 0.4.0 or newer) —
with the public address its batches arrive from underneath when the two
differ (a server behind NAT, a cloud VM whose public IP is mapped by the
provider). Hover for the other interfaces; container and bridge
interfaces are left out. Search matches the name, tags (`env=prod`), OS,
kernel, CPU model, both addresses and every interface address the agent
reports; the status filter and
the column sorting narrow it further; long lists are paged. The `⋯` menu on
a row opens the host, renames it, rotates its token or removes it — the same
actions as the host page.

## The host page

The host page has the same tabs as a monitor's page: **Overview**,
**Alerting**, **Incidents**, **Uptime SLA** and **Settings**. Deep links work
the same way too (`?tab=alerts`).

- **Status** — *Online* while batches arrive, *Stale* after three missed
  push intervals (never sooner than 90 seconds — 90 s on a 15-second plan,
  3 minutes on a 60-second one), *Offline* after 10 minutes. A host that was offline catches up:
  the agent spools batches on disk while it cannot reach WatchFor and replays
  them in order.
- **Overview** — first the same two cards a monitor opens with: uptime for
  the last 24 hours and the incident heatmap (one cell per hour — bright red
  or amber while a critical or warning incident is firing, the same colour
  faded once it resolved, green when the hour was clean; click a count to
  jump to those incidents). Then six cards from the latest batch, each a small instrument
  rather than a number: CPU (arc gauge plus the user / system / I/O wait /
  steal split), memory (used, cache + buffers and free as one bar, swap
  below), load (1, 5 and 15 minutes as bars that fill at the core count),
  disks (one bar per mount, fullest first), network (in and out now, per
  interface) and system (uptime, OS, kernel). Colours follow the same
  thresholds as the alert presets; hover the icon on each for what it
  means. Then
  charts for the last hour to the last 30 days: CPU breakdown, load, memory
  in gigabytes (used, cache + buffers, free and swap, each line at its own
  value against the installed total — used is total minus MemAvailable),
  disk usage per mount (the tooltip adds the gigabytes), disk I/O, network,
  process counts and sockets. Every chart's icon explains what is plotted and how it is
  computed; hovering one chart moves the cursor on all of them, so you read
  CPU, load, memory and I/O at the same moment. Below the charts: tables
  for filesystems, top processes by CPU and memory (with agent 0.9 and
  newer, a **container** badge on processes that run in one, shown by
  their numeric uid, and an **agent** badge on the agent's own process),
  and watched processes. Raw 15-second samples are kept for 7 days; minute and
  hour averages for 30 and 400 days.
- **Settings** — the name shown in WatchFor, free-form notes (what the
  server runs, who owns it — shown under the name) and the token card. The
  agent's hostname and tags come from its config file on the server.

## Hosts elsewhere in WatchFor

A host is a first-class citizen wherever monitors are:

- **Dashboard** — the Healthy number counts hosts that report, the Hosts tile
  shows online / stale / offline, and the unstable list ranks hosts next to
  monitors.
- **Status pages** — add a host to a [component](/docs/status-pages/components);
  it shows as online or down from its own alerts.
- **Reports** — a *Hosts* section with uptime, incidents and average CPU,
  memory and disk usage per host (see [Reports](/docs/reports)).
- **Search** — press <kbd>⌘K</kbd> / <kbd>Ctrl K</kbd> and type a host's name.
- **API, MCP and A2A** — [`/v1/hosts`](/docs/api/hosts), the `list_hosts`,
  `get_host` and `get_host_metrics` MCP tools and the `check-hosts` A2A skill.

## Alerting

Every host comes with rules switched on, routed to your default contact
group:

| Rule | Fires when | Severity |
| --- | --- | --- |
| Agent Silent | no batch arrived for three push intervals plus a minute (at least 2 minutes) | critical |
| Disk Almost Full | the fullest filesystem is at 80 % (warning) or 90 % (critical) in two consecutive batches | warning / critical |
| Memory Pressure | memory in use stays at 85 % (warning) or 95 % (critical) for eight consecutive batches | warning / critical |

The **Alerting** tab offers more, each with its own sliders: high CPU
(80 / 95 %), load per core (1 / 2), I/O wait, CPU steal, swap and inodes
all come as a **warning + critical pair** you can move independently, each
level with its own switch (keep only the critical page, only the warning
heads-up, or both);
blocked and zombie processes, network errors and "server rebooted" (uptime
under five minutes) take a single value and a severity of your choice; a
watched process that is not running is always critical. "Consecutive
batches" is the duration: 8 batches means 2 minutes on a 15-second plan
and 8 minutes on a 60-second one — adjust it when you change plans if that
matters to you.

Rules evaluate every batch the agent sends. A rule only sees a metric the
batch carries: if you turn a collector off in `agent.yml`, its rules stay
quiet rather than resolving or firing on a missing value.

**Incidents** and **Uptime SLA** work exactly as for monitors — an incident
opens when a rule fires, notifications go through your channels and on-call
rotations, maintenance windows suppress alerts and are excluded from
uptime (pick the host under **Hosts** in the window's picker, or use
**Schedule maintenance** in the host page's ⋯ menu), and uptime counts confirmed critical incidents only. A silent agent
is such an incident; a busy CPU on its own is not.

## Plans and the push interval

How many hosts you can add and how often each agent may report depend on
your plan — the Hosts page shows both, and [pricing](/pricing) lists them per
plan. The interval is not something you have to configure: the server
answers every batch with the shortest interval your plan allows, and the
agent adjusts to it on the spot. Upgrade, and running agents speed up
within a minute; downgrade, and they slow down the same way. Setting a
faster `interval` in `agent.yml` than the plan allows just makes the agent
stretch back to the plan value (it logs that once). The agent's own floor is
5 seconds.

If your organization has more hosts than a new plan includes, the oldest
ones (by when they were added) keep reporting and the newest above the
limit are **paused**: the Hosts page marks them *Paused — over plan limit*,
WatchFor refuses their batches and they raise no alerts. Nothing is deleted.
Upgrade or remove a host and the paused ones are let back in, oldest first.
Agent 0.9 and newer keep asking while paused (after 1, 5 and 15 minutes,
then every 30) and resume by themselves. Older agents stop asking after
about an hour and remember the refusal, so a plain restart stops again at
once; run `sudo watchfor-agent upgrade` on those servers — the new version
starts, asks again and from then on resumes by itself.

## Upgrading the agent

You do not have to watch the releases page. The server tells every running
agent about the newest release in its answer to each batch (the agent talks
to nothing but WatchFor, so no server of yours needs access to GitHub for
this). The agent logs it once, and the host page shows **update available**
next to the version badge, with the newer version and the command to run.

On the server, as root:

```sh
sudo watchfor-agent upgrade
```

That downloads the release from GitHub over HTTPS (only GitHub's release
hosts are accepted, redirects included), verifies the release signature
with the WatchFor key built into the agent, checks the archive against the
signed checksums, runs the new binary once to confirm it is the expected
version, swaps it in atomically and restarts the service. If any step
fails, nothing is replaced. `sudo watchfor-agent upgrade -check` only
reports. The **Update** row under Maintenance on the agent's screen does
the same, and re-running the one-line installer from **New host** does
the same upgrade with the same checks and keeps your token and `agent.yml`.

Servers that should keep themselves current can opt in to a daily timer:

```sh
curl --proto '=https' --tlsv1.2 -fsSL https://watchfor.io/agent/install.sh | sudo sh -s -- --auto-update
```

The timer installs a newer release only when the server has reported one,
once a day at a random moment within a six-hour window so a fleet does not
restart in lockstep. The running agent itself never replaces its own
binary; the swap is done by a separate root-run one-shot, and the agent
keeps running unprivileged.

Switch it later without the installer — the agent manages the timer and
records the choice in `agent.yml` (`updates.auto`):

```sh
sudo watchfor-agent auto-update on
sudo watchfor-agent auto-update off
watchfor-agent auto-update status
```

## Removing a host

**Remove** (in the host page's ⋯ menu, or the row menu in the list) revokes
the token and deletes the host's metric history, alert rules and incidents.
The agent on the server keeps running until you remove it — `sudo
watchfor-agent uninstall` takes the service, the update timer, the binary,
the state, the config with the token and the service user off the machine
(the installer's `--uninstall` does the same without the binary). Left
alone, the agent retries the rejected token with growing pauses and stops
by itself once the rejection has held for over an hour.

## Running as root

The service runs as an unprivileged user. Everything the four modules need
is readable that way on a standard kernel. Per-process I/O counters and
container sockets are not; when a module needs them, grant the capability
in a drop-in (`sudo systemctl edit watchfor-agent`):

```ini
[Service]
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
AmbientCapabilities=CAP_DAC_READ_SEARCH
```

Editing the unit file itself is undone by the next upgrade. That grants
read access everywhere without running as root.

---

Canonical page: https://watchfor.io/docs/hosts · All docs: https://watchfor.io/docs · Site guide: https://watchfor.io/llms.txt