---
title: Hosts (server monitoring)
description: Install watchfor-agent on a Linux server and see CPU, memory, disk, network and processes in WatchFor — pushed as often as every 15 seconds, no inbound ports.
canonical: https://watchfor.io/docs/hosts
---

# Hosts (server monitoring)

Install watchfor-agent on a Linux server and see CPU, memory, disk, network and processes in WatchFor — pushed as often as every 15 seconds, no inbound ports.

Monitors look at your systems from the outside. A **host** is the inside view:
a small open-source agent runs on the server, reads `/proc` and `/sys` every
few seconds, and pushes the numbers to WatchFor over HTTPS. Nothing listens on
the server, nothing to open in a firewall, and the agent never receives
commands — the traffic is one way, out.

The agent is [watchfor-io/agent](https://github.com/watchfor-io/agent) on
GitHub: Go, a single static binary, Apache-2.0, a handful of dependencies
(YAML, x/crypto and the Charm terminal libraries). Read it
before you run it — `watchfor-agent check` prints exactly what it would send,
without sending anything.

## Add a host

1. **Hosts → Add host**, give it a name (the server's hostname is a fine
   default).
2. Copy the install command. It checks that `curl`, `tar`, `sha256sum`
   and `useradd` are present — and, on a terminal, offers to install what
   is missing with your distribution's package manager — downloads the
   release for your architecture, verifies the sha256 checksum and the
   release signature, creates an unprivileged
   `watchfor-agent` user, writes the token to `/etc/watchfor-agent/token`
   (mode 0600), enables the systemd unit and asks whether to turn on daily
   auto-update. It needs no extra packages: the checksums it checks the
   download against are served by watchfor.io only after their signature
   verified there, and the agent then verifies the release signature
   itself with the key built into it. Run the same command again later to
   upgrade, reinstall or change the auto-update setting. Add `--token -`
   to type the token on the terminal instead of putting it on the command
   line:

   ```sh
   curl --proto '=https' --tlsv1.2 -fsSL https://watchfor.io/agent/install.sh | sudo sh -s -- --token wfh_…
   ```

3. Within a minute the host shows **Online** and the first charts fill in.

The token is shown once. If it is lost, **Rotate token** on the host page
issues a new one and invalidates the old one within a minute.

> **Info**
>
> Prefer not to pipe a script into a shell? Download the tarball from the
> releases page, put the binary anywhere, write `/etc/watchfor-agent/agent.yml`
> (the host page shows a complete one) and run `watchfor-agent run` as a service
> — or `watchfor-agent once` from cron on hosts that must not run a daemon.

## What is collected

| Module | Metrics |
| --- | --- |
| `system` | CPU usage (user, system, iowait, steal; per core optional), load 1/5/15, memory and swap, uptime, context switches |
| `disk` | Usage and inodes per filesystem; read/write throughput, ops, utilisation and await per disk |
| `network` | Bytes, packets, errors and drops per interface; TCP/UDP socket counts |
| `processes` | Process-state counts, top consumers of CPU and memory, and `watch` entries — "is `nginx` running, how much does it use" |

Host facts (OS, kernel, CPU model and count, memory, virtualisation, boot
time) are reported on start and once an hour. Everything is described in
`agent.yml`; unknown keys are an error, and secrets only ever come from a
0600 file or an environment variable.

The installer writes `agent.yml` with every key explained and the mounts,
disks and interfaces it found on the machine listed inside, so choosing
what to watch is editing a list. A box with forty mounts and six
interfaces where one of each matters:

```yaml
modules:
  system: { per_core: true }
  disk:
    mounts: [/, /data]              # only these mount points; default: every real filesystem
    devices: [nvme0n1]              # I/O rates for these whole disks
    ignore_fs: [nfs]
  network:
    interfaces: [eth0]              # default: all but lo, veth*, docker*, br-*, virbr*
  processes:
    top: 10
    watch:
      - name: nginx
      - cmdline: postgres -D
        user: postgres
```

`watchfor-agent` with no command opens a screen on the server that shows
all of this without editing the file: the machine's current CPU, memory,
disks, network and busiest processes, the exact batch the next push will
carry, every setting in an editable pane, the service with its log, and
the agent's own upkeep. Run it as root and the maintenance keys work too.

Arrow keys move, Enter opens, Esc goes back — there is nothing else to
learn, and everything the agent can do is a row you choose rather than a
key you have to know. That last group is worth knowing about:

- **Health check** — everything that has to be right for the agent to
  work: the config file and whether it parses, permissions on the binary
  and the token, the spool folder, the service and whether it starts at
  boot, the update timer, and the way out to WatchFor. Whatever can be
  repaired safely is repaired for you when you choose it, and one row
  further down compares the binary on disk with the release WatchFor
  signed.
- The same checks run from a script or over SSH as `watchfor-agent
  health` (exit 1 when something failed, `-fix` repairs what it safely
  can).
- **Reinstall & reset** — install this version again over a damaged
  binary, or write a fresh `agent.yml` with the defaults and this
  machine's mounts and interfaces. Both ask first, with the safe answer
  under the cursor; the old config is kept as `agent.yml.bak`.

Useful commands on the server:

- `watchfor-agent config detect` — what the collectors see: CPU, memory,
  hardware, every mount with its size, whole disks, interfaces.
- `sudo watchfor-agent config set interval 30s` — change one value in
  place (`config keys` lists them); the file is validated first.
- `watchfor-agent check` — print the batch that would be sent, without
  sending it.

Other settings worth knowing: `interval` (how often a batch is pushed;
your plan sets the floor and the server adjusts a running agent),
`spool.dir` (batches wait in `/var/lib/watchfor-agent` while WatchFor is
unreachable and are replayed later), `log.level: debug` (every push,
spooled batch, fact and cloud lookup in the journal), and
`facts.cloud_metadata: false` to stop the agent from ever asking a cloud
metadata service for the instance size and the public addresses the cloud
assigned to the host (it asks only the recognised provider, only those
paths, once an hour; a private or malformed answer is dropped). The full
list is in the agent's README.

## The hosts list

Every host with its status, the latest CPU, memory and disk readings, its
address, system and tags. **Address** is the host's own address — the one
it uses to reach WatchFor, as the agent sees it (agent 0.4.0 or newer) —
with the public address its batches arrive from underneath when the two
differ (a server behind NAT, a cloud VM whose public IP is mapped by the
provider). Hover for the other interfaces; container and bridge
interfaces are left out. Search matches the name, tags (`env=prod`), OS,
kernel, CPU model, both addresses and every interface address the agent
reports; the status filter and
the column sorting narrow it further; long lists are paged. The `⋯` menu on
a row opens the host, renames it, rotates its token or removes it — the same
actions as the host page.

## The host page

The host page has the same tabs as a monitor's page: **Overview**,
**Alerting**, **Incidents**, **Uptime SLA** and **Settings**. Deep links work
the same way too (`?tab=alerts`).

- **Status** — *Online* when a batch arrived in the last 90 seconds, *Stale*
  up to 10 minutes, *Offline* after that. A host that was offline catches up:
  the agent spools batches on disk while it cannot reach WatchFor and replays
  them in order.
- **Overview** — first the same two cards a monitor opens with: uptime for
  the last 24 hours and the incident heatmap (one cell per hour; click a
  count to jump to those incidents). Then six cards from the latest batch, each a small instrument
  rather than a number: CPU (arc gauge plus the user / system / I/O wait /
  steal split), memory (used, cache + buffers and free as one bar, swap
  below), load (1, 5 and 15 minutes as bars that fill at the core count),
  disks (one bar per mount, fullest first), network (in and out now, per
  interface) and system (uptime, OS, kernel). Colours follow the same
  thresholds as the alert presets; hover the icon on each for what it
  means. Then
  charts for the last hour to the last 30 days: CPU breakdown, load, memory
  in gigabytes (used, cache + buffers, free and swap, each line at its own
  value against the installed total — used is total minus MemAvailable),
  disk usage per mount (the tooltip adds the gigabytes), disk I/O, network,
  process counts and sockets. Every chart's icon explains what is plotted and how it is
  computed; hovering one chart moves the cursor on all of them, so you read
  CPU, load, memory and I/O at the same moment. Below the charts: tables
  for filesystems, top processes by CPU and memory, and watched processes. Raw 15-second samples are kept for 7 days; minute and
  hour averages for 30 and 400 days.
- **Settings** — the name shown in WatchFor, free-form notes (what the
  server runs, who owns it — shown under the name) and the token card. The
  agent's hostname and tags come from its config file on the server.

## Alerting

Every host comes with rules switched on, routed to your default contact
group:

| Rule | Fires when | Severity |
| --- | --- | --- |
| Agent Silent | no batch arrived for three push intervals plus a minute (at least 2 minutes) | critical |
| Disk Almost Full | the fullest filesystem is at 80 % (warning) or 90 % (critical) in two consecutive batches | warning / critical |
| Memory Pressure | memory in use stays at 85 % (warning) or 95 % (critical) for eight consecutive batches | warning / critical |

The **Alerting** tab offers more, each with its own sliders: high CPU
(80 / 95 %), load per core (1 / 2), I/O wait, CPU steal, swap and inodes
all come as a **warning + critical pair** you can move independently, each
level with its own switch (keep only the critical page, only the warning
heads-up, or both);
blocked and zombie processes, network errors and "server rebooted" (uptime
under five minutes) take a single value and a severity of your choice; a
watched process that is not running is always critical. "Consecutive
batches" is the duration: 8 batches means 2 minutes on a 15-second plan
and 8 minutes on a 60-second one — adjust it when you change plans if that
matters to you.

Rules evaluate every batch the agent sends. A rule only sees a metric the
batch carries: if you turn a collector off in `agent.yml`, its rules stay
quiet rather than resolving or firing on a missing value.

**Incidents** and **Uptime SLA** work exactly as for monitors — an incident
opens when a rule fires, notifications go through your channels and on-call
rotations, maintenance windows suppress alerts and are excluded from
uptime, and uptime counts confirmed critical incidents only. A silent agent
is such an incident; a busy CPU on its own is not.

## Plans and the push interval

How many hosts you can add and how often each agent may report depend on
your plan — the Hosts page shows both, and [pricing](/pricing) lists them per
plan. The interval is not something you have to configure: the server
answers every batch with the shortest interval your plan allows, and the
agent adjusts to it on the spot. Upgrade, and running agents speed up
within a minute; downgrade, and they slow down the same way. Setting a
faster `interval` in `agent.yml` than the plan allows just makes the agent
stretch back to the plan value (it logs that once). The agent's own floor is
5 seconds.

## Upgrading the agent

You do not have to watch the releases page. The server tells every running
agent about the newest release in its answer to each batch (the agent talks
to nothing but WatchFor, so no server of yours needs access to GitHub for
this). The agent logs it once, and the host page shows **update available**
next to the version badge, with the newer version and the command to run.

On the server, as root:

```sh
sudo watchfor-agent upgrade
```

That downloads the release from GitHub over HTTPS (only GitHub's release
hosts are accepted, redirects included), verifies the release signature
with the WatchFor key built into the agent, checks the archive against the
signed checksums, runs the new binary once to confirm it is the expected
version, swaps it in atomically and restarts the service. If any step
fails, nothing is replaced. `sudo watchfor-agent upgrade -check` only
reports. Re-running the one-line installer from **Add a host** does the
same upgrade with the same checks and keeps your token and `agent.yml`.

Servers that should keep themselves current can opt in to a daily timer:

```sh
curl --proto '=https' --tlsv1.2 -fsSL https://watchfor.io/agent/install.sh | sudo sh -s -- --auto-update
```

The timer installs a newer release only when the server has reported one,
once a day at a random moment within a six-hour window so a fleet does not
restart in lockstep. The running agent itself never replaces its own
binary; the swap is done by a separate root-run one-shot, and the agent
keeps running unprivileged.

Switch it later without the installer — the agent manages the timer and
records the choice in `agent.yml` (`updates.auto`):

```sh
sudo watchfor-agent auto-update on
sudo watchfor-agent auto-update off
watchfor-agent auto-update status
```

A sibling command changes other settings the same safe way (`sudo watchfor-agent
config set interval 30s`, `config set host.tags.env prod`, `config keys`);
it validates the file the way the agent does at start, so a bad value is
refused instead of written.

## Removing a host

**Remove** (in the host page's ⋯ menu, or the row menu in the list) revokes
the token and deletes the host's metric history, alert rules and incidents.
The agent on the server keeps running until you remove it — `sudo
watchfor-agent uninstall` takes the service, the update timer, the binary,
the state, the config with the token and the service user off the machine
(the installer's `--uninstall` does the same without the binary). Left
alone, the agent retries the rejected token with growing pauses and stops
by itself once the rejection has held for over an hour.

## Running as root

The service runs as an unprivileged user. Everything the four modules need
is readable that way on a standard kernel. Per-process I/O counters and
container sockets are not; when a module needs them, grant the capability
in a drop-in (`sudo systemctl edit watchfor-agent`):

```ini
[Service]
CapabilityBoundingSet=CAP_DAC_READ_SEARCH
AmbientCapabilities=CAP_DAC_READ_SEARCH
```

Editing the unit file itself is undone by the next upgrade. That grants
read access everywhere without running as root.

---

Canonical page: https://watchfor.io/docs/hosts · All docs: https://watchfor.io/docs · Site guide: https://watchfor.io/llms.txt