Linux monitoring

The XorMon NVIDIA agent (`xormon-agent-nvidia`) is the standard XorMon OS agent with NVIDIA GPU monitoring (NVML) compiled in.
It installs into the same location (`/opt/xormon-agent`) and uses the same `xormon-agent` service as the regular agent — it simply adds GPU metrics.

It is a replacement for the agent of the regular `xormon-agent`, not alongside it.
On a host with no NVIDIA driver the agent still runs normally and just reports no GPUs.

It is available for amd64 / x86_64 only.

Which download should I use?

    Three formats are published. Pick one:
    Format File Use when
    .deb xormon-agent-nvidia-<version>-amd64.deb Debian / Ubuntu
    .rpm xormon-agent-nvidia-<version>-x86_64.rpm RHEL / SUSE / Fedora
    Source bundle xormon-agent-nvidia-<version>-src-amd64.tar.gz Older systems, or when a package fails to start (see below)

Why is there a source package too?

    The prebuilt `.deb` and `.rpm` are the easy choice and work on most current systems — use them first.

    GPU support relies on the NVIDIA library, which means the prebuilt binary is linked against system libraries and therefore needs a reasonably up-to-date operating system.
    On an older OS the package may refuse to start with a library-version error.

    For exactly those cases there is the source bundle: it compiles the agent directly on your machine, so the result always matches your system.
    Everything needed to build is already inside the bundle (no internet, no extra tools) — the only requirement is a C compiler (`gcc`), which it can install for you.

    Most users will never need the source bundle — reach for it only if a prebuilt package won't start, or if you simply prefer a build guaranteed to fit your host.

Requirements

    • Linux with systemd, amd64 / x86_64
    • Root privileges for installation
    • For the source bundle only: a C compiler (`gcc`). The installer installs it automatically on an online host; on an offline host install it first (`apt-get install gcc` or `dnf install gcc`).

    Behaviour of all installers can be controlled with environment variables, set before the command (applied on a fresh install only):
    Variable Default Description
    TARGET_IP localhost XorMon server IP or hostname
    RUN_AS_USER xormon User the agent service runs as

Debian, Ubuntu from .deb

    If the plain `xormon-agent` is already installed, remove it first so the files don't clash: 
      
dpkg -r xormon-agent 2>/dev/null || true
    Then install the NVIDIA package: 
      
# Basic install (creates the xormon user, detects UUID, starts the service)
TARGET_IP=192.168.1.100 dpkg -i xormon-agent-nvidia-<version>-amd64.deb

# Run under an existing user instead of creating "xormon"
RUN_AS_USER=myuser TARGET_IP=192.168.1.100 dpkg -i xormon-agent-nvidia-<version>-amd64.deb
    You can change the server address at any time by editing `/opt/xormon-agent/agent.yaml` and restarting the service.

RHEL from .rpm

    The `.rpm` automatically replaces an existing plain `xormon-agent`. 
      
# Basic install
TARGET_IP=192.168.1.100 rpm -i --nodeps xormon-agent-nvidia-<version>-x86_64.rpm

# Under an existing user
RUN_AS_USER=myuser TARGET_IP=192.168.1.100 rpm -i --nodeps xormon-agent-nvidia-<version>-x86_64.rpm

Manuall from tar (any distro / architecture)

    Use this if a prebuilt package won't start, or you want a build guaranteed to match your system. The bundle compiles the agent on the machine; the Go toolchain, all dependencies, and generated code are already inside it.
    1. Extract the bundle: 
        
   tar xzf xormon-agent-nvidia-<version>-src-amd64.tar.gz
   cd xormon-agent-nvidia-<version>-src-amd64
    2. Run the installer as root 
        
   # Basic install
   sudo ./install.sh

   # With custom server IP and/or an existing user (variables BEFORE the command)
   sudo TARGET_IP=192.168.1.100 RUN_AS_USER=myuser ./install.sh
      The installer compiles the agent against your system, installs it to `/opt/xormon-agent`, creates the user, detects the machine UUID, and starts the `xormon-agent` service.
      On an offline host, install a C compiler first (e.g. `dnf install gcc` or `apt-get install gcc`). Everything else needed to build is in the bundle.

Verify GPU monitoring

      
systemctl status xormon-agent
ldconfig -p | grep libnvidia-ml      # confirms the NVIDIA driver / NVML is present
tail -f /var/log/xormon-agent.log
    If the NVIDIA driver isn't installed the agent still runs — it just won't report any GPUs.

Testing the connection (debug mode)

    To verify a fresh install can reach the XorMon server before relying on the background service, run the agent once with -d (or --debug).
    This mirrors the old lpar2rrd agent's -d option: it tests the connection to each server in agent.yaml, then collects one minute of metric samples and sends them immediately, printing the result — instead of running continuously.
    cd /opt/xormon-agent
./agent -d
    What it does, in order:
    1. Connects to every server listed in agent.yaml. If none are reachable it prints the failure and exits with a non-zero status — nothing is collected.
    2. On success, collects one minute of samples (the CPU collector samples over a full minute, so this step takes ~60 seconds).
    3. Sends the collected metrics, architecture, and configuration to the server(s) right away and exits.

    Example output: 
    [DEBUG] Debug mode enabled - will test connection and send data immediately
Testing connection to server(s)...
  - 192.168.1.100:8162: connected
Connection OK. Collecting one minute of metric samples...
Collected 12 metric entities and 3 architectures. Sending to server(s)...
Debug run complete.
    The agent runs in the foreground and the service does not need to be stopped — the debug run is a separate one-shot process.
    Debug mode also forces verbose logging to stdout so any send errors are shown.

Rest like agent Upgrade and Uninstall is same as for the agent without NVIDIA support.
The XorMon OS agent is a new agent that replaces the older LPAR2RRD one.
It has the same functionality, but is optimised to handle much lower loads on monitored hosts, and to handle data much more effectively on the XorMon server side.
This is a significant advantage in environments with thousands of OS agents installed.
This agent will be further enhanced to monitor selected processes and more.
It is written in "GO" and compiled for each hardware platform. The source code is also available.
Initial release v1.0.0 comes with XorMon v2.3.0.

Requirements

  • Linux, systemd
  • Architecture: x86_64, amd64, arm64, or ppc64le
  • Root privileges for installation

Debian, Ubuntu from .deb

    The .deb package handles everything automatically: configures the agent, detects the machine UUID, and starts the service.

    Behaviour can be controlled via environment variables set before the dpkg call:
    Variable Default Description
    TARGET_IP localhost XorMon server IP or hostname
    RUN_AS_USER xormon User the agent service runs as

    Basic install:
    TARGET_IP=192.168.1.100 dpkg -i xormon-agent-<version>-amd64.deb
    Creates a xormon system user automatically and starts the service.
    Install under an existing user (skip xormon user creation)
    RUN_AS_USER=myuser dpkg -i xormon-agent-<version>-amd64.deb
    The service will run as myuser. No new system user is created. Both variables can be combined:
    TARGET_IP=192.168.1.100 RUN_AS_USER=myuser dpkg -i xormon-agent-<version>-amd64.deb
    You can change the server address at any time by editing /opt/xormon-agent/agent.yaml and restarting the service.


RHEL from .rpm (x86_64 / amd64 / arm64 / ppc64le)

    The .rpm package works the same way as the .deb — it handles user creation, UUID detection, and service startup automatically.
    Variable Default Description
    TARGET_IP localhost XorMon server IP or hostname
    RUN_AS_USER xormon User the agent service runs as 
    # Basic install
rpm -i --nodeps xormon-agent-<version>-x86_64.rpm

# With custom server IP
TARGET_IP=192.168.1.100 rpm -i --nodeps xormon-agent-<version>-x86_64.rpm

# Under an existing user
RUN_AS_USER=myuser TARGET_IP=192.168.1.100 rpm -i --nodeps xormon-agent-<version>-x86_64.rpm
    Available packages: x86_64 (amd64), aarch64 (arm64), ppc64le.


Manuall from tar (any distro / architecture)

Use this method on RPM-based distributions (RHEL, SUSE, …) or non-amd64 architectures.
  1. Extract files
    su -
mkdir -p /opt/xormon-agent
tar xzf xormon-agent-<version>-<arch>.tar.gz -C /opt/xormon-agent
chmod 755 /opt/xormon-agent/xormon-agent /opt/xormon-agent/getsysuuid.sh
chmod 644 /opt/xormon-agent/agent.yaml
  2. Configure the agent
    Open /opt/xormon-agent/agent.yaml and set the XorMon server address: 
    servers:
  - host: "192.168.1.100"   # ← your XorMon server IP or hostname
    port: 8162
    tls: true
  3. Create system user
    Optionally names "xormon", but you can use your own user name: 
    useradd --system --no-create-home --shell /sbin/nologin xormon
chown -R xormon:xormon /opt/xormon-agent
  4. Detect machine UUID
    The UUID uniquely identifies this host in XorMon. Run as root: 
    /opt/xormon-agent/getsysuuid.sh
    The script reads the UUID from /sys/class/dmi/id/product_uuid, dmidecode, or /etc/machine-id (fallback) and writes it into agent.yaml automatically.

  5. Create systemd service
    Create /etc/systemd/system/xormon-agent.service (file), change the username if you did not use "xormon" above: 
    [Unit]
Description=Xormon Agent
After=network.target

[Service]
Type=simple
ExecStart=/opt/xormon-agent/xormon-agent
StandardOutput=file:/var/log/xormon-agent.log
StandardError=file:/var/log/xormon-agent.err
WorkingDirectory=/opt/xormon-agent
User=xormon
Group=xormon
Restart=always
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectHome=true
ReadWritePaths=/opt/xormon-agent

[Install]
WantedBy=multi-user.target
  6. Enable logrotate
    create this file /etc/logrotate.d/xormon-agent with this content: /var/log/xormon-agent.log /var/log/xormon-agent.err { 
        daily
    missingok
    rotate 7
    compress
    delaycompress
    notifempty
    copytruncate
}
  7. Enable and start
    systemctl daemon-reload
systemctl enable xormon-agent.service
systemctl start xormon-agent.service

Troubleshooting

  • Service management
    systemctl status xormon-agent     # current status
systemctl stop xormon-agent       # stop
systemctl restart xormon-agent    # restart after config change
  • Logs
    /var/log/xormon-agent.log         # Standard output 
/var/log/xormon-agent.err         # Errors

tail -f /var/log/xormon-agent.log
journalctl -u xormon-agent -f
    Logs rotate daily, keeping 7 days (configured via /etc/logrotate.d/xormon-agent).

Testing the connection (debug mode)

    To verify a fresh install can reach the XorMon server before relying on the background service, run the agent once with -d (or --debug).
    This mirrors the old lpar2rrd agent's -d option: it tests the connection to each server in agent.yaml, then collects one minute of metric samples and sends them immediately, printing the result — instead of running continuously.
    cd /opt/xormon-agent
./agent -d
    What it does, in order:
    1. Connects to every server listed in agent.yaml. If none are reachable it prints the failure and exits with a non-zero status — nothing is collected.
    2. On success, collects one minute of samples (the CPU collector samples over a full minute, so this step takes ~60 seconds).
    3. Sends the collected metrics, architecture, and configuration to the server(s) right away and exits.

    Example output: 
    [DEBUG] Debug mode enabled - will test connection and send data immediately
Testing connection to server(s)...
  - 192.168.1.100:8162: connected
Connection OK. Collecting one minute of metric samples...
Collected 12 metric entities and 3 architectures. Sending to server(s)...
Debug run complete.
    The agent runs in the foreground and the service does not need to be stopped — the debug run is a separate one-shot process.
    Debug mode also forces verbose logging to stdout so any send errors are shown.

Upgrade

  • Package upgrade (.deb / .rpm)

    Upgrades are handled automatically — just install the newer package over the existing one.
    Your agent.yaml is preserved: the previous configuration is backed up and merged on top of the new version's defaults (so config keys added in the new release are picked up automatically), the service user is retained, and the service is restarted. 
    # RPM (RHEL / SUSE)
rpm -U --nodeps xormon-agent-<new-version>-x86_64.rpm

# DEB (Debian / Ubuntu)
dpkg -i xormon-agent-<new-version>-amd64.deb
    TARGET_IP and RUN_AS_USER are applied only on a fresh install — on upgrade the existing server address and service user are kept.

  • Manual upgrade (tar install)
    The tarball ships a fresh agent.yaml, so a manual upgrade replaces only the binary and helper script and leaves your existing configuration in place. 
    # 1. Stop the service
systemctl stop xormon-agent

# 2. Back up the current config (safety)
cp -p /opt/xormon-agent/agent.yaml /opt/xormon-agent/agent.yaml.bak

# 3. Extract the new release to a temporary directory
mkdir -p /tmp/xormon-agent-new
tar xzf xormon-agent-<new-version>-<arch>.tar.gz -C /tmp/xormon-agent-new

# 4. Replace the binary and helper script (your agent.yaml is left untouched)
install -m 755 /tmp/xormon-agent-new/xormon-agent  /opt/xormon-agent/xormon-agent
install -m 755 /tmp/xormon-agent-new/getsysuuid.sh /opt/xormon-agent/getsysuuid.sh

# 5. Bump the version field in your config to match the new release
NEW_VER=$(grep -E '^[[:space:]]*version:' /tmp/xormon-agent-new/agent.yaml | awk -F'"' '{print $2}')
sed -i "s/^\([[:space:]]*version:\).*/\1 \"$NEW_VER\"/" /opt/xormon-agent/agent.yaml

# 6. Restore ownership and restart
chown -R xormon:xormon /opt/xormon-agent
systemctl restart xormon-agent

# 7. Clean up
rm -rf /tmp/xormon-agent-new
    New config options: keys added in a new release won't appear in your existing agent.yaml; the agent falls back to built-in defaults for any missing key. To adopt new options, compare before cleaning up in step 7: 
        diff /opt/xormon-agent/agent.yaml /tmp/xormon-agent-new/agent.yaml
    and copy over the sections you want.

  • Verify the upgrade:
    systemctl status xormon-agent
tail -f /var/log/xormon-agent.log

Uninstall

  • .rpm, .deb
    # RPM (RHEL)
rpm -e xormon-agent

# DEB (Debian / Ubuntu)
dpkg -r xormon-agent
  • Manual:
    systemctl stop xormon-agent
systemctl disable xormon-agent
rm -f /etc/systemd/system/xormon-agent.service
rm -rf /opt/xormon-agent
userdel xormon
systemctl daemon-reload
OS agent (LPAR2RRD agent) is add-on feature for monitoring from operating system level.
It is monitoring CPU, memory usage , paging, LAN and SAN traffic on all adapters.
It requires the OS agent deployment to every monitored LPAR.
The agent is written in Perl and calls basic OS commands to obtain required statistics like vmstat, lparstat and svmon.
You can even use already installed LPAR2RRD agents and direct them to XorMon host.

Operating systems

  • Linux on Power
  • Linux x86

Implementation

it is implemented as a simple client/server application.
There is XorMon daemon listening on the host where XorMon server is running on port 8162.
Each LPAR has a simple Perl-based agent installed. This agent is started every minute from the crontab and saves memory and paging statistics into a temporary file.
The agent contacts the server randomly every 10-20 minutes and sends all locally stored data for that period.

Usage

perl ./lpar2rrd-agent.pl [-d] [-x] <XorMon server hostname/IP>[:<PORT>]

 -d  forces sending out data immediately to check communication channel (DEBUG purposes)
 -m  using sudo for multipath (only root can run it): sudo multipath -l", put this into sudoers: xormon  ALL = (root) NOPASSWD: /usr/sbin/multipath -ll
 -x  use Transport Layer Security (TLS)

 no option - agent collects & sends standard OS agent data
Crontab entry for scheduling, use non-admin account preferably 
* * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <XorMon server hostname/IP> > /var/tmp/lpar2rrd-agent.out 2>&1
The agent collects data and sends them every 5 - 20 minutes to the XorMon server
If you use other than standard XorMon port, then add it after SERVER, separated by the ':' delimiter
* * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <XorMon server hostname/IP>:<PORT> > /var/tmp/lpar2rrd-agent.out 2>&1
If you want to send data to more XorMon server instances (number is not restricted) 
* * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <XorMon server 1 hostname/IP> <XorMon server 2 hostname/IP> > /var/tmp/lpar2rrd-agent.out 2>&1

Prerequisites

  • Perl on each VM (each OS already contains it, no special modules are necessary)

  • Opened TCP communication between each VM / LPAR and XorMon server on port 8162.
    Connections are initiated from VMs / LPARs side.

  • Create preferable dedicated user xormon on each host with minimum rights 
    # useradd -c "XorMon agent user" -m xormon

OS agent installation (client)

  • Get the latest OS agent from download page

  • Linux RedHat, Rocky
    # rpm -Uvh lpar2rrd-agent-7.60-0.noarch.rpm
# rpm -qa|grep lpar2rrd-agent
  lpar2rrd-agent-7.60-0
  • Linux Debian 
    # apt-get install ./lpar2rrd-agent_7.60-0_all.deb
  lpar2rrd-agent-7.60-0
  • Schedule its run every minute from the crontab on every LPAR.
    This line must be placed into xormon crontab: 
    # su - xormon  
$ crontab -e
* * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <XorMon_SERVER.your-domain.com> > /var/tmp/lpar2rrd-agent.out 2>&1
    Replace <XorMon_SERVER> by hostname of your XorMon server.
    Use preferably FDQN in XorMon hostname, hostname only might have a problem with resolving.

    In case you want to direct the agent data to more servers (or ports) add second or more hosts on the cmd line
    Bellow cfg will collect data once and sends it to 3 Hosts (Host1 port 8162, Host2 port 8162 and Host3 port 7162)
    * * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <Host1>  <Host2>  <Host3>:7162   > /var/tmp/lpar2rrd-agent.out 2>&1
  • You might need to add xormonuser into or /etc/cron.allow (Linux) under root user if above "crontab -e" fails. 
    # echo "xormon" >> /etc/cron.allow

Troubleshooting

Client (agent) side:
  • Test if communication through the LAN is allowed. 
    $ telnet  <XorMon_SERVER> 8162
  Connected to 192.168.1.1   .
  Escape character is '^]'.
    This is ok, exit either Ctrl-C or ^].

  • Check following agent files:
    data store: /var/tmp/lpar2rrd-agent-*.txt
    error log: /var/tmp/lpar2rrd-agent-*.err
    output log: /var/tmp/lpar2rrd-agent.out

  • run the agent from cmd line: 
    $ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <XorMon_SERVER.your-domain.com>
  ...
  Agent send     : yes : forced by -d
  Agent send slp: sending wait: 4
  OS/HMC agent working for server: <XorMon_SERVER>
  store file for sending is /var/tmp/lpar2rrd-agent-<XorMon_SERVER.your-domain.com>-lpar2rrd.txt
    It means that data has been sent to the server, all is fine
    Here is example when the agent is not able to sent data : 
    $ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <XorMon_SERVER.your-domain.com>
  ...
  Agent send     : yes : forced by -d
  Agent send slp: sending wait: 1
  OS/HMC agent working for server: <XorMon_SERVER>
  store file for sending is /var/tmp/lpar2rrd-agent-<XorMon_SERVER>-lpar2rrd.txt
  Agent timed out after : 50 seconds /opt/lpar2rrd-agent/lpar2rrd-agent.pl:265
    It means that the agent could not contact the server.
    Check communication, port, above telnet example, DNS resolution of the server etc.