Endurance
Get started

Monitoring & reporting

This chapter covers the ways in which the appliance and the services it load balances can be monitored from outside the WebUI, and the health information the WebUI itself presents. It describes the SNMP agent and the MIBs it exposes, the Prometheus metric collectors used to feed an external monitoring stack, and the resource utilization graphs shown on the Dashboard.

SNMP

The appliance runs a Net-SNMP agent that answers SNMP v2c and SNMP v3 requests. It is disabled by default - until it has been configured, the SNMP page shows only the message "Nothing configured."

SNMP is enabled and configured from Settings > Collectors & Integrations > SNMP, where the protocol version(s), community string, location, contact, Service IP and port are set. SNMP v3 additionally requires at least one user to be defined. For a full description of every option, see SNMP Configuration and SNMPv3 Users.

Several characteristics of the agent are worth noting before pointing a monitoring system at it:

  • The agent listens on the Service IP selected on the SNMP page and on the loopback address only - not on every address the appliance holds. The Service IP chosen must therefore be one that the monitoring system can reach. For more information on Service IPs, see Service IPs.

  • The Location and Contact values configured in the WebUI are returned as the standard sysLocation and sysContact objects.

  • The default port is 161 (UDP).

  • SNMP is a monitoring interface only. The SNMPv2 community is granted read-only access, and although an SNMPv3 user can be given Read/Write permission (see SNMPv3 Users), the appliance’s load balancing configuration cannot be changed over SNMP - use the WebUI or the API for that.

MIB Files

A Management Information Base file (MIB) tells the monitoring system how to interpret the values the agent returns. All of the MIB files used by the appliance are held in /usr/share/snmp/mibs, and the loadbalancer.org MIBs can also be downloaded directly from the links below - the published copies are identical to the ones shipped on the appliance.

Standard MIBs - the appliance includes the usual Linux MIB set, so network statistics, CPU load, memory and disk usage are all read in the normal way. The most commonly used are HOST-RESOURCES-MIB, IF-MIB, UCD-SNMP-MIB, IP-MIB, TCP-MIB, UDP-MIB and SNMPv2-MIB.

loadbalancer.org MIBs - three custom MIBs are provided:

MIB Purpose

LBO-MIB.txt

Defines the loadbalancer.org enterprise root OID, 1.3.6.1.4.1.54849, under which the two MIBs below are registered. It carries no objects of its own, but it must be loaded for the other two to resolve.

L7INFO-EXPERIMENTAL-MIB.txt

Process-level information about the Layer 7 service (HAProxy) - version, uptime, connection and session rates, SSL/TLS counters, memory pools and so on. This is the SNMP equivalent of the HAProxy show info output.

L7STAT-EXPERIMENTAL-MIB.txt

Defines the per-proxy Layer 7 statistics tables, together with three scalar objects reporting the state of the statistics poller itself.
Layer 4 services are not exposed through SNMP - there is no LVS or Layer 4 MIB on the appliance. To monitor the state of Layer 4 Virtual Services and their Real Servers, use the Dashboard (see Using the Dashboard) or the API (see Application Programming Interface (API)).

Running the Examples

The examples in the sections that follow can be run at the appliance console, over SSH, or from the WebUI using Settings > Terminal command. They use SNMP v2c with a community string of public against the local appliance - adjust the version, community string and target address to match your own configuration and to query the appliance remotely.

Commands run from the Terminal command page are terminated after 30 seconds. An snmpwalk over a large subtree can easily exceed this, so run the broader examples from the console or over SSH. The page also rejects any command containing shell metacharacters - including ;, &, |, <, >, * and ? - so pipes, redirection and wildcards cannot be used there.
The standard MIBs are loaded automatically, so the system statistics examples below resolve object names with no extra options. The appliance does not ship an /etc/snmp/snmp.conf, however, so the loadbalancer.org MIBs are not loaded automatically: querying them by name fails with Unknown Object Identifier, and querying them by numeric OID returns the unresolved form, for example SNMPv2-SMI::enterprises.54849.1.1.1.0. Add -m ALL to load every installed MIB - as the Layer 7 examples below do - or load a single MIB with -m +<MIB-NAME>, for example -m +L7INFO-EXPERIMENTAL-MIB.

Monitoring System Statistics Using SNMP

General system statistics - CPU load, memory usage, disk usage and network throughput - are read using the standard MIBs, and need no -m option.

To display the system group (appliance name, uptime, contact and location):

snmpwalk -v 2c -c public localhost system

SNMPv2-MIB::sysDescr.0 = STRING: Linux lb1 6.12.0-211.54.1.el10_2.x86_64 #1 SMP PREEMPT_DYNAMIC
Thu Sep 10 19:25:03 UTC 2026 x86_64
SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (2363) 0:00:23.63
SNMPv2-MIB::sysContact.0 = STRING: admin
SNMPv2-MIB::sysName.0 = STRING: lb1
SNMPv2-MIB::sysLocation.0 = STRING: Datacenter
...

The sysName value is the appliance hostname, while sysContact and sysLocation are the Contact and Location configured on the SNMP page.

Per-interface network statistics are available under the IF-MIB:

snmpwalk -v 2c -c public localhost IF-MIB::ifTable

Memory, CPU and disk statistics are available under the UCD-SNMP-MIB and HOST-RESOURCES-MIB, for example:

snmpwalk -v 2c -c public localhost HOST-RESOURCES-MIB::hrStorageTable
snmpwalk -v 2c -c public localhost UCD-SNMP-MIB::laTable

The laTable returns the 1, 5 and 15 minute load averages as Load-1, Load-5 and Load-15.

Monitoring Layer 7 Services Using SNMP

Layer 7 information is published under the loadbalancer.org enterprise root OID 1.3.6.1.4.1.54849, which is divided into two groups:

Group OID Contents

l7Info

1.3.6.1.4.1.54849.1.1

Process-level information about the running Layer 7 service.

l7Stat

1.3.6.1.4.1.54849.1.2

The state of the statistics poller, and the per-proxy statistics tables.

The examples below use -m ALL so that object names are resolved rather than returned as numeric OIDs (see the note above).

Layer 7 process information

The l7Info group returns 57 objects describing the running Layer 7 service, equivalent to the HAProxy show info output:

snmpwalk -v 2c -c public -m ALL localhost l7Info

L7INFO-EXPERIMENTAL-MIB::l7InfoLastPolled.0 = Timeticks: (4156506864) 481 days, 1:51:08.64
L7INFO-EXPERIMENTAL-MIB::l7InfoUpdateInterval.0 = INTEGER: 4000
L7INFO-EXPERIMENTAL-MIB::l7InfoLastError.0 = Gauge32: 5
L7INFO-EXPERIMENTAL-MIB::l7InfoName.0 = STRING: HAProxy
L7INFO-EXPERIMENTAL-MIB::l7InfoVersion.0 = STRING: 3.0.5-8e879a5
L7INFO-EXPERIMENTAL-MIB::l7InfoReleaseDate.0 = STRING: 2024/09/19
L7INFO-EXPERIMENTAL-MIB::l7InfoUptime.0 = STRING: 0d 0h00m34s
L7INFO-EXPERIMENTAL-MIB::l7InfoUptimeSec.0 = Counter32: 34
L7INFO-EXPERIMENTAL-MIB::l7InfoNode.0 = STRING: lb1
L7INFO-EXPERIMENTAL-MIB::l7InfoPid.0 = INTEGER: 2916634
L7INFO-EXPERIMENTAL-MIB::l7InfoNBThread.0 = Gauge32: 3
...

The first three objects describe the poller rather than the Layer 7 service itself: l7InfoLastPolled is the time the service was last polled, l7InfoUpdateInterval is the interval between polls (4000 ms by default) and l7InfoLastError is the last error encountered when polling.

The objects that describe the service itself fall into the groups below, and are fully documented in the MIB file:

Area Objects

Service identity

l7InfoName, l7InfoVersion, l7InfoReleaseDate, l7InfoNode, l7InfoPid, l7InfoUptime, l7InfoUptimeSec, l7InfoStopping

Capacity and limits

l7InfoMaxConn, l7InfoHardMaxconn, l7InfoUlimitN, l7InfoMaxSock, l7InfoMaxPipes, l7InfoListeners, l7InfoNBThread, l7InfoNBProc, l7InfoProcessNum

Connections and sessions

l7InfoCurrConns, l7InfoCumConns, l7InfoCumReq, l7InfoConnRate, l7InfoConnRateLimit, l7InfoMaxConnRate, l7InfoSessRate, l7InfoSessRateLimit, l7InfoMaxSessRate

SSL/TLS

l7InfoCurrSSLConns, l7InfoCumSSLConns, l7InfoMaxSSLConns, l7InfoSSLRate, l7InfoSSLRateLimit, l7InfoMaxSSLRate, l7InfoSSLCacheLookups, l7InfoSSLCacheMisses, l7InfoSSLFrontendKeyRate, l7InfoSSLFrontendMaxKeyRate, l7InfoSSLFrontendSessionReusePct, l7InfoSSLBackendKeyRate, l7InfoSSLBackendMaxKeyRate

Compression

l7InfoCompressBPSIn, l7InfoCompressBPSOut, l7InfoCompressBPSRateLim, l7InfoZlibMemUsage, l7InfoMaxZlibMemUsage

Resource usage

l7InfoIdlePct, l7InfoTasks, l7InfoRunQueue, l7InfoJobs, l7InfoMemMaxMB, l7InfoPoolAllocMB, l7InfoPoolUsedMB, l7InfoPoolFailed, l7InfoPipesUsed, l7InfoPipesFree

Individual objects are polled with snmpget, which is how a monitoring system would normally read them. Note that each object is a scalar, so the instance suffix .0 is required:

snmpget -v 2c -c public -m ALL localhost l7InfoCurrConns.0 l7InfoIdlePct.0

Layer 7 poller status

The l7Stat group exposes three scalar objects describing the statistics poller:

snmpwalk -v 2c -c public -m ALL localhost l7Stat

L7STAT-EXPERIMENTAL-MIB::l7StatLastPolled.0 = Timeticks: (4156515864) 481 days, 1:52:38.64
L7STAT-EXPERIMENTAL-MIB::l7StatUpdateInterval.0 = Gauge32: 4000
L7STAT-EXPERIMENTAL-MIB::l7StatLastError.0 = Gauge32: 7
The L7STAT-EXPERIMENTAL-MIB also defines per-proxy statistics tables - l7ProxyTable, l7FrontendTable, l7BackendTable and l7ServerTable - but the agent does not implement them: walking any of them returns No Such Object available on this agent at this OID, and l7StatLastError reports a non-zero value as shown above. Per-VIP and per-Real Server statistics are therefore not available over SNMP. Use the Dashboard (see Using the Dashboard), the service pages (see Viewing a Service), the Prometheus Layer 7 exporter (see Metric Collectors) or the API to monitor individual Virtual Services and Real Servers.

SNMPv3

Every SNMPv3 user created on the appliance is registered with the agent as an authPriv user, so each request must carry both the authentication and the privacy options - there is no authentication-only mode. The same queries shown above are made as follows:

snmpwalk -v 3 -u <USM-USERNAME> -l authPriv -a <AUTH-ALGORITHM> -A <AUTH-PASSPHRASE> \
  -x AES -X <PRIVACY-PASSPHRASE> -m ALL localhost l7Info

The user name and both passphrases must match a user defined on the SNMP page - see SNMPv3 Users. The authentication algorithm passed to -a must match the Authentication algorithm set for that user, MD5 or SHA.

Always use -x AES, whatever the user’s Privacy algorithm setting says. The Net-SNMP client shipped with the appliance does not support DES - passing -x DES is rejected outright with Invalid privacy protocol specified after -3x flag: DES - and a user created with a Privacy algorithm of DES is still queried successfully with -x AES.

Metric Collectors

As an alternative - or in addition - to SNMP, the appliance can expose its metrics in Prometheus format for collection by an external monitoring stack. Two independent exporters are provided and each can be enabled on its own:

Exporter Default port Metrics provided

Layer 7 exporter

9090

HAProxy’s built-in Prometheus exporter, publishing the frontend, backend and server statistics that the Layer 7 service collects.

Node exporter

9100

Local node metrics for the appliance itself - CPU, memory, disk, filesystem and network statistics, published as the standard node_* metric set.

Both exporters are disabled by default and, as with SNMP, are published on a Service IP that must exist before they can be enabled. They are configured from Settings > Collectors & Integrations > Prometheus - see Collectors & Integrations for the full procedure.

Once an exporter is enabled, the Prometheus server is pointed at the /metrics path on the selected Service IP and the exporter’s port, over plain HTTP - for example http://10.11.40.70:9100/metrics for the node exporter. The node exporter also serves a small landing page at / that links to /metrics and to a pair of /debug/pprof/ profiling endpoints; any other path returns 404 page not found. The Layer 7 exporter serves /metrics only - its root path returns a 503.

The Layer 7 exporter is provided by the Layer 7 service itself rather than by a separate daemon, so it is only reachable while that service is running - that is, while at least one Layer 7 Virtual Service is configured. On an appliance with no Layer 7 services, nothing listens on the Layer 7 exporter port even though the exporter is enabled. The node exporter is independent of this and is always available once enabled.
Where SNMP does not provide the required detail for Layer 7 services, the Layer 7 exporter is the alternative route for an external monitoring system.

Resource Utilization Graphs

For an at-a-glance view of appliance health without an external monitoring system, the WebUI Dashboard presents a set of live graph widgets:

  • Network Graph (kbs) - network throughput, showing the transmit (tx) and receive (rx) rate for each interface.

  • System Load Average (%) - the system load average over time.

  • System Memory Usage (Mb) - memory usage over time.

These sit in the lower portion of the Dashboard, alongside the CPU, Memory, Disks and Connections gauges at the top of the page. For a description of every widget, see Dashboard Widgets.

Clicking the eye icon in the top-right corner of any of these widgets opens the Dashboard details view, where a larger, historical graph can be displayed for a selected metric over a period of 1 Hour, 1 Day, 1 Week, 1 Month or 1 Year. The metric is pre-selected to match the widget whose icon was clicked. For more information, see Viewing Detailed Metrics.