Endurance

Tools & maintenance

Updated

The Tools & Maintenance group provides the operational tools used to run, inspect and recover the appliance - running commands, managing its services, viewing the generated configuration, collecting metrics, taking backups, serving fallback pages, locking the configuration and generating a support bundle.

Terminal command

  1. Using the WebUI, navigate to Settings > Terminal command.

    execute shell command
  2. Either select a command from the Common commands dropdown or enter your own using the Command input field.

  3. Click Submit to run the command. Command output will be displayed in the area below the Command input field, together with a Copy output link that copies the output to the clipboard.

Commands run from this page are subject to a 30 second timeout, after which they are terminated. Commands that take longer than this - for example an snmpwalk over a large MIB subtree - must be run from the console or over SSH instead. The page also rejects any command containing shell metacharacters - ;, &, |, >, <, *, ?, ` , (, ), {, }, [, ], !, # and ' - reporting Command had Dangerous Characters, so pipes, redirection and wildcards cannot be used here.

System services

The key services running on the appliance are listed here, and can be manually reloaded or restarted. This is normally only required when a new configuration is added or an existing configuration is modified.

Each service is listed along with its current State (Active or Inactive). The Action column shows the action outstanding for that service, as a clickable link - for example Restart where a configuration change is pending and the service must be restarted to pick it up. A service that the configuration no longer requires shows a greyed-out Disabled that cannot be clicked - for example the fallback pages service once the last fallback page has been deleted. Services with nothing outstanding leave the column empty.

To reload or restart a service:

  1. Using the WebUI, navigate to Settings > System services.

  2. Click the three dots menu next to the service.

  3. Select the required action. Depending on the service, one or more of Update, Reload and Restart will be available.

The Restart load balancer button (top right) restarts the load balancing services; the adjacent drop-down also provides a Shutdown load balancer option. When the appliance is part of a cluster, select the Propagate across cluster checkbox before performing an action to apply it to all appliances in the cluster.

When one or more services are reporting an error, a View error log button appears alongside it. It opens an Error log window listing the messages reported by any service that is in an error state, for example:

The service is running but is not accepting connections on /run/lb-websocket/websocket.sock.
Restart it to rebind the socket.

Click Copy to copy the text to the clipboard, or Close to dismiss the window.

View configs

The View configs menu provides a convenient, read-only view of the underlying configuration of every system component and load balancing service on the appliance. It’s useful for confirming that a service has been configured as expected and for diagnosing configuration or network issues, as it shows exactly how each component has been set up. For more information on using these views when troubleshooting, see Diagnostics & Troubleshooting.

The pages under View configs are read-only. Configuration changes are made using the relevant WebUI menu (for example Network or Services); any changes made are then reflected in the corresponding View configs page.

Viewing a Configuration

  1. Using the WebUI, navigate to Settings > View configs.

  2. From the submenu, select the component or service whose configuration you’d like to view, e.g. HAProxy.

    view configuration
  3. The configuration is displayed in the panel on the right. For most components the configuration is presented as it is stored in the appliance database, on the Database tab.

  4. Where a component also generates its own configuration file - or has additional configuration or live runtime state - one or more extra tabs are displayed. Use the tabs at the top of the panel to switch between the available views. For example, the HAProxy page provides an additional Config tab that shows the generated HAProxy configuration file:

    view configuration config
  5. Click Copy output to copy the currently displayed configuration to the clipboard.

Available Configuration Views

The following configuration views are available. Where the relevant setting is configured elsewhere in the WebUI, the applicable section of this manual is referenced.

View Tabs Description

Certificates

Database

The SSL/TLS certificates held on the appliance. See SSL Certificate.

Ciphers

Database

The configured SSL cipher lists. See Cipher lists.

Conntrack

Database, Config

The connection tracking (conntrackd) configuration, used to synchronize connection state between cluster nodes. See HA Node Communication.

Fallback pages

Database

The configured fallback pages. See Fallback pages.

FIPs

Database

The configured Floating IPs. See Floating IPs.

GSLB

Database, Polaris Lb, Polaris topology, Polaris Generic State, Polaris PowerDNS State, Polaris Heartbeat

The Global Server Load Balancing (Polaris) configuration, including the generated Polaris load balancing and topology files and the live state reported by Polaris. See GSLB Services.

HAProxy

Database, Config, State

The Layer 7 configuration, including the generated HAProxy configuration file (Config tab) and the server state the Layer 7 service has saved (State tab), which lists each Real Server with the operational and administrative state, weights and health check counters recorded for it. See Layer 7 Services.

Health checks

Database

The configured health checks. See Healthchecks.

Interfaces

Database, Addresses, IPv4 Routes, IPv4 Routes (Table 100), IPv6 Routes, IPv6 Routes (Table 100), IPv4 Rules, IPv6 Rules

The network interface configuration, together with the addresses, routes and routing rules currently in force on the appliance. See Interfaces.

IP groups

Database

The configured IP groups (subnets). See Subnets.

IP Virtual Service Director

Database, Config

The Layer 4 configuration, including the generated director configuration file (Config tab). See Layer 4 Services.

LDAP

Database

The LDAP authentication configuration. See Authentication.

Licence

Database

The appliance license. See Appliance Licensing. (The WebUI spells this view Licence, although the menu option it relates to is License & Billing.)

Local Users

Database

The locally held user accounts. See Users & Groups.

Logs

Database

The log retention configuration. See Log configuration.

NF Tables

Database, Loaded Rules

The firewall (nftables) configuration, together with the ruleset currently loaded on the appliance (Loaded Rules tab).

OIDC

Database

The OpenID Connect (OIDC) single sign-on configuration.

Prometheus

Database

The Prometheus collector configuration. See Prometheus.

PulseHA

Database, Status, Loaded Config

The high availability (clustering) configuration, together with the live cluster status (Status tab) and the configuration PulseHA currently has loaded (Loaded Config tab). See High availability.

RADIUS

Database

The RADIUS authentication configuration. See Authentication.

RBAC

Database

The users, roles and groups that control access to the appliance (role-based access control). See Users & Groups.

Regions

Database

The configured regions. See Regions.

Routes

Database

The appliance routing table. See Routing.

SAML

Database

The SAML single sign-on configuration.

Service IPs

Database

The configured Service IPs. See Service IPs.

Sidecar

Database, Doctor

The configuration of the internal appliance management service (sidecar), together with a self-diagnostic report (Doctor tab) listing each check with its result, a suggested fix where it has failed, and an overall status.

SIPs

Database

The configured Static IPs. See Static IPs.

SNMP

Database

The SNMP configuration. See SNMP Configuration.

System

Database

System-level node settings, including hostname, date & time and user interface settings. See System settings.

Collectors & Integrations

The Collectors & Integrations menu configures SNMP and the Prometheus metrics collector.

SNMP

A Service IPs must be available first before configuring SNMP.

Until SNMP has been configured, the page shows only the message "Nothing configured."

To configure SNMP:

  1. Using the WebUI, navigate to Settings > Collectors & Integrations > SNMP.

  2. Click Add SNMP settings.

    configure snmp
  3. Configure the settings as required:

    Setting Description

    Select protocol version

    The SNMP version(s) the agent should answer on - V2 and/or V3. Both are unchecked by default, and at least one must be selected. Selecting V3 alone hides the Community field, since community strings do not apply to SNMPv3.

    Community

    The SNMPv2 community string. Only shown when V2 is selected.

    Location

    A free-text description of the appliance’s physical location. This is returned as sysLocation. A value is required.

    Contact

    A free-text administrative contact. This is returned as sysContact. A value is required.

    Service IP

    The Service IP the SNMP agent listens on, selected from the drop-down.

    Port

    The UDP port the SNMP agent listens on. The default is 161.

  4. Click Submit.

Once SNMP has been configured the page shows a summary of the current settings - the protocols in use, and the Community, Location and Contact values - and the button changes from Add SNMP settings to Edit SNMP settings. Click Edit SNMP settings to change any of the values above.

There is no explicit delete option. To remove the SNMP configuration entirely, click Edit SNMP settings, clear both the V2 and V3 checkboxes and click Submit - the page returns to showing "Nothing configured."

SNMPv3 Users

When V3 is selected, a Users section is displayed beneath the summary. SNMPv3 authenticates each request against a named user rather than a shared community string, so at least one user must be added before SNMPv3 can be used.

configure snmp configured

The list shows the User, Authentication algorithm, Privacy algorithm and Permission for each user. The Search box filters the list and Download CSV exports it.

To add an SNMPv3 user:

  1. On the SNMP configuration page, click Add user.

    configure snmp user
  2. Configure the user:

    Setting Description

    Username

    The SNMPv3 user name (the USM user name). This field is required.

    (unlabelled)

    The hashing algorithm used to authenticate the user - MD5 or SHA. The default is MD5; SHA is the stronger choice. This pair of radio buttons sits directly below Username and currently has no caption on screen.

    Authorization passphrase

    The authentication passphrase for the user. This field is required.

    Privacy algorithm

    The cipher used to encrypt the payload - AES or DES. The default is AES.

    Privacy passphrase

    The encryption passphrase for the user. This field is required - SNMPv3 users are always registered as authPriv, so a privacy passphrase must be supplied even if the monitoring system is only going to authenticate.

    Permission

    The access level granted to the user - Read or Read/Write. The default is Read.

  3. Click Submit.

The username and both passphrases must match the values configured in your monitoring system - see SNMPv3 for a worked example.

Prometheus

The Prometheus collector can provide both Layer 7 (HAProxy) and local node metrics.

To enable the Prometheus Collector:

A Service IPs must be available first before configuring Prometheus.
  1. Using the WebUI, navigate to Settings > Collectors & Integrations > Prometheus.

    configure prometheus
  2. Enable (check) the Enabled layer 7 exporter checkbox to provide Layer 7 metrics. An Exporter port field is revealed, defaulting to 9090 - update the port if required.

  3. Enable (check) the Enabled node exporter checkbox to provide local node metrics. This also reveals its own Exporter port field, defaulting to 9100 - update the port if required.

  4. Select the required Service IP.

  5. Click Submit.

Backup & Restore

The current appliance configuration can be downloaded as a backup at any time, and uploaded again later to restore it. In addition, an automatic backup entry is recorded each time the configuration is changed, providing a history of all additions, modifications and deletions.

Downloading a backup

To download a backup of the current configuration:

  1. Using the WebUI, navigate to Settings > Backup & Restore.

    backup restore
  2. Click Download backup.

  3. A backup containing the current configuration is generated and downloaded to your browser’s default download location. It is a single JSON file named LB_Backup_<date>_<time>.json.

The backup holds the appliance configuration only - it is not a system image, and it does not include the appliance software. It is also not encrypted, so store it somewhere appropriate to its contents.

Uploading a backup

A backup must be uploaded to the appliance before any of it can be restored.

  1. Using the WebUI, navigate to Settings > Backup & Restore.

  2. Click Upload backup.

  3. Click Choose file and select the backup. The page accepts a single .json file - the one produced by Download backup.

  4. Click Upload.

The backup is added to the Uploaded backups list at the top of the page, which records the Date & time each one was taken, in local time with the UTC equivalent beneath it. The Search box filters the list and Download CSV exports it.

Restoring a backup

The actions (⋮) menu at the end of an uploaded backup’s row offers Part restore, Full restore and Delete.

backup restore actions
Action Description

Part restore

Restores selected sections of the backup, leaving the rest of the configuration as it is.

Full restore

Restores the entire configuration. The appliance logs you out and reloads while it is applied. This option is not available while the appliance is part of a cluster - it is greyed out, with the tooltip "A full restore is not available while this appliance is part of a cluster." Remove the node from the cluster first, as described in High availability.

Delete

Removes the uploaded backup from the appliance. A confirmation dialog names the file; click Confirm. The backup file itself is unaffected and can be uploaded again.

To restore part of a backup:

  1. Using the WebUI, navigate to Settings > Backup & Restore.

  2. Click the actions (⋮) menu next to the uploaded backup and select Part restore.

    backup part restore
  3. Select the section(s) to restore. Sections that cannot be restored on their own are greyed out, as the dialog notes - "Sections that cannot be partially restored are disabled."

  4. Leave Activate floating IPs selected to bring up every Floating IP held in the backup, or clear it to restore them all in a disabled state. It applies to every Floating IP in the backup, including any that are already active, and is selected by default.

  5. Click Restore. The button stays greyed out until at least one section has been selected.

The sections that can be restored on their own are Certificates, IP Groups, FIPs, Service IPs, Conntrack, Logs, RBAC, Prometheus, Fallback Pages, HAProxy, IP Virtual Service Director and GSLB.

The sections that cannot are Interfaces, Routes, NF Tables, SNMP, PulseHA, SIPs, Licence, System, Local Users, LDAP, RADIUS, SAML and OIDC. Only a full restore can put these back.

A restore overwrites the sections selected with the contents of the backup. Anything configured in those sections since the backup was taken is lost. Take a fresh backup first if the current configuration is worth keeping.

Auto backup

Each time the configuration is changed, an entry is automatically added to the Auto backup list. Every entry shows a Description of the change, the type of Change (Added, Edited or Deleted), and the Date & time it occurred.

Use the Search box to filter the list, or click Download CSV to export the entries.

Fallback pages

A fallback page is displayed to clients when all of the Real Servers associated with a Virtual Service are unavailable (i.e. down or taken offline). Multiple fallback pages can be configured and each one can be associated with one or more Virtual Services.

A fallback page can either be served directly by the appliance (Internal) or be provided by a separate server (External).

The Fallback pages list shows each page’s Name and Type, its IP (external pages only) and Port, and its Usage.

To add a fallback page:

  1. Using the WebUI, navigate to Settings > Fallback pages.

  2. Click Add fallback page.

    add fallback page
  3. Enter an appropriate Name for the fallback page.

  4. If the page should be served over HTTPS, select the relevant Certificate.

  5. Select the Type:

    • If Internal is selected, enter the required content in the Page content editor.

    • If External is selected, specify the IP address and Port of the external fallback server.

  6. Click Add.

To modify or delete a fallback page:

  1. Using the WebUI, navigate to Settings > Fallback pages.

  2. Click the three dots menu next to the relevant fallback page.

  3. Click Edit to change its settings (update the relevant setting(s), then click Update), or click Delete then Confirm to remove it.

System lock

The appliance can be locked to prevent configuration changes from being made, for example during planned maintenance or while updates are being applied. When locked, the configuration is protected and no changes can be made until the appliance is unlocked.

The lock state is also shown in the top bar of every page as Unlocked or Locked; clicking the indicator opens this page. See The Top Bar.

To lock the appliance:

  1. Using the WebUI, navigate to Settings > System lock.

  2. Select the System is locked checkbox.

  3. Select a Reason for lock: Under maintenance, Locked for updates or Custom message. If Custom message is selected, enter the message to be displayed (maximum 160 characters).

  4. Click Submit.

To unlock the appliance:

  1. Using the WebUI, navigate to Settings > System lock.

  2. Clear the System is locked checkbox.

  3. Click Submit.

Support download

The Support download page generates a diagnostics bundle that can be downloaded and sent to Loadbalancer.org support to assist with troubleshooting.

  1. Using the WebUI, navigate to Settings > Support download.

    The page reports the Recommended download size that has been selected for you, together with the current disk and memory usage, and the appliance’s Release version, Client version and API version.

  2. Click Download to generate and download the support bundle.