High Availability
High Availability Configuration
Loadbalancer.org recommends deploying multiple nodes in a cluster to provide a highly available, resilient load balancing solution and avoid a single point of failure. This section explains the clustering concepts and describes how to create and manage an HA cluster.
| A single appliance can be deployed, but this introduces a single point of failure. We always recommend deploying at least two appliances in a cluster. |
Clustering Concepts
A cluster operates in an Active-Passive manner and can contain up to four nodes — one active node and up to three passive nodes:
-
Under normal conditions the active node handles all traffic.
-
Each node runs PulseHA, which continually health checks the other nodes in the cluster. If the active node fails, one of the passive nodes automatically becomes active and takes over the floating IP addresses and traffic. This is known as fail-over.
Once the cluster has been formed, load balanced services and most other settings can be configured or modified on any node — the other nodes are automatically kept in sync. Configuration is replicated between the nodes over the network using SSH/SCP.
Clustering can be configured at any time. Clustering is always initiated from the appliance that is to become the active node. When a passive node is added to the cluster, the active node’s configuration is replicated to it, which erases any existing configuration on the passive node.
Non-Replicated Settings & Actions
Most configuration is automatically replicated between the nodes. The settings and actions listed below are not replicated and must be configured or performed on each node individually:
| WebUI Main Menu Option | Sub Menu Option | Description |
|---|---|---|
Local Configuration |
All sub menu options |
Date & time settings, WebUI settings, various physical advanced settings, SNMP, licensing, Prometheus collector setting and ADC Portal related settings |
Network |
Interfaces |
Network interface settings |
Network |
IP Groups (Subnets) |
Network subnet settings |
Network |
Static IP |
Static IPs |
Maintenance |
Backup & Restore |
Creation of backups |
Maintenance |
Software Update |
Appliance software updates |
| Make sure that the settings and actions listed above are configured / performed on each node. |
HA Node Communication
The nodes in a cluster communicate using PulseHA. By default, PulseHA uses unicast (ucast) on UDP port 6694 to exchange health checks between the nodes in the cluster.
For PulseHA to function correctly, all nodes must be able to reach each other as follows:
| Protocol / Port | Purpose |
|---|---|
ICMP (echo request / ping) |
Node reachability checks |
TCP 22 (SSH) |
Configuration replication (SSH/SCP) |
UDP 6694 |
PulseHA health check / heartbeat traffic |
TCP 9443 |
Inter-node API communication |
| If a firewall is deployed between the two appliances, ensure that the traffic listed above is permitted in both directions. |
Configuring HA
HA clustering is configured from the appliance that will become the active node. Adding a passive node replicates the active node’s configuration to it and erases the passive node’s existing configuration.
-
Deploy a second appliance and configure its network settings as described in Network Configuration. The second appliance must be reachable from the first over the network (see HA Node Communication).
-
Using the WebUI on the appliance that is to be the active node, navigate to Cluster Configuration > High availability configuration.
-
Click Add node.
-
Complete the Add Node form with the details of the peer — the appliance being added as a passive node:
Field Description Peer IP address
The IP address of the node being added to the cluster.
Peer port
The port that the peer’s WebUI/API is listening on (default 443).
Administrator email
The administrator username / email address of the peer.
Administrator password
The administrator password of the peer.
-
Click Confirm. A confirmation dialog is displayed, warning that the process will erase the peer’s configuration:
-
Click Confirm to start the clustering process.
The clustering process restarts services on both appliances and can take a short time to complete. During this time the WebUI may briefly become unavailable.
A cluster can contain up to three passive nodes in addition to the active node. To add further passive nodes, repeat the steps above on the active node for each additional appliance.
Viewing the HA Configuration and Status
Once clustering is complete, both nodes are listed on the Cluster Configuration > High availability configuration page, along with the current status of each node:
The table shows the following information for each node:
| Column | Description |
|---|---|
Hostname |
The hostname of the node. |
Bind address |
The IP address that PulseHA is bound to on the node. |
Latency |
The measured network latency to the peer node. |
Services |
The status of the load balanced services on the node. |
Status |
The clustering role of the node — Active or Passive. |
State |
The health state of the node — for example Online. |
Last health check |
The time of the last successful health check. |
In the example above, lb1 is the active node and lb2 is the passive node. Each node also provides a three-dots menu offering the following options:
-
Promote — make the selected node the active node. Selecting Promote on the passive node fails the cluster over so that it becomes active and the current active node becomes passive.
-
Remove — remove the selected node from the cluster, returning it to a standalone configuration.
PulseHA configuration
The clustering (PulseHA) behavior is tuned under Cluster Configuration > PulseHA configuration.
The PulseHA Global Configuration section provides the following settings:
| Field | Description |
|---|---|
Health check interval |
The rate (in milliseconds) at which PulseHA sends health checks to determine whether a node is available. This value should be consistent across all nodes. |
Failover interval |
The rate (in milliseconds) at which PulseHA checks confirmed health checks to determine whether a node has become unavailable and a fail-over recovery should be attempted. This value may vary per node depending on your configuration. |
Failover limit |
The total time (in milliseconds) allowed to elapse since the last confirmed health check before the node is considered unavailable and a fail-over is performed. |
Log level |
The granularity of the logs PulseHA writes to journalctl. The available options are debug, info, warn and error. |
Auto fail back |
Controls the cluster behavior when a failed node returns to service. When enabled, the recovered node automatically returns to active status, taking back the floating IP addresses from the node that took over. When disabled, the node that took over remains active and retains the floating IP addresses; fail-over back must then be performed manually using the Promote option (see Viewing the HA Configuration and Status). |
Click Submit to apply any changes.
| The PulseHA Configuration section at the foot of the page shows the address and port that PulseHA is listening on. While a node is part of a cluster these settings are read-only; to change them, the node must first be removed from the cluster. |