Diagnostics & Troubleshooting

Problems can occur for a number of reasons. Typically, these are due to misconfigured Virtual Services or network topology related issues. With high throughput deployments, issues can also occur due to appliance resources being overloaded.

The appliance aggregates the log output from all of its services into a single, searchable interface, and provides a range of resource utilization graphs. Both are useful when diagnosing issues. The logs are accessed via the Logs menu option in the WebUI, where they can be filtered by service, searched using the Search box and refreshed using the Update button. For more information, see Logs.

Don’t hesitate to contact support if you’re experiencing any issues or need assistance configuring your appliance.

Service Logs

As described in Logs, Endurance provides comprehensive logs across all system functionality, which can be very useful when diagnosing issues. The Services logs page (Logs > Services logs) aggregates the log output from every appliance service into a single searchable view - for full details on using this page, see Services Logs.

When diagnosing a specific problem, filter the display to the relevant service(s):

Service	Useful for
IP Virtual Service Director	Layer 4 health check activity, including when and why Real Servers were marked up or down.
HAProxy	Layer 7 service state, reloads and health checks and - if connection logging is enabled - individual client connections.
PulseHA	High availability (clustering) and failover events.
NF Tables	Firewall (nftables) activity.

Service

Useful for

IP Virtual Service Director

Layer 4 health check activity, including when and why Real Servers were marked up or down.

HAProxy

Layer 7 service state, reloads and health checks and - if connection logging is enabled - individual client connections.

PulseHA

High availability (clustering) and failover events.

NF Tables

Firewall (nftables) activity.

By default, Layer 7 (HAProxy) logging is set to Off. To enable it, navigate to Cluster Configuration > Advanced configurations, select the Layer 7 tab and set the Log level under Logging to the required level (Health Check, Errors or Debug), then click Submit. Note that verbose logging on a busy Virtual Service is likely to rapidly exhaust the available log space.

View Configuration

As described in View Configuration, the underlying configuration of all system components and services can easily be viewed. This is often useful when diagnosing an issue, as it shows exactly how a service has been configured.

To view a configuration:

Using the WebUI, navigate to View configuration.
Select the system component or service you’d like to view, e.g. HAProxy (Layer 7), IP Virtual Service Director (Layer 4) or Conntrack.
The related configuration is displayed. Where more than one view is available (for example Database and Config), use the tabs at the top of the page to switch between them. Click Copy output to copy the displayed configuration to the clipboard.

Appliance Resource Utilization

The graphs in the System Overview can be used to provide a quick overview of appliance resource utilization. The following graphs are provided:

Network Throughput
Load Average
Memory Usage

The graphs can be viewed by navigating to the lower portion of the System Overview, or by clicking the relevant "eye" icon in the top right corner of the CPU, Memory or Disks widget at the top of the screen. For more information, see Viewing Detailed Metrics.

Diagnosing Virtual Service Issues

Virtual Services may not function as expected for a variety of reasons. This section suggests various diagnostic methods that can help identify problems and also looks at potential misconfiguration issues.

Using the System Overview

The System Overview can be used to quickly check the state of layer 4 and layer 7 Virtual Services and their associated Real Servers.

Verify that all Real Servers are up. In the example below, the orange "1" indicates that one of the Real Servers associated with Web-Cluster has failed its health check.

Click into the Web-Cluster service.

The image above shows that RS-02 is failing its health check. This should be investigated and corrected - possible steps include:

Verify that the application/service is running on RS-02.

Make sure you can ping the Real Server from the load balancer - either from the console, via an SSH session or using the WebUI menu option: Local Configuration > Execute shell command.

If you run ping from the WebUI, use the form:

ping -c 4 10.11.40.20

The -c 4 means ping 4 times then stop. This is important when running the command from the WebUI to ensure it terminates cleanly.

Verify that the application/service is up and available when accessed from the load balancer - various methods can be used:
- Using nmap at the console, via an SSH session or using the WebUI option: Local Configuration > Execute shell command:
  nmap 10.11.40.20
  The following is displayed for a working server:
  Starting Nmap 7.92 ( https://nmap.org ) at 2025-08-08 08:20 UTC Nmap scan report for 10.11.40.20 Host is up (0.00030s latency). Not shown: 995 filtered tcp ports (no-response) PORT STATE SERVICE 80/tcp open http 135/tcp open msrpc 443/tcp open https 445/tcp open microsoft-ds 3389/tcp open ms-wbt-server MAC Address: 00:50:56:82:0B:D3 (VMware) Nmap done: 1 IP address (1 host up) scanned in 4.90 seconds
  This shows that the server is listening on port 80.
- Using curl at the console, via an SSH session or using the WebUI option: Local Configuration > Execute shell command:
  curl http://10.11.40.20 or curl http://host.mydomain.com
  For a working web server listening on port 80, the default page is returned.
Ensure that access to the Real Server is not blocked by a firewall.

VIPs & RIPs are up but Users Can’t Connect

If you’ve configured your VIPs and RIPs and everything looks good (green) in the System Overview but users still can’t connect, there are a number of potential causes for this as described below.

Layer 4 VIPs

Layer 4 DR mode and NAT mode have certain configuration requirements. It’s important to remember that the health checks performed by the load balancer verify that the load balancer can successfully access the server/service/application. This does not verify that each server has been configured correctly to enable client access. The following sections explain the configuration requirements for each mode. In both cases, a Real Server that passes its health check (and so appears green in the System Overview) but to which clients still cannot connect is a strong indicator that these requirements have not been met.

DR Mode

For layer 4 DR mode VIPs, the "ARP Problem" must be solved on all associated Real Servers - the exact steps required depend on the particular Real Server OS. If the "ARP Problem" has not been correctly solved, the load balancer’s health checks will pass but client connections will fail. For more information, see DR Mode Considerations.

NAT Mode

For layer 4 NAT mode VIPs, the default gateway on all associated Real Servers must be configured to be an IP address on the load balancer to ensure that client return traffic passes back via the load balancer. For an HA cluster, this should be a floating IP address to allow the IP address to float (move) between the nodes. If the default gateway is not set correctly, the load balancer’s health checks will pass but client connections will fail.

Layer 7 VIPs

Layer 7 mode - Have you configured the correct Layer 7 mode? The default mode for new layer 7 VIPs is HTTP mode. This is fine for web based traffic typically on port 80, but if you’ve configured your layer 7 VIP to load balance something else such as HTTPS on port 443 or RDP on port 3389 then you’ll need to set the Layer 7 mode to TCP mode.

TProxy - If you’ve enabled transparent proxy (TProxy) so that the Real Servers see the original client source IP address, the Real Servers must route their return traffic back via the load balancer - typically by setting their default gateway to a Floating IP owned by the load balancer. For more information on the Enable TProxy option, see Backend Configuration - Layer 7 advanced.

Multi-port VIPs

Does your VIP listen on multiple ports, for example 80,443? If so, make sure that the Real Server port field is left blank so that traffic is passed through on the correct port. If you specify for example 80, all traffic - including traffic intended to reach port 443 - will be passed to port 80.

DR mode does not have a Real Server port field since port translation is not possible when using this mode. Traffic is always passed through to the same port as received by the VIP.

Requests are not Load Balanced as Expected

If the System Overview reports that there is an imbalance in the number of connections, this may be caused by a number of reasons as listed below:

Are clients connecting from behind a NAT device? - If this is the case, then all requests will appear to come from the same source IP address. This will be an issue if source IP address persistence is used because all client sessions would be load balanced to the same Real Server.

Has a real server been brought back online? - If Balance mode has been set to Weighted Least Connection, when the server is brought online it will handle all new requests until balance is reached across all servers. This may occur quickly if lots of new requests are received. If not, the imbalance will continue for longer. If Balance mode has been set to Weighted Round Robin, requests will continue to be distributed equally (assuming the weight is the same). If the other Real Servers in the cluster were already processing requests there will again be an imbalance.

Are you testing using a single test client? - If persistence is enabled for the VIP then you’ll be load balanced to the same Real Server.

Useful Tools & Utilities

Useful Diagnostics Tools

The appliance includes a number of useful diagnostics tools that can be run at the console or via an SSH session. most commands can also be run via the WebUI - to run commands via the WebUI, see Execute shell command.

Netstat

Print network connections, routing tables, interface statistics, masquerade connections, and multicast memberships. Useful to check that services are listening on the correct IP/port.

e.g. netstat -anp

Command Output:

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address         Foreign Address     State       PID/Program name
tcp        0      0 127.0.0.1:7778        0.0.0.0:*           LISTEN      7218/haproxy
tcp        0      0 192.168.100.238:80    0.0.0.0:*           LISTEN      7218/haproxy
tcp        0      0 0.0.0.0:22            0.0.0.0:*           LISTEN      9638/sshd
tcp        0      0 0.0.0.0:9081          0.0.0.0:*           LISTEN      5136/nginx
tcp        0    336 192.168.100.237:22    10.10.0.26:61430    ESTABLISHED 9801/sshd
tcp        0      0 :::9443               :::*                LISTEN      732/httpd
tcp        0      0 :::22                 :::*                LISTEN      9638/sshd
tcp        0      0 :::9080               :::*                LISTEN      732/httpd
udp        0      0 192.168.100.238:123   0.0.0.0:*                       1650/ntpd
udp        0      0 192.168.100.237:123   0.0.0.0:*                       1650/ntpd
udp        0      0 127.0.0.1:123         0.0.0.0:*                       1650/ntpd
udp        0      0 0.0.0.0:123           0.0.0.0:*                       1650/ntpd
udp        0      0 0.0.0.0:161           0.0.0.0:*                       10089/snmpd
etc.

Tcpdump

Tcpdump enables network traffic to be dumped to a file for analysis. Filters can also be applied if required to select which traffic is captured. Very useful tool when diagnosing network issues. Note that this command should be run from the console or a terminal session rather than via the WebUI.

e.g. tcpdump -i any -s 0 -w tcpdump-file.pcap

This command captures all network traffic on all interfaces using the maximum packet size of 65535 bytes and dumps it to a file called tcpdump-file.pcap. To end the capture use CTRL+C.

Our support department may ask you to run this command and send the resulting output file to help them diagnose certain network issues.

Ethtool

Ethtool is used for querying settings of an Ethernet device and changing them.

e.g. ethtool eth0

Command output:

Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supports auto-negotiation: No
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Speed: 10000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        MDI-X: Unknown
        Supports Wake-on: uag
        Wake-on: d
        Link detected: yes

Nmap

Nmap (Network Mapper) can be used to scan a range of hosts or a single host to determine which ports are open and which services are listening on those ports.

e.g. nmap 192.168.110.241

Command output:

Starting Nmap 5.51 ( http://nmap.org ) at 2022-03-15 14:54 UTC
Nmap scan report for 192.168.110.241
Host is up (0.0010s latency).
Not shown: 996 filtered ports
PORT     STATE SERVICE
80/tcp   open  http
135/tcp  open  msrpc
443/tcp  open  https
3389/tcp open  ms-term-serv
MAC Address: 00:50:56:82:0B:D3 (VMware)

Nmap done: 1 IP address (1 host up) scanned in 4.53 seconds

Wireshark

Wireshark is an open source application that can be used to analyze tcpdump output files. It can be downloaded from here.

Windows Specific Tools

Microsoft Network Monitor

Network Monitor is a simpler alternative to Wireshark that has some nice features. It can be downloaded from here.

WinSCP

WinSCP is an open source application that allows files to be uploaded/downloaded to/from the load balancer using Windows. It can be downloaded from here.

PuTTy

PuTTy is an open source SSH client for Windows. It can be downloaded from here.