This is one post in a series focusing on load balancing various CyberArk (now Idira) components using HAProxy with a focus on application/service-based health checking.
With the experience we gained from load balancing CyberArk Privileged Vault Web Access with HAProxy, load balancing the Privileged Session Manager HTML5 GW (PSM HTML5 GW) while incorporating application-based health checking with HAProxy is simple. CyberArk already provides an endpoint we can use for the check so it is all about crafting our HAProxy configuration file.
Setting up HAProxy
Similar to how we did it with load balancing the PVWA we will use Docker and a HAProxy Docker image, passing our haproxy.cfg and TLS certificates in mounted volumes.
Lets create a directories to store our haproxy.cfg and certificate with it's private key.
tim@docker01:~$ mkdir -p haproxy-html5gw/haproxy
tim@docker01:~$ mkdir -p haproxy-html5gw/certificates
And generate a self-signed certificate. For any production environment we would make sure to use an issued certificate from a trusted certificate authority.
tim@docker01:~$ openssl req -x509 -newkey rsa:4096 -keyout haproxy-html5gw/certificates/tls.pem.key -out haproxy-html5gw/certificates/tls.pem -sha256 -days 365 -nodes
Generating a RSA private key
.....................................++++
.....................................................................................................................................++++
writing new private key to 'haproxy-html5gw/certificates/tls.pem.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (eg, city) []:
Organization Name (eg, company) [Internet Widgits Pty Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:192.168.0.115
Email Address []:
tim@docker01:~$
We created the certificate and it's private key under our personal user and with the file permissions the private key is created with the HAProxy process the user runs as cannot read the private key. Already knowing that the HAProxy process runs under the user ID 99 we go ahead and change ownership of the private key.
tim@docker01:~$ ls -la haproxy-html5gw/certificates/tls.pem.key
-rw------- 1 tim tim 3272 Feb 1 15:36 haproxy-html5gw/certificates/tls.pem.key
tim@docker01:~$ chown 99 haproxy-html5gw/certificates/tls.pem.key
tim@docker01:~$ ls -la haproxy-html5gw/certificates/tls.pem.key
-rw------- 1 99 tim 3272 Feb 1 15:36 haproxy-html5gw/certificates/tls.pem.key
tim@docker01:~$
Creating our haproxy.cfg
In haproxy.cfg we define our frontend and backend sections.
In the frontend section we define on which address and port HAProxy accepts traffic on and the TLS certificate used to encrypt traffic between HAProxy and the client. In the backend section we define the the individual servers HAProxy will relay traffic to. We can also define a load balancing strategy, health checking, and persistent sessions.
Lets start with a basic haproxy.cfg:
tim@docker01:~$ vi haproxy-html5gw/haproxy/haproxy.cfg
with the content
frontend html5gw_loadbalancer
bind *:443 ssl crt /usr/local/etc/certs/tls.pem
default_backend html5gws
backend html5gws
server html5gw1 192.168.0.113:443 ssl verify none
We define a frontend to listen on port 443 and tell it to use the certificate we generated earlier (HAProxy will look for a private key with the name <certificate filename>.key in the same directory.) With default_backend we define the name of the backend that traffic coming from the frontend should be relayed to.
In the backend section we define a single HTML5GW to relay the traffic to. In reality we will have multiple but we can define just one and still be able to implement our concepts. The server name is arbitrary and we can call it whatever we want but the backend must be named html5gws as we used it as the value for default_backend in the frontend.
After specifying the address and IP of the server we state to connect with ssl and again because this is a lab environment we allow for invalid TLS certificates on the backend HTML5 servers with verify none.
We are ready to start our container with the certificates folder and the folder containing our haproxy.cfg as mounted volumes.
tim@docker01:~$ sudo docker run -p 443:443 -v /home/tim/haproxy-html5gw/haproxy:/usr/local/etc/haproxy:ro -v /home/tim/haproxy-html5gw/certificates:/usr/local/etc/certs:ro haproxy:latest
[NOTICE] (1) : haproxy version is 2.5.1-86b093a
[NOTICE] (1) : path to executable is /usr/local/sbin/haproxy
[WARNING] (1) : config : missing timeouts for frontend 'html5gw_loadbalancer'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[WARNING] (1) : config : missing timeouts for backend 'html5gws'.
| While not properly invalid, you will certainly encounter various problems
| with such a configuration. To fix this, please ensure that all following
| timeouts are set to a non-zero value: 'client', 'connect', 'server'.
[NOTICE] (1) : New worker (8) forked
[NOTICE] (1) : Loading success.
Ignoring the warnings about the timeouts that we will fix later, we navigate to https://192.168.0.115/guac/direct and are shown an error:
This error is fine and is not indicative of anything in our haproxy.cfg being wrong. When browsing directly to https://192.168.0.115/guac/direct our browser is making a GET request, which is not supported by the HTML5 GW. Looking at the HTML5GW logs we can see the GET request being made:
Now that we verified that HAProxy is relaying traffic received from the frontend to the backend we can start implementing our application-based health check and sticky sessions.
Adding application health checking
Like we could have with the PVWA, we can enable basic health checking by adding the check keyword to our server defined in the backend section but we still run the risk of HAProxy relaying traffic to a HTML5 GW in a scenario where the Tomcat (or whatever web server you are using to host the HTML 5 GW Web App) server is running but the Apache Guacamole service that the HTML5 GW is based off of is not functional.
option httpchk enables us to have more complex health checking. It makes a request to the server and if the HTTP response code is either 2xx or 3xx then HAProxy considers the server healthy and keeps relaying traffic to it. Without defining a URI as part of option httpchk the health check is done against /. As the HTML5 GW Web App lives under the /gauc path, not defining another URI may not give us true application-based health checking.
The documentation for the PSM HTML5 GW specifies a URI we should use when making our health check.
We need to make sure we specify /guac/rest/healthcheck with our option httpchk resulting in our haproxy.cfg looking like:
frontend html5gw_loadbalancer
bind *:443 ssl crt /usr/local/etc/certs/tls.pem
default_backend html5gws
backend html5gws
option httpchk GET /guac/rest/healthcheck
server html5gw1 192.168.0.113:443 check ssl verify none
We also must add the check keyword to our server otherwise the parameters of the health check are defined via option httpchk but the health check will never happen and the server will always be considered healthy.
Sticky sessions
In the same way we did it with the PVWA we will use stick tables again to implement sticky sessions although HAProxy offers more than one way to accomplish sticky sessions.
Our stick table captures client IP addresses and associates them to a backend server. The first time a client connects, HAProxy relays the traffic to a server and associates the client IP address to the server in the stick table. The next time the client IP address connects, HAProxy looks up the associated server in the stick table and relays the traffic to that server. We set a long expiration date of 36 hours for this association as our PSMs are configured to close idle PSM sessions at this point.
frontend html5gw_loadbalancer
bind *:443 ssl crt /usr/local/etc/certs/tls.pem
default_backend html5gws
backend html5gws
stick-table type ip size 1m expire 36h
stick on src
option httpchk GET /guac/rest/healthcheck
server html5gw1 192.168.0.113:443 check ssl verify none
Adding stats
We already know how useful it is having a stats page that we can use to see numerous information for our frontends and backends, among that the health of the servers in the backend. We use it to see various statistics regarding our frontends and backends and as we specify stats admin if TRUE we can perform administrative tasks like disabling health checks on servers or marking individual servers as healthy or unhealthy.
defaults
timeout client 10s
timeout connect 10s
timeout server 10s
frontend html5gw_loadbalancer
bind *:443 ssl crt /usr/local/etc/certs/tls.pem
default_backend html5gws
backend html5gws
stick-table type ip size 1m expire 36h
stick on src
option httpchk GET /guac/rest/healthcheck
server html5gw1 192.168.0.113:443 check ssl verify none
frontend stats
mode http
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
stats admin if TRUE
We also define values under defaults for the timeouts that HAProxy was complaining about missing when we first created our haproxy.cfg.
Adding a second HTML5 GW
So far we have been working with a single HTML5 GW but for sake of better being able to test our new haproxy.cfg lets add a second HTML5 GW into the mix. Because I am managing my HTML5 GW containers with Docker Compose it is trivial to spin up a second HTML5 GW running on the same Docker host but with a different published port.
defaults
timeout client 10s
timeout connect 10s
timeout server 10s
frontend html5gw_loadbalancer
bind *:443 ssl crt /usr/local/etc/certs/tls.pem
default_backend html5gws
backend html5gws
stick-table type ip size 1m expire 36h
stick on src
option httpchk GET /guac/rest/healthcheck
server html5gw1 192.168.0.113:443 check ssl verify none
server html5gw2 192.168.0.113:8443 check ssl verify none
frontend stats
mode http
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
stats admin if TRUE
Testing our haproxy.cfg with the help of stats
We start our container once more:
tim@docker01:~$ sudo docker run \
-p 8404:8404 \
-p 443:443 \
-v /home/tim/haproxy-html5gw/haproxy:/usr/local/etc/haproxy:ro \
-v /home/tim/haproxy-html5gw/certificates:/usr/local/etc/certs:ro \
haproxy:latest
Browsing to the stats we see frontend and backend -- and a bunch of green, which is a good sign!
We navigate to https://192.168.0.115/gauc/direct and though we get an error as expected, refreshing the stats page shows all our connections being relayed to a single server.
Doing more with HAProxy
Our haproxy.cfg only scratches the surface in terms of the functionality HAProxy offers. We can adjust how often the health check happens by specifying the inter parameter for our servers if we find the default not aggressive enough and explicitly pick a load balancing method with the balance keyword.
As HAProxy is a Layer 4 proxy we can even load balance the Privileged Session Manager and the Privileged Session Manager for SSH!
