# Load balancing CyberArk (now Idira) Privileged Session Manager HTML5 Gateway with HAProxy *This is one post in a* [*series focusing on load balancing various CyberArk (now Idira) components using HAProxy with a focus on application/service-based health checking.*](https://timschindler.blog/series/cyberark-haproxy) With the experience we gained from [load balancing CyberArk Privileged Vault Web Access with HAProxy](https://timschindler.blog/application-health-checking-and-load-balancing-cyberark-privileged-vault-web-access-with-haproxy), load balancing the Privileged Session Manager HTML5 GW (PSM HTML5 GW) while incorporating application-based health checking with HAProxy is simple. CyberArk already provides an endpoint we can use for the check so it is all about crafting our HAProxy configuration file. # Setting up HAProxy Similar to how we did it with [load balancing the PVWA](https://timschindler.blog/application-health-checking-and-load-balancing-cyberark-privileged-vault-web-access-with-haproxy#heading-setting-up-haproxy) we will use Docker and a HAProxy Docker image, passing our `haproxy.cfg` and TLS certificates in mounted volumes. Lets create a directories to store our `haproxy.cfg` and certificate with it's private key. ```bash tim@docker01:~$ mkdir -p haproxy-html5gw/haproxy tim@docker01:~$ mkdir -p haproxy-html5gw/certificates ``` And generate a self-signed certificate. For any production environment we would make sure to use an issued certificate from a trusted certificate authority. ```bash tim@docker01:~$ openssl req -x509 -newkey rsa:4096 -keyout haproxy-html5gw/certificates/tls.pem.key -out haproxy-html5gw/certificates/tls.pem -sha256 -days 365 -nodes Generating a RSA private key .....................................++++ .....................................................................................................................................++++ writing new private key to 'haproxy-html5gw/certificates/tls.pem.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]: State or Province Name (full name) [Some-State]: Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]: Organizational Unit Name (eg, section) []: Common Name (e.g. server FQDN or YOUR name) []:192.168.0.115 Email Address []: tim@docker01:~$ ``` We created the certificate and it's private key under our personal user and with the file permissions the private key is created with the HAProxy process the user runs as cannot read the private key. Already knowing that the HAProxy process runs under the user ID 99 we go ahead and change ownership of the private key. ```bash tim@docker01:~$ ls -la haproxy-html5gw/certificates/tls.pem.key -rw------- 1 tim tim 3272 Feb 1 15:36 haproxy-html5gw/certificates/tls.pem.key tim@docker01:~$ chown 99 haproxy-html5gw/certificates/tls.pem.key tim@docker01:~$ ls -la haproxy-html5gw/certificates/tls.pem.key -rw------- 1 99 tim 3272 Feb 1 15:36 haproxy-html5gw/certificates/tls.pem.key tim@docker01:~$ ``` # Creating our `haproxy.cfg` In `haproxy.cfg` we define our frontend and backend sections. In the frontend section we define on which address and port HAProxy accepts traffic on and the TLS certificate used to encrypt traffic between HAProxy and the client. In the backend section we define the the individual servers HAProxy will relay traffic to. We can also define a load balancing strategy, health checking, and persistent sessions. Lets start with a basic `haproxy.cfg`: ```bash tim@docker01:~$ vi haproxy-html5gw/haproxy/haproxy.cfg ``` with the content ```plaintext frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws server html5gw1 192.168.0.113:443 ssl verify none ``` We define a frontend to listen on port 443 and tell it to use the certificate we generated earlier (*HAProxy will look for a private key with the name* `.key` *in the same directory.*) With `default_backend` we define the name of the backend that traffic coming from the frontend should be relayed to. In the backend section we define a single HTML5GW to relay the traffic to. In reality we will have multiple but we can define just one and still be able to implement our concepts. The server name is arbitrary and we can call it whatever we want but the backend must be named `html5gws` as we used it as the value for `default_backend` in the frontend. After specifying the address and IP of the server we state to connect with ssl and again because this is a lab environment we allow for invalid TLS certificates on the backend HTML5 servers with `verify none`. We are ready to start our container with the certificates folder and the folder containing our `haproxy.cfg` as mounted volumes. ```bash tim@docker01:~$ sudo docker run -p 443:443 -v /home/tim/haproxy-html5gw/haproxy:/usr/local/etc/haproxy:ro -v /home/tim/haproxy-html5gw/certificates:/usr/local/etc/certs:ro haproxy:latest [NOTICE] (1) : haproxy version is 2.5.1-86b093a [NOTICE] (1) : path to executable is /usr/local/sbin/haproxy [WARNING] (1) : config : missing timeouts for frontend 'html5gw_loadbalancer'. | While not properly invalid, you will certainly encounter various problems | with such a configuration. To fix this, please ensure that all following | timeouts are set to a non-zero value: 'client', 'connect', 'server'. [WARNING] (1) : config : missing timeouts for backend 'html5gws'. | While not properly invalid, you will certainly encounter various problems | with such a configuration. To fix this, please ensure that all following | timeouts are set to a non-zero value: 'client', 'connect', 'server'. [NOTICE] (1) : New worker (8) forked [NOTICE] (1) : Loading success. ``` Ignoring the warnings about the timeouts that we will fix later, we navigate to `https://192.168.0.115/guac/direct` and are shown an error: ![Screenshot 2022-02-01 at 17.14.05.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1643732064831/qX22TZJwH.png align="center") This error is fine and is not indicative of anything in our `haproxy.cfg` being wrong. When browsing directly to `https://192.168.0.115/guac/direct` our browser is making a [GET request](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/GET), which is not supported by the HTML5 GW. Looking at the HTML5GW logs we can see the GET request being made: ![Screenshot 2022-02-01 at 17.17.46.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1643732279878/WPvNQoWHC.png align="center") Now that we verified that HAProxy is relaying traffic received from the frontend to the backend we can start implementing our application-based health check and sticky sessions. # Adding application health checking Like we could have with the PVWA, we can enable basic health checking by adding the `check` keyword to our server defined in the backend section but we still run the risk of HAProxy relaying traffic to a HTML5 GW in a scenario where the Tomcat (or whatever web server you are using to host the HTML 5 GW Web App) server is running but the Apache Guacamole service that the HTML5 GW is based off of is not functional. `option httpchk` enables us to have more complex health checking. It makes a request to the server and if the HTTP response code is either 2xx or 3xx then HAProxy considers the server healthy and keeps relaying traffic to it. Without defining a URI as part of `option httpchk` the health check is done against `/`. As the HTML5 GW Web App lives under the `/gauc` path, not defining another URI may not give us true application-based health checking. The [documentation for the PSM HTML5 GW](https://docs.cyberark.com/Product-Doc/OnlineHelp/PAS/Latest/en/Content/PAS%20INST/Install_PSM_HTML5.htm?tocpath=Installation%7CInstall%20PAM%7CInstall%20PSM%7CAdvanced%20PSM%20Implementations%7CInstall%20PSM%20HTML5%20Gateway%7C_____0#HealthCheck) specifies a URI we should use when making our health check. We need to make sure we specify `/guac/rest/healthcheck` with our `option httpchk` resulting in our `haproxy.cfg` looking like: ```plaintext frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws option httpchk GET /guac/rest/healthcheck server html5gw1 192.168.0.113:443 check ssl verify none ``` We also must add the `check` keyword to our server otherwise the parameters of the health check are defined via `option httpchk` but the health check will never happen and the server will always be considered healthy. # Sticky sessions In the same [way we did it with the PVWA](https://timschindler.blog/application-health-checking-and-load-balancing-cyberark-privileged-vault-web-access-with-haproxy#heading-sticky-sessions) we will use [stick tables](https://www.haproxy.com/blog/introduction-to-haproxy-stick-tables/) again to implement sticky sessions although HAProxy offers more than one way to accomplish sticky sessions. Our stick table captures client IP addresses and associates them to a backend server. The first time a client connects, HAProxy relays the traffic to a server and associates the client IP address to the server in the stick table. The next time the client IP address connects, HAProxy looks up the associated server in the stick table and relays the traffic to that server. We set a long expiration date of 36 hours for this association as our PSMs are configured to close idle PSM sessions at this point. ```bash frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws stick-table type ip size 1m expire 36h stick on src option httpchk GET /guac/rest/healthcheck server html5gw1 192.168.0.113:443 check ssl verify none ``` # Adding `stats` We [already know how useful it is having a stats page](https://timschindler.blog/application-health-checking-and-load-balancing-cyberark-privileged-vault-web-access-with-haproxy#heading-finishing-it-off-with-stats) that we can use to see numerous information for our frontends and backends, among that the health of the servers in the backend. We use it to see various statistics regarding our frontends and backends and as we specify `stats admin if TRUE` we can perform administrative tasks like disabling health checks on servers or marking individual servers as healthy or unhealthy. ```bash defaults timeout client 10s timeout connect 10s timeout server 10s frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws stick-table type ip size 1m expire 36h stick on src option httpchk GET /guac/rest/healthcheck server html5gw1 192.168.0.113:443 check ssl verify none frontend stats mode http bind *:8404 stats enable stats uri /stats stats refresh 10s stats admin if TRUE ``` We also define values under `defaults` for the timeouts that HAProxy was complaining about missing when we first created our `haproxy.cfg`. # Adding a second HTML5 GW So far we have been working with a single HTML5 GW but for sake of better being able to test our new `haproxy.cfg` lets add a second HTML5 GW into the mix. Because I am [managing my HTML5 GW containers with Docker Compose](https://timschindler.blog/cyberark-html5-gateway-and-docker-compose) it is trivial to spin up a second HTML5 GW running on the same Docker host but with a different published port. ```bash defaults timeout client 10s timeout connect 10s timeout server 10s frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws stick-table type ip size 1m expire 36h stick on src option httpchk GET /guac/rest/healthcheck server html5gw1 192.168.0.113:443 check ssl verify none server html5gw2 192.168.0.113:8443 check ssl verify none frontend stats mode http bind *:8404 stats enable stats uri /stats stats refresh 10s stats admin if TRUE ``` # Testing our `haproxy.cfg` with the help of `stats` We start our container once more: ```bash tim@docker01:~$ sudo docker run \ -p 8404:8404 \ -p 443:443 \ -v /home/tim/haproxy-html5gw/haproxy:/usr/local/etc/haproxy:ro \ -v /home/tim/haproxy-html5gw/certificates:/usr/local/etc/certs:ro \ haproxy:latest ``` Browsing to the stats we see frontend and backend -- and a bunch of green, which is a good sign! ![Screenshot 2022-02-02 at 20.39.23.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1643830775273/YXOfdFurXh.png align="center") We navigate to `https://192.168.0.115/gauc/direct` and though we get an error as expected, refreshing the stats page shows all our connections being relayed to a single server. ![Screenshot 2022-02-02 at 21.33.45.png](https://cdn.hashnode.com/res/hashnode/image/upload/v1643834071411/uZA0o8BO4.png align="center") # Doing more with HAProxy Our `haproxy.cfg` only scratches the surface in terms of the functionality HAProxy offers. We can adjust how often the health check happens by specifying the [`inter` parameter](https://cbonte.github.io/haproxy-dconv/2.5/configuration.html#5.2-inter) for our servers if we find the default not aggressive enough and explicitly pick a load balancing method with the [`balance` keyword](https://cbonte.github.io/haproxy-dconv/2.5/configuration.html#4-balance). As HAProxy is a Layer 4 proxy we can even load balance the [Privileged Session Manager](https://cbonte.github.io/haproxy-dconv/2.5/configuration.html#option%20forwardfor) and the [Privileged Session Manager for SSH](https://docs.cyberark.com/Product-Doc/OnlineHelp/PAS/Latest/en/Content/PAS%20INST/Installing-the-Privileged-Session-Manager-SSH-Proxy.htm?tocpath=Installation%7CInstall%20PAM%7CInstall%20PSM%20for%20SSH%7C_____0)!