This is one post in a series focusing on load balancing various CyberArk (now Idira) components using HAProxy with a focus on application/service-based health checking.
Load balancing CyberArk Privileged Session Manager for SSH (often referred to as PSMP) with application-based health checking using HAProxy is a bit more complicated than doing the same with CyberArk Privileged Web Access Vault or the PSM HTML5 Gateway but still accomplishable with the advanced features HAProxy provides.
Using Expect we can build a simple script that tests the functionality of the PSMP by proxying a connection to a target server and ensuring we get the expected password prompt, providing a known, good credential to authenticate with, and ultimately seeing the expected bash prompt of the target server. We configure can HAProxy to execute it as part of it's health check in order to achieve application-based health checking.
As we are already familiar with HAProxy from load balancing other CyberArk components we can focus our energy on building the script used as part of the health check.
What is Expect?
Expect is '... a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.' It '... "talks" to other interactive programs according to a script. Following the script, Expect knows what can be expected from a program and what the correct response should be. An interpreted language provides branching and high-level control structures to direct the dialogue. In addition, the user can take control and interact directly when desired, afterward returning control to the script.' [0] In fact, Expect is used by CyberArk Central Password Manager's Terminal Plugin Controller as part of some of it's plugins.
Put it more straightforwardly: we can use Expect and it's commands to spawn a ssh connection to a target server using a PSMP, expect the login prompt as a response, and send keystrokes to complete the authentication. If we see the bash prompt of the target server then we can reasonably assume the PSMP service is running and healthy
[0] https://www.tcl.tk/man/expect5.31/expect.1.html
Before introducing Expect lets first lets login manually to see what prompts we get. This will help us outline the steps we need to take in a script that will operate as our health check.
tim@docker01:~$ ssh tim@admin01@linux01.iosharp.lab@192.168.121
The authenticity of host '192.168.0.121 (192.168.0.121)' can't be established.
ECDSA key fingerprint is SHA256:FuAf0xfImgbvwwn3l4wTvwNUc6K7ZpdQsZmJrjUb2jc.
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.121' (ECDSA) to the list of known hosts.
Vault Password:
This session is being recorded
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-97-generic x86_64)
Last login: Tue Feb 8 14:25:40 2022 from 192.168.0.122
admin01@linux01:~$ exit
logout
Connection to linux01.iosharp.lab closed.
Connection to 192.168.0.121 closed.
tim@docker01:~$
What is important to take note is what we are prompted for so that we can include them as expect commands in a script and where we need to provide input to move the process along so we can provide input in send commands.
If we run through the login process again, keeping in mind how it would look with Expect commands, we:
spawn a SSH connection to the target server proxying through a PSMP
expect to be prompted when connecting to a machine for the first time
send the word 'yes' to continue the process where we
expect to be prompted for our credentials in order to authenticate to the Vault
send the credential for the Vault user we are authenticating
expect the prompt of the target system we are connecting to
send exit in order to close the connection
We have our steps, now we just need to put it together in a script.
Creating our Expect script
After installing Expect using apt (my Docker host is running Ubuntu - depending on what your OS is your package manager may differ) we create login.check and store it in our home directory.
tim@docker01:~$ touch login.expect
tim@docker01:~$ vi login.expect
With the steps above as a basis our login.check should look like
#!/usr/bin/expect
spawn ssh tim@admin01@linux01.iosharp.lab@192.168.0.121
expect "yes/no"
send "yes\r"
expect "Vault Password:"
send "Password1234!\r"
expect "admin01@linux01:~$ "
send "exit\r"
At the top, similar to a Bash script, we define a shebang so that when we run the script the OS knows to invoke it with Expect.
Afterwards, the first thing we do is spawn a connection to a target server using the PSMP whose health we want to validate. In anticipation of an unknown SSH host key, we expect "yes\no" and then send "yes\r". We do not have to define the exact string we expect to receive as Expect allows us to use regular expressions in our expect commands.
When using send we need to remember to provide a carriage return using \r when we 'emulating' pressing Enter. We do the same thing with the prompt Vault Password: where we provide our password.
Finally we expect to see the bash prompt of the target server we are connecting to and then end the script by ending the SSH connection.
Executing our script we can see the results:
tim@docker01:~$ ./login.expect
spawn ssh tim@admin01@linux01.iosharp.lab@192.168.0.121
Vault Password:
Vault Password:tim@docker01:~$
It doesn't work. The SSH connection to the PSMP was successful as we received the Vault Password: prompt however we were not able to authenticate and eventually the timeout was reached and the script quit.
Because we were never prompted to accept the unknown SSH host key our script got hung up at that step. We can ensure we are never prompted regarding an unknown SSH host key by passing -o StrictHostKeyChecking=no as an argument to our SSH command and we can remove the expect/send from our script:
#!/usr/bin/expect
spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121
expect "Vault Password:"
send "Password1234!\r"
expect "admin01@linux01:~$ "
send "exit\r"
We execute our script once more:
tim@docker01:~$ ./login.expect
spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121
Vault Password:
This session is being recorded
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-97-generic x86_64)
Last login: Tue Feb 8 15:47:06 2022 from 192.168.0.122
admin01@linux01:~\( tim@docker01:~\)
It works! We have a functioning basis for our health check but we should extend our script a bit more to handle some known behavior when the PSMP is not working.
Handling known, non-working behavior in Expect
Expect allows us to throw exit codes. These exit codes can be used by other processes, like HAProxy, to know if the script completed fine or with an error.
Our script already handles the situation where the PSMP is working fine but we should attempt to capture the behavior of a non-functional PSMP and account for it in our script. We can then throw an exit code other than 0, which signifies the login process and thus our script encountered a problem.
Like we did with crafting our Expect script with a working PSMP, lets see what the behavior of a PSMP is when the service is shut off.
With the PSMP service disabled, lets try to connect to a target server:
tim@docker01:~$ ssh tim@admin01@linux01.iosharp.lab@192.168.0.121
Vault Password:
Vault Password:
Vault Password:
tim@admin01@linux01.iosharp.lab@192.168.0.121: Permission denied (publickey,keyboard-interactive).
tim@docker01:~$
Despite putting in the correct password, we were prompted two more times and finally the connection closed.
Lets update our script to reflect this behavior when the PSMP service is not running:
#!/usr/bin/expect
spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121
expect {
"Vault Password:" { send "Password1234!\r" }
}
expect {
"admin01@linux01:~$ " { exit 0 }
"Vault Password:" { exit 1 }
}
We refactor our script a bit when extending it to handle the case where the PSMP service is not running by organizing the statements into expect blocks.
Within our first expect block we handle the first prompt -- Vault Password: and send our password when prompted.
In the second expect block we handle what comes after the first prompt. If we receive the target server's expected bash prompt -- our success case -- we close with exit 0. If instead we receive Vault Password: once more, indicating that the login process with our known, good password failed, then we throw exit 1. We need these error codes as HAProxy will use them to determine the health of the server.
With the PSMP service still stopped, lets invoke our script and check the error code:
tim@docker01:~$ ./login.expect
spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121
Vault Password:
Vault Password:tim@docker01:~\( echo \)?
1
tim@docker01:~$
We use $? to get the exit code of the last command executed. As any exit code other than 0 indicates an error, this is exactly what we want. For good measure we should re-enable the PSMP service, run our script, and check the exit code:
tim@docker01:~$ ./login.expect
spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121
Vault Password:
Last failed login: Tue Feb 8 20:57:39 CET 2022 from 192.168.0.115 on ssh:notty
There were 3 failed login attempts since the last successful login.
This session is being recorded
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-97-generic x86_64)
Last login: Tue Feb 8 20:00:27 2022 from 192.168.0.122
admin01@linux01:~\( tim@docker01:~\) echo $?
0
tim@docker01:~$
Both success and failure behavior is covered and return the appropriate exit codes. With our script in a suitable place lets introduce HAProxy into the mix.
HAProxy and haproxy.cfg
What HAProxy is, it's capabilities, and what a basic and functioning haproxy.cfg looks like is well covered in the posts where we load balance the PVWA and HTML5 Gateway.
Once again we use Docker to run HAProxy however unlike done previously we cannot simply use the an existing image and mount our haproxy.cfg as we intend to use a script that leverages Expect and the OpenSSH client -- two programs not found in the HAProxy Docker images provided by the HAProxy team. We need to built our own Docker image that includes our needed dependencies.
This this done by creating our own Dockerfile which is nowhere as daunting as sounds as we can use an existing HAProxy Docker image as a base.
In a new working directory, move our script and create our Dockerfile:
tim@docker01:~$ mkdir haproxy-psmp
tim@docker01:~$ mv login.expect haproxy-psmp/
tim@docker01:~$ vi haproxy-psmp/Dockerfile
with the content
FROM haproxy:latest
USER root
RUN apt-get update && apt-get install -y expect openssh-client && apt-get clean
USER haproxy
COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
COPY login.expect /var/lib/haproxy/login.expect
Before we build an image based on our Dockerfile we need to create the haproxy.cfg in our working directory with frontends and backends:
defaults
timeout client 10s
timeout connect 10s
timeout server 10s
global
insecure-fork-wanted
external-check
frontend psmp_loadbalancer
bind *:22
default_backend psmp_servers
backend psmp_servers
option tcp-check
tcp-check expect string SSH-2.0-
option external-check
external-check command /var/lib/haproxy/login.expect
server psmp01 192.168.0.122:22 check inter 31s
server psmp02 192.168.0.121:22 check inter 31s
frontend stats
mode http
bind *:8404
stats enable
stats uri /stats
stats refresh 10s
stats admin if TRUE
Focusing on what is in our backend, we have two health checks: a tcp-check expect that works similar to our Expect script in that it expects to receive a certain string -- the SSH banner -- when first connecting to a SSH server and an option external-check that invokes the script we created. If the first check fails, most likely to the OpenSSH service not running, the HAProxy marks the server as unhealthy and doesn't bother with subsequent ones.
As optional external-check invokes processes and thus poses a security risk, we also need to explicitly add insecure-fork-wanted and external-check under the global section to tell HAProxy we really want to use it.
We still are not ready to build our image as our script has the target PSMP hard coded. external-check command passes arguments four arguments to the command, the third being the address of the server the health check is being ran against. We need to update our script to take this argument.
As we already updating our script, we also take the opportunity to disable logging to Stdout by setting user_log 0, create a CyberArk user healthcheck to explicitly authenticate to the Vault with for the purpose of our health check, and define the full path to the ssh binary. Our script now looks like:
#!/usr/bin/expect
user_log 0
set psmp [lindex $argv 2]
spawn /usr/bin/ssh -o StrictHostKeyChecking=no healthcheck@admin01@linux01.iosharp.lab@$psmp
expect {
"Vault Password:" { send "Password1234!\r" }
}
expect {
"admin01@linux01:~$ " { exit 0 }
"Vault Password:" { exit 1 }
}
Building our Docker image and running our container
Building the image is simple. We just need to run docker build with a tag we will remember:
tim@docker01:~/haproxy-psmp$ sudo docker build -t haproxy-psmp .
Sending build context to Docker daemon 4.608kB
Step 1/6 : FROM haproxy:latest
---> 4313a6c47541
Step 2/6 : USER root
---> Using cache
---> 974bba5aa714
Step 3/6 : RUN apt-get update && apt-get install -y expect openssh-client && apt-get clean
---> Using cache
---> 625afeb7fa22
Step 4/6 : USER haproxy
---> Using cache
---> 2bba1cf696a7
Step 5/6 : COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg
---> 8ee4cfac7f22
Step 6/6 : COPY login.expect /var/lib/haproxy/login.expect
---> 4672cc1f4c34
Successfully built 4672cc1f4c34
Successfully tagged haproxy-psmp:latest
tim@docker01:~/haproxy-psmp$
Now we just start the container, publishing the ports for our backend and stats. As we include our haproxy.cfg and login.expect as part of the image we do not need to mount them.
tim@docker01:~/haproxy-psmp$ sudo docker run \
-p 8404:8404 \
-p 23:22 \
haproxy-psmp:latest
As we already have our Docker host's SSH daemon running on port 22, we need to have Docker publish HAProxy on 23.
Our container starts successfully and in the HAProxy logs we see processes exiting with code 0, indicating the health check using our script is firing and ending successfully.
tim@docker01:~/haproxy-psmp$ sudo docker run -p 8404:8404 -p 23:22 haproxy-psmp:latest
[NOTICE] (1) : New worker (8) forked
[NOTICE] (1) : Loading success.
[NOTICE] (1) : haproxy version is 2.5.1-86b093a
[NOTICE] (1) : path to executable is /usr/local/sbin/haproxy
[WARNING] (1) : Process 12 exited with code 0 (Exit)
[WARNING] (1) : Process 17 exited with code 0 (Exit)
[WARNING] (1) : Process 22 exited with code 0 (Exit)
[WARNING] (1) : Process 27 exited with code 0 (Exit)
[WARNING] (1) : Process 32 exited with code 0 (Exit)
Checking our stats page, we see both PSMPs as green.
Shutting off one of the PSMP services it goes red.
We are still able to login through our HAProxy:
tim@docker01:~/haproxy-psmp$ ssh tim@admin01@linux01.iosharp.lab@192.168.0.115 -p 23
Vault Password:
This session is being recorded
Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-97-generic x86_64)
Last login: Wed Feb 9 09:55:44 2022 from 192.168.0.122
admin01@linux01:~$ whoami
admin01
admin01@linux01:~$ hostname
linux01
admin01@linux01:~$ exit
logout
Connection to linux01.iosharp.lab closed.
Connection to 192.168.0.115 closed.
tim@docker01:~/haproxy-psmp$
Going forward
Though the health checking is more complicated than with the PVWA and HTML5 Gateway due to requiring the usage of an external script leveraging Except but being able to run HAProxy and the script in it's own Docker container makes the entire setup more manageable.
We can also add additional, non-working behavior in our script, a good candidate being when we receive Password: instead of Vault Password:, indicating there maybe something wrong with the pluggable authentication module stack.
How are you checking the health of your PSMPs? Provide some feedback in the comments!
