Tim Schindler's Blog

Building Idira (CyberArk) ISP PowerShell tooling with Claude Code, Context7 MCP, and SecretStore

Tim Schindler — Wed, 01 Jul 2026 07:37:40 GMT

Coding agents write competent PowerShell against the CyberArk (now Idira) REST APIs but often fall short with Idira endpoints. Left to its own devices, the model will invent a token endpoint, treat the Identity tenant ID and the ISP subdomain as one string, and hardcode a client secret into the script the model just produced. Those are context failures, and no prompt fixes the gap one message at a time.

So we give the model the context up front. We will build one small PowerShell tool that authenticates to the Idira Identity Security Platform (ISP) and proves a token works, and the tool itself is almost incidental. What earns the reliability is the work we do around it: we feed the model the real API documentation with Context7, we write the platform's quirks into a CLAUDE.md so the model stops guessing them, we give it the tenant's non-secret details as config, then we keep the service account secret where the model never gets the cleartext. The harness here is Claude Code, but the technique is about agentic coding in general, so the pieces port to whatever harness you run.

Why letting the model guess is the wrong default

The failure modes here are not subtle, but each one looks correct until something returns a 404 or a malformed request error. The model derives the Idira service host from the login domain, because for most SaaS products the two match.

For Idira ISP they do not. The model also typically reaches for the PAM self-hosted logon flow when the tenant is really ISP. PAM self-hosted's logon API endpoint is hosted by the PVWA, while ISP gets a platform token from Idira Identity with a client-credentials grant, so a script that worked against a PAM self-hosted instance fails when authenticating against an ISP tenant. The model writes the secret straight into the script, or into a .env that the model then reads back, which in an agentic loop puts the secret into the model's context, or the model's logs.

We will not fix any of this one prompt at a time. The way out is to make the right information cheap for the model to pull, and to keep the dangerous information out of the files and context the model touches -- and, where that is not enough, behind a hard stop it cannot cross.

Feeding the model real docs with Context7

Context7 is an up-to-date documentation source for coding agents: give it a library name and it returns that library's current docs as snippets the model can read on demand. It plugs into a harness as an MCP (Model Context Protocol) server or as a Skill; here we call it as an MCP server. For us the value is simple: the model stops guessing the ISP auth flow and reads the flow straight from the source documentation. Pointed at the Idira API docs, the library resolves cleanly:

Context7:resolve-library-id  "api-docs.cyberark.com"
 -> /websites/api-docs_cyberark   (CyberArk API, 80 snippets, source reputation: High)

When we query that library for the platformtoken flow, we get the actual endpoint contract rather than a plausible reconstruction:

POST https://.id.cyberark.cloud/oauth2/platformtoken
content-type: application/x-www-form-urlencoded

grant_type=client_credentials
client_id=
client_secret=

200 OK
{ "access_token": "eyJh...", "token_type": "Bearer", "expires_in": 900 }

The model pulling the platformtoken contract from Context7's /websites/api-docs_cyberark library rather than generating it.

The request body is an ordinary OAuth 2.0 client-credentials grant, the same call we would make with cURL. The only parts specific to ISP are the host we send the grant to and where the token works afterwards, and those are the two details the model gets wrong from memory. Pulling them from api-docs.cyberark.com removes the guesswork.

Context7 helps beyond the language we happen to be writing in. It also indexes the libraries next to our task:

/pspete/pspas               psPAS               (2305 snippets, High)
/pspete/identitycommand     IdentityCommand     (75 snippets,   High)
/cyberark/idsec-sdk-golang  CyberArk's official Go SDK

We write PowerShell here, but the model can learn from more than PowerShell, because the platformtoken call is the same HTTP whether we express the call in Go, in PowerShell, or as a cURL example sitting in the documentation.

The cleanest reference implementation of the ISP authenticator lives in the idsec-sdk-golang NewIdsecISPAuth flow, so we let the model read how that SDK models authentication and translate the pattern into the Invoke-RestMethod call we want. psPAS and IdentityCommand serve the same purpose closer to home: proven patterns the model can lift instead of improvise.

None of this means we should pipe every library through Context7, since coverage varies and what comes back is only ever as good as the underlying documentation. We still give the model the real reference instead of hoping the model remembers, and Context7 makes that cheap.

Encoding the idiosyncrasies in CLAUDE.md

Context7 gives the model the facts. CLAUDE.md gives the model the rules that hold for this platform and this tenant, the ones no generic documentation will surface, because they are the assumptions a careful engineer would otherwise make and get wrong. Claude Code reads CLAUDE.md from the project root at the start of every session; this is an agentic-coding technique rather than a Claude Code one, so other harnesses read an equivalent file -- Codex and several others look for AGENTS.md -- and the rules port even where the filename does not.

The most valuable rule we can write is the one about hosts. The Identity tenant ID is not an Idira ISP's subdomain. The portal might be acme.cyberark.cloud and the Idira ISP's APIs acme.dpa.cyberark.cloud, both built from the same subdomain, but the Identity host that serves the token endpoint is something like abc1234.id.cyberark.cloud, built from a different identifier entirely. The model assumes one identifier covers all three hosts, and because the token host uses its own, deriving it from the subdomain produces a DNS failure long before we ever reach the credentials.

Here is the CLAUDE.md we hand the model:

# Idira (formerly CyberArk) ISP — project rules

## Branding
- CyberArk is now Idira. URLs and APIs are UNCHANGED. Do not rename `*.cyberark.cloud`, `api-docs.cyberark.com`, or the `cyberark/*` SDK repos.

## Hosts — two identifiers, three hostnames. Never derive one from another.
- IDENTITY_TENANT_ID and SUBDOMAIN are non-secret config; read them from .env. They are not secrets: do not hardcode them and do not put them in the vault.
- IDENTITY_TENANT_ID  -> token endpoint host: .id.cyberark.cloud
- SUBDOMAIN           -> service API hosts:   ..cyberark.cloud
- DO NOT assume IDENTITY_TENANT_ID == SUBDOMAIN. **They are ALWAYS DIFFERENT.** Treat both as required inputs; never infer the service host from the login domain.

## Authentication — client_credentials, service user
- This is an ISP (SaaS) tenant. Authenticate with the OAuth2 client_credentials grant. Pull the exact token endpoint, re quest body, and response shape from the api-docs.cyberark.com reference through Context7; do not write them from memory.
- The credential is a SERVICE USER (an OAuth confidential client): its login name is the client_id and its password is the client_secret. A regular, interactive Cloud Directory user authenticates through a different CyberArk Identity flow, so do not use that flow here.
- Tokens are short-lived. Re-authenticate on expiry; never cache a token for the life of a long-running job.

## API endpoints — read them from Context7, never from memory
- Look up EVERY Idira API endpoint you call (path, method, query string, and request/response shape) from api-docs.cyberark.com through Context7 — not just the token endpoint, every service API too.
- api-docs.cyberark.com does not cover every Idira service. When an endpoint is not there, pull it from another Context7 library that does — `psPAS`, `IdentityCommand`, or the `idsec-sdk-golang` SDK — instead of guessing it.
- The docs are not infallible: what Context7 returns can still be wrong or incomplete. Use the retrieved form, but flag any endpoint you cannot confirm for live verification against the tenant -- never commit an unverified path silently.

## Do NOT cross the streams
- DO NOT use the self-hosted Vault flow `/PasswordVault/API/auth/CyberArk/Logon`. This is a SaaS (ISP) tenant; the two flows are not interchangeable and will fail against the wrong base URL.

## Secrets
- The service-account client_id/secret come from the SecretManagement vault at runtime via Get-Secret. NEVER hardcode them, write them to a file, echo them, or commit them. Do not read the SecretStore vault file.

The host rules and the ISP-vs-self-hosted guardrail are the two the model gets wrong most often.

Every block in there exists because I have watched a model make that mistake. The branding note stops the model from helpfully renaming working URLs, and the host rules are there because the Identity-tenant-ID-equals-Idira-subdomain assumption is the single most common way these scripts fail. The endpoints rule is there for the same reason: a model will otherwise reach for a plausible API path from memory and get it subtly wrong, and a wrong path only fails when the script runs, so every endpoint has to come from the docs through Context7 instead of from recall. The cross-the-streams rule stops the model from authenticating an ISP tenant as though the tenant were a PAM self-hosted deployment.

One caveat matters here: the official docs are careful to say that Claude Code treats CLAUDE.md as context, not enforced configuration, which means every rule in the file guides the model without binding it. The host rules make the model far less likely to conflate the two identifiers, but the rules are guidance, not a guarantee, and the same holds for the "do not read the vault file" line.

When we need a hard stop instead of a strong nudge, and for a credential store we should, the docs point to a PreToolUse hook that denies the matching tool call in code rather than by persuasion. A hook is only as tight as the patterns you give it: match every path to the credential, or the gap you leave is the bypass. Hooks are Claude Code's enforcement mechanism; other harnesses expose their own, or none, in which case the boundary has to live outside the harness. CLAUDE.md shapes the model's behavior, and a hook enforces the limit in code.

Keeping the service-account secret out of the model's reach

The credential we are protecting is an Idira Cloud Directory service user -- Identity's name for a service account -- configured as an OAuth confidential client. We create the service user in the Identity Administration portal, tick Is OAuth confidential client, and grant the least privilege the integration needs. The Add service users docs walk through the steps, after which the service user's login name becomes the client_id and the password becomes the client_secret.

The service user configured as an OAuth confidential client.

A .env file does not protect the secret here, and the reason is specific to agentic coding rather than plain hygiene: the model can read the file. Claude Code opens files and writes code that prints whatever the code loaded. A plaintext secret then sits squarely inside the model's reach.

💡 The risk is not that the secret leaks to git, since we already gitignore .env. The risk is the model itself reading, printing, or committing the secret while doing exactly what we asked, landing the cleartext in its context or a transcript. So we keep the secret out of any file the model has reason to open, and we lean on the harness permission controls so the model will not read the credential store or run the authenticated command in a shell whose environment the model then dumps.

I use one of two options here, depending on what the project already runs.

The first-party choice is PowerShell's own SecretManagement with the SecretStore extension (two Microsoft modules, no third-party service). SecretStore keeps secrets in a local file that is encrypted with .NET cryptographic APIs and unlocked by a vault password. The model can see the vault file, but the file is ciphertext, and the model never holds the password that unlocks the vault, which is the whole reason this arrangement survives an agent with read access to the project. We set this up once:

Install-Module Microsoft.PowerShell.SecretManagement -Scope CurrentUser
Install-Module Microsoft.PowerShell.SecretStore       -Scope CurrentUser

Register-SecretVault -Name idira -ModuleName Microsoft.PowerShell.SecretStore -DefaultVault

# Stored once, interactively. The values never appear in the script.
Set-Secret -Vault idira -Name idira-client-id     -Secret 'svc-automation@cyberark.cloud'
Set-Secret -Vault idira -Name idira-client-secret -Secret (Read-Host -AsSecureString)

The alternative I reach for, when a project already runs it, is the 1Password CLI. With op run, we store op://vault/item/field references, and 1Password resolves them into a child process's environment for the life of that process alone. The real values stay out of your project files, and op run redacts them on a best-effort basis when they surface in stdout — redaction, not containment: a child process can still read, transform, or log the resolved value. For an agentic workflow that is the better fit, because the secret stays out of any project file entirely.

Injecting the secret as an environment variable still does not isolate it from the model: on Linux, another process running as our user can usually read our processes' environment, and the model runs as our user. When the model's process must not reach the credential at all, use a PreToolUse hook, or move the credential behind a boundary the model's user cannot cross — a separate OS identity, a broker process, or a container it has no access to.

For the example below we use SecretStore: it is a Microsoft module that runs anywhere PowerShell 7 does and needs no account. It does prompt for a vault password to unlock.

What the context changes

With Context7 connected and the CLAUDE.md in the project root, we hand the model one plain instruction:

Write a PowerShell script that gets all the Secure Infrastructure Account strong accounts for databases. Make it AS SIMPLE AS POSSIBLE. Do not go overboard with multiple functions, parameterization, etc. The simpler the better.

The difference from a cold start shows up in the first draft.

Without the context, a model tends to produce something like this -- a composite of the failure modes I have watched, not one captured run:

# context-free draft -- hardcoded secret, self-hosted flow, hosts collapsed
$secret = 'svc-Pa55w0rd!'                                                       # hardcoded
$logon  = 'https://acme.cyberark.cloud/PasswordVault/API/auth/CyberArk/Logon'   # self-hosted flow
\(token  = Invoke-RestMethod -Method Post -Uri \)logon -Body @{ username = 'svc'; password = $secret }

# prove the token against a service -- but on the bare subdomain, not the .dpa service host
Invoke-RestMethod -Uri 'https://acme.cyberark.cloud/dpa/api/...' -Headers @{ Authorization = $token }

It hardcodes the secret, reaches for the PAM self-hosted /PasswordVault logon flow, and collapses the token host and the service host into the one subdomain it was handed. The script fails on the first call, and the error is a DNS or 404 that points at the wrong layer.

With the context, the model has what it needs before it writes a line: the platform-token contract as facts from Context7, the host and secret rules from CLAUDE.md, and the ISP tenant's subdomain and Identity tenant ID as non-secret config in a .env. The first draft already sends the grant to platformtoken on the .id host, proves it against the .dpa host built from the tenant subdomain, and reads the credentials from the vault instead of inventing them.

That draft is the script below. Run that prompt yourself and the exact lines will differ -- the model is non-deterministic, and more than one script is correct here -- but the parts the context pins down come out the same: the two hosts, the platform-token grant, and the secret read from the vault by name.

The same request with the context in place: the model honors the host rules and reads the secret from the vault.

The script

The script gets a token from the Identity platformtoken, then proves it against a service on the tenant-subdomain host. We point the second call at a non-Identity service, Secure Infrastructure Access (SIA), so the call exercises the second host and confirms the model honored the CLAUDE.md host rule instead of reusing the token endpoint's host.

The ISP tenant subdomain and Identity tenant ID are configuration, not secrets, so they go in a plain .env. The tenant is the same on every run, which is why the script takes no parameters -- we write the two values down once and gitignore the file:

# .env -- tenant identifiers, not secrets
IDENTITY_TENANT_ID=abc1234
SUBDOMAIN=acme

The client_id and client_secret stay in the vault, looked up by name; the names are not sensitive either. With the config in .env, the secret in the vault, and the rules and contract already in place, the model has what it needs to write a script that authenticates and reaches both hosts:

#Requires -Modules Microsoft.PowerShell.SecretManagement

# Non-secret config from .env (IDENTITY_TENANT_ID and SUBDOMAIN are different hosts).
Get-Content .env | ForEach-Object {
    if (\(_ -match '^\s*([^#=]+?)\s*=\s*(.*)\)') { Set-Item "env:\((\)Matches[1].Trim())" $Matches[2].Trim() }
}

# Service-user (OAuth confidential client) credentials from the vault — never hardcoded.
$clientId     = Get-Secret -Name 'idira-client-id'     -AsPlainText
$clientSecret = Get-Secret -Name 'idira-client-secret' -AsPlainText

# OAuth2 client_credentials token from the identity tenant. Short-lived; fetched per run.
$token = (Invoke-RestMethod -Method Post `
    -Uri "https://$env:IDENTITY_TENANT_ID.id.cyberark.cloud/oauth2/platformtoken" `
    -ContentType 'application/x-www-form-urlencoded' `
    -Body @{ grant_type = 'client_credentials'; client_id = \(clientId; client_secret = \)clientSecret }
).access_token

# SIA (formerly DPA) database strong accounts. Verify this path against your tenant's
# SIA OpenAPI — api-docs.cyberark.com lists it inconsistently as /databases/strong-accounts.
(Invoke-RestMethod -Method Get `
    -Uri "https://$env:SUBDOMAIN.dpa.cyberark.cloud/databases/strong-accounts" `
    -Headers @{ Authorization = "Bearer $token" }
).accounts

The two Invoke-RestMethod calls deliberately hit two different hosts, $env:IDENTITY_TENANT_ID.id.cyberark.cloud and $env:SUBDOMAIN.dpa.cyberark.cloud, and that split is the point. Collapse the two hosts into one value, and the second call fails the moment we run it.

Get-Secret -AsPlainText does put the secret into a variable in process memory, where it lives until the process exits. That is unavoidable, since the HTTP body needs the cleartext. We never write the secret to a file, never log the secret, never hand the secret to anywhere the model captures output. Keep it that way.

Context7 changes the failure mode; it does not remove the need to verify. Pulling the endpoint from the docs stops the model inventing one from memory, but the docs themselves can be wrong, and we hit exactly that on the strong-accounts call: the model pulled /databases/strong-accounts from Context7 as the rule demands and flagged it right there in the script above, but that path did not match what works against the tenant, which is /api/database-strong-accounts. So whatever the model writes from a retrieved contract gets checked against the live tenant before we trust it. The docs raise the floor; they are not the last word.

Limitations

The script authenticates and proves the two-host split, but it is deliberately minimal, so a few limits come with it:

We add no token caching or refresh here. For a long-running job, re-authenticate on expiry instead of holding a token past expires_in.
A service account that authenticates with a client secret has no MFA in front of it, so treat the service user's permissions as the blast radius and keep them minimal.
Runtime secret injection is not a sandbox; a PreToolUse hook (or an equivalent boundary outside the harness) is still required to keep the model's process from reaching the credential.
SecretStore unlocks with a vault password; for unattended runs you configure that password, which is one more secret to place carefully.
A real integration needs error handling the example leaves out.
Context7 can reproduce errors in the upstream docs, so verify the retrieved contract.

Make it a skill

I built grant-cli, a just-in-time (JIT) cloud-role-elevation tool that wraps the idsec-sdk-golang ISP authenticator, on this exact loop, and the pattern carries well beyond a single language or service.

The CLAUDE.md rules and the Context7 lookups are not specific to that tool or this script. The host rules and the service-user auth note hold on every ISP project we touch, and so does the instruction to pull the contract from Context7. That is the signal that a section of CLAUDE.md has outgrown being a fact and become a procedure worth packaging as a skill: a skill loads only when it is relevant and follows the Agent Skills standard across harnesses, so the tenant knowledge travels into the next tool instead of being retyped. Context7 itself already ships as a skill and the ISP rules can too.

Creating a CyberArk (now Idira) Central Policy Manager plugin for an API using the new REST API framework

Tim Schindler — Mon, 17 Mar 2025 17:55:39 GMT

CyberArk (now Idira) has released their new framework for creating Central Policy Manager (CPM) plugins that interact with REST APIs. Unlike WSChains, this framework is documented and supported by CyberArk, making it worthwhile to become familiar with it. CyberArk is already using it, as it is the framework on which the new CyberArk Digital Vault platform is built, among others.

Additionally, it offers an alternative to the Credential Management .NET SDK for storing credentials generated by target systems (or “access keys”), which is becoming increasingly common.

To complete our CyberArk Shopizer Administrator extensions series, we will use it to build a CPM plugin that manages Shopizer users through the Shopizer API.

Developing the REST API plugin

The documentation for the REST API framework is quite comprehensive so there is no need to repeat it. After giving it a read-through and ensuring we’ve met the prerequisites for the framework, we can jump in.

Creating a new platform

After we import the REST API framework CPM plugin all that needs to be done is cloning it for use with our Shopizer platform and under Additional Policy Settings updating two parameters:

Defining 8080 as the value for Port
Changing the value for ConfigurationFilePath under to the filename that will contain the Shopizer configuration: ShopizerAdministratorConfiguration.xml

Creating the parameters file for testing

When developing the CPM plugin, we will test it by invoking it manually. Our .ini will be similar to what we used for WSChains with slight modifications:

[targetaccount]
username=shopizeradministrator@timschindler.dev
newpassword=Password2
password=Password2
safename=Safename
foldername=Foldername
objectname=Objectname
PolicyID=PolicyID
; The machine where the Shopizer instance is running.
Address=192.168.178.61
; The port the Shopizer REST API is running on.
Port=8080

[extrapass3]
username=reconcile@timschindler.dev
password=Password1

[extrainfo]
; Path to the configuration file.
ConfigurationFilePath=ShopizerAdministratorConfiguration.xml
AddressValidationPattern=.*
Debug=Yes
ManagementType=password

After adding ConfigurationFilePath and AddressValidationPattern we can invoke it from the root of the Central Policy Manager’s installation folder with .\bin\CANetPluginInvoker.exe shopizerrestplugin.ini CyberArk.Extensions.Generic.Plugin.RestAPI.dll True

Writing the configuration file

Luckily, we've already done most of the hard work when developing the WSChains plug-in. We know the necessary API calls for each operation, their URLs, and the request and possible response bodies. We just need to format the data to match what the REST API framework expects. We can refer to the Shopizer API documentation as needed.

To begin, we copy the sample configuration file provided with the framework, SampleRestAPIPlatformConfiguration.xml, and rename it to ShopizerAdministratorConfiguration.xml.

Verify

We’ll leverage the Parameters section for the sake of cleaner code, removing almost everything but opting to leave AddressWithValidation as it is.

We add the parameter AddressWithValidationAndPort that combines the address (host) and port properties along with the scheme, which is hard-coded to be http as the Shopizer instance we are developing against does not have a HTTPS listener.


	{{Address}}
	http://{{AddressWithValidation}}:{{Port}}

We only need a single APICall in the APICalls section:


	
		{{AddressWithValidationAndPort}}/api/v1/private/login
		
			application/json
		
		
					{
					"username": "{{Username}}",
					"password": "{{Password|JsonEscape}}"
					}

The Request element does not have any magic to it. It is exactly what we would see if we were making the call using a tool such as cURL or PowerShell’s Invoke-RestMethod. The only part that is specific to the REST API framework is how we can JSON escape placeholders using placeholder modifiers.

We have two Response child elements in the Responses element: one representing a successful logon response and a failed logon due to an incorrect password. In the successful logon response, we parse out the returned id and token as we can reuse the Response in other APICall elements. In the case of a failed login, we parse out the message so it can be displayed in the PVWA.

The last piece is defining the Request name in the verifypass chain in the Chains element. With that, our ShopizerAdministratorConfiguration.xml reads as below:



	
		{{Address}}
		http://{{AddressWithValidation}}:{{Port}}
	

	

	
		
			
				{{AddressWithValidationAndPort}}/api/v1/private/login
				
					application/json
				
				
					{
					"username": "{{Username}}",
					"password": "{{Password|JsonEscape}}"
					}

Even though we are not using ErrorMessages, we need to at least define as the REST API framework expects the element to be there.

Testing the plugin, we see positive progress:

PS > .\bin\CANetPluginInvoker.exe shopizerrestplugin.ini verifypass CyberArk.Extensions.Generic.Plugin.RestAPI.dll True
The plugin ended successfully (RC = 0)
PS >

Change

We need one more APICall that is used to change the password.

The additional APICall in the APICalls element for our change operation is:


	
		{{AddressWithValidationAndPort}}/api/v1/private/user/{{id}}/password
		
			application/json
			Bearer {{token}}
		
		
					{
					"password": "{{password|JsonEscape}}",
					"changePassword":"{{newpassword|JsonEscape}}"
					}

The Request element is simple. We include the Bearer token from the LogonAsTargetAccount Request as an Authorization header.

A successful password change gives a status code 200 with no extra details to process. We have a response for when the API returns a 500 — this is the only code the Shopizer API seems to give when the new password doesn't meet complexity requirements.

Adding the below Chain element to Chains and a successful test has us moving on to the reconcile operation.

PS > .\bin\CANetPluginInvoker.exe shopizerrestplugin.ini changepass CyberArk.Extensions.Generic.Plugin.RestAPI.dll True
The plugin ended successfully (RC = 0)
PS >

Reconcile

Compared to the verify and change operation, reconcile is more complicated. Our reconcile operation needs to:

Log in as the reconciliation account.
Get the details of the target account.
Set the new password for the target account, using the details from step 2.

We can create a new APICall element just for logging in as the reconciliation account, but since the request and responses would be the same as our current APICall for logging in as the target account during the verify operation, this would lead to a lot of repetition.

By using dynamic account types for placeholders, we can modify the existing APICall to work for logging in as either the target account or the reconciliation account.

The APICall element is updated to:


	
		{{AddressWithValidationAndPort}}/api/v1/private/login
		
			application/json
		
		
					{
					"username": "{{reconcileaccountORtargetaccount\username}}",
					"password": "{{reconcileaccountORtargetaccount\password|JsonEscape}}"
					}

Besides updating the Body element of the Request, we change the name attribute to Logon to show that it's not used for only logging in as the target account anymore. Be sure to update the verifypass and changepass chains with the new name of the Request element!

How do dynamic account type placeholders work? The documentation states:

The plugin searches for the ParameterName in the accounts declared from left to right and takes the first occurrence it finds. If the parameter is not found in any account, the replacement fails. For example, {{logonaccountORextrapass2ORtargetaccount\username}} tries to retrieve the username in the logon account. If the plugin cannot find it, it tries to retrieve the username from the extrapass2 account. If not found there as well, the plugin tries to retrieve the username from the target account. If the username parameter is not found in any of the declared accounts, the plugin fails.

The CPM only gives the reconciliation account's details during a reconcile operation. Since we list reconcileaccount first in the dynamic type placeholder, the plugin uses the reconcile account details for the Logon Request during the reconcile operation and uses the target account details for verify and change operations.

We still need the APICall elements for retrieving the target account’s details and the actual password change.

To retrieve the target account’s details needed as part of the password change process:


	
		{{AddressWithValidationAndPort}}/api/v1/private/users
		
			application/json
			Bearer {{token}}

The only thing we are doing for the first time is parsing the response using JSONPath.

We used XPath before with the Shopizer PSM connection component, and JSONPath works in a similar way. We use it to extract user details from the response. Since the APICall element calls a URL that returns all Shopizer users, we use JSONPath to find the information we need for the target account, using the username placeholder in the expression.

In the final APICall element, we include the new password along with the details extracted from the previous APICall:


	
		{{AddressWithValidationAndPort}}/api/v1/private/user/{{targetaccount\id}}
		
			application/json
			Bearer {{token}}
		
		
					{
					"active": {{targetaccount\active}},
					"defaultLanguage": "{{targetaccount\defaultLanguage}}",
					"firstName": "{{targetaccount\firstName}}",
					"groups:": {{targetaccount\groups}},
					"lastName": "{{targetaccount\lastName}}",
					"password": "{{newpassword}}",
					"repeatPassword": "{{newpassword}}",
					"store": "{{targetaccount\merchant}}",
					"emailAddress": "{{targetaccount\emailAddress}}",
					"userName": "{{targetaccount\username}}"			
					}

Finally, our Chains element looks like:

💡

You can find the complete platform, including the configuration constructed above, in GitHub.

In the end, what matters is that the plugin works when invoked by the CPM so we would be remiss not to ensure it works as we expect. Invoking the reconcile in the PVWA versus invoking it manually we see success:

Conclusion

Creating a CyberArk Central Policy Manager plugin using the new REST API framework offers a robust and supported method for managing credentials through REST APIs. This approach not only aligns with CyberArk's latest platform developments but also provides a documented alternative to older methods like WSChains.

By following the documentation and leveraging the flexibility of the REST API framework, we can efficiently build plugins tailored to specific applications, such as managing Shopizer administrator users.

See the below GitHub repository for all the platform files.

https://github.com/aaearon/cyberark-shopizer-admin-extensions

Creating a CyberArk (now Idira) Central Policy Manager plugin for an API using WSChains

Tim Schindler — Fri, 02 Aug 2024 15:39:40 GMT

In a previous article, we created a CyberArk (now Idira) Central Policy Manager plugin for the web application Shopizer using the Web Application CPM Plugin Framework and while it works, we had to employ an awkward workaround to get reconcile functioning. As Shopizer's web interfaces leverage its APIs, we should skip the middleman and have our CPM plugin interface those.

WSChains is an undocumented, unsupported, used-by-CyberArk-interally framework to build CPM plugins that interact with APIs. Using our existing knowledge of Shopizer, examples of WSChains-based plugins from the CyberArk Marketplace, we will build a WSChains-based CPM plugin to manage Shopizer users.

💡

This is article is one in a series of articles around creating PSM connectors and CPM plugins.

❗

Interested in seeing WSChains becoming a documented and supported option? Vote on this enhancement request!

Developing the WSChains plugin

Armed with Shopizer REST API documentation and a bit of knowledge about WSChains, we begin. We will re-use the Shopizer administrator users we previously created.

Creating a new web application platform

We can clone the web application framework-based platform and make some changes:

Under Automatic Password Management -> CPM Plug-in, change DLLName to Cyberark.Extensions.Plugin.WSChains.dll.
Under Automatic Password Management -> Generate Password, update PasswordForbiddenChars to include <>&\"', which are characters that need to be escaped inside of XML documents.
Under Automatic Password Management -> Additional Policy Settings, set Port to 8080 as this is the default port the Shopizer REST API listens on. Debug can be set to Yes but a majority of testing will be done outside of CyberArk.
Under Automatic Password Management -> Additional Policy Settings -> Parameters, we only need a single, additional parameter named ChainsFilePath with the value of bin\ShopizerAdministratorChains.xml -- the relative path to the XML Chains file that holds the operations of the CPM plugin.

❗

The WSChains DLL Cyberark.Extensions.Plugin.WSChains.dll is not available as part of the CPM's base installation. You need to acquire the DLL as part of a platform package from the CyberArk Marketplace or another source.

Preparing our parameters file for testing

We can test a WSChains plugin the same way we can test any other and we will do this to facilitate faster and easier development of our plugin. The Parameters file looks like the following:

[targetaccount]
username=shopizeradministrator@timschindler.dev
newpassword=Password2
password=Password2
safename=Safename
foldername=Foldername
objectname=Objectname
PolicyID=PolicyID
; The machine where the Shopizer instance is running.
Address=192.168.178.61
; The port the Shopizer REST API is running on.
Port=8080

[extrapass3]
username=reconcile@timschindler.dev
password=Password1

[extrainfo]
; Path to the chains file.
ChainsFilePath=bin\ShopizerAdministratorChains.xml
Debug=Yes

This is saved as an .ini file in the root of the CPM folder.

Crafting `ShopizerAdministratorChains.xml`

Like a WebFormFieldsFile, the Chains file will hold the details for all three operations of our CPM plugin. For each operation, we will need to define the necessary requests that will be used by the links in the chains that represent each CPM operation.

We start with the bare minimum and some Fails.

Save the file as ShopizerAdministratorChains.xml in the bin folder of the CPM.

Verify

Like the web application framework plugin, we will simply authenticate to the Shopizer REST API. Authenticating with a correct username and password will result in an authentication token.

In the Requests element, the following RequestWrapper needs to be added based on the Shopizer REST API documentation:


	
		
			2
			
				/api/v1/private/login
				POST
			
			
				application/json
				application/json
				Integration/1.0 (CyberArk, CPM, 1)
			
			
				
					{
						"username":"{{username}}",
						"password":"{{pmpass}}"
					}

Inside the request element, the name attribute is used to refer to the Request in our Chain's links. {{username}} and {{pmpass}} will be replaced with the username and password of the account when the request is executed.

Under the Chains element, we create a Chain with the name of verifypass. It contains only a single Link:

The Link object is mostly self-explanatory: the request attribute defines what Request is invoked and the different StatusCode elements determine how the response to the request is handled.

❗

There is not a hard requirement to having a child Parse element under the StatusCode that represents the end of the CPM operation however not parsing the returned authentication token will result in it being shown in plaintext in the debug logs!

When testing, we quickly get positive feedback:

PS C:\Program Files (x86)\CyberArk\Password Manager> .\bin\CANetPluginInvoker.exe shopizerwschains.ini verifypass Cyberark.Extensions.Plugin.WSChains.dll True
The plugin ended successfully (RC = 0)

Looking in the debug logs, we get a nice picture of what the plugin is doing.

Change

Change is made simple for us as in the same response that provides the authentication token, it provides the id of the user the token is for. This is important as we need to specify the id of the user whose password we are changing as part of the request URL.

As part of the change operation, at the end, we are going to authenticate with the new password to be sure of the change so we need three requests in total. We can re-use the GetToken request but we need one represent the change operation itself as well as logging in with the new password:


	
		
			2
			
				/api/v1/private/user/{{parsed/id}}/password
				PATCH
			
			
				application/json
				application/json
				Integration/1.0 (CyberArk, CPM, 1)
				Bearer {{parsed/token}}
			
			
				
				  {
				  "password": "{{pmpass}}",
				  "changePassword": "{{pmnewpass}}"
				  }
				
			
		
	
	
		
			2
			
				/api/v1/private/login
				POST
			
			
				application/json
				application/json
				Integration/1.0 (CyberArk, CPM, 1)
			
			
				
					{
						"username":"{{username}}",
						"password":"{{pmnewpass}}"
					}

We add the Authorization header with the Bearer token we received from the GetToken request to the ChangePassword request. GetTokenWithNewPass looks exactly like GetToken but we use {{pmnewpass}} as the password.

Under the Chains element, we create a Chain with the name of changepass with three Links:

The first Link is copied from our verify operation's Chain. The only difference is the addition of the Parse element that captures the id of the account so we can use it in the next Link. The next attribute of the Link tells the plugin the name of the Link in the Chain to execute.

ChangePasswordLink invokes the ChangePassword request and when executed successfully (the response's status code is 200), VerifyNewPassLink is called to login with the password just set. Again we parse the result so that we can mask the authentication token from the response in the logs.

Testing shows success:

PS C:\Program Files (x86)\CyberArk\Password Manager> .\bin\CANetPluginInvoker.exe shopizerwschains.ini changepass Cyberark.Extensions.Plugin.WSChains.dll True
The plugin ended successfully (RC = 0)

Reconcile

Reconcile is complex. We need to use a different endpoint when changing the password of another user. The request URL includes the id of the user to be changed -- requiring us to either store it as an account property or look this up as part of the CPM operation -- and because it is a PUT operation, we need to provide all the existing details of the user along with the new password, otherwise they will be deleted from the user's profile.

There is an endpoint that will list all users and their id so we will need to call this as a link in our reconcile operation's chain and parse the response. We will then retrieve the user's profile in order to get the information we need to pass along as part of the change password request.

We need the following, additional requests:


	
		
			2
			
				/api/v1/private/login
				POST
			
			
				application/json
				application/json
				Integration/1.0 (CyberArk, CPM, 1)
			
			
				
					{
						"username":"{{extrapass3\username}}",
						"password":"{{pmextrapass3}}"
					}
				
			
		
	
	
		
			2
			
				/api/v1/private/users
				GET
			
			
				application/json
				application/json
				Integration/1.0 (CyberArk, CPM, 1)
				Bearer {{parsed/token}}
			
			
				
			
		
	
	
		
			2
			
				/api/v1/private/users/{{parsed/id}}
				GET
			
			
				application/json
				application/json
				Integration/1.0 (CyberArk, CPM, 1)
				Bearer {{parsed/token}}
			
			
				
			
		
	
	
		
			2
			
				/api/v1/private/user/{{parsed/id}}
				PUT
			
			
				application/json
				application/json
				Integration/1.0 (CyberArk, CPM, 1)
				Bearer {{parsed/token}}
			
			
				
				{	
					
					"active": {{parsed/active}},
					"defaultLanguage": "{{parsed/defaultLanguage}}",
					"firstName": "{{parsed/firstName}}",			
					
					"groups:": {{parsed/groups}},
					"id": "{{parsed/id}}",
					"lastName": "{{parsed/lastName}}",
					"password": "{{pmnewpass}}",
					"repeatPassword": "{{pmnewpass}}",
					"store": "{{parsed/merchant}}",
					"emailAddress": "{{parsed/emailAddress}}",
					"userName": "{{parsed/userName}}"
				}

The RecGetToken request is like GetToken but uses the linked reconcile account's username and password. The GetUserByID and GetUserProfileByID requests facilitate retrieving the needed details from the user's profile while RecChangePassword is used to change the password, including the parsed details.

The reconcilepass Chain will have five links:

The heavy lifting is done in GetUserByIDLink and GetUserProfileByIDLink.

Regular expression is used to get the id of the user we are trying to reconcile in GetUserByIDLink. In GetUserProfileByIDLink, we can easily parse all the details except for the groups as this is a JSON array and again we use regular expression to grab the array as a string.

Again, our testing validates the chain.

PS C:\Program Files (x86)\CyberArk\Password Manager> .\bin\CANetPluginInvoker.exe shopizerwschains.ini reconcilepass Cyberark.Extensions.Plugin.WSChains.dll True
The plugin ended successfully (RC = 0)

Conclusion

WSChains is a great framework and our CPM plugin uses only a small amount of its features. I previously said it being unsupported by CyberArk makes it unsuitable for production use but I no longer agree with that statement. The maturity of WSChains and the functionality of it makes it a no-brainer for developing CPM plugins that interact with REST APIs.

The platform files, Chain file, and parameter file can be found in my GitHub repository. If you are interested in developing your own CPM plugin for Shopizer administrator users, you can use this docker-compose.ymlto quickly stand up a containerized Shopizer environment. Check out the README for more information. The same GitHub repository contains the platform files, Chain file, and a WSChains DLL.

Integrating CyberArk (now Idira) self-hosted PAM as a Secret Store in StrongDM

Tim Schindler — Sun, 26 May 2024 16:44:30 GMT

StrongDM bills itself as a Zero Trust PAM solution. It offers a pretty slick way to access resources through a Desktop client as well as a CLI. With either installed, you can easily connect to resources using native tooling. It is a nice alternative to the Privileged Session Manager (PSM).

The credentials for accessing resources can be stored in StrongDM's Strong Vault (a "simple" secret store) or StrongDM can integrate with various third party stores such as CyberArk — now Idira — (self-hosted PAM and Conjur), AWS Secrets Manager, Azure Key Vault, and a few more.

This is a quick article on integrating CyberArk self-hosted PAM as a secret store in StrongDM.

Reading through the documentation

StrongDM provides documentation for integrating CyberArk self-hosted PAM as a secret store but it is largely incomplete, and in one spot even incorrect, but depending on your familiarity with CyberArk, you can read between the lines as to what you need to do.

The first is creating the necessary users in CyberArk and configuring the gateways and relays to use them.

First, any gateway(s) and relay(s) that you intend to use to access resources with via CyberArk PAM must be configured to authenticate with CyberArk. Due to the manner in which CyberArk identifies users and manages seats, each of those gateways and relays must be set up in CyberArk as a user with its own credentials. Once this is done, those gateways and relays are capable of authenticating to CyberArk in order to fetch the required credentials to connect a user to a protected resource.

On the gateway or relay, set the environment variables PAM_USERNAME and PAM_PASSWORD with the user’s corresponding credentials as the value. You can also set PAM_TLS_SKIP_VERIFY=true to skip certificate verification if the CyberArk instance doesn’t have a valid certificate.

Without explicitly saying it, we can safely assume the integration uses the Password Vault Web Access (PVWA) REST API.

The justification for needing a CyberArk user per gateway or relay isn't clear to me but this requirement could also be in part due to not setting the concurrentSession parameter to True when calling the Logon endpoint.

The documentation doesn't mention how the CyberArk user authenticates. What is also missing are the needed safe memberships the users must have but they can already be assumed based on the nature of the integration.

Afterwards we set up the Secret Store in StrongDM.

In the Admin UI, go to Network > Secret Stores.

Click Add secret store.

Give the secret store a name that is recognizable within your organization, such as “CyberArk PAM.”

Choose CyberArk PAM as the secret store type.

For the application URL, enter the IP address of the Windows server that hosts your Central Credential Provider

Straight forward however step 5 looks to be incorrect. As we need to create CyberArk users for each gateway and relay and set the PAM_USERNAME and PAM_PASSWORD environmental variables, it must mean the PVWA and not the Central Credential Provider. Note: I submitted a case to StrongDM support about CCP vs PVWA and they've already replied they are passing it on to the Docs team.

There is no option to choose how the gateways and relays should authenticate with their CyberArk user so it likely only works with the users doing Vault authentication.

The last part is configuring a StrongDM resource to use an account in CyberArk. The example below uses a database.

From the Admin UI, go to Infrastructure > Datasources.

Click Add datasource.

Enter the properties for your database resource.

From the Secret Store dropdown menu, select the CyberArk PAM option.

In the Username (path) field, add the path to retrieve the username, in the format ?key=username. Use the CyberArk account ID of your resource, followed by ?key=username.

In the Private Key (path) field, add the path to retrieve the password, in the format ?key=password. Use the CyberArk account ID of your resource, followed by ?key=password.

When all required fields are complete, click Create.

The need to use an account ID cements the fact that the integration is using the PVWA REST API. An account ID is a construct only used in the PVWA.

Integrating CyberArk self-hosted PAM

The documentation is enough to get us started and we can fill in the gaps along the way.

As my lab environment only has a single gateway, I create a single CyberArk Vault user named strongdm. Despite the documentation saying nothing about safe memberships, we know the user needs something and List accounts together with Retrieve accounts on the safes containing accounts we want to use with StrongDM resources makes the most sense to start with.

I created a gateway on an existing Linux server in my lab and need to set the PAM_USERNAME and PAM_PASSWORD environmental variables.

PAM_TLS_SKIP_VERIFY is set to true as my PVWA's certificate's issuer is not trusted by my Linux server.

Adding CyberArk as a Secret Store is effortless. We just need to keep in mind that, despite the instructions, we provide the URL of the PVWA and not a Central Credential Provider instance. We also do not need to add the /PasswordVault/ path.

Once the secret store is created, it is marked as healthy as it is reachable by at least one gateway.

If we needed any more proof, we can check the IIS logs. 192.168.0.4 is the Linux server with the StrongDM gateway installed.

We can follow the steps for configuring a StrongDM resource to use an account stored in CyberArk self-hosted PAM quite literally. We just need the ID of the account we want to use.

A browser's developer tools is the quickest way to find an account's ID but if we needed to automate the adding of StrongDM resources using CyberArk accounts, the Get Accounts endpoint would be more appropriate.

With the account ID in hand, adding the resource looks like the following:

After resource creation, it is marked as healthy for our Linux gateway.

In activities for the account, we see the retrieval by the strongdm user with the reason "StrongDM Connection".

And for the ultimate test: a screenshot of the recording connecting through StrongDM.

Even with the documentation for the integration not being complete, it was still pretty quick and easy.

Closing thoughts

For what it is, StrongDM's CyberArk self-hosted PAM integration works nicely. A few thoughts:

Being that it works with the PVWA REST API, it doesn't require any Central Credential Provider or Credential Provider licenses, which -- together with removing the need for any PSMs and their associated RDS CAL costs -- can be a nice cost savings.
Having to use the ID of an account can make the initial creation of a resource a bit of a pain but with how fast connecting to a resource with StrongDM is, even when StrongDM needs to retrieve the password from the PVWA, it makes up for it.
If the accounts you intend to use with StrongDM resources will also be used by users for password retrieval or PSM sessions in CyberArk, you need to be mindful of the platforms having exclusive usage active as StrongDM will not check-in an account after a StrongDM connection closes.
Don't forget to provide Access Safe without confirmation in addition to List accounts and Retrieve accounts to StrongDM's CyberArk users when accounts have dual control enforced.
Central Credential Provider integration would be a nice option. StrongDM does not seem to retrieve the password for an account for each connection so you could run the CP without cache in order to not hit the 10,000 mark. To avoid the CP having access to more than 50,000 accounts, a gateway or relay could use a CyberArk Application that has access to a certain scope of accounts.
Expanding support to CyberArk's Privilege Cloud would be a big plus as together with StrongDM it could greatly reduce the amount of infrastructure you need to manage for a PAM solution.

Let me know your experience with integrating CyberArk self-hosted PAM and StrongDM.

Creating a CyberArk (now Idira) Central Policy Manager plugin for a web application

Tim Schindler — Sat, 30 Dec 2023 16:46:09 GMT

In a previous article, we developed a Microsoft Edge-based CyberArk (now Idira) Privileged Session Manager connection component for the web application Shopizer and with the knowledge we gained, we will create a Central Policy Manager plugin for a web application to manage the password of Shopizer administrator users.

Note: You can find the platform files for the Shopizer administrator user CPM plugin at https://github.com/aaearon/cyberark-shopizer-admin-extensions, including a docker-compose.yml file that can be used to quickly stand up your own Shopizer environment.

💡

This is article is one in a series of articles around creating PSM connectors and CPM plug-ins.

Understanding what a CPM plugin for a web app is doing

A CPM plugin for a web application works fundamentally the same as a PSM connection component for a web application.

The Web Application CPM Plugin Framework uses a WebDriver to identify and interact with elements in a web page's DOM based on a set of commands that represent the verify, change, and reconcile operations. We already covered WebDrivers and the DOM in more depth when we created a PSM connection component.

But instead of using the WebFormFields property, we use WebFormFieldsFile -- an .ini file -- that will hold the commands for each CPM operation. This file will live in the bin folder of the Central Policy Manager and the CPM plugin framework will parse and perform the commands contained.

A closer look at a `WebFormFieldsFile`

With an out of the box installation of the CPM there is no WebFormFieldsFile in the bin folder but the documentation tells us what it looks like (step 6): an .ini file with three sections -- verify, change, and reconcile.

[Verify]
# Verify commands go here, i.e. username > {username} (searchby=name)
[Change]
# Change commands go here
[Reconcile]
# Reconcile commands go here

The .ini file can be named anything so we want to make sure it is something unique and meaningful.

Developing our Shopizer CPM plugin

We already understand how to identify and select the best DOM elements so we can jump right in with creating our CPM plugin. The Create CPM plugins for Web applications documentation has pretty explicit instructions for creating the platform.

Setting the stage

Before creating the platform, we will create two administrator users in Shopizer for the purpose of development: shopizeradministrator@timschindler.dev and reconcile@timschindler.dev -- both with full administrator rights. The shopizeradministrator user will be the shared user and reconcile will serve as the reconciliation account for it.

Creating a new web application platform

Picking up at step 5 of the documentation for configuring platforms for web applications, we need to define VerifyURL, ChangeURL, ReconcileURL, and WebFormFieldsFile for our newly created platform.

The VerifyURL, ChangeURL, and ReconcileURL parameters function the same as to that of LogonURL for PSM connection components for web apps: before executing the commands in the WebFormFieldsFile, the framework navigates to the URL for the relevant section. We can use dynamic parameters corresponding to account properties as part of the parameters, too ({Address}, {Port}, etc.)

The WebFormFieldsFile property will hold the name of the file with all the operations' commands. For our CPM plugin used to manage Shopizer administrators, we will name the file ShopizerAdministrator.ini.

We will keep the default values for the parameters, changing only BrowserPath and adding the parameter Browser with a value of Edge to reflect the fact we will use Microsoft Edge. We set EnforceCertificate to No as our Shopizer development environment does not have a valid TLS certificate (but we would want to change this to Yes for production.) We will leave the VerifyURL, ChangeURL, and ReconcileURL parameters for the time being as we will set them at same time we define the commands for the CPM operations in ShopizerAdministrator.ini. Our last change will be defining the value for the Port parameter as 4200 -- the port our Shopizer administrator portal is running as -- under Additional Policy Settings.

With our (incomplete) platform created, we can onboard our two Shopizer accounts into CyberArk.

Crafting `ShopizerAdministrator.ini`

For each operation, we need to determine:

All the DOM elements that need to be interacted with to perform the operation. We need these for the commands.
The appropriate URL to start the commands at for the particular operation (VerifyURL, ChangeURL, and ReconcileURL.)

Important to note: Unlike a PSM connection component where we can disable validations, CPM plugins for web applications require at least one validation per operation so that the CPM knows if it was successful or not.

Verify section

As a verify operation is simply logging into the website with the vaulted credentials, we can simply re-use what we already have from our Shopizer PSM connection component plus add the required validation.

This results in our ShopizerAdministrator.ini file looking like:

[Verify]
username>{Username} (SearchBy=name)
password>{Password} (SearchBy=name)
//button[@type="submit"]>(Button) (SearchBy=XPath)
Informations sur le magasin>(Validation) (SearchBy=text)
Store information>(Validation) (SearchBy=text)

Note that, unlike the WebFormFields for a PSM connection component, we must not escape any brackets. And because the default Shopizer language is French, we search for the text 'Store information' or 'Informations sur le magasin' to validate we have logged in successfully.

We can also use the same value for LogonURL that we used in our PSM connection component for VerifyURL in our platform however we will not hard code the port and instead use a dynamic parameter, resulting in the final value being http://{Address}:{Port}/#/auth .

Clicking Verify in the PVWA, we see a successful verify operation.

Change section

For a Shopizer administrator user to change their own password, they need to:

Login to the Shopizer administrator portal.
Navigate to their profile.
Click Change Password from the pulldown.
Enter the current password, the new password, the new password again, and hit save.

To simplify the commands in our Change section, we will use http://{Address}:{Port}/#/auth as ChangeURL and then use the Navigate command to have the browser go directly to where we can change the password, http://{Address}:{Port}/#/pages/user-management/change-password.

This results in our Change section looking like:

[Change]
username>{Username} (SearchBy=name)
password>{Password} (SearchBy=name)
//button[@type="submit"]>(Button) (SearchBy=XPath)
Informations sur le magasin>(Validation) (SearchBy=text)
Store information>(Validation) (SearchBy=text)
(Wait=1)
(Navigate=http://{Address}:{Port}/#/pages/user-management/change-password)
password>{password} (SearchBy=id)
newPassword>{newpassword} (SearchBy=id)
confirmNewPassword>{newpassword} (SearchBy=id)
//button[text()=' Save ' or text()=' Sauvegarder ']>(Click) (SearchBy=xpath)
correctement>(Validation) (SearchBy=text)
Password successfully changed>(Validation) (SearchBy=text)

There is a lot going on so lets break it apart.

username>{Username} (SearchBy=name)
password>{Password} (SearchBy=name)
//button[@type="submit"]>(Button) (SearchBy=XPath)
Informations sur le magasin>(Validation) (SearchBy=text)
Store information>(Validation) (SearchBy=text)

The first lines are a copy and paste of our Verify section as we need to login first.

(Wait=1)
(Navigate=http://{Address}:{Port}/#/pages/user-management/change-password)

We use a Wait to break the validation context and then use Navigate to go directly to the profile page where we can change the password.

password>{password} (SearchBy=id)
newPassword>{newpassword} (SearchBy=id)
confirmNewPassword>{newpassword} (SearchBy=id)
//button[text()=' Save ' or text()=' Sauvegarder ']>(Click) (SearchBy=xpath)

After entering the current password and the new password twice we search for a button that has either the text Save or Sauvegarder , which is the Save button for changing the password, because the site may load in either English or French.

correctement>(Validation) (SearchBy=text)
Password successfully changed>(Validation) (SearchBy=text)

Our validations cover the Toast notification that briefly pops up when the password is changed successfully. What isn't visible in our WebFormFieldsFile is a necessary tweak to the ActionTimeout value.

ActionTimeout determines how long to wait for a command or an action to complete. When the command is clicking on a button, the framework expects the button to disappear from the page as part of the action completing. The framework will wait up to the value of ActionTimeout before proceeding with the next command.

Because a successful change does not result in the Save button disappearing and the Toast disappears after 3 or 4 seconds, we have to adjust ActionTimeout in our platform to 2 so that the validations can catch the Toasts before they are gone.

The account's password is rotated after clicking Change in the PVWA.

Reconcile section

Reconcile will be similar to change in that for Shopizer administrators to change the password for another user, they need to:

Login to the Shopizer administrator portal.
Navigate to the Users list.
Find the user in the list and edit their details.
Enter the new password, repeat the new password again, and hit Save.

Like the Change section, our Reconcile section will use http://{Address}:{Port}/#/auth as ReconcileURL and then use Navigate to navigate directly to the Users list, http://{Address}:{Port}/#/pages/user-management/users.

Our Reconcile section results in:

[Reconcile]
username>{reconcileaccount\username} (SearchBy=name)
password>{reconcileaccount\password} (SearchBy=name)
//button[@type="submit"]>(Button) (SearchBy=XPath)
Informations sur le magasin>(Validation) (SearchBy=text)
Store information>(Validation) (SearchBy=text)
(Wait=1)
(Navigate=http://{Address}:{Port}/#/pages/user-management/users)
//tbody//tr//td[3]//div[text()="{username}"]/ancestor::tr/td[5]//i>(Click) (Searchby=XPath)
password>{newpassword} (SearchBy=id)
repeatPassword>{newpassword} (SearchBy=id)
//button[text()=' Save ' or text()=' Sauvegarder ']>(Click) (SearchBy=xpath)
utilisateur>(Validation) (SearchBy=text)
User updated>(Validation) (SearchBy=text)

Like for Change, lets break it down into manageable pieces:

username>{reconcileaccount\username} (SearchBy=name)
password>{reconcileaccount\password} (SearchBy=name)
//button[@type="submit"]>(Button) (SearchBy=XPath)
Informations sur le magasin>(Validation) (SearchBy=text)
Store information>(Validation) (SearchBy=text)

The first lines are a copy and paste of our Verify and Change sections as however we need to login with the reconcile account so we use {reconcileaccount\username} and {reconcileaccount\password} instead of {username} and {password}.

(Wait=1)
(Navigate=http://{Address}:{Port}/#/pages/user-management/users)

Again, we use a Wait to break the validation context and then use Navigate to go directly to the Users list.

//tbody//tr//td[3]//div[text()="{username}"]/ancestor::tr/td[5]//i>(Click) (Searchby=XPath)

This is the most complex part of our CPM plugin.

The XPath expression identifies the Edit user "button" (in actuality it is an image) used to navigate to a user's details by looking for a

element with text content that matches the account's username within the third column () of a table row in the User's list table. Then, it goes up to the parent row () and selects the fifth column (), and finally, it searches for an element anywhere within that fifth column ().

Once found, it clicks on the image to navigate to that user's details page where the password can be updated.

password>{newpassword} (SearchBy=id) repeatPassword>{newpassword} (SearchBy=id) //button[text()=' Save ' or text()=' Sauvegarder ']>(Click) (SearchBy=xpath)

The new password is entered twice and the Save button is clicked.

utilisateur>(Validation) (SearchBy=text) User updated>(Validation) (SearchBy=text)

Similar to changing a password, once we click Save, a toast pops up informing us that the user is updated. As ActionTimeout is already set to 2, the framework spots the notification and the reconciliation should be considered successful.

Let's give it a shot.

Unlike our Verify and Change operations, Reconcile isn't working on the first attempt.

The error message says it cannot press on the Save button, which is different from being unable to find the button. Digging in the Debug logs, we see the following error:

30/12/2023 11:06:47.438 | ERROR -> WebElementWrapper :: Click -> Failed to press on button. Exception Details: OpenQA.Selenium.ElementClickInterceptedException: element click intercepted: Element is not clickable at point (1036, 20). Other element would receive the click:
...
(Session info: MicrosoftEdge=120.0.2210.91)

The significant part is at the end: Other element would receive the click:
...
.

The framework is telling us that it could not click the Save button that it found at an earlier position because now another element is on top of it.

At this point, we need to see how the framework is interacting with the page and unlike with PSM connection components, we cannot easily flip on switch to see the operations (EnableTrace=yes) but we can test the plugin manually without having to invoke an operation in CyberArk.

Troubleshooting Reconcile by testing the CPM plugin

Testing a CPM plugin is already well documented by CyberArk. As we already have our WebFormFieldsFile, we just need to create a user.ini file that will hold the parameters sent to the CPM plugin as part of our testing.

Our user.ini file -- which I've named shopizeruser.ini as there is no requirement around the filename -- looks like:

[targetaccount] username=shopizeradministrator@timschindler.dev newpassword=Password2 password=Password1 safename=Safename foldername=Foldername objectname=Objectname PolicyID=Policyid Address=192.168.178.61 Port=4200 [extrapass3] username=reconcile@timschindler.dev password=Password1 [extrainfo] VerifyURL=http://{Address}:{Port}/#/auth ChangeURL=http://{Address}:{Port}/#/auth ReconcileURL=http://{Address}:{Port}/#/auth WebFormFieldsFile=ShopizerAdministrator.ini RunVerifyAfterChange=No RunVerifyAfterReconcile=No ActionTimout=2 PageLoadTimeout=30 EnforceCertificate=No Debug=Yes BrowserPath=C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe Browser=Edge

The targetaccount section holds parameters related to the account we are testing with, extrapass3 has parameters relating to a reconcile account, and extrainfo has the same parameters and values as defined in our platform.

Note: The values for the safename, foldername, objectname, and policyid parameters are irrelevant for our testing but are required as they are used to form the name of the log file found in the Logs\ThirdParty folder.

With shopizeruser.ini saved to the root of the Password Manager folder, we can test the CPM plugin. Let's start with an operation we know works -- verify:

.\bin\CANetPluginInvoker.exe shopizeruser.ini verifypass CyberArk.Extensions.Plugin.WebApp.dll True

The plugin ends with error code 0, which is the error code returned when the operation is successful.

We know we have a working test setup so let's proceed with reconcile, with the expectation that it will fail:

Reconcile works when we test but not when the CPM invokes the plugin.

It's a nice confirmation that our commands for the CPM plugin are correct but we need to understand why we get different results.

Because the error message we received mentions that another element would receive the click, it's worth opening the User details page in a normal Edge browser and experiencing the page when navigating around it. It is also important to change the browser window size as we are not sure what the window size is when the CPM invokes the plugin.

At the resolution we are connected to the CPM with and the browser maximized, at no point is the Save button not visible. It's only when we put the browser in windowed mode is the Save button not visible after we focus on the Password and Repeat Password fields.

Resizing our RDP window to a smaller resolution and testing the plugin once more, we can reproduce the error that we originally received:

Checking the Debug log, we see:

30/12/2023 12:44:47.066 | ERROR -> WebElementWrapper :: Click -> Failed to press on button. Exception Details: OpenQA.Selenium.ElementClickInterceptedException: element click intercepted: Element is not clickable at point (792, 20). Other element would receive the click:
...

The message details differ slightly but it still complains about another element receiving the click. We can reasonably assume that when the framework focuses on the Password and New Password fields to enter text, the browser window no longer shows the Save button because the floating header is covering it.

How can we move the browser window to the top of the page?

With the framework, there is no command we can define in our WebFormFieldsFile to have the browser scroll to the top of the page. Furthermore, we cannot send keys outside of an input field so we could not simply instruct the framework to hit the Home key.

Fixing the Reconcile operation

The solution to fixing the Reconcile operation isn't elegant.

We know the browser scrolls to an input field when text is being entered so we need to enter text in the First name field. This will result in the browser 'scrolling up' and the header not covering the Save button.

Because we use Shopizer as the First name for our Shopizer users and the value of this user property is irrelevant, we can have commands to send Shopizer to the First name field before trying to click the Save button. The relevant part of our Reconcile section transforms to:

password>{newpassword} (SearchBy=id) repeatPassword>{newpassword} (SearchBy=id) firstname>Shopizer (SearchBy=id) //button[text()=' Save ' or text()=' Sauvegarder ']>(Click) (SearchBy=xpath)

Testing our reconcile operation, we see a success:

But what matters is if it is successful when the CPM invokes the plugin.

Adding the command to populate the First name field so that the browser would have the Save button in view worked!

Our final Reconcile section is:

[Reconcile] username>{reconcileaccount\username} (SearchBy=name) password>{reconcileaccount\password} (SearchBy=name) //button[@type="submit"]>(Button) (SearchBy=XPath) Informations sur le magasin>(Validation) (SearchBy=text) Store information>(Validation) (SearchBy=text) (Wait=1) (Navigate=http://{Address}:{Port}/#/pages/user-management/users) //tbody//tr//td[3]//div[text()="{username}"]/ancestor::tr/td[5]//i>(Click) (Searchby=XPath) password>{newpassword} (SearchBy=id) repeatPassword>{newpassword} (SearchBy=id) firstname>Shopizer (SearchBy=id) //button[text()=' Save ' or text()=' Sauvegarder ']>(Click) (SearchBy=xpath) utilisateur>(Validation) (SearchBy=text) User updated>(Validation) (SearchBy=text)

Next steps

Our CPM plugin for Shopizer administrator users works but there are some improvements that can be made before rolling out to production:

Each operation should include Failure commands so that meaningful error messages can be shown in the PVWA.

We should change EnforceCertificate to Yes after securing all our Shopizer administrator portals with trusted TLS certificates.

Evaluate if using the Web App framework is the best solution. The Shopizer administrator portal leverages REST APIs and our CPM plugin would be less prone to breaking if we call the endpoints directly.

If you are interested in developing your own CPM plugin for Shopizer administrator users, you can use this docker-compose.yml to quickly stand up a containerized Shopizer environment. Check out the README for more information. The same GitHub repository contains the platform files and WebFormFieldsFile.

Generating TOTPs for CyberArk (now Idira) Privileged Session Manager Webapp Connectors using a PreConnect DLL

Tim Schindler — Thu, 12 Oct 2023 15:59:35 GMT

With version 13.0 of CyberArk (now Idira)'s Privileged Session Manager, as part of a Webapp PSM connector, you can inject a time-based one-time password. You can even store the secret for the TOTP in a separate account and link it as a logon account to secure it.

As the latest LTS version is 12.6, many customers are 'stuck' without TOTP injection support but by using a custom PreConnect DLL for Webapp PSM connectors -- a feature introduced in PSM version 12.2-- we can achieve the same functionality and have TOTP injection with PSMs as early as version 12.2

What is a PreConnect DLL?

A PreConnect DLL is a piece of code that is run before a Webapp PSM connector's WebFormFields is executed. Account properties can be passed to the DLL as parameters and return values can be used in the connector's WebFormFields' input fields. CyberArk provides straightforward documentation on how to develop and configure a PreConnect DLL.

Creating the PreConnect DLL

We will keep the PreConnect DLL simple.

It will take a single parameter -- logonaccount_password -- being the TOTP secret, generate a six-digit TOTP using it, and return the TOTP. We will use &MfaCode& to represent the TOTP for use in the WebFormFields.

As version 12.2 of the PSM comes with Otp.NET out of the box in the Components folder, we will re-use the library to do the OTP generating and leave our PreConnect DLL to handle just the receiving of the parameter and the returning of the TOTP. With this, we have only a single file we need to distribute to each PSM.

using CyberArk.PSM.WebAppDispatcher.PreconnectUtils; using OtpNet; using System; using System.Collections.Generic; using System.Net; using System.Security; using System.Text; namespace CyberArk.PSM.GenerateTOTPPreconnect { public class GenerateTOTP : IPreconnectContract { private readonly string TOTP_OUTPUT_PARAMETER = "MFACode"; public Dictionary GetParameters(Dictionary parameters, LogUtils.WriteToLogHandler writeToLogMethod) { string totpCode; try { byte[] totpSecretBytes = Encoding.UTF8.GetBytes(new NetworkCredential(string.Empty, parameters["logonaccount_password"]).Password); writeToLogMethod("Generating TOTP.", Consts.LOG_LEVEL_INFO); var totp = new Totp(totpSecretBytes); totpCode = totp.ComputeTotp(); } catch (KeyNotFoundException ex) { writeToLogMethod(string.Format("Key not found in parameters dictionary: {0}", ex.ToString()), Consts.LOG_LEVEL_ERROR); throw new PreconnectException("Key not found in parameters dictionary"); } catch (Exception ex) { writeToLogMethod(string.Format("Error while generating TOTP: {0}", ex.ToString()), Consts.LOG_LEVEL_ERROR); throw new PreconnectException("Error while generating TOTP"); } writeToLogMethod("Successfully generated TOTP.", Consts.LOG_LEVEL_INFO); return new Dictionary { { TOTP_OUTPUT_PARAMETER, new NetworkCredential(string.Empty, totpCode).SecurePassword } }; } } }
Note: I am not a C# developer so there is room for improvement!
After compiling the DLL and moving it to the Components folder of our PSM, we are ready to leverage it with the Webapp PSM connector.

You can find the code and Visual Studio project, including a compiled DLL, in my GitHub repository below:

https://github.com/aaearon/CyberArk.PSM.GenerateTOTPPreconnect

Configuring the Webapp PSM Connector

💡

This was tested in a PAM self-hosted environment with Vault version 13.0 and PSM 12.2. The 12.2 PSM is the second PSM installed, after one running version 13.0.

With the compiled DLL copied to all PSMs,

Ensure that the Webapp PSM connector has LogonAccount added under Component Parameters > Target Settings > Supported Capabilities.

Add the required parameters PreConnectDllName and PreConnectParameters as described in step three. For PreConnectDllName, specify the name of the DLL (CyberArk.PSM.GenerateTOTPPreconnect.dll if downloaded from my GitHub repository) and for PreConnectParameters provide just logonaccount_password.

In WebFormFields, use &MfaCode& to inject the TOTP in the desired input field.

Connecting with our Webapp PSM connector, the TOTP is generated based on a secret that is stored as the password for the account's linked logon account.

The code and a compiled DLL can be found at https://github.com/aaearon/CyberArk.PSM.GenerateTOTPPreconnect.

Exploring CyberArk (now Idira) Central Policy Manager's WSChains

Tim Schindler — Tue, 29 Aug 2023 19:36:50 GMT

If you have downloaded newer CPM plug-ins from the CyberArk (now Idira) Marketplace, you may have noticed that some deviate from the typical TPC or C#-based implementations and instead leverage a new WSChains DLL with an accompanying XML file. From the plug-in descriptions and the XML file content, WSChains is used when communicating with a REST API.

Scouring the CyberArk technical community and official documentation results in practically zero information about WSChains but studying the XML files of the Marketplace plug-ins utilizing it provides a fair amount of information about this extremely powerful and flexible plug-in type.

First look at WSChains

Like the Secure Web App Framework, WSChains is declarative in nature. It consists of two files: an XML file and the WSChains DLL that acts on it and the contents within.

In the XML, there are four top elements:

GlobalSettings

Requests

Fails

Chains

GlobalSettings

Of the three plug-ins I looked at, only one had a child GlobalSetting elements in GlobalSettings.

In the Cisco ISE Internal Users via API plug-in, the GlobalSetting element and the comment imply that the Address property of the account is used as the base URL for all the subsequently defined Request elements.

Requests

The Requests section is made up of RequestWrapper child elements with each RequestWrapper element representing an API call that can be made as part of a later-defined chain.

Within each RequestWrapper there is a Request element where the relative URL, HTTP method, headers, and request body can be defined as child elements. Seemingly, the base URL and the relative URL are automatically concatenated to create the full URL.

Account properties -- including those of the linked logon and reconcile accounts -- are available to use inside the elements of the Request element.

The Tenable.io - CPM plug-in shows that the body can contain JSON when inside a json element.

When the HTTP POST request sends form data and not json, the Request element can look like:

2 /authentication/users/{{username}} POST application/x-www-form-urlencoded Basic {{username}}:{{pmpass}} oldpassword={{pmpass}}&password={{pmnewpass}}

It is important to remember when sending multiple key-value paris in the body element, that the ampersand must be escaped as &!

Fails

The Fails section is made up of one or more child Fail elements, which are similar to what you would find in a TPC-based plug-in's Process file.

Chains

The Chains section is where it all comes together.

Each CPM operation has its own Chain element consisting of one or more child Link elements. Each Link element has name and request attributes, where the request attribute's value is the name of a Request element to be executed.

Link elements contain child StatusCode elements, each with a value attribute whose value corresponds to a HTTP status code and a next attribute's value that corresponds to the name of a Fail element or a sibling Link element in the same Chain to be executed when the HTTP status code is matched.

In the case that the value for next is END, similar to a Process file, the operation is considered to have been completed successfully.

StatusCode

The StatusCode elements are particularly interesting.

The content returned in a response's body can be parsed and stored as a variable to be used in Request elements, such as in the case where a session token must first be generated and used as part of subsequent requests.

In the case that the next Link depends not just on the HTTP status code returned but also on the content of a response's body, StatusCode elements can have child Parse elements that in turn have child Equals elements. The Equals elements have next attributes that operate the same as they do with StatusCode elements.

Equals elements are also powerful as the Hitachi Administrator Link CS plug-in shows that they can be used to query an asynchronous job's status.

Wrap-up

With an understanding of CPM plug-ins and REST APIs, it is relatively easy to create your own WSChains-based plug-ins using existing plug-ins as references, even without official documentation. However, since it appears to be an internal-use-only framework for CyberArk, it is not suitable for use in a production setting.

Hopefully, CyberArk will choose to support WSChains in the same way they do TPC and the Web App Frameworks, and provide official documentation. WSChains is powerful in its current state and could potentially replace the need for C# or TPC-based plug-ins developed to interact with REST APIs.

Creating a Universal iDRAC CyberArk (now Idira) PSM Connector with if-else Conditional Statements

Tim Schindler — Sun, 09 Jul 2023 20:38:16 GMT

With the version 13.2 release of the Secure Web Application Connectors Framework for the CyberArk (now Idira) Privileged Session Manager, if-else conditional statements can be used in WebForm fields which can be used to build more resilient connectors and allow a single PSM connector to be used for different versions of an application that have different web elements.

Using if-else conditional statements, we will build a PSM connector that will cover both iDRAC versions 8 and 9.

💡

Version 13.2 of the Web Application CPM Plugin Framework offers the same functionality.

The Framework's Conditional Statements

The documentation for conditional statements in the Secure Web Application Connectors Framework is pretty comprehensive, if not a bit overwhelming in parts.

It offers the standard conditional statements found in other programming languages (if, else-if, else) as well as conditional operators (or, and) that can be used between conditions.

It allows the following actions:

Exists - verifies if an element exists on the page with operators true and false.

Count - counts how many elements exist on the page with operators eq, ne, gt,ge, lt, and le.

Placeholder - allows for usage of account properties or Preconnect return values with the single operator eq.

Using a conditional statement in WebFormFields would look something like:

if ( (Login > (Condition) (searchby=text) (exists eq true)) ) username > {Username} password > {password} submit > (Click) if-end

(Condition) is similar to (Click), (Validation), and {accountProperty} in that it informs the Framework on how to interpret the statement.

Creating a Universal iDRAC PSM Connector

A Universal iDRAC PSM connector makes for a simple example of using conditional statements but is also practical as iDRACs can be upgraded without CyberArk administrators being informed and while an account can be assigned a platform with PSM connectors for all possible iDRAC versions, it may not be known to the end user which one to use.

Our Universal iDRAC PSM Connector will work for versions 8 and 9 and due to the nature of iDRACs (being a part of enterprise-grade, physical hardware), you may not have all or any of the versions available so we will use publically exposed (!) iDRAC interfaces:

iDRAC8 -> https://103.215.176.114/

iDRAC9 -> https://www.deos.lr.tudelft.nl

These are not iDRACs we have access to and therefore we do not have valid credentials for but it is enough for us to build our PSM connector. We will know if our Universal iDRAC connector works or not because we will not receive any messages about elements not being found but rather error messages that the login or password is invalid.

Connectors for each version iDRAC version

As the point is to use conditional statements, we will assume we already have a functioning PSM connector for both iDRAC versions and that all we need to do is to combine them into a single one.

If you don't have working PSM connectors for each version, you can import from the Marketplace or use the following WebFormFields (taken from the Marketplace connectors minus validations) when creating your own:

iDRAC8

user > {username}(searchby=id) password > {password}(searchby=id) submit_lbl > (Button)(searchby=id)

iDRAC9

username > {username}(searchby=name) (wait=2) password > (Button)(searchby=name) password > {password}(searchby=name) cux-button > (Button)(searchby=class)

Creating the Universal iDRAC Connector

Before defining our WebFormFields using the if-else statements, we need to determine the logic we will use.

As we are only considering two versions, we need only a single if statement to identify a specific iDRAC version and an else statement to handle the other. We only need to develop a condition for the if statement as else does not take one.

Version 8 of iDRAC uses a lot of JavaScript that updates the DOM and elements may not be found by the Framework so we will build a condition for our if statement that identifies version 9 of iDRAC and if the condition is not met, assume version 8 in our else section.

In addition, versions 8 and 9 of iDRAC have different paths for their login forms (8: /login.html, 9: /restgui/start.html ) so when the if condition is not met and the version is 8, we need to first navigate to the correct path in our else section.

With all the above considered, the WebFormFields for our Universal iDRAC connector looks like:

if ( (Integrated Dell Remote Access Controller 9 > (Condition) (searchby=text) (exists eq true)) ) username > {username}(searchby=name) (wait=2) password > (Button)(searchby=name) password > {password}(searchby=name) cux-button > (Button)(searchby=class) end-if else (Navigate=https://{address}/login.html) user > {username}(searchby=id) password > {password}(searchby=id) submit_lbl > (Button)(searchby=id) end-else

The condition for our if statement is simple: we search for the text Integrated Dell Remote Access Controller 9 and if found, treat it as iDRAC version 9. If not, treat it as version 8 but navigate to the version 8-specific login form first.

After cloning the PSM-DellDRAC9 PSM connector, setting a unique PSM connector ID and appropriate connector display name, RunValidations to No under Client Specific Target Settings and EnforceCertificateValidation to No in Web Form Settings, we can set the above as the WebFormFields value.

Testing the Universal iDRAC Connector

I've onboarded two accounts both with the username and password test but with different addresses: 103.215.176.114 (iDRAC8) and www.deos.lr.tudelft.nl (iDRAC9). Both have the same platform with only our Universal iDRAC Connector.

Connecting to 103.215.176.114, the result is an iDRAC error message indicating the username and password are invalid -- a success in our case!

And with www.deos.lr.tudelft.nl is the same situation.

The conditional statements introduced as part of version 13.2 PSM and CPM web application Frameworks are a nice addition. They provide flexibility that is both beneficial to CyberArk administrators and end users.

💡

Looking for more content around CyberArk PSM Web application connectors? Check out my other post Creating a CyberArk Privileged Session Manager connection component for a web application.

Generating time-based one-time passwords in Privileged Session Manager WebApp Connectors

Tim Schindler — Sat, 10 Jun 2023 20:40:57 GMT

Edit: After conversing with a contact, this requires version 13.0 of the PSM no matter the version of the Secure Web Application Connectors Framework!

Edit 2: This is not officially supported but there exists an enhancement request you can vote on.

Like possible with CyberArk (now Idira) Central Policy Manager plugins for Web applications, you can generate time-based one-time passwords (TOTP) as part of a Privileged Session Manager (PSM) connector.

The Web Application CPM Plugin Framework enables you to generate a TOTP as part of the WebFormFields but this functionality is not advertised for PSM connectors as part of its Secure Web Application Connectors Framework. Despite this, it is still possible.

Furthermore, we can generate a TOTP with a secret stored as a Logon Account instead of storing the secret as part of the PSM connector's WebFormFields.

Generating a TOTP in WebFormFields

Generating a TOTP for a Webapp PSM connector is done the same way as with a CPM plugin for a Webapp. You can copy and paste right out of Example 3 in the Web Form Fields section of the CPM plugin for Web applications documentation:

Launching the PSM connector, the OTP is generated and injected successfully:

But hardcoding the TOTP secret as shown in the example doesn't make sense as the secret will be different for each account and the secret itself should be kept as secure as the password.

The TOTP secret as a Logon Account

The best way to securely store the TOTP secret is to onboard it as an account. Afterward, we will use that account as the Logon Account from which we will retrieve the TOTP secret.

The account the TOTP secret is onboarded with can have any platform, address, username, etc. CPM management can be disabled unless you have a CPM plugin that can rotate the TOTP secret itself.

After linking the TOTP secret as a Logon Account, we can tweak our WebFormFields to generate the TOTP using the Logon Account's password using the {logonaccount\password} property instead of a hardcoded value:

Before we test our new PSM connector, we need to add LogonAccount to the Supported Capabilities under the Target Settings of the PSM connector:

If we neglect to add LogonAccount, then the Logon Account's properties and password will not be available to the PSM connector and we receive an error:

With the WebFormFields updated and the capability added, the PSM connector launches, generates a TOTP based on the secret stored in the password of the Logon Account, and injects it successfully:

We see the username of the Logon Account in the details of the PSM Connect activity:

And we see a Connect in the activities of the Logon Account:

It's not clear why this PSM connector functionality is undocumented by CyberArk. Maybe it is not supported or the documentation was simply overlooked?

Either way, just like generating a TOTP as part of a CPM plugin may be needed in some scenarios, the same goes for PSM connectors.

Solution to Network Connectivity Issues after the Hardening of the CyberArk (now Idira) PAM self-hosted's Vault on Virtualized Hardware

Tim Schindler — Fri, 05 May 2023 17:12:46 GMT

The CyberArk (now Idira) PAM self-hosted Vault is not typically installed on virtualized hardware but some cases make the scenario justified -- one of them being when standing up a lab environment.

When using Hyper-V (and potentially other hypervisors), after hardening the Vault during the installation process, the Vault may lose network connectivity upon subsequent reboot. This results in the inability to establish inbound or outbound connections for all services.

This issue arises because the Windows Firewall rules created by the CyberArk Vault servers do not take effect, necessitating minor adjustments to two registry keys to resolve the problem.

Determining if your Vault is impacted

The easiest way to tell if you are being impacted by this is by checking the Vault machine's network connectivity. This can be done using Test-NetConnection and seeing that outbound connections to all services such as Active Directory and the NTP server fail.

Another way is checking the Firewall pane under the Monitoring section in the Windows Defender Firewall snap-in.

This pane shows the rules in effect and when the Vault services are running, four rules representing the allowing of outbound and inbound Vault-related traffic should be showing. If it is empty, then there are no effective rules and no inbound or outbound traffic is allowed.
A screenshot of the Firewall pane when there is no network connectivity.
The Fix

The fix is straightforward. You simply need to change the two following Windows Defender Firewall-related registry keys from 0 to 1:

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\WindowsFirewall\PublicProfile\AllowLocalIPsecPolicyMerge

HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\WindowsFirewall\PublicProfile\AllowLocalPolicyMerge

Afterward, reboot the Vault machine. Rules will show up in the Firewall pane and network connectivity will be restored.
A screenshot of the Firewall pane when network connectivity is working.
What is happening as part of the hardening?

These keys control if locally defined Windows Defender Firewall rules -- such as the ones created by the Vault when the service starts -- should take effect or not and as part of the Vault hardening, these registry keys are explicitly set to 0 for the Public profile and 1 for the Private profile. It is likely that the intention is for the Vault to take on the Private profile but is for an unknown reason assuming the Public profile.

Even CyberArk is uncertain about why this occurs, as when the Vault hardening process identifies that the Vaults are in AWS, CyberArk explicitly sets both keys to 1.
An exerpt from the HardenWindowsFireWall.ps1 script found in Server-Rls-x.x/Hardening/Scripts

Effortlessly Setting Up A CyberArk (now Idira) PAM Self-Hosted Lab with AutomatedLab

Tim Schindler — Mon, 24 Apr 2023 06:09:30 GMT

If you want to learn CyberArk (now Idira) PAM self-hosted without risking your production environment or experiment with different configurations and test scenarios, setting up a personal lab environment is a great option. However, installing the Vault and its components can take up valuable time. Fortunately, with AutomatedLab and custom CyberArk roles, you can automate the entire process and focus on more important tasks. In this article, you will see how to deploy a CyberArk PAM self-hosted lab environment using AutomatedLab with Active Directory Domain Services.

What is AutomatedLab?

"AutomatedLab is a provisioning solution and framework that lets you deploy complex labs on HyperV and Azure with simple PowerShell scripts. It supports all Windows operating systems from 2008 R2 to 2022, some Linux distributions and various products like AD, Exchange, PKI, IIS, etc." [0]

AutomatedLab will provision virtual machines as well as install and configure the operating system. It provides PowerShell cmdlets by which you can easily install your software.

You can assign roles to machines which will result in services like Active Directory Domain Services, Exchange, or IIS being installed. AutomatedLab also provides methods for creating your own custom roles.

[0] https://github.com/AutomatedLab/AutomatedLab

Deploying CyberArk PAM self-hosted with AutomatedLab

AutomatedLab has a bit of a learning curve but provides documentation to help you get started. The remainder of this article assumes you have successfully installed your first lab as described in the documentation.

In addition, you need the installation archives from the CyberArk Marketplace as well as Operator/Master keys and a license file. These can be placed anywhere on your local machine but I have stored them under $LabSources\CyberArkInstallFiles.

Installing the custom CyberArk roles

Installation is nothing more than copying the folders representing the roles to $LabSources\CustomRoles .

Check out the repository at https://github.com/aaearon/AutomatedLab.CyberArk

https://github.com/aaearon/AutomatedLab.CyberArk

Copy all the folders starting with PAM to $LabSources\CustomRoles . On Windows, unless you changed it during the installation of AutomatedLab, C:\LabSources is used for $LabSources.

The Vault and each component have their folder. PAMCommon contains common functions used among all the roles.

Creating the lab definition

We will use Hyper-V running locally and our lab will consist of four machines:

DC01 - the domain controller with Active Directory Domain Services.

VAULT01 - the CyberArk Vault.

COMP01 - the CyberArk Password Vault Web Access (PVWA) and CyberArk Central Policy Manager (CPM.)

PSM01 - the CyberArk Privileged Session Manager (PSM.)

We will have a switch with a defined address space to segregate our lab from other Hyper-V machines.

First, we create the lab definition, add our switch, and set some defaults so we do not need to repeat ourselves when defining the machines:

New-LabDefinition -Name PAM -DefaultVirtualizationEngine HyperV -VmPath C:\AutomatedLab-VMs Add-LabVirtualNetworkDefinition -Name PAM -AddressSpace 192.168.0.0/24 $PSDefaultParameterValues = @{ 'Add-LabMachineDefinition:OperatingSystem' = 'Windows Server 2019 Datacenter Evaluation (Desktop Experience)' 'Add-LabMachineDefinition:Memory' = 4GB 'Add-LabMachineDefinition:Network' = 'PAM' }

Then define our domain controller with the 'RootDC' role and set the DomainName parameter to a domain name of our choice:

$DC01MachineProperties = @{ Name = 'DC01' Roles = 'RootDC' DomainName = 'acme.corp' } Add-LabMachineDefinition @DC01MachineProperties

Defining the PAM-self hosted machines

For each of our CyberArk PAM self-hosted lab machines, we need to define role properties that will be used when installing the CyberArk software and the properties for the machine itself -- where we will specify the custom role as a PostInstallationActivity.

All the roles require the installation archive path to be defined. The Vault necessitates two folders to be specified – one for each of the Master and Operator key sets. The license in XML format is also needed. Furthermore, we can designate the Master and Administrator account passwords, but if we do not, they will both default to Cyberark1.

The other components require at least the Vault's IP address to be defined, and optionally, the username and password of the user for installation. If neither is specified, the default values of Administrator and Cyberark1 will be used.

All component servers will have DomainName defined, directing AutomatedLab to join them to our domain.

Our Vault is defined with:

$PAMVaultRoleProperties = @{ InstallationArchivePath = 'C:\LabSources\CyberArkInstallFiles\Server-Rls-v13.0.zip' OperatorKeysFolder = 'C:\LabSources\CyberArkInstallFiles\DemoOperatorKeys' MasterKeysFolder = 'C:\LabSources\CyberArkInstallFiles\DemoMasterKeys' LicensePath = 'C:\LabSources\CyberArkInstallFiles\nfr_license.xml' } $PAMVaultRole = Get-LabPostInstallationActivity -CustomRole PAMVault -Properties $PAMVaultRoleProperties $PAMVaultMachineProperties = @{ Name = 'VAULT01' IpAddress = '192.168.0.100' PostInstallationActivity = $PAMVaultRole } Add-LabMachineDefinition @PAMVaultMachineProperties

COMP01 hosts both the PVWA and CPM and is defined by:

Add-LabMachineDefinition @PAMVaultMachineProperties $PAMPvwaRoleProperties = @{ InstallationArchivePath = 'C:\LabSources\CyberArkInstallFiles\Password Vault Web Access-Rls-v13.0.zip' VaultIpAddress = '192.168.0.100' } $PAMPvwaRole = Get-LabPostInstallationActivity -CustomRole PAMPvwa -Properties $PAMPvwaRoleProperties $PAMCpmRoleProperties = @{ InstallationArchivePath = 'C:\LabSources\CyberArkInstallFiles\Central Policy Manager-Rls-v13.0.zip' VaultIpAddress = '192.168.0.100' } $PAMCpmRole = Get-LabPostInstallationActivity -CustomRole PAMCpm -Properties $PAMCpmRoleProperties $COMP01MachineParameters = @{ Name = 'COMP01' DomainName = 'acme.corp' PostInstallationActivity = $PAMPvwaRole,$PAMCpmRole } Add-LabMachineDefinition @COMP01MachineParameters

Finally, our PSM is:

$PAMPsmRoleProperties = @{ InstallationArchivePath = 'C:\LabSources\CyberArkInstallFiles\Privileged Session Manager-Rls-v13.0.1.zip' VaultIpAddress = '192.168.0.100' } $PAMPsmRole = Get-LabPostInstallationActivity -CustomRole PAMPsm -Properties $PAMPsmRoleProperties $PSM01MachineParameters = @{ Name = 'PSM01' DomainName = 'acme.corp' PostInstallationActivity = $PAMPsmRole } Add-LabMachineDefinition @PSM01MachineParameters

Installing the lab

As the Vault hardening removes the ability for AutomatedLab to communicate with the machine over WinRM, we cannot simply invoke Install-Lab rather we need to install in stages.

First, we do everything but the post-installation activities:

Install-Lab -BaseImages -NetworkSwitches -VMs -Domains -NoValidation

After, we have use Invoke-LabCommand to install CyberArk PAM-self hosted installed.

# Perform the Vault installation. The hardening part of the installation will kill the ability to use WinRM so it will timeout and throw an error. We want to ignore that error. Invoke-LabCommand -ComputerName VAULT01 -PostInstallationActivity -ActivityName 'CyberArk Vault installation' -ErrorAction SilentlyContinue Invoke-LabCommand -ComputerName COMP01 -PostInstallationActivity -ActivityName 'CyberArk Pvwa and CPM installation' Invoke-LabCommand -ComputerName PSM01 -PostInstallationActivity -ActivityName 'CyberArk PSM installation'

The entire lab definition can be found below:

https://gist.github.com/aaearon/d584a11f09efa21091785b1bf129c6a9

A little less than two hours later (depending on the resources allocated to each machine), a complete CyberArk PAM self-hosted lab environment is fully operational and ready for use.

Elevating our lab

We did little more than the bare minimum to get our lab environment off the ground however with all the different functionality AutomatedLab provides plus the ability to write custom roles, we can take it much further.

We could have our domain controller serve as a certificate authority and then issue a certificate and configure the HTTPS binding in IIS. We could also execute a script that creates safes and onboard accounts after the installation of the components.

Installing and configuring Utimaco's SecurityServer Simulator

Tim Schindler — Thu, 03 Nov 2022 20:06:58 GMT

In a previous blog where I integrated a HSM with the CyberArk (now Idira) Vault using Utimaco's SecurityServer simulator, I did not touch on at all how to install and configure it but this is arguably the toughest part of the integration.

Though the documentation from Utimaco is comprehensive, the sheer amount can be overwhelming so in this post I want to show how the Utimaco SecurityServer simulator can installed and configured. We will then configure Utimaco PKCS#11 provider DLL on a client, using a CyberArk Vault as an example.

Installation and configuration of the SecurityServer simulator

Requirements

You will need:

The Utimaco SecurityServer simulator. You can only download it after you register for an account in the Utimaco portal. As the simulator is meant for commercial evaluation, you should sign up with your business contact details. You may need to wait up to 48 hours until the account is activated.

A Windows server with a Java Development Kit installed with the Zip archive downloaded from the Utimaco portal.

Installing the simulator

Installing the simulator is as simple as using the .msi provided in the Zip archive but be sure to choose a Complete setup type otherwise the tool needed to initialize the HSM slots will not be installed.

After installation, you will have three new icons on your desktop:

We only need the CryptoServer Simulator and the PKCS#11 CryptoServer Administration applications. If you do not see the PKCS#11 CryptoServer Administration then you did not install the Complete setup type as part of the installation.

Configuration the provider and initializing the slots

The simulator is our HSM. Double clicking the CryptoServer Simulator icon starts the simulator and this needs to run continuously. The PKCS#11 CryptoServer Administration tool is only used to initialize the token and slots.

Just like a client uses the PKCS#11 provider DLL to communicate with the HSM, so does the PKCS#11 CryptoServer Administration tool. The PKCS#11 provider DLL uses a configuration file -- cs_pkcs11_R3.cfg -- to know the location of the HSM. We need to change the cs_pkcs11_R3.cfg to point to the simulator running on the same server plus enable some logging for easier troubleshooting.

The provider DLL knows where to find cs_pkcs11_R3.cfg based on the value of the system environmental variable CS_PKCS11_R3_CFG. This is automatically created as part of the installation and the default location is C:\ProgramData\Utimaco\PKCS11_R3\cs_pkcs11_R3.cfg.

Opening C:\ProgramData\Utimaco\PKCS11_R3\cs_pkcs11_R3.cfg we want to change three things: enable the PKCS#11 provider to write logs in the same directory as the configuration file, change the log level to INFO, and define the address of the ‘device’ the provider connects to -- in our case, the locally-running simulator.

Change the following settings to have the following values, uncommenting the settings as necessary:

Logpath = C:/ProgramData/Utimaco/PKCS11_R3

Logging = 3

Under a [CryptoServer] section that is uncommented, set Device = 3001@127.0.0.1

Save the changes.

With the the CryptoServer Simulator running, open the PKCS#11 CryptoServer Administration tool. Ten slots should be shown on the left.

Click on the first slot, Login/Logout at the top, and click Login Generic. Type in ADMIN as the username, select Keyfile as the authentication token, and define C:\Program Files\Utimaco\SecurityServer\Administration\ADMIN.key as the Keyfile. Do not define a password.

NOTE: In the latest versions of the SecurityServer, complexity requirements are being enforced for the SO and User PIN. Use strong passwords as the PINs for both the SO and User.

Once logged in, the token needs to be initialized by setting the SO PIN. Click Slot Management at the top, expand Init Token, and set the SO PIN and Confirm SO PIN and click Init Token. In the Status pane, you will see the message Slot 0000 0000: Token initialized.

With the slot's token initialized, we then login with the SO PIN and initialize (another, separate PIN.)

As you can only be logged in with a single user at a time, hit Login/Logout, click the Logout All button, click on Login/Logout once more, and login with the SO PIN. After successful login with the SO PIN, click on Slot Management at the top and expand Init PIN. Define both Normal User PIN and Confirm Normal User PIN. This is the PIN that will be secured with CAVaultManager SecureSecretFiles when defining HSMPinCode in the dbparm.ini and used clients to access the Server key. Non-CyberArk Vault clients will also use a PIN when needing to access keys.

After initializing the PIN, log out once more by clicking Login/Logout and then clicking Logout All. Login as User by clicking on the Login/Logout button, expanding Login User, and logging in with the Normal User PIN you defined previously.

After successful authentication with the Normal User PIN, the Object Management button at top is clickable. Any cryptographic information (symmetric/asymmetric keys) will show up as an object in this pane but as we just initialized the PIN, it is empty.

We can generate our own keys by clicking Generate. In the case of integration with the CyberArk Vault, we will do it as part of CAVaultManger GenerateKeyOnHSM /ServerKey.

Setting up the CS PKCS#11 Provider on the CyberArk Vault

In order to use the HSM from another client, we need the PKCS#11 provider DLL and the configuration file -- just like we have it on the server running the simulator.

With the CyberArk Vault as our client, in order to use the SecurityServer simulator as our HSM, we need to:

On the server running the SecurityServer simulator, allow inbound traffic on port 3001/tcp.

Copy cs_pkcs11_R3.dll to the Vault. This can be found in the Zip archive in the Software\Windows\x86-64\Crypto_APIs\PKCS11_R3\lib\ folder.

Copy cs_pkcs11_R3.cfg to the Vault.

Update cs_pkcs11_R3.cfg to reflect the proper paths on the Vault.

Define the system environmental variable CS_PKCS11_R3_CFG.

Configuring the CyberArk Vault

On the server running the SecurityServer simulator, add a rule that allows inbound traffic over 3001/tcp. In a PowerShell opened as Administrator, run New-NetFirewallRule -DisplayName "SecurityServerSimulator" -Direction Inbound -LocalPort 3001 -Protocol TCP -Action Allow.

On the Vault server, create the HSM folder under C:\Program Files (x86)\PrivateArk\Server. Copy cs_pkcs11_R3.dll, and cs_pkcs11_R3.cfg into C:\Program Files (x86)\PrivateArk\Server\HSM.

Open cs_pkcs11_R3.cfg and set the following:

Logpath = C:\Program Files (x86)\PrivateArk\Server\HSM

Under a [CryptoServer] section that is uncommented, set Device = 3001@

SlotCount = 1

Now edit the system environmental variables and add CS_PKCS11_R3_CFG with a value of C:\Program Files (x86)\PrivateArk\Server\HSM\cs_pkcs11_R3.cfg

At this point the PKCS#11 provider is configured. The easiest way to test connectivity is trying to generate a Server key with CAVaultManger GenerateKeyOnHSM /ServerKey after performing the initial Vault configurations for HSM integration.

Closing notes

If you are having trouble, do not forget to look in the cs_pkcs11_R3.log file. Together with error messages returned by the client using the PKCS#11 provider DLL, the log contains useful information.

Utimaco is the only vendor I have seen provide something like the SecurityServer simulator to the public. If you are in search of a HSM solution, consider checking out their offerings as a way to say 'thanks.'

Creating a CyberArk (now Idira) Privileged Session Manager connection component for a web application

Tim Schindler — Tue, 01 Nov 2022 18:14:51 GMT

After helping someone in a CyberArk Discord that I frequent with correctly identifying a button as part of a Privileged Session Manager connection component for a web application, I realized that I've never made a connection component or a CPM plugin for a web application myself.

From experience I have from 'previous lives', I have already worked with the Document Object Model (DOM) and Selenium -- both used by the PSM and CPM web application frameworks -- so the fundamentals are already familiar but I've never done it in the context of CyberArk (now Idira).

In this post, we will develop a Microsoft Edge-based PSM connection component for the web application Shopizer -- an open-source eCommerce platform that is available as a container so we can focus our time on developing the connection component.

💡

This is article is one in a series of articles around creating PSM connectors and CPM plug-ins.

Understanding what a PSM connection component for a web app is doing

Before we start thinking about creating a PSM connection component for a web app, it makes sense to understand how they work.

The CyberArk Web applications for PSM documentation does not explicitly explain how they function but reviewing the page gives us important hints: the need to have the proper (Web)Driver for the browser and the syntax and examples for defining Web Form Fields.

What is a driver and what does it do? A (Web)Driver ".. is a remote control interface that enables introspection and control of user agents. It provides a platform- and language-neutral wire protocol as a way for out-of-process programs to remotely instruct the behavior of web browsers. Provided is a set of interfaces to discover and manipulate DOM elements in web documents and to control the behavior of a user agent."

We can extract another hint from the description of a WebDriver: the DOM or Document Object Model. The DOM "... is a programming API for HTML and XML documents. It defines the logical structure of documents and the way a document is accessed and manipulated. ... With the Document Object Model, programmers can create and build documents, navigate their structure, and add, modify, or delete elements and content."

From the information we gleaned, we can assume that the PSM uses a WebDriver to identify and interact with elements in the DOM. It will inject usernames and passwords into input fields, click buttons to submit forms, and read resulting DOMs to perform validations to determine if the login was successful or not. The connection component's WebFormFields property is how we instruct the PSM to identify elements and what to do with them.

Identifying elements in the DOM

We can tell CyberArk to identify elements in the DOM several ways: using the name, class, text, or XPath. XPath can be used to navigate through elements and attributes in an XML document. XPath can also be used with an HTML document.

Explaining HTML elements and XPath is outside the scope of this post however let us quickly see the multiple ways we can identify elements in a connection component's WebFormsField using its syntax with a simplified DOM:

Identifying elements with id, name, class, or text in WebFormsField

With the above DOM as an example, we can identify the elements in our WebFormsField effortlessly. Identifying by the elements' id and name, it would look something like:

i_username > {Username} (SearchBy=id) password > {Password} (SearchBy=name) s_login > (Click) (SearchBy=id)

We can mix and match what we search by. Depending on the elements, it may lend itself to one way or the other.

Identifying elements with XPath in WebFormsField

We can accomplish the same as we did purely with XPath:

//*^[@id="i_username"^] > {Username} (SearchBy=XPath) //*/form/input^[2^] > {Password} (SearchBy=XPath) //*^[@class="submit_button"^] > (Click) (SearchBy=XPath)

With XPath we can use the same identifiers as we previously had and do not need to specify element attributes at all like in the case of the password field -- which could be useful if an element we are working at has attributes with dynamic values or cannot be defined uniquely by it's attributes.

Note: The way we defined the XPath for the password field is not recommended. It is simply selecting the second input element on the page. If the page is later updated and an additional input is added before the two existing ones (for example: a search field in a header of the page) then our password field is no longer the second input but rather the third.

Creating our Shopizer connection component

With a better understanding of how PSM connection components for web apps work and how we can identify DOM elements in the WebFormsField, the creation of the connection component should be pretty straightforward.

Shopizer offers two portals: a customer-facing one and one meant for administrators. We will create a connection component for the Shopizer administrator portal.

With our local Shopizer containers running, a local test Shopizer administrator user account created, and our PSM equipped with Edge, the Edge Driver and the latest Secure Web Application Connectors Framework, let's create our first connection component for a web application.

Creating a new web connection component

As instructed in the Configuration part of the documentation, we first need to clone the PSM-WebAppSample and change the Id of it. We will set PSM-ShopizerAdministrator as the new Id.

Under Target Settings, we set the ClientApp to Edge.

Under Client Specific, we add new two new parameters: BrowserPath and DriverPath. As we are using Edge, the paths are C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe and C:\Program Files (x86)\CyberArk\PSM\Components\msedgedriver.exe respectively. While in the area, we set the value for RunValidations to No -- this is something we should do later.

Then under Web Form Settings, we adjust the LogonURL to http://{address}:4200/#/auth. The administrator web interface runs on port 4200 and it would be better to add Port as a connection parameter but as we are not creating this for production use, we hard-code the port.

Defining our WebFormFields

The WebFormFields is the key to this connection component working. As we did in the example above, we need to define the username, password, and submit button elements.

But unlike in our example, we do not have the DOM clearly laid out in front of us so how can we identify those elements? This is when your favorite browser's Developer Tools come into play. I am using Firefox but other browsers have similar functionality.

Assuming the Shopizer containers are running on your local machine, navigate to http://localhost:4200/#/auth and you should be greeted with the login page.

From there, we can right-click the username field and click Inspect to be brought directly to the element in the DOM. When we mouse over the element in the Inspector tool, the element is highlighted above in the web page.

Now it is just about how we want to identify the input element that represents the Username field.

In addition to XPath, as the input element has an id, class, and name attributes defined, we can use whatever we want in our WebFormFields to identify the element. But which way is best?

Ultimately, it depends. As we are typically developing connection components for web applications we are not developing ourselves, we are at the will of the application's developers.

An element that has a class defined one day may not have it defined the next day if the site goes through a re-vamp. Some attributes may have values with random digits at the end giving us the clue that the attribute's values are not explicitly set and may be auto-generated with each deployment. XPath can be pretty resilient if our path expression does not include too many nodes.

For the username field of our Shopizer login page, we have a few options. The id and name values both look explicitly set and feel as if they have a relatively low risk of changing in the future. Let's go with name.

Back in our WebFormFields we update it to identify the field to input the account username using the attribute name with the value of username. We use Inspect in Firefox to identify the password and submit buttons and define those too, resulting in:

Now after adding our new connection component to the platform assigned to the onboarded test Shopizer administrator account we created, let's give it a whirl.

It's in French -- this seems to be the default language for Shopizer -- but it worked! We are logged into the Shopizer administrator portal.

Troubleshooting

Depending on the complexity of the web app, you may run into problems. With complex DOMs -- especially ones that involve iFrames -- it may not be straightforward when identifying elements.

Thankfully, though not well documented at all, you can enable very verbose trace levels in .ClientDispatcher.log that are created in the Logs\Components\ folder. To do so, set TraceLevels=1,2 in the PVWA options under Privileged Session Management > General Settings > Client Connection Settings.

After a restart of the PSM service or waiting for the PVWA configuration refresh on the PSM, you will see the log file with helpful content such as:

Another helpful debugging 'tool' is setting EnableTrace to Yes. This shows the entire login process that the PSM obscures.

Improving our connection component

We developed our connection component 'quick and dirty' which is fine for a lab environment but nothing we would want to introduce into production. We have a lot of areas to improve on:

We should not hardcode the Port in the LogonURL, rather allow this to be defined at the account level to handle the case where the Shopizer administrator portal is running on a port other than the default.

We have no validations. After we login, we should use an element unique to a successfully-authenticated Shopizer administrator portal to inform the PSM the login was successful.

Similar to a validation of a successful login, we should set a validation for login failures. Like many applications, Shopizer returns an error message when a login was unsuccessful and we can use that element to clue the PSM into the fact that it failed.

With the above points addressed, the connection component moves closer to being production-ready and with our new knowledge of how a connection component for a web app works, we likely have a lot of the information we need to develop a CPM plugin for a web app, too.

Using a hardware security module to secure the CyberArk (now Idira) Vault's Server Key

Tim Schindler — Mon, 31 Oct 2022 10:42:23 GMT

The CyberArk (now Idira) Vault allows for the Server key to be stored in a hardware security module (HSM). The Server key is used as a key-encryption-key so it is appropriate to use a HSM as they provide the highest level of protection for the Server key.

HSM integration with CyberArk is actually well-documented. CyberArk provides information on loading an existing Server key into an HSM as well as generating a new Server key directly in the HSM but as with anything, hands-on experience goes a long way to demystifying a concept.

In a lab environment, we will generate a new Server key directly on the HSM then perform a re-key of the Vault using that key. The SecurityServer simulator from Utimaco will be used as our HSM.

What are hardware security modules and what do they offer?

Hardware security modules provide a secure location to store cryptographic information (typically keys) as well as provide a secure place to perform cryptographic operations (such as using the keys to encrypt or decrypt objects.) HSMs are commonly physical, hardware devices that have tamper proof protections. They can be in the form of a network appliance or a PCI device.

Applications interact with HSMs through different APIs -- the PKCS#11 standard being one of the most common and the one CyberArk uses to talk to HSMs.

As HSMs store cryptographic information and provide the place to perform operations involving that information, applications need a constant connection to the HSM. When the connection is broken, the ability to use the cryptographic information is lost. The same applies to CyberArk: when it cannot connect to the HSM, critical functionality such as retrieving credentials from the Vault is does not work!

The Vault and it's encryption keys

Before doing any sort of HSM integration, it first makes sense to understand the encryption keys used by the Vault to secure Vault objects. The CyberArk Technical Community has a nice article on all the Vault keys and their purpose. We will focus on the ones used to encrypt Vault objects.

There are three 'tiers' of keys in the Vault encryption hierarchy: Vault keys, safe keys, and object keys. Each object is encrypted by a unique key specific to the object, object keys are encrypted by a safe key, and Vault keys encrypt the safe keys.

There are two types of Vault keys when referring to the encryption of Vault objects: the Recovery key (really it is a pair as the Recovery 'key' is asymmetric) and the Server key (symmetric.) Safe keys are encrypted and decrypted by both leading to two copies of the encrypted safe key. The Server key must be accessible to the Vault in order for it to start.

Only the Server key can be stored on a HSM and the private key for the Recovery key pair should be kept securely offline.

The CyberArk Technical Community has an excellent knowledge article regarding hierarchical key management.

HSM and CyberArk integration

To better understand HSM integration with CyberArk, in our CyberArk 12.6 Primary-DR lab environment running on Windows Server 2019, we will:

Perform initial Vault configurations

Generate a new Server key directly on the HSM

Re-encrypt all the Vault data and metadata with the new Server key on the HSM

The Server key will be generated on the HSM from the DR Vault. We will then re-encrypt the DR Vault before re-encrypting the Primary vault.

Note: In the lab environment, the HSM (Utimaco's SecurityServer simulator) is already configured with a slot initialized to store our new Server key. On both the Primary and DR Vaults, the PKCS#11 provider used to communicate with the HSM is configured and functional. See this blog post dedicated to configuring Ultimaco's SecurityServer simulator.

Initial Vault configurations

On both Vaults, we need to directly edit the dbparm.ini to define the path to our HSM's PKCS11 provider DLL via the PKCS11ProviderPath as well as use AllowNonStandardFWAddresses to allow outbound communication to the HSM -- the last part being optional if your Vaults are version 12.6 and running Windows Server 2019 due to hardening changes.

Afterwards we need to define the PIN that the Vault will use to access the slot the Server key is located using CAVaultManager.exe SecureSecretFiles /SecretType HSM /Secret 1111

Opening dbparm.ini we see HSMPinCode defined with the value of our encrypted PIN.

Now we are in a position to generate the Server key.

Generate the new Server key

We will generate the Server key on the DR Vault. This only needs to be done once as the same Server key will be used for both the Primary and DR Vault. There is no requirement to generate it on one Vault versus the other -- we just do it on the DR Vault as we will re-encrypt the DR Vault with it first.

After stopping the CyberArk Vault Disaster Recovery service, we use CAVaultManger GenerateKeyOnHSM /ServerKey to generate the Server key. CAVaultManager uses the parameters defined in dbparm.ini to communicate with the HSM. Successful Server key generation lets you know the Vault will be able to as well.

We need to note down the KeyID as we will use it later.

Re-encrypt the DR Vault with our new Server key

At this point it makes sense to ensure the Primary Vault's Vault services have been stopped as after the re-encryption on the DR Vault the two will be using different Server keys and replication between them will not be possible until both are using the same, new Server key.

With the Master private key at the location defined at RecoveryPrvKey in the dbparm.ini, ChangeServerKeys is used to re-encrypt the Vault data.

We run ChangeServerKeys.exe C:\Keys\DemoOperatorKeys\ C:\Keys\DemoOperatorKeys\VaultEmergency.pass HSM#1. C:\Keys\DemoOperatorKeys\ refers to the location of the keys that will be used to re-encrypt the Vault -- including our existing Master private key. The Server key used will be the one on the HSM, however.

The changing of keys may take awhile depending on the specifications of your Vault and the HSM as well as the amount of data in the Vault. As this process creates new safe and object keys based on the new Server key, this could take days.

Like the output says, now we need to update the ServerKey parameter in the dbparm.ini to refer to the HSM.

And quickly start the Vault services.

The final verification is logging into PrivateArk.

After logging out of PrivateArk and stopping the Vault services on the DR Vault, we are ready to move on to the Primary.

Re-encrypt the Primary Vault with our new Server key

We run the exact same ChangeServerKeys command on the Primary Vault.

Like the DR Vault, we change the dbparm.ini on the Primary Vault to have the ServerKey parameter point to HSM#1, start the Vault services, and login.

At this point, we can start the CyberArk Vault Disaster Recovery service on the DR Vault and observe successful replication.

Vault behavior when the HSM is unavailable

Our Server key being stored on a HSM increases the security posture of our environment but also introduces complexity and another point of failure. We should ensure we have a good understanding on what the behavior of the Vault is when the HSM is unavailable and what we need to do.

We should consider the following:

Vault behavior when the Vault tries to start and the HSM is unavailable.

Vault behavior when the Vault is already running and the HSM is unavailable.

What needs to be done on the Vault side when the HSM becomes available.

Starting the Vault with the HSM unavailable

This is a pretty straightforward case. As the Server key is needed to start the Vault services and the Server key lives on the HSM, the HSM being unavailable prevents the Vault services from starting.

Looking in the log file for the PKCS#11 provider we are using, we see the device cannot be found which could give us a hint where to start our troubleshooting.

An already running Vault with the HSM unavailable

If the Vault is already running and the HSM becomes unavailable, the Vault does not become completely unavailable. Some functionality -- user authentication, report generation, user management, changing of safe memberships, etc. -- may still work and give a false sense of security.

What is guaranteed not to work is the retrieval of Vault objects (credentials.) When attempting to retrieve a credential while the HSM is unavailable, there will be an error about the Safe key being incorrect.

Peeking in the trace logs, we can even see the decryption failing:

Based on our understanding of the Vault's encryption hierarchy, this makes sense. We know each Vault object is encrypted by an individual key, which is then encrypted by a Safe key where the Safe key is then encrypted by the Server key. With the Server key being unavailable, the Safe key cannot be decrypted in order to be used to decrypt the object's key.

In our current setup, the HSM becoming available again is enough for the Vault to assume normal operations. The first retrieval of a credential takes awhile -- the Vault is probably doing whatever it needs to do to recover from an HSM outage -- while the subsequent ones go much quicker:

Recovery of the Vault with ReconnectHSMOnErrorCodes

In the case the Vault does not assume normal operations automatically after the HSM becoming unavailable again then we can use ReconnectHSMOnErrorCodes to have the Vault reconnect to the HSM.

DBPARM.sample.ini has an example of the values for ReconnectHSMOnErrorCodes (ReconnectHSMOnErrorCodes=48,50,179) but there does not seem to be a complete list of possible HSM error codes however we can see them in the trace logs with debug levels CRYPT(1,2).

Wrapping it up

HSMs provide the the best security for our Vaults' Server key by providing a secure location for them to reside and to perform cryptographic operations with. Generating a Server key and re-encrypting our Vaults' data is easy with the available documentation from CyberArk.

Once the integration with an HSM is complete, together with an understanding of the Vault's encryption hierarchy, Vault debug levels and trace logs give us important insight into when and how the Vault interacts with it, demystifying both HSMs and HSM integration with CyberArk's Vault.

Are you planning to or have you already integrated your CyberArk environment with a HSM? What HSM vendor are you using? Provide some feedback in the comments.

Managing objects in CyberArk (now Idira) with PowerShell Desired State Configuration

Tim Schindler — Mon, 19 Sep 2022 10:07:14 GMT

PowerShell Desired State Configuration (DSC) is a tool similar to Ansible, Puppet, and Chef that enables declaratively setting how an environment is configured. PowerShell DSC can be used to ensure resources such as environmental variables, software, files, Active Directory users, and application-specific settings are either present or absent through the use of configurations.

A few of the most obvious, CyberArk (now Idira)-related use cases for PowerShell DSC include making sure that the required software is installed after importing a Universal Connector package or after deploying a CPM plugin as part of a continuous deployment pipeline. It can be used to manage the rules within an PSMConfigureAppLocker.xml file or invoke the automated installation or upgrade processes for components.

Though with the CyberArkDsc module, PowerShell DSC can also be used to manage CyberArk Vault objects such as safes, safe memberships, and accounts.

https://github.com/aaearon/CyberArkDsc

Why use PowerShell DSC to manage Vault objects?

Using PowerShell DSC to manage Vault objects makes sense for the same reasons configuration management makes sense for infrastructure.

Combat configuration drift - Ensure accounts have the intended configurations long after they are created -- particularly useful in a test or lab environment where experimentation may lead to false settings and thus unintended consequences.

Idempotency - Safes, safe memberships, and accounts can be created and configured using scripts, the Password Upload Utility, or PACLI but running executing these when the objects already exist may produce different results or fail entirely.

Treat Vault objects as code - Like with infrastructure as code, Vault objects can be defined in configurations and these configurations can be stored in version control, be included in continuous deployment pipelines, or reviewed by other team members.

Included in PowerShell 5.1 and above - PowerShell DSC is included in PowerShell 5.1 and above which absolves the need to install to install more software, such as Python when dealing with Ansible.

Using PowerShell DSC and CyberArkDsc to populate an empty Vault

One of the leading use cases for using PowerShell DSC together with CyberArk is that it can be used to quickly create accounts on target systems and onboard them into the Vault.

In the following section, a DSC configuration will be used to create Active Directory users, safes to store them in the Vault, assign the needed safe memberships to make the accounts accessible, and finally onboard the users into the safes with the needed reconcile accounts.

Note: For the sake of brevity, knowledge of PowerShell DSC concepts and terminology such as resources and configurations is assumed.

Installing the CyberArkDsc and ActiveDirectoryDsc modules

Installing the ActiveDirectoryDsc module is straightforward as it can be installed from the PowerShell Gallery like any other PowerShell module. The ActiveDirectoryDsc wiki has information on getting started.

CyberArkDsc is not available in the PowerShell Gallery and needs to be manually installed. As by default configurations are executed under the SYSTEM account, CyberArkDsc needs to, along with it's dependency psPAS, exist in a non-user-specific folder location defined in $env:PSModulePath -- for example: $env:ProgramFiles\WindowsPowerShell\Modules.

We can use Get-DscResource to ensure that both DSC modules are installed correctly.

PS > Get-DscResource | Where-Object {$_.Name -eq 'CYA_Account' -or $_.Name -eq 'ADUser'} ImplementedAs Name ModuleName Version Properties ------------- ---- ---------- ------- ---------- PowerShell ADUser ActiveDirectoryDsc 6.2.0 {DomainName, UserName, AccountNotDelegated, All... PowerShell CYA_Account CyberArkDsc 0.0.2 {Address, AuthenticationType, Credential, Ensur... PS >

Writing our configuration

We will keep our DSC configuration relatively short and sweet. We already have LDAP integration configured for the Vault, a user that is in both Vault Admins and Domain Admins groups, and platforms created in the Vault that the accounts will be onboarded with so this just leaves us with:

Creating two users in Active Directory, one to be used as a reconcile account.

Adding the reconcile account as a member to the Domain Admins group in Active Directory so it can reconcile the other.

Creating two safes to store the accounts in: one for the reconcile account, the other.

Assigning a membership to each safe enabling an existing Active Directory group named CyberArk Administrators full permissions to the reconcile account safe and another, existing Active Directory group named Windows Administrators with list, retrieve, and use accounts to the other.

Onboarding both accounts: The reconcile account will be onboarded with the password defined when creating it in Active Directory and the other account will be onboarded with the reconcile account defined as a linked account.

Defining the base of our configuration

Each DSC resource in CyberArkDsc takes at least four parameters: PvwaUrl, AuthenticationType, Credential, and SkipCertificateCheck. We define these as parameters our configuration accepts. In addition, we define an InitialPassword parameter that will be used when creating the reconcile account's Active Directory user and when onboarding into CyberArk.

We will have a single node -- localhost -- as we will run the DSC configuration on the same machine we compile it on. Our resources will go inside it.

Both Credential and InitialPassword will be passed PSCredential objects. As this is a lab environment I will not go through the effort to have the credentials be properly secured in the resulting, compiled configuration so I set a switch to allow the passwords to be stored plaintext.

Configuration CreateLab_Configuration { param( $PvwaUrl, $AuthenticationType, $Credential, $SkipCertificateCheck, $InitialPassword ) Node localhost { } } $ConfigData = @{ AllNodes = @( @{ NodeName = 'localhost' PSDscAllowDomainUser = $true PSDscAllowPlainTextPassword = $true } ) } $VaultCredential = New-Object System.Management.Automation.PSCredential('allison', (ConvertTo-SecureString 'Password!' -AsPlainText -Force)) $Password = New-Object System.Management.Automation.PSCredential('banana', (ConvertTo-SecureString 'P@$$W0RD12345' -AsPlainText -Force)) $ConfigurationParameters = @{ ConfigurationData = $ConfigData PvwaUrl = 'https://192.168.137.101' AuthenticationType = 'LDAP' SkipCertificateCheck = $true Credential = $VaultCredential InitialPassword = $Password } CreateLab_Configuration @ConfigurationParameters

Defining resources

Our configuration will have nine resources. Using documentation, our familiarity with managing Vault objects via psPAS and Get-DscResource -Name ADUser,ADGroup,CYA_Account,CYA_Safe,CYA_SafeMember -Syntax to know how to define each resource, our complete configuration looks like:

Configuration CreateLab_Configuration { param( $PvwaUrl, $AuthenticationType, $Credential, $SkipCertificateCheck, $InitialPassword ) Import-DscResource -ModuleName 'ActiveDirectoryDsc' -Name 'ADUser', 'ADGroup' Import-DscResource -ModuleName 'CyberArkDsc' Node 'localhost' { ADUser 'iosharp\windowsReconcile50' { Ensure = 'Present' UserName = 'windowsReconcile50' DomainName = 'iosharp.dev' Password = $InitialPassword PasswordNeverResets = $true Credential = $Credential } ADUser 'iosharp\windowsAdmin50' { Ensure = 'Present' UserName = 'windowsAdmin50' DomainName = 'iosharp.dev' Password = $InitialPassword PasswordNeverResets = $true Credential = $Credential } ADGroup 'Domain Admins' { Ensure = 'Present' GroupName = 'Domain Admins' MembersToInclude = 'windowsReconcile50' Credential = $Credential DependsOn = '[ADUser]iosharp\windowsReconcile50' } CYA_Safe 'WinRec50' { Ensure = 'Present' SafeName = 'WinRec50' Description = 'Windows Reconcile Safe' ManagingCPM = 'PasswordManager' NumberOfDaysRetention = '1' PvwaUrl = $PvwaUrl AuthenticationType = $AuthenticationType SkipCertificateCheck = $SkipCertificateCheck Credential = $Credential } CYA_Safe 'WinAdmins50' { Ensure = 'Present' SafeName = 'WinAdmins50' Description = 'Windows Admins Safe' ManagingCPM = 'PasswordManager' NumberOfDaysRetention = '1' PvwaUrl = $PvwaUrl AuthenticationType = $AuthenticationType SkipCertificateCheck = $SkipCertificateCheck Credential = $Credential } CYA_SafeMember 'WinRec\CyberArk Administrators' { Ensure = 'Present' SafeName = 'WinRec50' MemberName = 'CyberArk Administrators' SearchIn = 'iosharp.dev' UseAccounts = $true RetrieveAccounts = $true ListAccounts = $true AddAccounts = $true UpdateAccountContent = $true UpdateAccountProperties = $true InitiateCPMAccountManagementOperations = $true SpecifyNextAccountContent = $true RenameAccounts = $true DeleteAccounts = $true UnlockAccounts = $true ManageSafe = $true ManageSafeMembers = $true BackupSafe = $true ViewAuditLog = $true ViewSafeMembers = $true AccessWithoutConfirmation = $true CreateFolders = $true DeleteFolders = $true MoveAccountsAndFolders = $true RequestsAuthorizationLevel1 = $true RequestsAuthorizationLevel2 = $false PvwaUrl = $PvwaUrl AuthenticationType = $AuthenticationType SkipCertificateCheck = $SkipCertificateCheck Credential = $Credential DependsOn = '[CYA_Safe]WinRec50' } CYA_SafeMember 'WinAdmins\Windows Administrators' { Ensure = 'Present' SafeName = 'WinAdmins50' MemberName = 'Windows Administrators' SearchIn = 'iosharp.dev' UseAccounts = $true RetrieveAccounts = $true ListAccounts = $true ViewSafeMembers = $true PvwaUrl = $PvwaUrl AuthenticationType = $AuthenticationType SkipCertificateCheck = $SkipCertificateCheck Credential = $Credential DependsOn = '[CYA_Safe]WinAdmins50' } CYA_Account 'windowsReconcile50' { Ensure = 'Present' UserName = 'windowsReconcile50' Address = 'iosharp.dev' PlatformId = 'ioSHARPWindowsDomainReconcileAccount' SafeName = 'WinRec50' Name = 'windowsReconcile50@iosharp.dev' Password = $InitialPassword PvwaUrl = 'https://192.168.137.101' AuthenticationType = 'LDAP' SkipCertificateCheck = $true Credential = $VaultCredential DependsOn = '[CYA_Safe]WinRec50' } CYA_Account 'windowsAdmin50' { Ensure = 'Present' UserName = 'windowsAdmin50' Address = 'iosharp.dev' PlatformId = 'ioSHARPWindowsDomainAccount' SafeName = 'WinAdmins50' Name = 'windowsAdmin50@iosharp.dev' ReconcileAccount = @{ Name = 'windowsReconcile50@iosharp.dev' Safe = 'WinRec50' Folder = 'root' } PvwaUrl = 'https://192.168.137.101' AuthenticationType = 'LDAP' SkipCertificateCheck = $true Credential = $VaultCredential DependsOn = '[CYA_Account]windowsReconcile50', '[CYA_Safe]WinAdmins50' } } } $ConfigData = @{ AllNodes = @( @{ NodeName = 'localhost' PSDscAllowDomainUser = $true PSDscAllowPlainTextPassword = $true } ) } $VaultCredential = New-Object System.Management.Automation.PSCredential('allison', (ConvertTo-SecureString 'Password!' -AsPlainText -Force)) $Password = New-Object System.Management.Automation.PSCredential('banana', (ConvertTo-SecureString 'P@$$W0RD12345' -AsPlainText -Force)) $ConfigurationParameters = @{ ConfigurationData = $ConfigData PvwaUrl = 'https://192.168.137.101' AuthenticationType = 'LDAP' SkipCertificateCheck = $true Credential = $VaultCredential InitialPassword = $Password } CreateLab_Configuration @ConfigurationParameters

Compiling and running our configuration

We dot source our saved configuration file to compile it:

PS C:\Users\Tim.IOSHARP\Desktop\DSC> . .\CreateLab.ps1 Directory: C:\Users\Tim.IOSHARP\Desktop\DSC\CreateLab_Configuration Mode LastWriteTime Length Name ---- ------------- ------ ---- -a---- 9/16/2022 7:43 PM 15494 localhost.mof PS C:\Users\Tim.IOSHARP\Desktop\DSC>

We can use Test-DscConfiguration to see if our resources in their desired state.

PS C:\Users\Tim.IOSHARP\Desktop\DSC> Test-DscConfiguration .\CreateLab_Configuration\ | Select-Object -ExpandProperty ResourcesNotInDesiredState | Select-Object ResourceId ResourceId ---------- [ADUser]iosharp\windowsReconcile50 [ADUser]iosharp\windowsAdmin50 [ADGroup]Domain Admins [CYA_Safe]WinRec50 [CYA_Safe]WinAdmins50 [CYA_SafeMember]WinRec\CyberArk Administrators [CYA_SafeMember]WinAdmins\Windows Administrators [CYA_Account]windowsReconcile50 [CYA_Account]windowsAdmin50 PS C:\Users\Tim.IOSHARP\Desktop\DSC>

No surprise, none of our resources are as they do not even exist.

From there, we can start the configuration using Start-DscConfiguration and passing the Wait parameter.

PS C:\Users\Tim.IOSHARP\Desktop\DSC> Start-DscConfiguration .\CreateLab_Configuration\ -Wait PS C:\Users\Tim.IOSHARP\Desktop\DSC>

We do not get any output from Start-DscConfiguration which is exactly what we want. Running Test-DscConfiguration but this time looking at resources in the desired state shows us all nine of them.

PS C:\Users\Tim.IOSHARP\Desktop\DSC> Test-DscConfiguration .\CreateLab_Configuration\ | Select-Object -ExpandProperty ResourcesInDesiredState | Select-Object ResourceId ResourceId ---------- [ADUser]iosharp\windowsReconcile50 [ADUser]iosharp\windowsAdmin50 [ADGroup]Domain Admins [CYA_Safe]WinRec50 [CYA_Safe]WinAdmins50 [CYA_SafeMember]WinRec\CyberArk Administrators [CYA_SafeMember]WinAdmins\Windows Administrators [CYA_Account]windowsReconcile50 [CYA_Account]windowsAdmin50 PS C:\Users\Tim.IOSHARP\Desktop\DSC>

But what matters is what we see in CyberArk. Browsing to the PVWA, we see our two newly onboarded accounts in our newly created safes. windowsAdmin50 has windowsReconcile50 defined as it's reconcile just as we defined in our DSC configuration.

Changing existing Vault objects with PowerShell DSC

Our configuration creates the objects when they are missing but what happens if they already exist in some form?

We make small changes to our accounts -- change their names, unlink the reconcile account, different platform -- with the goal that running Start-DscConfiguration will adjust them back to what is defined in our configuration.

A quick Test-DscConfiguration shows that DSC knows the accounts in the Vault are incorrect based on the configuration:

PS C:\Users\Tim.IOSHARP\Desktop\DSC> Test-DscConfiguration .\CreateLab_Configuration\ | Select-Object -ExpandProperty ResourcesNotInDesiredState | Select-Object ResourceId ResourceId ---------- [CYA_Account]windowsReconcile50 [CYA_Account]windowsAdmin50 PS C:\Users\Tim.IOSHARP\Desktop\DSC>

Simply running Start-DscConfiguration 'silently' adjusts the accounts in CyberArk to their desired configuration.

There we have it: the same configuration we used to create the Vault objects was used to correct them without having to do any changes.

The CyberArkDsc module can be found on GitHub. Currently it only has resources for safes, safe memberships, and accounts. Pull requests are welcome!

https://github.com/aaearon/CyberArkDsc

SecretManagement.CyberArk: An extension for the SecretManagement PowerShell module

Tim Schindler — Mon, 22 Aug 2022 13:00:00 GMT

The SecretManagement PowerShell module provides a common interface to interact with a wide array of secret vaults enabled through SecretManagement extensions. There are a handful of SecretManagement extensions, including for Azure KeyVault, KeePass, and LastPass.

The SecretManagement module supports having multiple secret vault types registered at the same time -- allowing you to access secrets in Azure KeyVault, KeePass, LastPass, etc. through the same cmdlets. The same secrets vault type can also be registered multiple times, allowing you to easily retrieve secrets from multiple KeePass databases.

SecretManagement.CyberArk is a SecretManagement extension that brings support to connecting to CyberArk (now Idira) using the Central Provider/Central Credential Provider (using CredentialRetriever) or REST API (using psPAS).

Using SecretManagement.CyberArk with the Central Credential Provider

The SecretManagement.CyberArk README provides the information we need to start working with the module.

The first step is to install and import both the Microsoft.PowerShell.SecretManagement and SecretManagement.CyberArk modules.

PS > Install-Module Microsoft.PowerShell.SecretManagement PS > Install-Module SecretManagement.CyberArk PS > Import-Module Microsoft.PowerShell.SecretManagement PS > Import-Module SecretManagement.CyberArk

Next is to register a secret vault with the module SecretManagement.CyberArk. We give it a name to refer to when working with a secret and specify that we want to use the Central Credential Provider and pass the relevant details in VaultParameters.

PS > $VaultParameters = @{ ConnectionType = 'CentralCredentialProvider' AppID = 'windowsScript' URL = 'https://comp01' SkipCertificateCheck = $true } PS > Register-SecretVault -Name CyberArk -ModuleName SecretManagement.CyberArk -VaultParameters $VaultParameters

Get-SecretInfo returns information about a secret: the name of the account, the data type of the secret when using Get-Secret and the SecretManagement Vault the secret is found in.

It also returns a Metadata property that shows information in CyberArk about the account.

PS > Get-SecretInfo -VaultName CyberArk -Filter windowsAdmin01 Name Type VaultName ---- ---- --------- Operating System-ioSHARPWindowsDomainAccount-iosharp.lab-windowsAdmin01 SecureString CyberArk PS > Get-SecretInfo -VaultName CyberArk -Filter windowsAdmin01 | Select-Object -ExpandProperty Metadata Key Value --- ----- PolicyID ioSHARPWindowsDomainAccount Folder Root LastTask ChangeTask CreationMethod PVWA Safe Windows CPMStatus success UserName windowsAdmin01 PasswordChangeInProcess False Address 192.168.0.10 Name Operating System-ioSHARPWindowsDomainAccount-iosharp.lab-windowsAdmin01 LastSuccessVerification 1660836856 SequenceID 116 LastSuccessReconciliation 1660834970 DeviceType Operating System LastSuccessChange 1660923260 RetriesCount -1 PS >

We retrieve the secret in a similar way through Get-Secret, returned as a SecureString.

PS > $Secret = Get-Secret -VaultName CyberArk -Name 'Operating System-ioSHARPWindowsDomainAccount-iosharp.lab-windowsAdmin01' PS > $Secret.GetType() IsPublic IsSerial Name BaseType -------- -------- ---- -------- True False SecureString System.Object PS > $Secret | ConvertFrom-SecureString -AsPlainText UTyga8!ZOEDvPhQsrwlI(xjb2K$c PS >

The SecretManagement module supports adding and removing secrets through Set-Secret and Remove-Secret however as the Central Provider and Central Credential Provider do not support this functionality, these two cmdlets are only implemented for the REST ConnectionType.

Working with multiple SecretManagement.CyberArk secret vaults

We can register multiple SecretManagement.CyberArk secret vaults. These secret vaults can use different connection types.

PS > $TestEnvironmentVaultParameters = @{ ConnectionType = 'CredentialProvider' AppID = 'linuxApp' ClientPath = 'C:\Path\To\CLIPasswordSDK.exe' } PS > Register-SecretVault -Name CyberArkTest -ModuleName SecretManagement.CyberArk -VaultParameters $TestEnvironmentVaultParameters PS > $ProductionEnvironmentVaultParameters = @{ ConnectionType = 'CentralCredentialProvider' AppID = 'windowsScript' URL = 'https://comp01' } PS > Register-SecretVault -Name CyberArkProduction -ModuleName SecretManagement.CyberArk -VaultParameters $ProductionEnvironmentVaultParameters

Executing Get-SecretVault shows us all our register secret vaults and their names so we can specify which Vault to get the secret from.

PS > Get-SecretVault Name ModuleName IsDefaultVault ---- ---------- -------------- CyberArkProduction SecretManagement.CyberArk False CyberArkTest SecretManagement.CyberArk False PS >

Conclusion

The SecretManagement.CyberArk extension allows you to retrieve secrets from multiple CyberArk environments using different connection methods through the same cmdlets provided by the SecretManagement module.

SecretManagement.CyberArk can be found in the PowerShell Gallery. The code is available on GitHub.

https://github.com/aaearon/SecretManagement.CyberArk

Continuous Deployment of CyberArk (now Idira) Platforms using GitHub Actions

Tim Schindler — Wed, 13 Jul 2022 16:43:55 GMT

Management of CyberArk (now Idira) platforms can be difficult. They have numerous settings that are integral to how privileged accounts are used and their credentials managed. An incorrect configuration could leave an account in an unusable state.

Platforms are typically staged across multiple environments. Whereas the REST API allows for easy export and import of platforms from environments, it lacks endpoints to comprehensively manage their settings once they are created.

Furthermore, it is common to have platforms recorded outside of CyberArk -- the XML-based visual representation of them in the PVWA often shoehorned inside a table in Confluence or Word -- with the necessity to keep the documentation accurate, which can be challenging in an environment with a large number of platforms and people who can make edits.

Being that platforms are represented by both an XML and INI file we can ditch the awkward tables and stash them in a Git repository while reaping the rewards version control offers. From there, when it is a new platform, we can create a platform package with the help of the New-PASExtensions module and import it into a CyberArk environment but how can we streamline this in the case where the platform already exists in the Vault and thus importing is not an option?

We could still keep manually updating platforms in CyberArk based on what is documented in our Git repository but this is still time-intensive and error prone. After understanding how the importing of platform packages works, we may be able to use part of or all of the functionality in order to develop a continuous deployment workflow.

Platforms and platform packages

The first step to seeing how our continuous deployment could leverage CyberArk's platform deployment mechanism is understanding how the entire import platform package process works. And that starts with understanding a platform package.

According to the documentation a platform package is straightforward: it is just a Zip file that contains a CPM policy file (an INI file with the name Policy-$PolicyId.ini), a PVWA settings file (a XML file with the name Policy-$PolicyId.xml), and (optionally) the CPM plugin files (executables, PowerShell scripts, TPC prompts and process files, or a combination.)

Once created, the Zip file is imported through the PVWA and the CPM plugin files are copied automatically to the CPM.

Whereas the documentation does a good job telling us what to do, it does not provide enough details on how it is exactly done and this "how" is what we need to really understand if we want or can use this process in our continuous deployment.

Understanding the import process of a platform's package

The best place to start is downloading a plugin from the CyberArk Marketplace, importing it, and observing the behavior.

I chose the RealVNC Service Mode as it is a TPC plugin using a PowerShell script with the classic prompts and process files, in addition to the required CPM policy and PVWA setting files, so we will see how those files are distributed.

Upon importing the platform package for the first time we see that it is done successfully and is added under Imported Platforms:

With the knowledge we already have from programmatically changing CyberArk platform properties using PowerShell we know that the PVWA settings file Policy-RealVNC.xml was likely merged into Policies.xml in the PVWAConfig safe and the CPM policy file Policy-RealVNC.ini was dropped into the Policies folder of the PasswordManagerShared safe but what happened to the CPM plugin files made up of the PowerShell script and the prompts and process files?

As with anything to do with platforms and their files, the first place to look should be the PasswordManagerShared safe.

First lets take a quick look to make sure the CPM policy file Policy-RealVNC.ini landed in the Policies folder like we expected:

What is interesting to note is that as part of the import process the CPM policy file was renamed from Policy-RealVNC.ini to Policy-RealVNCServiceMode.ini, which is the PolicyID of the plugin.

Moving on to finding the CPM plugins files, inside the same safe we see the ImportedPlatforms folder and a sub-folder named Policy-RealVNCServiceMode that contains our CPM plugin files. The sub-folder seems to be named based off the convention Policy-$PolicyId, which is something we should keep in mind for our CD workflow:

Step 3 of creating a platform package explains that the CPM plugin files will be dumped into the bin folder of the CPM. Checking the folder and pm.log we see that is the case:

The distribution of CPM plugin files is similar to that of what we see with the deployment of Universal Connectors on multiple PSMs. This works for the first deployment of a new platform but what about consecutive deployments for updated platforms or CPM plugin files?

Updating platforms and CPM plugin files

Although we manually imported the platform through the PVWA we know that an API endpoint to do the same exists. The best idea when needing to update a platform's settings or it's CPM plugin files would be to make the needed updates, re-create the platform package, and use the import platform package API endpoint to deploy the platform.

Unfortunately trying to import a package for a platform that shares the PolicyID or PolicyName of an existing platform is not possible. As the platform of an account is defined through an account property you could first delete the platform and then import it but at the very least any Master Policy exceptions are also deleted and must be manually re-created.

We saw that as part of the import process our platform's CPM plugin files were stashed in their own sub-folder under ImportedPlatforms in the PasswordManagerShared safe where they then ended up in the bin folder of the CPM. What happens when we upload a new version of an existing file directly into the Vault?

Uploading new CPM plugin files into the Vault

For the CPM plugin files we should download one of the files out of PrivateArk, make an inconsequential change to it, and observe the results.

Inside of RealVNCProcess.ini we add a new comment to the file:

and then upload it back into the safe:

After waiting some minutes, searching the pm.log for any log entry comes up with nothing:

and inside the bin folder RealVNCProcess.ini is still the same:

Out of curiosity, lets see what happens when we restart the CPM service:

Kicking the CPM service worked but having to restart the CPM service every time we want to deploy new CPM plugin files -- which could be countless times per day with our continuous deployment workflow -- across all our CPMs probably isn't what we want.

Looking again in pm.log when the plugin files are downloaded successfully, we always seeing a proceeding log message about a new policy being found. Can we trigger the downloading of new CPM plugin files by changing a superfluous value in the CPM policy file Policy-RealVNCServiceMode.ini and uploading that?

Manually triggering the download of new CPM plugin files by the CPM

We make another change to RealVNCProcess.ini:

and upload it to the Vault. We then head to the Policies folder in the PasswordManagerShared safe and download a copy of Policy-RealVNCServiceMode.ini.

But before we make any change to Policy-RealVNCServiceMode.ini lets simply upload that same exact copy back into the safe. This is worth trying because more than likely CyberArk is not comparing the content of files to know if there is an update rather just looking at the Last Modified date of the file -- which changes when the file is uploaded again.

Simply re-uploading Policy-RealVNCServiceMode.ini and thus it having a new Last Modified date was enough to trigger CyberArk to distribute our updated CPM plugin file.

Putting it all together in a continuous deployment workflow

I am using GitHub Actions with a self-hosted Windows runner as PACLI/PoShPACLI needs to communicate with the Vault.

FYI: All the code can be found in the below repos.

https://github.com/aaearon/cyberark-extensions-continuous-deployment

https://github.com/aaearon/Deploy-PASExtensions

Setting up the GitHub repository

Still staying with the RealVNC CPM plugin, we create a repository on GitHub to store the platform files. All the plugin's files are stored in a folder that matches the platform's ID -- RealVNCServiceMode in our case -- in the platforms folder.

PS C:\Users\Tim\Projects\cyberark-extensions-continuous-delivery> Get-ChildItem Directory: C:\Users\Tim\Projects\cyberark-extensions-continuous-delivery Mode LastWriteTime Length Name ---- ------------- ------ ---- d---- 7/13/2022 4:27 PM .git d---- 6/19/2022 1:40 PM .github d---- 7/10/2022 11:19 AM platforms -a--- 7/13/2022 4:27 PM 621 deploy.ps1 -a--- 6/19/2022 12:16 PM 1070 LICENSE -a--- 7/13/2022 4:27 PM 1950 README.md PS C:\Users\Tim\Projects\cyberark-extensions-continuous-delivery> Get-ChildItem .\platforms\RealVNCServiceMode\ Directory: C:\Users\Tim\Projects\cyberark-extensions-continuous-delivery\platforms\RealVNCServiceMode Mode LastWriteTime Length Name ---- ------------- ------ ---- -a--- 7/10/2022 11:19 AM 235 LICENSE -a--- 7/13/2022 4:27 PM 6642 Policy-RealVNCServiceMode.ini -a--- 7/11/2022 2:40 PM 673 Policy-RealVNCServiceMode.xml -a--- 7/10/2022 11:19 AM 2566 realvnc.ps1 -a--- 7/10/2022 11:19 AM 4177 RealVNCProcess.ini -a--- 7/10/2022 11:19 AM 1432 RealVNCPrompts.ini

Note that Policy-RealVNC.ini was renamed to Policy-RealVNCServiceMode.ini!

Inside the root of the repository, I have created a small PowerShell script deploy.ps1 to be invoked by our GitHub Actions workflow that calls a custom function I wrote -- Update-PASPlatformFiles.ps1, part of the Deploy-PASExtensions PowerShell module -- that does the heavy lifting and uses PACLI/PoShPACLI to get the platform files -- regardless if they were changed -- in the Vault as part of the continuous deployment workflow:

param ( [string]$PACLIClientPath, [PSCredential]$VaultCredential, [string]$VaultAddress, [string]$BasePlatformFolder ) Import-Module PoShPACLI Import-Module .\deploy-pasextensions\Deploy-PASExtensions\Deploy-PASExtensions.psd1 Set-PVConfiguration -ClientPath $PACLIClientPath Start-PVPacli New-PVVaultDefinition -vault $VaultAddress -address $VaultAddress -ErrorAction SilentlyContinue Connect-PVVault -user $($VaultCredential.UserName) -password $($VaultCredential.Password) (Get-ChildItem $BasePlatformFolder).FullName | Update-PASPlatformFiles Disconnect-PVVault Stop-PVPacli

The script is pretty simple. It imports the needed modules, builds up the PACLI session, and for each platform in $BasePlatformFolder updates it's files and then tears down the PACLI session.

The last step is adding the password of the user you will connect to the Vault via PACLI with as a secret to the GitHub repository. Use VAULT_PASSWORD as the secret name otherwise you will need to adapt the workflow below with the correct secret name.

Installing the GitHub Actions self-hosted runner

Using the GitHub Actions documentation we install a self-hosted runner and add it to the repository we created above.

It also makes sense to make sure PACLI and PoShPACLI are installed and working on the same machine the self-hosted runner is on.

Creating the GitHub Actions workflow

This is where the magic happens. In the workflow is where we define our jobs, the steps they consist of, and which actions trigger the workflow. Whereas GitHub Actions is extremely versatile and would allow us to trigger a workflow upon a merged pull request, for now we keep it simple and have the workflow trigger on each push.

name: Deploy platform files to CyberArk on: push: workflow_dispatch: jobs: deploy: name: Deploy platform files to the Vault runs-on: self-hosted steps: - uses: actions/checkout@v3 - name: Checkout Deploy-PASExtensions repo uses: actions/checkout@v3 with: repository: aaearon/deploy-pasextensions path: deploy-pasextensions - name: Uploads platform files to the Vault shell: powershell run: | $env:HOMEDRIVE = "" $env:HomePath = "${{ runner.temp }}" $VaultCredential = New-Object System.Management.Automation.PSCredential ("Administrator", (ConvertTo-SecureString -String ${{ secrets.VAULT_PASSWORD }} -AsPlainText -Force)) .\deploy.ps1 -PACLIClientPath 'C:\PACLI\Pacli.exe' -VaultCredential $VaultCredential -VaultAddress 192.168.0.50 -BasePlatformFolder '.\platforms\'

We need to overwrite $env:HOMEDRIVE and $env:HomePath as these do not seem to exist inside the runner and thus thrown an error. A PSCredential object is created with the VAULT_PASSWORD secret stored in the GitHub repository and is passed as an argument to deploy.ps1 along with other arguments that match my target CyberArk environment.

At this point we can push everything to GitHub.

Making a change that will kick off the continuous deployment workflow

With the GitHub repository set up, self-hosted runner installed, and GitHub Actions workflow defined we are ready to see if it works.

We make two small changes to the RealVNCServiceMode platform, commit them, and then push: Change ImmediateInterval to 6 and add the Banana property as optional.

PS C:\Users\Tim\Projects\cyberark-extensions-continuous-delivery> git show 36aa8ad2ec2fb1afe12c6f34bcbb794551ab0ae4 commit 36aa8ad2ec2fb1afe12c6f34bcbb794551ab0ae4 (HEAD -> main, origin/main, origin/HEAD) Author: Tim Schindler Date: Wed Jul 13 17:01:15 2022 +0200 add banana property and change immediateinterval diff --git a/platforms/RealVNCServiceMode/Policy-RealVNCServiceMode.ini b/platforms/RealVNCServiceMode/Policy-RealVNCServiceMode.ini index 93fbcbf..b78d453 100644 --- a/platforms/RealVNCServiceMode/Policy-RealVNCServiceMode.ini +++ b/platforms/RealVNCServiceMode/Policy-RealVNCServiceMode.ini @@ -2,7 +2,7 @@ PolicyID=RealVNCServiceMode ;Mandatory, unique identifier for PolicyName=Real VNC Service Mode via PowerShell ;Short description SearchForUsages=No ;Expected values: yes/no PolicyType=Regular ;Expected values: regular, usage, group -ImmediateInterval=5 ;In minutes +ImmediateInterval=6 ;In minutes Interval=1440 ;In minutes MaxConcurrentConnections=3 ;Expected values: integer AllowedSafes=.* ;Regular expression of Safes pattern. Must be set if Reconciliation is enabled. diff --git a/platforms/RealVNCServiceMode/Policy-RealVNCServiceMode.xml b/platforms/RealVNCServiceMode/Policy-RealVNCServiceMode.xml index 549e6ba..0097911 100644 --- a/platforms/RealVNCServiceMode/Policy-RealVNCServiceMode.xml +++ b/platforms/RealVNCServiceMode/Policy-RealVNCServiceMode.xml @@ -7,7 +7,7 @@ - + PS C:\Users\Tim\Projects\cyberark-extensions-continuous-delivery>

Upon seeing the successful execution of the GitHub Actions workflow lets look in the PVWA to see if our changes are reflected:

Without having to do anything -- no restart of a CPM, no iisreset on the PVWAs, no changes to any platform in an attempt to force a 'refresh' of Policies.xml -- the changes in our commit take effect.

Going forward

We can see that it is (relatively) straightforward to have continuous delivery for platforms, including their CPM plugin files. Leveraging how imported platforms' plugin files are distributed automatically to all CPMs means we only have a single target system for our workflow -- the Vault.

Improving the GitHub Actions workflow to be more flexible and only deploying those files that have changed or only deploy upon a closed pull request and not using the Vault Administrator account when connecting via PACLI can lead to mature continuous deployment for all of our platforms -- enabling the ability to treat them "as code."

Load balancing CyberArk (now Idira) Privileged Session Manager for SSH (PSMP) with HAProxy and Expect

Tim Schindler — Mon, 14 Feb 2022 15:37:23 GMT

This is one post in a series focusing on load balancing various CyberArk (now Idira) components using HAProxy with a focus on application/service-based health checking.

Load balancing CyberArk Privileged Session Manager for SSH (often referred to as PSMP) with application-based health checking using HAProxy is a bit more complicated than doing the same with CyberArk Privileged Web Access Vault or the PSM HTML5 Gateway but still accomplishable with the advanced features HAProxy provides.

Using Expect we can build a simple script that tests the functionality of the PSMP by proxying a connection to a target server and ensuring we get the expected password prompt, providing a known, good credential to authenticate with, and ultimately seeing the expected bash prompt of the target server. We configure can HAProxy to execute it as part of it's health check in order to achieve application-based health checking.

As we are already familiar with HAProxy from load balancing other CyberArk components we can focus our energy on building the script used as part of the health check.

What is Expect?

Expect is '... a tool for automating interactive applications such as telnet, ftp, passwd, fsck, rlogin, tip, etc.' It '... "talks" to other interactive programs according to a script. Following the script, Expect knows what can be expected from a program and what the correct response should be. An interpreted language provides branching and high-level control structures to direct the dialogue. In addition, the user can take control and interact directly when desired, afterward returning control to the script.' [0] In fact, Expect is used by CyberArk Central Password Manager's Terminal Plugin Controller as part of some of it's plugins.

Put it more straightforwardly: we can use Expect and it's commands to spawn a ssh connection to a target server using a PSMP, expect the login prompt as a response, and send keystrokes to complete the authentication. If we see the bash prompt of the target server then we can reasonably assume the PSMP service is running and healthy

[0] https://www.tcl.tk/man/expect5.31/expect.1.html

Formulating our health check

Before introducing Expect lets first lets login manually to see what prompts we get. This will help us outline the steps we need to take in a script that will operate as our health check.

tim@docker01:~$ ssh tim@admin01@linux01.iosharp.lab@192.168.121 The authenticity of host '192.168.0.121 (192.168.0.121)' can't be established. ECDSA key fingerprint is SHA256:FuAf0xfImgbvwwn3l4wTvwNUc6K7ZpdQsZmJrjUb2jc. Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added '192.168.0.121' (ECDSA) to the list of known hosts. Vault Password: This session is being recorded Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-97-generic x86_64) Last login: Tue Feb 8 14:25:40 2022 from 192.168.0.122 admin01@linux01:~$ exit logout Connection to linux01.iosharp.lab closed. Connection to 192.168.0.121 closed. tim@docker01:~$

What is important to take note is what we are prompted for so that we can include them as expect commands in a script and where we need to provide input to move the process along so we can provide input in send commands.

If we run through the login process again, keeping in mind how it would look with Expect commands, we:

spawn a SSH connection to the target server proxying through a PSMP

expect to be prompted when connecting to a machine for the first time

send the word 'yes' to continue the process where we

expect to be prompted for our credentials in order to authenticate to the Vault

send the credential for the Vault user we are authenticating

expect the prompt of the target system we are connecting to

send exit in order to close the connection

We have our steps, now we just need to put it together in a script.

Creating our Expect script

After installing Expect using apt (my Docker host is running Ubuntu - depending on what your OS is your package manager may differ) we create login.check and store it in our home directory.

tim@docker01:~$ touch login.expect tim@docker01:~$ vi login.expect

With the steps above as a basis our login.check should look like

#!/usr/bin/expect spawn ssh tim@admin01@linux01.iosharp.lab@192.168.0.121 expect "yes/no" send "yes\r" expect "Vault Password:" send "Password1234!\r" expect "admin01@linux01:~$ " send "exit\r"

At the top, similar to a Bash script, we define a shebang so that when we run the script the OS knows to invoke it with Expect.

Afterwards, the first thing we do is spawn a connection to a target server using the PSMP whose health we want to validate. In anticipation of an unknown SSH host key, we expect "yes\no" and then send "yes\r". We do not have to define the exact string we expect to receive as Expect allows us to use regular expressions in our expect commands.

When using send we need to remember to provide a carriage return using \r when we 'emulating' pressing Enter. We do the same thing with the prompt Vault Password: where we provide our password.

Finally we expect to see the bash prompt of the target server we are connecting to and then end the script by ending the SSH connection.

Executing our script we can see the results:

tim@docker01:~$ ./login.expect spawn ssh tim@admin01@linux01.iosharp.lab@192.168.0.121 Vault Password: Vault Password:tim@docker01:~$

It doesn't work. The SSH connection to the PSMP was successful as we received the Vault Password: prompt however we were not able to authenticate and eventually the timeout was reached and the script quit.

Because we were never prompted to accept the unknown SSH host key our script got hung up at that step. We can ensure we are never prompted regarding an unknown SSH host key by passing -o StrictHostKeyChecking=no as an argument to our SSH command and we can remove the expect/send from our script:

#!/usr/bin/expect spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121 expect "Vault Password:" send "Password1234!\r" expect "admin01@linux01:~$ " send "exit\r"

We execute our script once more:

tim@docker01:~$ ./login.expect spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121 Vault Password: This session is being recorded Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-97-generic x86_64) Last login: Tue Feb 8 15:47:06 2022 from 192.168.0.122 admin01@linux01:~$ tim@docker01:~$

It works! We have a functioning basis for our health check but we should extend our script a bit more to handle some known behavior when the PSMP is not working.

Handling known, non-working behavior in Expect

Expect allows us to throw exit codes. These exit codes can be used by other processes, like HAProxy, to know if the script completed fine or with an error.

Our script already handles the situation where the PSMP is working fine but we should attempt to capture the behavior of a non-functional PSMP and account for it in our script. We can then throw an exit code other than 0, which signifies the login process and thus our script encountered a problem.

Like we did with crafting our Expect script with a working PSMP, lets see what the behavior of a PSMP is when the service is shut off.

With the PSMP service disabled, lets try to connect to a target server:

tim@docker01:~$ ssh tim@admin01@linux01.iosharp.lab@192.168.0.121 Vault Password: Vault Password: Vault Password: tim@admin01@linux01.iosharp.lab@192.168.0.121: Permission denied (publickey,keyboard-interactive). tim@docker01:~$

Despite putting in the correct password, we were prompted two more times and finally the connection closed.

Lets update our script to reflect this behavior when the PSMP service is not running:

#!/usr/bin/expect spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121 expect { "Vault Password:" { send "Password1234!\r" } } expect { "admin01@linux01:~$ " { exit 0 } "Vault Password:" { exit 1 } }

We refactor our script a bit when extending it to handle the case where the PSMP service is not running by organizing the statements into expect blocks.

Within our first expect block we handle the first prompt -- Vault Password: and send our password when prompted.

In the second expect block we handle what comes after the first prompt. If we receive the target server's expected bash prompt -- our success case -- we close with exit 0. If instead we receive Vault Password: once more, indicating that the login process with our known, good password failed, then we throw exit 1. We need these error codes as HAProxy will use them to determine the health of the server.

With the PSMP service still stopped, lets invoke our script and check the error code:

tim@docker01:~$ ./login.expect spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121 Vault Password: Vault Password:tim@docker01:~$ echo $? 1 tim@docker01:~$

We use $? to get the exit code of the last command executed. As any exit code other than 0 indicates an error, this is exactly what we want. For good measure we should re-enable the PSMP service, run our script, and check the exit code:

tim@docker01:~$ ./login.expect spawn ssh -o StrictHostKeyChecking=no tim@admin01@linux01.iosharp.lab@192.168.0.121 Vault Password: Last failed login: Tue Feb 8 20:57:39 CET 2022 from 192.168.0.115 on ssh:notty There were 3 failed login attempts since the last successful login. This session is being recorded Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-97-generic x86_64) Last login: Tue Feb 8 20:00:27 2022 from 192.168.0.122 admin01@linux01:~$ tim@docker01:~$ echo $? 0 tim@docker01:~$

Both success and failure behavior is covered and return the appropriate exit codes. With our script in a suitable place lets introduce HAProxy into the mix.

HAProxy and haproxy.cfg

What HAProxy is, it's capabilities, and what a basic and functioning haproxy.cfg looks like is well covered in the posts where we load balance the PVWA and HTML5 Gateway.

Once again we use Docker to run HAProxy however unlike done previously we cannot simply use the an existing image and mount our haproxy.cfg as we intend to use a script that leverages Expect and the OpenSSH client -- two programs not found in the HAProxy Docker images provided by the HAProxy team. We need to built our own Docker image that includes our needed dependencies.

This this done by creating our own Dockerfile which is nowhere as daunting as sounds as we can use an existing HAProxy Docker image as a base.

In a new working directory, move our script and create our Dockerfile:

tim@docker01:~$ mkdir haproxy-psmp tim@docker01:~$ mv login.expect haproxy-psmp/ tim@docker01:~$ vi haproxy-psmp/Dockerfile

with the content

FROM haproxy:latest USER root RUN apt-get update && apt-get install -y expect openssh-client && apt-get clean USER haproxy COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg COPY login.expect /var/lib/haproxy/login.expect

Before we build an image based on our Dockerfile we need to create the haproxy.cfg in our working directory with frontends and backends:

defaults timeout client 10s timeout connect 10s timeout server 10s global insecure-fork-wanted external-check frontend psmp_loadbalancer bind *:22 default_backend psmp_servers backend psmp_servers option tcp-check tcp-check expect string SSH-2.0- option external-check external-check command /var/lib/haproxy/login.expect server psmp01 192.168.0.122:22 check inter 31s server psmp02 192.168.0.121:22 check inter 31s frontend stats mode http bind *:8404 stats enable stats uri /stats stats refresh 10s stats admin if TRUE

Focusing on what is in our backend, we have two health checks: a tcp-check expect that works similar to our Expect script in that it expects to receive a certain string -- the SSH banner -- when first connecting to a SSH server and an option external-check that invokes the script we created. If the first check fails, most likely to the OpenSSH service not running, the HAProxy marks the server as unhealthy and doesn't bother with subsequent ones.

As optional external-check invokes processes and thus poses a security risk, we also need to explicitly add insecure-fork-wanted and external-check under the global section to tell HAProxy we really want to use it.

We still are not ready to build our image as our script has the target PSMP hard coded. external-check command passes arguments four arguments to the command, the third being the address of the server the health check is being ran against. We need to update our script to take this argument.

As we already updating our script, we also take the opportunity to disable logging to Stdout by setting user_log 0, create a CyberArk user healthcheck to explicitly authenticate to the Vault with for the purpose of our health check, and define the full path to the ssh binary. Our script now looks like:

#!/usr/bin/expect user_log 0 set psmp [lindex $argv 2] spawn /usr/bin/ssh -o StrictHostKeyChecking=no healthcheck@admin01@linux01.iosharp.lab@$psmp expect { "Vault Password:" { send "Password1234!\r" } } expect { "admin01@linux01:~$ " { exit 0 } "Vault Password:" { exit 1 } }

Building our Docker image and running our container

Building the image is simple. We just need to run docker build with a tag we will remember:

tim@docker01:~/haproxy-psmp$ sudo docker build -t haproxy-psmp . Sending build context to Docker daemon 4.608kB Step 1/6 : FROM haproxy:latest ---> 4313a6c47541 Step 2/6 : USER root ---> Using cache ---> 974bba5aa714 Step 3/6 : RUN apt-get update && apt-get install -y expect openssh-client && apt-get clean ---> Using cache ---> 625afeb7fa22 Step 4/6 : USER haproxy ---> Using cache ---> 2bba1cf696a7 Step 5/6 : COPY haproxy.cfg /usr/local/etc/haproxy/haproxy.cfg ---> 8ee4cfac7f22 Step 6/6 : COPY login.expect /var/lib/haproxy/login.expect ---> 4672cc1f4c34 Successfully built 4672cc1f4c34 Successfully tagged haproxy-psmp:latest tim@docker01:~/haproxy-psmp$

Now we just start the container, publishing the ports for our backend and stats. As we include our haproxy.cfg and login.expect as part of the image we do not need to mount them.

tim@docker01:~/haproxy-psmp$ sudo docker run \ -p 8404:8404 \ -p 23:22 \ haproxy-psmp:latest

As we already have our Docker host's SSH daemon running on port 22, we need to have Docker publish HAProxy on 23.

Our container starts successfully and in the HAProxy logs we see processes exiting with code 0, indicating the health check using our script is firing and ending successfully.

tim@docker01:~/haproxy-psmp$ sudo docker run -p 8404:8404 -p 23:22 haproxy-psmp:latest [NOTICE] (1) : New worker (8) forked [NOTICE] (1) : Loading success. [NOTICE] (1) : haproxy version is 2.5.1-86b093a [NOTICE] (1) : path to executable is /usr/local/sbin/haproxy [WARNING] (1) : Process 12 exited with code 0 (Exit) [WARNING] (1) : Process 17 exited with code 0 (Exit) [WARNING] (1) : Process 22 exited with code 0 (Exit) [WARNING] (1) : Process 27 exited with code 0 (Exit) [WARNING] (1) : Process 32 exited with code 0 (Exit)

Checking our stats page, we see both PSMPs as green.

Shutting off one of the PSMP services it goes red.

We are still able to login through our HAProxy:

tim@docker01:~/haproxy-psmp$ ssh tim@admin01@linux01.iosharp.lab@192.168.0.115 -p 23 Vault Password: This session is being recorded Welcome to Ubuntu 20.04.2 LTS (GNU/Linux 5.4.0-97-generic x86_64) Last login: Wed Feb 9 09:55:44 2022 from 192.168.0.122 admin01@linux01:~$ whoami admin01 admin01@linux01:~$ hostname linux01 admin01@linux01:~$ exit logout Connection to linux01.iosharp.lab closed. Connection to 192.168.0.115 closed. tim@docker01:~/haproxy-psmp$

Going forward

Though the health checking is more complicated than with the PVWA and HTML5 Gateway due to requiring the usage of an external script leveraging Except but being able to run HAProxy and the script in it's own Docker container makes the entire setup more manageable.

We can also add additional, non-working behavior in our script, a good candidate being when we receive Password: instead of Vault Password:, indicating there maybe something wrong with the pluggable authentication module stack.

How are you checking the health of your PSMPs? Provide some feedback in the comments!

Load balancing CyberArk (now Idira) Privileged Session Manager HTML5 Gateway with HAProxy

Tim Schindler — Tue, 08 Feb 2022 12:11:28 GMT

This is one post in a series focusing on load balancing various CyberArk (now Idira) components using HAProxy with a focus on application/service-based health checking.

With the experience we gained from load balancing CyberArk Privileged Vault Web Access with HAProxy, load balancing the Privileged Session Manager HTML5 GW (PSM HTML5 GW) while incorporating application-based health checking with HAProxy is simple. CyberArk already provides an endpoint we can use for the check so it is all about crafting our HAProxy configuration file.

Setting up HAProxy

Similar to how we did it with load balancing the PVWA we will use Docker and a HAProxy Docker image, passing our haproxy.cfg and TLS certificates in mounted volumes.

Lets create a directories to store our haproxy.cfg and certificate with it's private key.

tim@docker01:~$ mkdir -p haproxy-html5gw/haproxy tim@docker01:~$ mkdir -p haproxy-html5gw/certificates

And generate a self-signed certificate. For any production environment we would make sure to use an issued certificate from a trusted certificate authority.

tim@docker01:~$ openssl req -x509 -newkey rsa:4096 -keyout haproxy-html5gw/certificates/tls.pem.key -out haproxy-html5gw/certificates/tls.pem -sha256 -days 365 -nodes Generating a RSA private key .....................................++++ .....................................................................................................................................++++ writing new private key to 'haproxy-html5gw/certificates/tls.pem.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]: State or Province Name (full name) [Some-State]: Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]: Organizational Unit Name (eg, section) []: Common Name (e.g. server FQDN or YOUR name) []:192.168.0.115 Email Address []: tim@docker01:~$

We created the certificate and it's private key under our personal user and with the file permissions the private key is created with the HAProxy process the user runs as cannot read the private key. Already knowing that the HAProxy process runs under the user ID 99 we go ahead and change ownership of the private key.

tim@docker01:~$ ls -la haproxy-html5gw/certificates/tls.pem.key -rw------- 1 tim tim 3272 Feb 1 15:36 haproxy-html5gw/certificates/tls.pem.key tim@docker01:~$ chown 99 haproxy-html5gw/certificates/tls.pem.key tim@docker01:~$ ls -la haproxy-html5gw/certificates/tls.pem.key -rw------- 1 99 tim 3272 Feb 1 15:36 haproxy-html5gw/certificates/tls.pem.key tim@docker01:~$

Creating our haproxy.cfg

In haproxy.cfg we define our frontend and backend sections.

In the frontend section we define on which address and port HAProxy accepts traffic on and the TLS certificate used to encrypt traffic between HAProxy and the client. In the backend section we define the the individual servers HAProxy will relay traffic to. We can also define a load balancing strategy, health checking, and persistent sessions.

Lets start with a basic haproxy.cfg:

tim@docker01:~$ vi haproxy-html5gw/haproxy/haproxy.cfg

with the content

frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws server html5gw1 192.168.0.113:443 ssl verify none

We define a frontend to listen on port 443 and tell it to use the certificate we generated earlier (HAProxy will look for a private key with the name .key in the same directory.) With default_backend we define the name of the backend that traffic coming from the frontend should be relayed to.

In the backend section we define a single HTML5GW to relay the traffic to. In reality we will have multiple but we can define just one and still be able to implement our concepts. The server name is arbitrary and we can call it whatever we want but the backend must be named html5gws as we used it as the value for default_backend in the frontend.

After specifying the address and IP of the server we state to connect with ssl and again because this is a lab environment we allow for invalid TLS certificates on the backend HTML5 servers with verify none.

We are ready to start our container with the certificates folder and the folder containing our haproxy.cfg as mounted volumes.

tim@docker01:~$ sudo docker run -p 443:443 -v /home/tim/haproxy-html5gw/haproxy:/usr/local/etc/haproxy:ro -v /home/tim/haproxy-html5gw/certificates:/usr/local/etc/certs:ro haproxy:latest [NOTICE] (1) : haproxy version is 2.5.1-86b093a [NOTICE] (1) : path to executable is /usr/local/sbin/haproxy [WARNING] (1) : config : missing timeouts for frontend 'html5gw_loadbalancer'. | While not properly invalid, you will certainly encounter various problems | with such a configuration. To fix this, please ensure that all following | timeouts are set to a non-zero value: 'client', 'connect', 'server'. [WARNING] (1) : config : missing timeouts for backend 'html5gws'. | While not properly invalid, you will certainly encounter various problems | with such a configuration. To fix this, please ensure that all following | timeouts are set to a non-zero value: 'client', 'connect', 'server'. [NOTICE] (1) : New worker (8) forked [NOTICE] (1) : Loading success.

Ignoring the warnings about the timeouts that we will fix later, we navigate to https://192.168.0.115/guac/direct and are shown an error:

This error is fine and is not indicative of anything in our haproxy.cfg being wrong. When browsing directly to https://192.168.0.115/guac/direct our browser is making a GET request, which is not supported by the HTML5 GW. Looking at the HTML5GW logs we can see the GET request being made:

Now that we verified that HAProxy is relaying traffic received from the frontend to the backend we can start implementing our application-based health check and sticky sessions.

Adding application health checking

Like we could have with the PVWA, we can enable basic health checking by adding the check keyword to our server defined in the backend section but we still run the risk of HAProxy relaying traffic to a HTML5 GW in a scenario where the Tomcat (or whatever web server you are using to host the HTML 5 GW Web App) server is running but the Apache Guacamole service that the HTML5 GW is based off of is not functional.

option httpchk enables us to have more complex health checking. It makes a request to the server and if the HTTP response code is either 2xx or 3xx then HAProxy considers the server healthy and keeps relaying traffic to it. Without defining a URI as part of option httpchk the health check is done against /. As the HTML5 GW Web App lives under the /gauc path, not defining another URI may not give us true application-based health checking.

The documentation for the PSM HTML5 GW specifies a URI we should use when making our health check.

We need to make sure we specify /guac/rest/healthcheck with our option httpchk resulting in our haproxy.cfg looking like:

frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws option httpchk GET /guac/rest/healthcheck server html5gw1 192.168.0.113:443 check ssl verify none

We also must add the check keyword to our server otherwise the parameters of the health check are defined via option httpchk but the health check will never happen and the server will always be considered healthy.

Sticky sessions

In the same way we did it with the PVWA we will use stick tables again to implement sticky sessions although HAProxy offers more than one way to accomplish sticky sessions.

Our stick table captures client IP addresses and associates them to a backend server. The first time a client connects, HAProxy relays the traffic to a server and associates the client IP address to the server in the stick table. The next time the client IP address connects, HAProxy looks up the associated server in the stick table and relays the traffic to that server. We set a long expiration date of 36 hours for this association as our PSMs are configured to close idle PSM sessions at this point.

frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws stick-table type ip size 1m expire 36h stick on src option httpchk GET /guac/rest/healthcheck server html5gw1 192.168.0.113:443 check ssl verify none

Adding stats

We already know how useful it is having a stats page that we can use to see numerous information for our frontends and backends, among that the health of the servers in the backend. We use it to see various statistics regarding our frontends and backends and as we specify stats admin if TRUE we can perform administrative tasks like disabling health checks on servers or marking individual servers as healthy or unhealthy.

defaults timeout client 10s timeout connect 10s timeout server 10s frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws stick-table type ip size 1m expire 36h stick on src option httpchk GET /guac/rest/healthcheck server html5gw1 192.168.0.113:443 check ssl verify none frontend stats mode http bind *:8404 stats enable stats uri /stats stats refresh 10s stats admin if TRUE

We also define values under defaults for the timeouts that HAProxy was complaining about missing when we first created our haproxy.cfg.

Adding a second HTML5 GW

So far we have been working with a single HTML5 GW but for sake of better being able to test our new haproxy.cfg lets add a second HTML5 GW into the mix. Because I am managing my HTML5 GW containers with Docker Compose it is trivial to spin up a second HTML5 GW running on the same Docker host but with a different published port.

defaults timeout client 10s timeout connect 10s timeout server 10s frontend html5gw_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend html5gws backend html5gws stick-table type ip size 1m expire 36h stick on src option httpchk GET /guac/rest/healthcheck server html5gw1 192.168.0.113:443 check ssl verify none server html5gw2 192.168.0.113:8443 check ssl verify none frontend stats mode http bind *:8404 stats enable stats uri /stats stats refresh 10s stats admin if TRUE

Testing our haproxy.cfg with the help of stats

We start our container once more:

tim@docker01:~$ sudo docker run \ -p 8404:8404 \ -p 443:443 \ -v /home/tim/haproxy-html5gw/haproxy:/usr/local/etc/haproxy:ro \ -v /home/tim/haproxy-html5gw/certificates:/usr/local/etc/certs:ro \ haproxy:latest

Browsing to the stats we see frontend and backend -- and a bunch of green, which is a good sign!

We navigate to https://192.168.0.115/gauc/direct and though we get an error as expected, refreshing the stats page shows all our connections being relayed to a single server.

Doing more with HAProxy

Our haproxy.cfg only scratches the surface in terms of the functionality HAProxy offers. We can adjust how often the health check happens by specifying the inter parameter for our servers if we find the default not aggressive enough and explicitly pick a load balancing method with the balance keyword.

As HAProxy is a Layer 4 proxy we can even load balance the Privileged Session Manager and the Privileged Session Manager for SSH!

Application health checking and load balancing CyberArk (now Idira) Privileged Vault Web Access with HAProxy

Tim Schindler — Sat, 29 Jan 2022 17:36:57 GMT

The Privileged Vault Web Access is the most straightforward component to load balance in CyberArk (now Idira)'s Privileged Access Security solution but despite this it is still easy to misconfigure one of the most important aspects: the health check.

It is possible to use whatever your load balancing solution's 'default' health check is but this typically leaves you with a health check that is server-based health check versus application-based. This could result in your load balancer still relaying traffic to a server where IIS is running but the PVWA service is not functional.

Whereas CyberArk provides documentation for other components on leveraging various endpoints to accomplish application-based health checks (Privileged Session Manager, Privileged Session Manager HTML5 Gateway), for the PVWA it offers only the most basic load balancing requirements.

Using HAProxy we will set up load balancing with a health check that evaluates the health of the PVWA service (our 'application') and not just IIS.

What is HAProxy?

From the official HAProxy website it is '... a free, very fast and reliable reverse-proxy offering high availability, load balancing, and proxying for TCP and HTTP-based applications.' It can meet the basic load balancing requirements for the PVWA and also the 'notable' load balancer settings found in this knowledge article. And just like the description implies we can use it to load balance in more complex scenarios like the load balancing PSMs.

Setting up HAProxy

Like most software HAProxy is provided as a Docker image. With Docker we can quickly get HAProxy up and running, passing our haproxy.cfg and TLS certificates in mounted volumes.

We will need a place to store our haproxy.cfg which configures HAProxy and a place to store our certificate and it's private key so lets first create those directories.

tim@docker01:~$ mkdir haproxy tim@docker01:~$ mkdir haproxy/haproxy tim@docker01:~$ mkdir haproxy/certificates

We will generate a self-signed certificate as this is a lab environment but in production we would want a certificate issued from a trusted certificate authority.

tim@docker01:~$ cd haproxy/certificates/ tim@docker01:~/haproxy/certificates$ openssl req -x509 -newkey rsa:4096 -keyout tls.pem.key -out tls.pem -sha256 -days 365 -nodes Generating a RSA private key ..........++++ .......................++++ writing new private key to 'tls.pem.key' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Country Name (2 letter code) [AU]: State or Province Name (full name) [Some-State]: Locality Name (eg, city) []: Organization Name (eg, company) [Internet Widgits Pty Ltd]: Organizational Unit Name (eg, section) []: Common Name (e.g. server FQDN or YOUR name) []:192.168.0.115 Email Address []: tim@docker01:~/haproxy/certificates$

With the file permissions and ownership the private key is created with the HAProxy process inside the container will not be able to read it. We will change the file to be owned by the user ID the HAProxy runs as inside the container.

tim@docker01:~/haproxy/certificates$ chown 99 tls.pem.key

Creating our haproxy.cfg

In a haproxy.cfg you will see two primary sections: one to represent the frontend and one to represent the backend.

In the frontend section, we specify settings around HAProxy receiving traffic. This includes configurations like the addresses and ports HAProxy is listening for traffic on, the TLS certificate used to secure communication between HAProxy and the client, as well as options to redirect HTTP traffic to HTTPS.

In the backend section, we specify the servers HAProxy will relay traffic to as well as options like load balancing strategy, more sophisticated health checking, and persistent sessions.

We will start with something simple just to ensure it works:

tim@docker01:~/haproxy/haproxy$ tim@docker01:~/haproxy/haproxy$ vi haproxy.cfg

with the content

frontend pvwa_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend pvwas backend pvwas server pvwa1 192.168.0.102:443 ssl verify none server pvwa2 192.168.0.116:443 ssl verify none

Our haproxy.cfg is extremely basic. We define a frontend to listen on port 443 and tell it to use the certificate we generated earlier (we can just define the certificate and HAProxy will look for a private key with the name .key in the same directory, which in our case is named tls.pem.key.) With default_backend we define the name of the backend that traffic coming from the frontend should be relayed to.

In the backend section we define the individual servers traffic will be relayed to. The names pvwa1 and pvwa2 for the servers are arbitrary and we can call them whatever we want but the backend must be named pvwas as we used it as the value for the default_backend in the frontend. After specifying the address and IP of the server we state to connect with ssl and again because this is a lab environment we allow for invalid TLS certificates on the backend PVWA servers with verify none.

With our certificates generated and our initial haproxy.cfg we can start our container, making sure to mount the local volumes.

tim@docker01:~/haproxy$ sudo docker run -p 443:443 -v /home/tim/haproxy/haproxy:/usr/local/etc/haproxy:ro -v /home/tim/haproxy/certificates:/usr/local/etc/certs:ro haproxy:latest [NOTICE] (1) : haproxy version is 2.5.1-86b093a [NOTICE] (1) : path to executable is /usr/local/sbin/haproxy [WARNING] (1) : config : missing timeouts for frontend 'pvwa_loadbalancer'. | While not properly invalid, you will certainly encounter various problems | with such a configuration. To fix this, please ensure that all following | timeouts are set to a non-zero value: 'client', 'connect', 'server'. [WARNING] (1) : config : missing timeouts for backend 'pvwas'. | While not properly invalid, you will certainly encounter various problems | with such a configuration. To fix this, please ensure that all following | timeouts are set to a non-zero value: 'client', 'connect', 'server'. [NOTICE] (1) : New worker (8) forked [NOTICE] (1) : Loading success.

Starting the container we see a few warnings about the lack of timeouts but navigating to the PVWA through our load balancer it loads!

But after logging in we immediately get an error about our session expiring.

Our basic haproxy.cfg is a bit too basic. There is no sticky sessions and we do not have a load balancing method configured so HAProxy is rotating through the PVWAs, sending the next request to the next server. Plus: we do not have ANY health checking, let alone something that could be considered an application-based health check.

Adding sticky sessions and application health checking

We can add basic health checks by simply adding the check keyword to each of servers defined in the backend section but this alone will not check the health of the PVWA rather just of IIS. We need to also add the option httpchk keyword to our backend section.

option httpchk allows us to define a URI. It will make a request to that URI and if the response returns a HTTP code of 2xx or 3xx then HAProxy considers the server healthy.

As we want HAProxy to not relay traffic to a PVWA where the PVWA service is not working or cannot connect to the Vault even though IIS is running, we need to find the right URI that will return something that is NOT 2xx or 3xx under those conditions. To find this, lets start with looking at requests made in Firefox when we browse to a functional PVWA with the idea that one or more return should something other than 2xx or 3xx when something is wrong with the PVWA.

Finding a URI for our health check

Lets approach it how a customer would by browsing directly to one of the PVWAs in Firefox. With the Network Monitor tab in focus of Web Developer tools open and filtering on just HTTP and XHR type of requests we navigate to one of the PVWAs:

Looking at all the requests we want to focus on the ones that return 200. In theory all of these would be suitable for the health check but it could be that 200 is still returned even though the PVWA is not working. We should purposely break the PVWA and see what is returned then.

Lets use the Windows Firewall and create a rule on the PVWA blocking outbound traffic on port 1858:

Ensuring that our rule is enabled we refresh the PVWA:

Even without a connection to the Vault the PVWA still loads -- sort of. In a non-functioning state IIS still returned a 200 for the first few requests which loaded the logon screen minus the fields to put in credentials. The request to https://192.168.0.102/PasswordVault/api/settings/authentication failed, however, so it seems like a perfect candidate to use for our health check.

Lets add it as part of a option httpchk in our backend in haproxy.cfg while at the same time adding values for the timeout configurations we saw warnings for when starting HAProxy the first time:

defaults timeout client 10s timeout connect 10s timeout server 10s frontend pvwa_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend pvwas backend pvwas option httpchk GET /PasswordVault/api/settings/authentication server pvwa1 192.168.0.102:443 check ssl verify none server pvwa2 192.168.0.116:443 check ssl verify none

In addition we had to add the check keyword to each of the servers in our backend that we want it's health checked. We need this otherwise our option httpchk is effectively worthless.

Sticky sessions

Sticky sessions can be accomplished a multitude of ways including via cookies or with hashing-based methods. We will use stick tables.

We will use a stick table to capture the information of client IP addresses and associate it to a backend server. Every time a client IP address connects HAProxy will look in the stick table to see the previously-associated server and relay the traffic to it. We also set a time for this association to expire so a client is not eternally 'stuck' with a particular server.

defaults timeout client 10s timeout connect 10s timeout server 10s frontend pvwa_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend pvwas backend pvwas stick-table type ip size 1m expire 15m stick on src option httpchk GET /PasswordVault/settings/authentication server pvwa1 192.168.0.102:443 check ssl verify none server pvwa2 192.168.0.116:443 check ssl verify none

Finishing it off with stats

HAProxy provides a convenient stats page that we can use to see numerous information for our frontends and backends, among that the health of the servers in the backend. As a last act before we start our container with our new haproxy.cfg lets add a frontend for the stats page.

defaults timeout client 10s timeout connect 10s timeout server 10s frontend pvwa_loadbalancer bind *:443 ssl crt /usr/local/etc/certs/tls.pem default_backend pvwas backend pvwas option httpchk GET /PasswordVault/api/settings/authentication server pvwa1 192.168.0.102:443 check ssl verify none server pvwa2 192.168.0.116:443 check ssl verify none frontend stats mode http bind *:8404 stats enable stats uri /stats stats refresh 10s stats admin if TRUE

We run the stats page on it's own frontend. It will auto refresh every 10 seconds and because we are still in a lab environment we use the stats admin to always return TRUE (if we want this production we should protect it with at least basic authentication like shown in the HAProxy examples) so we can perform administrative tasks like disabling health checks on servers or the entire backend, marking individual servers as healthy or unhealthy.

We also need to remember to publish the port for our stats page when we start our container.

Running and testing our new haproxy.cfg

First we start our container like in the beginning but with the port 8404 published for our stats page:

docker run -p 443:443 -p 8404:8404 -v /home/tim/haproxy/haproxy:/usr/local/etc/haproxy:ro -v /home/tim/haproxy/certificates:/usr/local/etc/certs:ro haproxy:latest

When we load the stats page we can see both the frontends and the backend we defined along with various statistics about them. On our backend we are able to administrator the individual servers or the backend as a whole:

Looking at one of the PVWAs we can observe the IIS logs to see the health check:

Stopping the IIS service or re-enabling the Windows Firewall rule we create to simulate the inability for the PVWA to connect to the Vault, the server in the backend goes red in our stats page and traffic is no longer relayed to it:

Going forward

With a simple configuration we have a powerful load balancer with an application-based health check. We can effortlessly add more functionality like the X-Forwarded-For header using the option forwardfor keyword to capture the real client IP address when accessing the PVWA behind our load balancer, HTTP to HTTPS auto redirection with redirect scheme, and defining a load balance method with the balance method.

As HAProxy is a Layer 4 load balancer it is not limited to just HTTP use cases. It can be used to load balance Privileged Session Manager and Privileged Session Manager for SSH.

Tim Schindler's Blog

Building Idira (CyberArk) ISP PowerShell tooling with Claude Code, Context7 MCP, and SecretStore

Why letting the model guess is the wrong default

Feeding the model real docs with Context7

Encoding the idiosyncrasies in CLAUDE.md

Keeping the service-account secret out of the model's reach

What the context changes

The script

Limitations

Make it a skill

Creating a CyberArk (now Idira) Central Policy Manager plugin for an API using the new REST API framework

Developing the REST API plugin

Creating a new platform

Creating the parameters file for testing

Writing the configuration file

Verify

Change

Reconcile

Conclusion

Creating a CyberArk (now Idira) Central Policy Manager plugin for an API using WSChains

Developing the WSChains plugin

Creating a new web application platform

Preparing our parameters file for testing

Crafting ShopizerAdministratorChains.xml

Verify

Change

Reconcile

Conclusion

Integrating CyberArk (now Idira) self-hosted PAM as a Secret Store in StrongDM

Reading through the documentation

Integrating CyberArk self-hosted PAM

Closing thoughts

Creating a CyberArk (now Idira) Central Policy Manager plugin for a web application

Understanding what a CPM plugin for a web app is doing

A closer look at a WebFormFieldsFile

Developing our Shopizer CPM plugin

Setting the stage

Creating a new web application platform

Crafting ShopizerAdministrator.ini

Verify section

Change section

Reconcile section

Troubleshooting Reconcile by testing the CPM plugin

Fixing the Reconcile operation

Next steps

Generating TOTPs for CyberArk (now Idira) Privileged Session Manager Webapp Connectors using a PreConnect DLL

What is a PreConnect DLL?

Creating the PreConnect DLL

Configuring the Webapp PSM Connector

Exploring CyberArk (now Idira) Central Policy Manager's WSChains

First look at WSChains

GlobalSettings

Requests

Fails

Chains

StatusCode

Wrap-up

Creating a Universal iDRAC CyberArk (now Idira) PSM Connector with if-else Conditional Statements

The Framework's Conditional Statements

Creating a Universal iDRAC PSM Connector

Connectors for each version iDRAC version

iDRAC8

iDRAC9

Creating the Universal iDRAC Connector

Testing the Universal iDRAC Connector

Generating time-based one-time passwords in Privileged Session Manager WebApp Connectors

Generating a TOTP in WebFormFields

The TOTP secret as a Logon Account

Solution to Network Connectivity Issues after the Hardening of the CyberArk (now Idira) PAM self-hosted's Vault on Virtualized Hardware

Determining if your Vault is impacted

The Fix

What is happening as part of the hardening?

Effortlessly Setting Up A CyberArk (now Idira) PAM Self-Hosted Lab with AutomatedLab

What is AutomatedLab?

Deploying CyberArk PAM self-hosted with AutomatedLab

Installing the custom CyberArk roles

Creating the lab definition

Defining the PAM-self hosted machines

Installing the lab

Elevating our lab

Crafting `ShopizerAdministratorChains.xml`

A closer look at a `WebFormFieldsFile`

Crafting `ShopizerAdministrator.ini`

Identifying elements with id, name, class, or text in `WebFormsField`

Identifying elements with XPath in `WebFormsField`

Defining our `WebFormFields`

Recovery of the Vault with `ReconnectHSMOnErrorCodes`

Using PowerShell DSC and `CyberArkDsc` to populate an empty Vault

Installing the `CyberArkDsc` and `ActiveDirectoryDsc` modules

Working with multiple `SecretManagement.CyberArk` secret vaults

HAProxy and `haproxy.cfg`

Creating our `haproxy.cfg`

Adding `stats`