Troubleshooting RADIUS In VMware Horizon Environments

VMware Horizon has supported RADIUS for 6 years now so it's a fairly mature and proven capability. That said, the initial integration of a RADIUS solution can be challenging.  One reason for this is the lack of description provided by the Horizon Client for failed RADIUS connections.  For a wide range of issues you get the very same error message with no specifics on what's going wrong.   Anything from a fat fingered password to a busted RADIUS server leads to the same old, "Access Denied," message box.

For day to day usage with a RADIUS solution that's validated and working, this message is specific enough to get by.  But for someone in the midst of troubleshooting an integration it's vague and can lead to confusion.  Making matters worse, while further info can be gleaned from the debug logs on your Horizon server,  there are still blind spots from the perspective of a VDI admin.   When a Horizon Connection server, acting as a RADIUS client, has a RADIUS request rejected it's not necessarily entitled to any explanation from the RADIUS server why the request was rejected.   Sometimes all that's received from the RADIUS server is a terse Access-Reject packet, with no specifics on why the rejection occurred.   This can cause things to get dicey in the context of a RADIUS integration, particularly because VDI admins don't necessarily have access to the RADIUS server or its logs.   Lots of fingered pointing can ensue. 

This post will detail a few strategies for troubleshooting RADIUS integrations with Horizon.   After providing a brief overview of how RADIUS authentication works, I'm going to detail the following strategies:

• Using network captures from Wireshark and Tcpdump  
• Leveraging utilities to verify RADIUS connectivity
• Standing up a 2nd RADIUS server

RADIUS Overview 

You can leverage packet sniffers to gain additional insight into the short but sweet authentication process RADIUS utilizes.  This process starts off with the RADIUS client sending an Access-Request packet to the RADIUS server.   This packet includes several pieces of info, including a shared secret and user credentials.   Assuming the packet arrives, the shared secret is correct and the RADIUS client is trusted, there are 3 possible responses from the RADIUS server.  It can respond back with an Access-Accept, Access-Reject or Access-Challenge.

An Access-Accept is sent when authentication has fully completed and the user is granted access.  The Access-Accept packet includes a list of parameters, in the form of attribute-value pairs, that are required for access to the service.

An Access-Reject is issued when you're request for access is rejected.  Your username or password might be wrong or you're just not authorized.  Whatever the reason, you get nothing but the terse rejection of an Access-Reject, often without the courtesy of any explanation why.

An Access-Challenge is issued if the RADIUS server requires another piece of information, like a secondary password, PIN, token, or card.

Cisco offers an excellent primer on RADIUS titled, "How Does RADIUS Work."  Here's a link: https://www.cisco.com/c/en/us/support/docs/security-vpn/remote-authentication-dial-user-service-radius/12433-32.html   Also, if you're looking for some really exciting Friday night reading, here's a description of the standard by the Internet Engineering Task Force: https://tools.ietf.org/html/rfc2865

Using Network Captures From Wireshark And Tcpdump 

This entire protocol exchange can be easily followed using a packet sniffer.   In the case of connections made against a Horizon Connection server, we can run Wireshark on the system, set the RADIUS filter, then start the dump.  At that point you'll get the play by play action of the RADIUS protocol in real time.

Using PAP, here's an example of a successful authentication with an Access-Request quickly followed by and Access-Accept:

Note that in this Access-Accept are a bunch of attribute value pairs that assist the RADIUS client in obtaining access to the service. 

Next, here's an example of an Access-Reject being issued due to an incorrect password:

As mentioned earlier, you don't necessarily get a lot of information with an Access-Reject.

Here's what it looks like when a shared secret is misconfigured on the RADIUS authenticator used by the Horizon Connection server.

After failing to receive a response to the initial Access-Request, the Horizon server attempts the request 4 more times according to the configured retry settings.  Just 5 lonely Access-Request with no responses.

Using Tcpdump on UAG To Capture RADIUS Packets

If you're integrating a RADIUS solution with Unified Access Gateway, we can run tcpdump while logged in as root on the UAG console.   Once logged in, Tcpdump is easily installed with this command:

/etc/vmware/gss-support/install.sh



After getting tcpdump installed, we can monitor RADIUS traffic with the following command:

tcpdump -i eth0 -n –v udp port 1812

With tcpdump running on the console, here's what a successful authentication looks like: 

And here's what a bad password looks like:

And once again, here's what it looks like when an incorrect shared secret is used: 

My UAG instance is configured to execute a RADIUS request a maximum of 2 times.   So when an incorrect shared secret is configured, we see the 2 request go out, but no responses from the RADIUS server. 

Confirming Password Used For Request

If you're using PAP, I've found you can confirm the correct password is used by your RADIUS client by entering the shared secret under Wireshark Preferences.   Doing this allows Wireshark to display the password being used by the RADIUS client.  With Wireshark open, click on Edit, then Preferences and then select RADIUS.   Entered in the shared secure for the RADIUS traffic you're sniffing.

After making these changes you can now see the password being sent in the RADIUS request.  In the image below you can see I've used the password VMware1!dfadfafdf, instead of the proper one.

Leveraging Utilities To Verify RADIUS Connectivity

I ran across several utilities that can make RADIUS requests as a client against a RADIUS server for testing purposes.  This is a great way to remove Horizon out of the equation and independently verify we're sending the correct info to the RADIUS server.   A very simple to use Windows based utility is NTRadping.   A similar tool available on Linux is Radtest, which installs as part of freeRADIUS.  Finally, with the installation of ClearBox Enterprise RADIUS server, you get the RADIUS Client Tool that can be used to test MSCHAPv2 authentication.   I'm going briefly walk through the usage of each utility below.

Just a friendly reminder: Don't forget that any endpoint you run these tools from will need to be configured as a RADIUS client on your RADIUS server.

NTRadping

I found a copy of NTRadping here: https://www.novell.com/coolsolutions/tools/14377.html

Here's what an Access-Accept response looks like when using NTRadping to successfully make a request using PAP. 

As opposed to an Acess-Reject caused by an incorrect password: 

For a fast and loose test, this is a nice and simple solution.   However, it is limited to PAP or CHAP authentication.

Radtest

Another alternative to NTRadping is Radtest, a utility that comes with FreeRADIUS.  As soon as you install FreeRADIUS on ubuntu you have access to this utility.  For guidance on setting up FreeRADIUS on Ubuntu, check out this recent post: http://www.evengooder.com/2018/04/providing-two-factor-authentication-for.html

Here's the basic syntax for radtest:

radtest {username} {password} {hostname} 10 {radius_secret}

So, for a radius server with an ip address of 10.0.0.238 and a shared secret of "letmein", and the credentials for justin, you have a command like this:

radtest justin VMware1! 10.0.0.238:1812 10 letmein

Here's a screenshot of what that looks like on my system:

In this example we can see that the RADIUS sever has accepted the request and has issued an Access-Accept in return.

Radtest can be used to issue RADIUS requests using CHAP, PAP, MSCHAP and EAP-MD5 as the authentication method.

Radius Client Tool

For testing out MSCHAPv2 I found a utility included with a RADIUS solution called ClearBox RADIUS Enterprise Server.    The Radius Client Tool is installed as part of the ClearBox RADIUS Enterprise Server.   It's available from here: http://xperiencetech.com/ 

Here's what a successful MSCHAPv2 based RADIUS connection looks like:

Versus on a failed password

Unfortunately, this is not a free tool and the only way to get access to it is via the 30 day eval.   Not the end of the world though given the fact that most of us won't be troubleshooting MSCHAPv2 RADIUS connections on a daily basis.

Standing Up A 2nd RADIUS Server  

I think standing up a 2nd RADIUS server to test against is a great way to add more context and insight while troubleshooting an integration.   It serves as a sanity check, providing us with a known success to compare with.

One option I detailed in a previous post is to leverage Ubuntu, FreeRADIUS and Google Authenticator.   While standing up this solution is a fairly complex task, I think a VDI admin who's handy with a Linux editor could have a working solution built out in a couple hours following this recipe: http://www.evengooder.com/2018/04/providing-two-factor-authentication-for.html  Once this process is completed you'll have a free and fully functional 2FA RADIUS solution to test your Horizon environment against.

For those who are looking for a simpler Windows based RADIUS solution, I've come across 2 options recently.  First, there's the solution I touched upon earlier, ClearBox Enterprise RADIUS. (http://xperiencetech.com/)  Within 15 minutes of installing this solution I was able to get it properly configured using the built in wizards and fumbling around through the interface.   After adding it as a RADIUS server to my Horizon Connection server I'm prompted once for my AD credentials from the RADIUS solution, then prompted a 2nd time from Horizon. 

A 2nd Windows based option is to leverage Microsofts Network Policy Server as a RADIUS server.  This entails adding the NPS server role on your Windows server, creating a network policy on this server and adding your Horizon Connection server as a client.  I've created a fast and loose guide doing this here: http://www.evengooder.com/2018/05/using-microsoft-network-policy-server.html