Thursday, May 10, 2018

Troubleshooting RADIUS In VMware Horizon Environments


VMware Horizon has supported RADIUS for 6 years now so it's a fairly mature and proven capability. That said, the initial integration of a RADIUS solution can be challenging.  One reason for this is the lack of description provided by the Horizon Client for failed RADIUS connections.  For a wide range of issues you get the very same error message with no specifics on what's going wrong.   Anything from a fat fingered password to a busted RADIUS server leads to the same old, "Access Denied," message box.



For day to day usage with a RADIUS solution that's validated and working, this message is specific enough to get by.  But for someone in the midst of troubleshooting an integration it's vague and can lead to confusion.  Making matters worse, while further info can be gleaned from the debug logs on your Horizon server,  there are still blind spots from the perspective of a VDI admin.   When a Horizon Connection server, acting as a RADIUS client, has a RADIUS request rejected it's not necessarily entitled to any explanation from the RADIUS server why the request was rejected.   Sometimes all that's received from the RADIUS server is a terse Access-Reject packet, with no specifics on why the rejection occurred.   This can cause things to get dicey in the context of a RADIUS integration, particularly because VDI admins don't necessarily have access to the RADIUS server or its logs.   Lots of fingered pointing can ensue. 

This post will detail a few strategies for troubleshooting RADIUS integrations with Horizon.   After providing a brief overview of how RADIUS authentication works, I'm going to detail the following strategies:


  • Using network captures from Wireshark and Tcpdump  
  • Leveraging utilities to verify RADIUS connectivity
  • Standing up a 2nd RADIUS server



RADIUS Overview 


You can leverage packet sniffers to gain additional insight into the short but sweet authentication process RADIUS utilizes.  This process starts off with the RADIUS client sending an Access-Request packet to the RADIUS server.   This packet includes several pieces of info, including a shared secret and user credentials.   Assuming the packet arrives, the shared secret is correct and the RADIUS client is trusted, there are 3 possible responses from the RADIUS server.  It can respond back with an Access-Accept, Access-Reject or Access-Challenge.













An Access-Accept is sent when authentication has fully completed and the user is granted access.  The Access-Accept packet includes a list of parameters, in the form of attribute-value pairs, that are required for access to the service.

An Access-Reject is issued when you're request for access is rejected.  Your username or password might be wrong or you're just not authorized.  Whatever the reason, you get nothing but the terse rejection of an Access-Reject, often without the courtesy of any explanation why.

An Access-Challenge is issued if the RADIUS server requires another piece of information, like a secondary password, PIN, token, or card.

Cisco offers an excellent primer on RADIUS titled, "How Does RADIUS Work."  Here's a link: https://www.cisco.com/c/en/us/support/docs/security-vpn/remote-authentication-dial-user-service-radius/12433-32.html   Also, if you're looking for some really exciting Friday night reading, here's a description of the standard by the Internet Engineering Task Force: https://tools.ietf.org/html/rfc2865


Using Network Captures From Wireshark And Tcpdump 


This entire protocol exchange can be easily followed using a packet sniffer.   In the case of connections made against a Horizon Connection server, we can run Wireshark on the system, set the RADIUS filter, then start the dump.  At that point you'll get the play by play action of the RADIUS protocol in real time.

Using PAP, here's an example of a successful authentication with an Access-Request quickly followed by and Access-Accept:









Note that in this Access-Accept are a bunch of attribute value pairs that assist the RADIUS client in obtaining access to the service. 

Next, here's an example of an Access-Reject being issued due to an incorrect password:





As mentioned earlier, you don't necessarily get a lot of information with an Access-Reject.

Here's what it looks like when a shared secret is misconfigured on the RADIUS authenticator used by the Horizon Connection server.





After failing to receive a response to the initial Access-Request, the Horizon server attempts the request 4 more times according to the configured retry settings.  Just 5 lonely Access-Request with no responses.


Using Tcpdump on UAG To Capture RADIUS Packets


If you're integrating a RADIUS solution with Unified Access Gateway, we can run tcpdump while logged in as root on the UAG console.   Once logged in, Tcpdump is easily installed with this command:

/etc/vmware/gss-support/install.sh



After getting tcpdump installed, we can monitor RADIUS traffic with the following command:

tcpdump -i eth0 -n –v udp port 1812

With tcpdump running on the console, here's what a successful authentication looks like: 












And here's what a bad password looks like:












And once again, here's what it looks like when an incorrect shared secret is used: 

















My UAG instance is configured to execute a RADIUS request a maximum of 2 times.   So when an incorrect shared secret is configured, we see the 2 request go out, but no responses from the RADIUS server. 

Confirming Password Used For Request


If you're using PAP, I've found you can confirm the correct password is used by your RADIUS client by entering the shared secret under Wireshark Preferences.   Doing this allows Wireshark to display the password being used by the RADIUS client.  With Wireshark open, click on Edit, then Preferences and then select RADIUS.   Entered in the shared secure for the RADIUS traffic you're sniffing.


After making these changes you can now see the password being sent in the RADIUS request.  In the image below you can see I've used the password VMware1!dfadfafdf, instead of the proper one.








Leveraging Utilities To Verify RADIUS Connectivity


I ran across several utilities that can make RADIUS requests as a client against a RADIUS server for testing purposes.  This is a great way to remove Horizon out of the equation and independently verify we're sending the correct info to the RADIUS server.   A very simple to use Windows based utility is NTRadping.   A similar tool available on Linux is Radtest, which installs as part of freeRADIUS.  Finally, with the installation of ClearBox Enterprise RADIUS server, you get the RADIUS Client Tool that can be used to test MSCHAPv2 authentication.   I'm going briefly walk through the usage of each utility below.

Just a friendly reminder: Don't forget that any endpoint you run these tools from will need to be configured as a RADIUS client on your RADIUS server.


NTRadping


I found a copy of NTRadping here: https://www.novell.com/coolsolutions/tools/14377.html

Here's what an Access-Accept response looks like when using NTRadping to successfully make a request using PAP. 





As opposed to an Acess-Reject caused by an incorrect password: 



















For a fast and loose test, this is a nice and simple solution.   However, it is limited to PAP or CHAP authentication.


Radtest


Another alternative to NTRadping is Radtest, a utility that comes with FreeRADIUS.  As soon as you install FreeRADIUS on ubuntu you have access to this utility.  For guidance on setting up FreeRADIUS on Ubuntu, check out this recent post: http://www.evengooder.com/2018/04/providing-two-factor-authentication-for.html

Here's the basic syntax for radtest:

radtest {username} {password} {hostname} 10 {radius_secret}

So, for a radius server with an ip address of 10.0.0.238 and a shared secret of "letmein", and the credentials for justin, you have a command like this:

radtest justin VMware1! 10.0.0.238:1812 10 letmein

Here's a screenshot of what that looks like on my system:
















In this example we can see that the RADIUS sever has accepted the request and has issued an Access-Accept in return.

Radtest can be used to issue RADIUS requests using CHAP, PAP, MSCHAP and EAP-MD5 as the authentication method.

Radius Client Tool


For testing out MSCHAPv2 I found a utility included with a RADIUS solution called ClearBox RADIUS Enterprise Server.    The Radius Client Tool is installed as part of the ClearBox RADIUS Enterprise Server.   It's available from here: http://xperiencetech.com/ 

Here's what a successful MSCHAPv2 based RADIUS connection looks like:

























Versus on a failed password





Unfortunately, this is not a free tool and the only way to get access to it is via the 30 day eval.   Not the end of the world though given the fact that most of us won't be troubleshooting MSCHAPv2 RADIUS connections on a daily basis.


Standing Up A 2nd RADIUS Server  


I think standing up a 2nd RADIUS server to test against is a great way to add more context and insight while troubleshooting an integration.   It serves as a sanity check, providing us with a known success to compare with.

One option I detailed in a previous post is to leverage Ubuntu, FreeRADIUS and Google Authenticator.   While standing up this solution is a fairly complex task, I think a VDI admin who's handy with a Linux editor could have a working solution built out in a couple hours following this recipe: http://www.evengooder.com/2018/04/providing-two-factor-authentication-for.html  Once this process is completed you'll have a free and fully functional 2FA RADIUS solution to test your Horizon environment against.

For those who are looking for a simpler Windows based RADIUS solution, I've come across 2 options recently.  First, there's the solution I touched upon earlier, ClearBox Enterprise RADIUS. (http://xperiencetech.com/)  Within 15 minutes of installing this solution I was able to get it properly configured using the built in wizards and fumbling around through the interface.   After adding it as a RADIUS server to my Horizon Connection server I'm prompted once for my AD credentials from the RADIUS solution, then prompted a 2nd time from Horizon. 

A 2nd Windows based option is to leverage Microsofts Network Policy Server as a RADIUS server.  This entails adding the NPS server role on your Windows server, creating a network policy on this server and adding your Horizon Connection server as a client.  I've created a fast and loose guide doing this here: http://www.evengooder.com/2018/05/using-microsoft-network-policy-server.html

Using Microsoft Network Policy Server For Basic RADIUS Testing With VMware Horizon 7.3.1

In order to troubleshoot RADIUS integrations with Horizon, you can setup Microsoft Network Policy Server as a RADIUS server. This isn't something I'd necessarily charge into production with, but it can be a useful way for VDI admins to double check the functionality of Horizon. Particularly when you don't have access to a RADIUS solution or need a sanity check.

One disclaimer: I have yet to see this NPS server handle password changes properly on users with expired passwords.  If anyone has made it happen I would love to hear about it. 


Setting Up Network Policy Server As A RADIUS Solution For Horizon


To begin, go to server manager and add Network Policy and Access Services as a role. Then select Network Policy Server as the role service to install.




When the install is complete, open up the Network Policy Server management console. Navigate to the Network Policies folder, right click and select, "New."



Give your new policy a descriptive name and stick with, "unspecified," as the network access server type.



Next you need to specify a condition for applying the policy. Below, I've gone with the option to base the condition on the IP address of the RADIUS client.




Go with the option to grant access if the condition matches.



On the Configure Authentication Methods screen, select MS-CHAP-v2, MS-CHAP, CHAP or PAP as a supported authentication method.




Next your way through the Configure Constraints and the Configure settings screens, accepting the defaults. As a final configuration step on this new RADIUS server, we have to create a RADIUS Client entry for our Horizon Connection server. With the Network Policy Server snap in still open, right click on RADIUS Clients and select, "New."



And here's what that basic client configuration looks like:



At this point, you can configure the Horizon connection server to act as a client for this RADIUS server. Here's an example configuration from my Horizon environment:



Now enable these new client settings as an authenticator from the Horizon server.



And here's what it looks like to authenticating into my Horizon environment after making this integration:



After doing the initial authentication against the NPS Radius using my AD credentials I'm presented with the normal dialog from the Horizon server:



While this isn't terribly sexy and exciting on it's own, it's a useful setup for testing and worth having in your back pocket.