NetScaler SSL offload with firmware nCore 10 build 69.4

One excellent feature of Citrix NetScaler is the SSL Offload capabilities.  Based on the MPX specs (Physical appliance), they all include a Cavium SSL accelerator card, this card has the ability to handle SSL encryption/decryption cycles using a hardware card, rather than consuming valuable CPU resources.

After reading and speaking with fellow Citrix engineers, the VPX (Virtual appliance) can also have the SSL offload feature enabled (I always thought it didn’t since there is no SSL card), however as there is no Cavium card, the SSL offload performance is not as high as an MPX appliance, and generally consumes a ton of CPU, hence it can severely impact the scalability of servers that host content requiring SSL encryption (be careful when you are sizing your NetScaler)

Here are some real world examples of how this is used:

  • Microsoft Outlook Web Access/Client Access (Ah that Source IP Mike!)
  • Inspection of SSL traffic and switching based on content
  • SSL based websites (no payment information exchanged*) – Been looking at PCI initiatives very closely lately

“*When payment information is exchanged, it is usually mandated that the data cannot be viewed in transit, this is a requirement of PCI-DSS compliance” – Citrix Tech

The NetScaler can instead use SSL-Bridge for these types of transactions, and not utilize the SSL Offload technology.  What this means is the NetScaler will simply pass the SSL traffic directly to your back-end servers, and not act as the SSL terminator.

Now… let me thank Citrix and Simon Barnes from ITVCE for sharing this information.

SSL offload is designed to function in a similar manner to the below image:

SSL-Offload

In essence all encryption/decryption between the client and server is handled by the NetScaler SSL offload vServer. Once the data has been decrypted it is then sent on the destination service in plain text HTTP.

Configuration:

This configuration assumes that an SSL certificate already exists on the NetScaler for the chosen FQDN. The SSL Certificate is named “SSLapp”.

1. Create a back-end HTTP service

The first step in creating a new service is to create a server object, This is achieved by using “SSL Offload -> Servers” and then select “add”. The resulting item is purely an object that the NetScaler references in further configuration options. The name field does not need to match the hostname of the server, but it is recommended. In this example the webserver is named WinSvr01 and has an IP address of 10.99.0.5

1_AddSvr

Now that a server object exists, you can create a service object to reflect the HTTP service that is running on this web server.

A NetScaler service consists of a server object (created in the previous step), a protocol, port and a monitor. The monitor is used to determine if the service is available, in the event that the service is unavailable the NetScaler will mark the service as down, removing it from load balancing decisions.

In this example we are running a HTTP service (using default of port 80) on the previously created server object, whilst using a standard HTTP monitor to confirm that HTTP services are responding with a response code of “200” – this is HTTP for “ok”.

2_AddSvc

2. Create your vServer

Now that a service exists, you are ready to present the service to the outside world. This is achieved using a NetScaler virtual server (vServer). The first step is to create the object and provide the following information:

  • Name
  • IP Address
  • Port
  • Bound services

In this example we only have a single service to bind to the vServer, however in a production installation there may be 20 backend webservers all delivering the content, if this was the case these could be added in to a service group and then bound to the vServer, load balancing would then be used to ensure that the load is distributed to services that are UP and are able to service the client’s request.

The IP address that is added to the vServer here will be used by clients to connect to the backend services:

3_SSLvs

A certificate must also be bound to the vServer, this is the certificate that the vServer will present for client connections.

4_SSLcert

Client connections should now be directed to the vServer’s IP address – 10.99.0.231. The vServer will present the SSL certificate when a connection is made using HTTPS (TCP 443), any encryption/decryption of data will be processed using the NetScaler’s built in Cavium card.

If the NetScaler is configured as a HA pair and the primary unit’s Cavium card fails, then HA failover will be invoked and the secondary unit will become the primary.

NetScaler 10 build 69.4 Bug

The GUI appears to incorrectly report the status of a feature, offering only the “disable” option for each feature. I have seen scenarios whereby the above configuration has been completed and the vServer immediately appears down, even though everything appears correctly configured.

The first thought is that SSL may not be enabled; checking in the GUI will only show you the option to disable the feature:

To view the reason that the vServer is down you can simply connect using SSH and then issue the following command:

7_SSLdisabledCLI

As you can see in the above screenshot the SSL feature is disabled, causing the vServer to appear down. Enable the SSL feature using the following command:

The vServer will immediately come online:

9_SSLVSup

XenApp 6.5 Desktop Director 2.1 installation and configuration

Looks like Citrix will be getting rid of the AppCenter console (I still call it CMC :)) for  application management.

They are finally looking into centralizing both application and virtual desktop management with a single interface.  Specially with the next version of XenApp, you will no longer need AppCenter for your app deployments.  You can read more on Project (Avalon) Excalibur Technology on one of my previous posts

Citrix has released a nice install guide to install and configure Desktop Director 2.1 to use with XenApp 6.5 and would like to share it with you.

Requirements

  • A XenApp 6.5 Farm
  • An available server to install Desktop Director, it can also be added to the XenApp 6.5 Controller
  • An install Media for XenDesktop 5.6 Feature pack 1
  • IIS 7 installed on the server that hosts the Desktop Director
  • .NET Framework 3.5 Sp1
  • Adobe Flash 10.x or above
  • Firewall exceptions for port 80, 2513 and 5985

Procedure

  • Ensure you have downloaded the ISO for XenDesktop 5.6 Feature pack 1; it is needed this for the DesktopDirector install.
  • If it is not already installed on the operating system, install .NET 3.5 Sp1 and Adobe Flash 10.x or above
  • Mount the XenDesktop ISO and launch the autorun option from the media and select the XenDesktop Option. Agree to the Licensing Agreement on the next page.

  • Select Desktop Director on the Select Components to Install and input the XenApp 6.5 controller FQDN in the Enter the address of the XenDesktop Controller field. Note: If the XenDesktop Controller option is not displayed, you are installing on a XenApp 6.5 server.

  • Click Next, at the install prompt, click the Install tab. When the installation completes, click Close.

  • Open IIS Manager on the Desktop Director Server to be able to log on to the Desktop Director Console. Select the Desktop Director site under Default Web Site

  • Locate Service.AutoDiscoveryAddresses in Application Settings. Double click this option and rename it to Service.AutoDiscoveryAddressesXA and ensure the controller FQDN is present. Note: If you have installed this on a XenApp 6.5 server and you are not able to enter the FQDN of the controller at installation, you can change it from Localhost to Your 6.5 Controller FQDN here, refer following image:

  • Run an IIS reset in the command prompt after the above options are updated. Note: You will be able to log in now, however you will not be able to view user-session specific details yet. You must follow the subsequent steps to be able to log in and view user-session specific details.
  • On the XenApp 6.5 controller, setup the WinRM permissions using a tool in the XenDesktop 5.6 FP 1 ISO (previously called ConfigRemoteMgmt.exe which was used to access information about session).
  • Open the command prompt window, browse to the following directory and run the application <Drive Letter>:\x64\Virtual Desktop Agent\ConfigRemoteMgmt.exe. Use the switch – ConfigRemoteMgmt.exe /configwinrm Domain\User.

The user-session specific information can be viewed if the XenApp 6.5 Controller has its permissions set for remote management.

Note: If you are not setting up certificates on the server for SSL, you can disable the SSL verification by changing the UI.EnableSslCheck to false.

View the original article for more information.

XenApp 6.5 Universal Print Server

Traditionally… one of the biggest hurtles for me over the last 12 years have been around ICA print management (Policies, Universal Printer Driver, native drivers, 3rd party tools, native print spool, Citrix print manager, etc).

Although printing via ICA is now pretty darn good with a combination of native drivers and the Universal Printer technology, I always felt there was a need for a 3rd party tool such as ThinPrint or Tricerat for those special cases.  As you are aware, Citrix released the Universal Printer Server to make printing a lot easier with XenApp 6.5 Hotfix Rollup Pack (HRP01) and now for XenDesktop 5.6 FP1

What is UPS?

The Universal Print Server introduces the EMF functionality to the print server as the client.  Sweet!

That means, again, no native or 3rd party printer driver must be installed on the XenApp server for any printer on the Windows print server. In many environments that should help reduce the installation of printer drivers on the XenApp server dramatically. Having only the Citrix EMF would keep the XenApp farm stable and help with user logon times.

This also resolves two common issues that administrators are faced with today. Leveraging the UPS means the no printer drivers have to be managed on the VDA or XenApp servers at all. Universal Print Servers also can be placed in branch offices to optimize print WAN traffic.

You can read further information on UPS by clicking here as well as Thomas koetzing’s FAQ

Installed correctly?

With the release of Citrix Universal Print Server, Citrix released a new KB article which clearly describes how to check whether Citrix Universal Print Server has been installed and enabled correctly on your XenApp 6.5 HRP01 environment and would like to share with you

  • The Universal Print Server feature comprises a client component – Universal Print Client (UPClient) that needs to be installed on the XenApp server along with Hotfix Rollup Pack 1 (HRP01) that provides the necessary updates implementing support for the UPS. When installed, the two items will appear in the installed programs list in the server Control Panel->Programs->Programs and Features

Note: The Citrix Universal Print Server package also contains Group Policy Management software that Administrators can install on the system from where they wish to manage the UPS policies. It can be a XenApp server (see Citrix Group Policy Management (x64) item).

  • After configuring the use of the UPS and the Citrix Universal Print Driver (UPD) in the relevant Citrix policy rules:

scription: UPS comp policies.png

Review the following registry locations on the XenApp server to verify the implementation:

scription: Ups enable reg3.png

scription: Ups enable reg2.png

In this example, value 2 indicates UPS has been enabled with no fallback to Windows native remote printing.

Installing the UPClient software modifies the print provider order list to be as expected.

escription: Untitled.png

The Citrix universal printing provider module filename and location can be checked here.

  • It is also possible to verify if the Citrix UpProv.dll is in use and loaded into the Print Spooler by issuing the following command in the command line window.

tasklist /m /fi “imagename eq spoolsv.exe”

scription: E:UPS Screen-shotsSpooler loaded dlls.png

Here’s the link to the original document.

Citrix VDI-in-a-Box – 1030 Connection Error

Issue:

Was helping out a friend with a deployment of Citrix VDI-in-a-Box for their company.  After setting up the environment we kept receiving a 1030 Connection error when accessing the virtual desktops from an external connection which utilized CAG 5.04.  After thinking that some ACL in the firewall was missing and waited around for the network folks to return emails, I noticed a very important step you need to configure inside vdiMgr.

Checked all the usual places
  • Is the STA generated from the vdiMrg in the CAG.
  • Used an SSL checker to see if the SSL was created correctly.
  • Checked that the vDesktop DHCP range is in the ICA access control list on the CAG.
  • CHecked that the correct ports are opened up on the firewall.

I found the issue by looking at the default.ica file from WI, and noticing the “Internal HDX gateway IP Addresses” is wrong inside the ICA file and seeing if it has been marked with the internal IP address.

Fix:
If you log in to the vdiMgr console and go to advance properties and look under gateways ensure that you have specified the “Internal HDX gateway IP Addresses” which HAS TO point to the internal IP address of the CAG.

Server Name Indication (SNI) and SSL VPN

Been working a lot around PCI Citrix initiatives for the last couple of months.  After speaking with some very smart Citrix NetScaler folks, they were kind enough to provide a lot of information to me around security and SSL VPN which I am sharing with you.  Take notice that Windows XP and IE 7 is not supported and it puts so many companies at risk.

Many People are unfamiliar with Server Name Indication (SNI).  In a nutshell, when client computer browsers or SSL based VPNs are negotiating encryption with a server, there is no information which can be gleaned by the server in order to determine which virtual host the client is actually requesting.  This is due to the fact that the HOSTNAME of the subsequent request is contained in the encrypted header which would not be visible until after the received data was decrypted as it made its way down the stack.  This is problematic with respect to virtual hosting since each server or appliance can serve many hosts through the same address.  If it is desired to secure the data of that host through SSL, then a 1:1 mapping of hostname to IP address is currently required (not so good)

Client: (TLS Handshake) Hello, I support XYZ Encryption.
Server
: (TLS Handshake) Hi There, Here is my Public Certificate, and lets use this encryption algorithm.
Client
: (TLS Handshake) Sounds good to me.
Client
: (Encrypted) HTTP Request
Server
: (Encrypted) HTTP Reply

What about ‘STARTTLS’ or TLS ‘Upgrade’ in HTTP/1.1?

STARTTLS is another standard which is commonly used by protocols such as SMTP, POP, IMAP, and LDAP.  Back in the day, it was common practice to have parallel secure ports for most protocols.  For example,  with SMTP, POP, IMAP, and LDAP, and HTTP you have 25/465 110/995 145/993 389/636, and 80/443 respectively. The idea of  STARTTLS was born when the IETF which governs internet assigned numbers and ports decided back in 1997 at some meeting that the issuing of paralell “secure” ports for all protcols should be depricated.   With STARTTLS, when the connection to the server host is established, the client sends a plantext command with the virtual host name.  This has enough information for the server to decide which certificate to offer for the SSL/TLS handshake.

Client: (TLS Handshake) Hello, I support XYZ Encryption.
Client: (Cleartext) I am using server ‘access.mycompany.com
Server: (Cleartext) By The Way, I also support TLS Encryptionn.
Client: (Cleartext) Lets use Encryption, aka ‘STARTTLS’.
Client: (TLS Handshake) Hello, I support XYZ Encryption.
Server: (TLS Handshake) Hi There, Here is my Public Certificate, and lets use this encryption algorithm.
Client: (TLS Handshake) Sounds good to me.
Client & Server: (Encrypted) Exchange Data
 
 A similar method for web browsers, and SSL VPN clients was derived in the HTTP/1.1 specification and is called TLS Upgrade. HTTP/1.1 TLS Upgrade method can be applied to upgrade an open HTTP connection. In a nutshell, the client would include this in a request:
 
GET http://access.mycompany.com/securestuff HTTP/1.1
Host: access.mycompany.com
Upgrade: TLS/1.0
Connection: Upgrade
 
The server in turn might respond with:
HTTP/1.1 101 Switching Protocols
Upgrade: TLS/1.0, HTTP/1.1
Connection: Upgrade
 
The main benefit with these methods are that you can have both naked and secure traffic traversing through the same  port.  Main problems to this and likely why these methods have not been adopted are that all methods would require that any proxy servers in between the client and server also support this method.  A proxy server that did not acknowledge it or perhaps strips the command (could also happen on a legacy firewall), would potentially present a man-in-the-middle attack.  A lesser issue might be that you have a user perception issue as there is a certain familiarity with port 443 being the “secure” port.  On the server end of things, you would also need to have the unsecure port open for the application in question which may not be the case.

The Solution

An extension to SSL/TLS called Server Name Indication (SNI) addresses this issue by sending the name of the virtual host as part of the SSL/TLS negotiation. This enables the server to bind the correct virtual host early and present the browser with the certificate containing a CN matching that in the SNI header.  This method also has far fewer complications associated with it as compared to TLS Upgrade or STARTTLS.  The SNI extension is described in gross detail here. With SNI, you would have a sequence like:

Client: (TLS Handshake) Hello, I support XYZ Encryption, and I am trying to connect to
access.mycompany.com‘.
Server: (TLS Handshake) Hi There, Here is my Public Certificate, and lets use this encryption algorithm.
Client: (TLS Handshake) Sounds good to me.
Client: (Encrypted) HTTP Request
Server: (Encrypted) HTTP Reply
 

But Don’t Browser’s and Servers need to support this extension in order for this to work?

Yup, that is the idea.  As with any RFC, extension, or modification you have to have adoption by the software developers, and hardware vendors which in turn are driven by the adoption or request of the technology by the IT community.  The latter is only done through education and practical application examples which is one of my main drivers for writing this blog post.  At the time of this writing, there are no known Remote Access Appliances which support this RFC extension.  Below is the list of known browsers, servers, and SSL application libraries which do support the SNI extension:

Browsers

Servers

  • Apache with mod_gnutls or mod_ssl
  • Cherokee if compiled with TLS support
  • New versions of lighttpd 1.4.x and 1.5.x
  • Nginx with an accompanying OpenSSL built with SNI support
  • OS X 10.5.6

Libraries

  • Mozilla NSS
  • OpenSSL
    • 0.9.8f – not compiled in by default, can be compiled in with config option ‘–enable-tlsext’.
    • Unreleased 0.9.9 is likely to include SNI compiled in by default.
  • GNU TLS

Unsupported Operating Systems Browsers, and Libraries

The following combinations do not support SNI.

  • Windows XP and Internet Explorer 6 or 7
  • Konqueror/KDE in any version
  • Microsoft Internet Information Server IIS
  • Sun Java System Web Server
  • Microsoft.Net
  • Sun Java JSEE

What SNI could add to SSL-based VPN Solutions?

So what does this mean with respect to Remote Access Solutions such as Citrix Access GatewayF5 Firepass, or Juniper Secure Access remote access solutions?  The benefits of adopting support for SNI are wide an varying, but here is my first pass at a few:

  • Since the SNI would be presented to the access appliance before the SSL negotiation finalized, specific SSL policies such as supported cipher suites, could be bound to the session.   This would be useful where you needed to meet more stringent security requirements such as FIPS level 1/2 for specific hosts, or where you had a client application which used a specific type of encryption that you needed to utilize.
  • As cloud computing is becoming more prevalent, service providers are going to need a means to offer customers secure access to their applications and content.  Since many cloud services are based on anycast addresses (floating IP), CNAME records, and also servicing 1000′s of users, a 1:1 option for customer:IP is not practical or possible. SNI presents a cheap, workable alternative to having no secure offering.
  • Enterprises who wish to publicly expose their intranet or line of business applications securely may want to do so through a remote access appliance, but not want to allocate multiple public IP addresses.
  • Businesses who have only been allocated a single IP address and are using Port Address Translation (PAT) to serve up multiple applications.  This is actually pretty common since many businesses are provisioned with ADSL which uses DHCP assign IP addresses. Most companies use a remote access device as an all-in-one solution for outbound RNAT, inbound VPN, and line of business applications, and firewall.

Outlook’s taskbar icon occassionally shows as PowerPoint

While working on a XenApp 6.5 project for a large company, I heard from one of the IT Directors that Outlook 2010 was showing the PowerPoint icon in the taskbar… Naturally I immediately attempted to reproduce the issue and was not able to.

Microsoft confirms this bug can affect you in the following scenario.

  • This is only happening on Windows 2008 R2 (Std or Ent Editions)
  • This is happening over RDP and ICA sessions, regardless
  • This is happening to Office 2010 (x64), Office 2007, and Office 2003 suites.

What I noticed after scratching my head a bit, this bug only occurred on the first launch of Outlook in an RDS/XenApp Desktop session.  Once the app was closed and reopened the icon showed the correct Outlook icon.  That is until you logged out and back in again and the PowerPoint icon came back for the initial launch of the app, of course completely randomly.

After doing some research, I was able to find a private hot fix from our friends at Microsoft, after applying the patch the issue was resolved.

Incorrect Outlook icon

Provisioning Services antivirus exclusions

Citrix wrote an excellent article about the PVS Antivirus exclusions and thought I will share it with you.  In my experience with PVS, this is very crucial step as it will ensure you don’t interrupt the streaming process and/or slow things down.

If you are like me and decide to run the TFTP boot process on separate servers, you can create TFTP HA utilizing the NetScalers, if you decide to go this route which I recommend, you will need to exclude the TFTP directory on the separate TFTP hosts.  You can read about how to HA the PVS  TFTP boot process via the NetScalers from a previous post I wrote, which let me tell you it was no easy task.

To read the entire Citrix article about the Antivirus exclusions, click here.

A few recommended Server Side file exclusions.

C:\Windows\System32\drivers\CVhdBusP6.sys => (PVS 6.1)
C:\Windows\System32\drivers\CVhdBus2.sys => (PVS 5.6)
C:\Windows\System32\drivers\CFsDep2.sys => (PVS 5.6 and PVS 6.1)
C:\Program Files\Citrix\Provisioning Services\BNTFTP.EXE => (PVS 5.6 and PVS 6.1)
C:\ProgramData\Citrix\Provisioning Services\Tftpboot\ARDBP32.BIN => (PVS 5.6 and PVS 6.1)
D:\Store => ( i.e. local vdisk store)

A few recommended Server Side processes to be excluded.
C:\Program Files\Citrix\Provisioning Services\StreamService.exe => (All versions)
C:\Program Files\Citrix\Provisioning Services\StreamProcess.exe => (All versions)
C:\Program Files\Citrix\Provisioning Services\soapserver.exe => (All versions)

A few recommended Target Device exclusions.
C:\Windows\System32\drivers\bnistack.sys => (Only targets, Win2003/XP)
C:\Windows\System32\drivers\bnistack6.sys => (Only targets, 2008/Win7)
C:\Windows\System32\drivers\BNNF.sys => (Only targets)
C:\Windows\System32\drivers\BNNS.sys => (Only targets, Win2003/XP)
C:\Windows\System32\drivers\BNNS6.sys => (Doesn’t exist anymore with PVS6.1 Agent)
C:\Windows\System32\drivers\BNPort.sys => (Only targets)
C:\Windows\System32\drivers\CFsDep2.sys => (Only targets, Win2003/XP)
C:\Windows\System32\drivers\CVhdBusP52.sys => (Only targets, Win2003/XP)
C:\Program Files\Citrix\Provisioning Services\BNDevice.exe => (Only targets, 2008/Win7)
C:\Program Files\Citrix\Provisioning Services\TargetOSOptimizer.exe => (Only targets, 2008/Win7)

 

Follow

Get every new post delivered to your Inbox.

Join 59 other followers