PVS Gold image WMI issues fix

As part of a PVS image issue discovery project, I was able to determine that WMI was not working on several gold images which was causing several memory leaks, as well as event viewer complaining just about every 30 seconds.

Problems escalated whenXenApp hosts would completely run our of virtual memory which would end up affecting the overall user experience.

Environment:

  • Windows 2003 SP3
  • Citrix XenApp 5.0
  • PVS 6.1.16

Issue:

WMI not working on several Golden images.  My guess is this image was copied in a broken state and was replicated to many different images.

Fix:

Run the following in command line

  • Regsvr32 %SystemRoot%\System32\wbem\wmidcprv.dll
  • cd /d %windir%\system32\wbem
  • for %i in (*.dll) do RegSvr32 -s %i
  • for %i in (*.exe) do %i /RegServer

The Windows Management Instrumentation Tester window may appear, this is normal and we can go ahead to close it.

If it does not work, I also suggest you run the following commands to repair WMI namespace:

  • net stop winmgmt
  • wmic /NAMESPACE:\\root path “__namespace.name=’wmi'” delete
  • mofcomp %windir%\system32\wbem\wmi.mof
  • net start winmgmt

Restart the computer to check the result. If the issue persists, try the following steps:

  • winmgmt /verifyrepository
  • winmgmt /salvagerepository

XenApp 6.5 – Seamless applications always on top

I noticed that intermittently seamless XenApp 6.5 published applications would stay on top of other windows even when they were not active.  In my case I been utilizing the Citrix Receiver Enterprise 13.1.0.89 to take advantage of the Pre-Launch feature.

Was able to replicate the issue with all Receiver versions 3.x, even with the new 3.4 client which also works with the Pre-Launch feature of XenApp 6.5

I did not experience the same problem with legacy clients 11.2 and 12.1 (Don’t use these clients with XenApp 6.5)

Environment

  • Windows 2008 R2
  • XenApp 6.5
  • UPM 4.1
  • vSphere 4.0 <-  What the heck? I know, I know :P
  • Citrix Receiver 3.x

Cause:

The issue exact cause is unknown, but seems to be related to changes in the gain focus behavior of the new Citrix Receiver 3.x.  I contacted Citrix support on this one, and after they thought I was crazy, I was able to replicate the problem while on the phone.

It seems that a Seamless Application Window does not become the active window if you make a local window active while the seamless connection initiates.

Resolution:

After speaking to a senior Citrix support specialist, he directed me to an old Citrix article.  Applying the Registry key changes mentioned solves the issue (TWISeamlessFlag = 1)

For x86 systems:

1. Edit the registry:
HKEY_LOCAL_MACHINE\Software\Citrix\ICA Client\Engine\Configuration\Advanced\Modules\WFClient

2. Create a string value, REG_SZ, called TWISeamlessFlag with a value data of 1.

For x64 Systems:

1. Edit the registry:
HKEY_LOCAL_MACHINE\Software\ Wow6432Node\Citrix\ICA Client\Engine\Configuration\Advanced\Modules\WFClient

2. Create a string value, REG_SZ, called TWISeamlessFlag with a value data of 1.

Provisioned XenApp servers halt when the license server is unavailable

Ever hear Citrix Engineers say, “If our Citrix License Server goes down, we still have 30 days to recover it?”  Well that is somewhat true.  However if you are running Citrix Provisioning and are not careful, Provisioned XenApp servers stop accepting connections if they are rebooted and the License Server is unavailable.  Read this previous article I wrote on how to HA your Citrix License server with the NetScalers.

You will notice this behavior on your farm by looking at the XenApp Server load and you see a load of 20000, even when there are no users connected.  You can check the load via PowerShell on XenApp 6.x hosts as it comes with the PSSnapin preinstalled or via some GUI tools.

Some examples on how to check your Farm’s Load via PS

A simple qfarm.exe /load” will display your current server load

simple_load

 

Another more complex way is to run the syntax below, which will allow you to see CPU/Memory utilization

Add-PSSnapin Citrix.XenApp.CommandsGet-XAZone | Get-XAServer -onlineonly | Get-XAServerLoad | format-table -auto

better_load

 

Some GUI options are:

  • Farm Nanny (Free) by Michel Stevelmans
  • Control Up (Paid) by the folks from SmartX which I highly recommend.

control_up

Now let’s fix this…

Environment

  • Windows 2008 R2
  • XenApp 6.5
  • UPM 4.1
  • vSphere 4.0 <-  What the heck? I know, I know :P
  • License Server 11.6.1 (or later)

Cause

The root cause for this problem is that the license server cache file MPS-WSXICA_MPS-WSXICA.ini is saved in the install directory \program files (x86)\Citrix\system32\cache, which could be unavailable when the server image is shut down.

Solution

Redirect the MPS-WSXICA_MPS-WSXICA.ini file to a network share. Read CTX131202 for additional instructions, please note this requires HRP01 for XenApp 6.5

  • On the XenApp Server Image, set the following registry value to a file share location with a UNC Path:
  • HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Citrix\Install\
    CacheLocation
  • Create the file share with the following NTFS permissions.
    Note
    : In this example the XA6LIC is the XenApp Server Computer Account not the Citrix License Server.

  • Set the following Share Permissions: 
    Note
    : In this example the XA6LIC is the Xenapp Server Computer Account not the Citrix License Server. If there are more than one computer, an active directory computer group can be used instead of specifying individual computer accounts.

 

Unable to Logoff XenApp 6.5 ICA session

While growing our XenApp 6.5 environment I noticed certain users were not be able to log off and would receive the message “Waiting for System Event Notification Service”.  Immediately when this occurred, other users would start seeing similar issues from the same XenApp server.

Environment:

  • Windows 2008 R2
  • XenApp 6.5
  • UPM 4.1
  • vSphere 4.0 <-  What the heck? I know, I know :P
  • Office Communicator 2007 R2

Cause

The screen remains unresponsive with the “Waiting for System Event Notification Service” pop-up. On the server user has a Communicator (OCS) running and one of the instances is pegging an entire core resulting in users unable to log off sessions.

This will occur when multiple users try to log off a terminal server at the same time while signed in to Office Communicator 2007 R2

Resolution

For an immediate fix, terminate the pegged OCS processes on the XenApp server(s) where users are not able to log off, this instantly fixes the issue for all other users logged on the server.

The permanent solution is to apply this fix (2755391) from Microsoft, however head over and read this article to get more information on the issue.

Original post at the Citrix knowledgebase.

Windows 2008 R2 BSOD error STOP: 0x0000006B

None of us like it when Windows servers fail and give you a nice BSOD, specially when the issue happens after they are patched.

I cannot state in strong enough terms to make sure YOU as an Admin of your environment controls/creates the process and procedures of patching.  Patching folks simply patch, their job is to pull the trigger.  Yes they do best effort to test, but it is never certain that production fully mimics the test environment.

Take a look at this article from an excellent engineer/blogger that I personally know who wrote some of his thoughts on patching.

Now lets fix this BSOD issue… 🙂

Environment:

  • Windows 2008 R2
  • XenApp 6.5
  • UPM 4.1
  • vSphere 4.0 <-  What the heck? I know, I know 😛

Issue:

You have a host that is running Windows Server 2008 R2.  You receive a Stop error message that resembles the following: STOP: 0x0000006B ( Parameter1, Parameter2, Parameter3, Parameter4)
PROCESS1_INITIALIZATION_FAILED Note The four parameters in the Stop error message may vary, depending on the configuration of the computer.

BSOD

Cause:

This issue occurs because the Bootcat.cache file located under %SystemRoot%\system32\codeintegrity is corrupted or because the size of the Bootcat.cache file is changed since the last successful start.

Solution:

  • Check out this MSFT support article, it is mentioned here to delete the Bootcat.cache file and reboot.  This did not work for me.
  • Copy the Bootcat.cache file from another system that is similar to yours (In my case a VM with the same level of virtual hardware), reboot, reinstall VMtools and you are in good shape.

Steps: If option 1 does not work

  • Copy Bootcat.cache file from your VM template
  • Create an ISO or put it in a USB stick.  You can create your ISO with a free tool like ImgBurn
  • Mount the ISO to your VM by clicking on the CD/DVD Drive (in this case VMware), and select “Connect to ISO image on local disk” then select the ISO containing the Bootcat.cache file

mount_iso

 

mount_boot_cache

  • Now reboot the VM and let the System Recovery Process start, select your desired language, then click on “Command Prompt”

system_recovery_reboot

recovery_options

  • Authenticate as one of your local Admin users

admin_log_in

  • Confirm you see the file on the ISO you mounted to your VM (In my case the E: drive)

bootcat_cache_on_iso

  • Now copy the bootcat.cache file form the mounted ISO to %SystemRoot%\system32\codeintegrity\ and verify the file is copied successfully

copy_bootcat_cache

  • Now reboot the VM… and welcome back Windows 🙂

welcome_back_windows

  • When you log back in, I notice some of my drivers were out of whack, make sure you take care of them, a VMTools reinstall did it for me.

post_fix_driver_issues

  • Once VMTools is reinstalled, reboot.
  • When the system is back up, verify your drivers are all fine, patch it, reboot and you are all set 🙂

device_manager_with_hidden

windows_update

 

  • Once the system is back up, just verify all your Citrix Services below.

Check the following services are set to auto and are started:
Citrix System Monitoring Agent Service
Citrix System Monitoring Agent Firebird service
Citrix CPU Rebalancing Service
Citrix CPU Rebalancing Service
XenApp Health Monitoring Service
XenApp XML Service
XenApp MFCOM Service
Citrix WMI Service
Citrix Independent Management Architecture and dependent services
Citrix Print Manager Service
Print Spooler Service
CitrixTools Cloning Service
Citrix Profile Management Service

Check your Citrix Profile Management Service Cache
C:\Program Files x86\Citrix\User Profile Manager\