Most Access Manager deployments are virtualized. There are many advantages to virtualization but there are some disadvantages. The NetIQ documentation provides the basic guidance that you should ensure that the same resources that a physical machine would provide are available to your virtual machine. This article will discuss how to do that for memory.
I’ll be publishing future articles dealing with other Access Manager performance considerations but please contact me if you have a pressing issue or question. Access Manager 3.2 (and later) is extremely scalable but each situation is different so it’s best to work with someone who has had real-world experience.
Large scale Access Manager implementations benefit from large amounts of memory and paging memory to disk rapidly impacts performance. Disk IO bandwidth is usually already an issue due to the gigabytes of logs that a busy site can generate. Adding more disk operations as memory is paged in and out will just slow everything down. For this reason it is highly recommended that you never operate Access Manager in a configuration that requires swapping memory to disk. On a physical machine this can be done by simply ensuring you have enough physical memory installed. On a virtual machine making sure that paging never happens is a bit more complex.
VMWare has several sophisticated methods for managing memory. These methods are used primarily to allow virtual machines to utilize more total memory than is physically installed in the host. The first method that VMWare uses when memory resources are low on the host is called Ballooning. The purpose of Ballooning is to transfer the impact of limited memory resources to the virtual machines. This is done because the host has limited insight on how memory is being used within the guest operating system. The idea is that the guest operating system is best able to decide which memory pages should be swapped to disk. When the host needs to free up some memory it will notify a special driver that is installed as part of the VMTools. This driver then consumes some of the memory that the guest OS believes is physically present in the virtual machine. The guest OS will then swap memory to disk reducing the load on the hosts physical memory. The host will then reallocate that memory to other virtual machines.
This method works very well for many workloads as the guest operating systems are often using more memory than they really need simply because they believe it’s available. However, for some workloads this does not work so well. The first type of workload where this behavior is often detrimental is for Java server applications. Memory for Java application is managed by the Java Virtual Machine (JVM) instead of by the operating system. This means the guest operating system, like the VM host, does not have good information about which memory pages should be swapped to disk. The second type of application that is heavily impacted by memory paging is high throughput servers that use a lot of sophisticated caching. Applications like Exchange servers, databases, and Access Manager benefit from every bit of cache they can get. So since Access Manager is both a Java application and a high throughput server, memory paging is a very bad idea.
So, how do you know if VMWare memory ballooning is affecting your Access Manager servers? The first indication, on Linux but similar principals apply to Windows, is that memory will begin to “disappear” in the “top” and “free” utilities. Lets say you have 25 gigabytes allocated to the JVM on your Identity Provider server. This results in the JVM process using 27.4GB of virtual memory and on my example server actually using about 9.9GB of memory. The 9.9GB is called reserved memory is the amount the guest operating system believes is actually currently in use. If I add up all the reserved memory used by all the processes on my system and then subtract it from the total memory allocated the virtual machine, the result is what top usually reports as “free” memory. On my test box the calculation is:
32GB – 13.2GB = 18.8GB
This is very close to what top and free report as free memory when there is no VMWare ballooning happening. But take a look at the top output below:
Notice that top says I have only 168MB free and that a whopping 31.9GB is being used. If you look through the process list there is no indication where this memory went. What is happening here is that the VMWare balloon driver has consumed over 20GB of memory because this virtual machine is on a VM host that is over committed on memory. Where the memory went is essentially invisible to the standard utilities because it has been allocated directly in the kernel instead of to a normal process.
If you have access to the VMWare console you can see how much memory the VMWare host has asked the guest to balloon but it would be really nice to be able to see this from the guest since often the group managing Access Manager does not have access to the host console. Fortunately, there is a VMTools utility, vmware-toolbox-cmd, that lets you see how much ballooning is taking place.
Note that this value will go up and down as memory conditions on the host change but that Linux will continue to show the memory as used until the guest is rebooted. It’s also possible that the VMWare administrators have set a hard limit on the amount of host memory that the VM can use. This can be seen using the following command:
For more information on VMWare memory management and it’s impact see:
VSphere Memory Management
Configuring Virtual Machines for Java Workloads
Challenges with ESX Memory Management and Java Applications
Special thanks to William Vera of Novell Mexico for leading me to the vmare-toolbox-cmd utility.