Understanding VMware ESX Server Security Profiles

Introduction

VMware ESX Server's built-in software firewall is called the "security profile" for the host server. To be clear, this firewall is the firewall for the entire host - including the service console (if it is not an ESXi server) but not the virtual guests running on the host. Personally, I wish that it was just called the "firewall", but the term "security profile" has "grown on me". I hope that after you read this article it will stick in your mind as well. Let's learn how it works, how to configure it in the GUI & CLI, and why it is important to you as a VMware Admin.

How does the VMware ESX Server Security Profile work?

As the VMware ESX Server security profile is the software firewall of the ESX Server its job is to monitor both inbound and outbound TCP & UDP ports to and from the ESX server. This is done in order to protect the server from network attack.

By default, only specific inbound connections are allowed to a VMware ESX Server. Specifically, (on an ESX 3.5 Server) only SSH and ports related to the VMware Infrastructure & Virtual Center management services are allowed inbound. If you want to access the server with any other applications, inbound, you will have to open that specific port.

Why is the VMware ESX Server Security Profile so important to you as an ESX Server Admin?

The VMware ESX Server Security Profile is important to you, as an ESX Server Admin for a few reasons:

• So that you can understand how your ESX Server is protected from attack and so that you can properly secure your server.
• If there are ESX services that you want to enable, such as FTP or NTP, you will need to open security profile ports.
• If you install any 3rd party applications on the server, you may need to open ports.

How do I configure Security Profiles in the VMware ESX Server VI Client?

To configure security profiles in the VMware Infrastructure Client (VI Client), open the client, log in, and click on an ESX Server, as you see in Figure 1 below.

Figure 1: Accessing the VMware ESX Server Security Profile

Next, you would click on the Configuration tab, then on Security Profile (under Software), as you see in Figure 1.

From here, you can see (on the left) what security profile (firewall) ports are opened on your server (both inbound and outbound). For example, on this server, you can see that SSH and CIM services (used for the VI Client and Virtual Center) are all opened, inbound. Outbound, SSH, Virtual center, VMware License server, iSCSI, NTP, and VCB are all open.

Figure 2: Viewing Security Profile Status and Configuring Security Profile Properties

So how do you change what ports are open, inbound and outbound? The answer is to click on the Properties for the security profile, as you see in Figure 2, above.

Once you click on the security profile properties, you will get a new window that looks like this:

Figure 3: Configuring Security Profile Properties

From the Security Profile properties window, you can enable the preconfigured applications & ports.

Let’s say that we wanted to enable SNMP services inbound and outbound. To do this, just check the checkbox next to that service. In our case, I enabled the SNMP Server port, allowing UDP traffic on port 161 inbound and UDP traffic on port 161. Notice that SNMP is not connected to a particular daemon, as the SSH server is. To apply changes, click OK.

There are times when you need to open a port in the firewall for various applications. For example, if you want to use iSCSI.

If a port is connected to a daemon and you select that port, you can click on the Option button for that port and see the services associated with it, like this:

Figure 4: Daemon / Service Properties

As I did not want to make any changes to the service, I just clicked OK.

Notice that you are limited to the preconfigured applications and whatever their specific inbound or outbound port that is preconfigured for that application. Plus, from the GUI interface, you cannot add any new ports or applications.

How do I configure the ESX Server security profile from the command line (CLI)?

To configure the security profile from the command line, use the esxcfg-firewall command. You would first, of course, have to SSH to the ESX Server and log in first before you can use this command.

The command syntax is simple. To see all the command options, just type the esxcfg-firewall command by itself, and press enter (see Figure 5, below).

Figure 5: esxcfg-firewall command syntax

To view open ports, use the esxcfg-firewall -q command line option.

To open a specific port, you would type a command similar to this:

[root@ESX3 root]# esxcfg-firewall -o 1000,tcp,in,test

However, don’t expect your CLI change to show up in the GUI interface.

You can also configure a port range, like this:

[root@ESX3 root]# esxcfg-firewall -o 1000:1050,tcp,in,test

Conclusion

In this article, you learned about VMware ESX Server’s built in software firewall - the “security profile”. You found out about how the security profile is the firewall for the entire host – including the service console (if it is not an ESXi server) but not the virtual guests running on the host. After that, we covered how the security profile works, how to configure it in the GUI & CLI, and why it is important to you, as a VMware Admin.

The Art of Patching your Virtual Infrastructure (Part 1)

Introduction

Updating systems is an essential step in an enterprise environment. Today's environments demand that the solution to managing patches be an automated controlled process. It is thus vital to have a patch management system (such as Microsoft’s Windows Server Update Services aka WSUS or VMware’s Update Manager) so the updates can be rolled out in batch to a (select) group of systems in your environment.

In part 1, we will discuss the old school way of patching an ESX host, but also look at VMware’s own Update Manager (introduced in VI 2.5) to baseline and patch your ESX hosts.

Patching the ESX host the old way

Before Update Manager was available, a plethora of tools were used to patch an ESX host. The most basic tool used is the built-in command esxupdate. We will not go in to details on how to use it. Suffice to say that it needs a tarred patch (called ESX-1002969.tgz or similar) to work and you need to execute it on the Service Console command prompt once the host is put into maintenance mode.

Because this is a rather manual process there were (and still are) a number of community tools freely available such as:

• VMTSPatchManager
• ESX-autopatch
• My own autopatching method using an IIS repository

Next to these more complex patching systems, it is also possible to write a (simple) bash script to run and update your system such as the one below:

#!/bin/bash
# esxAllPatches.sh -- auto unpack and update esx 3.0.2
# by Vincent Vlieghe
# Version 14/01/2008
mkdir /var/updates
function installPatch
{
tar -xvzf /tmp/ESX-$1.tgz
mv /tmp/ESX-$1 /var/updates
esxupdate -n -r file:/var/updates/ESX-$1 update
rm -rf /var/updates/ESX-$1
rm /tmp/ESX-$1.tgz
}
# Patches 15/11/07
installPatch 1002424
installPatch 1002425
installPatch 1002429
# Patches 30/11/07
installPatch 1002431
installPatch 1002435
installPatch 1002434
installPatch 1002426
installPatch 1002430
installPatch 1002428

#remove remaining entries & directories

rm -rf /var/updates
reboot

Update manager: what is it?

VMware’s Update Manager is used to create baselines and install software patches/upgrades for ESX hosts and Windows and Linux virtual machines running on them. It is a simple (Jetty) web server service and a download client. The update manager component is installed on a Windows 2003 or Windows XP machine that has access to the internet. It uses port 80 (443) to connect to the VMware (ESX) or Shavlik (guest) to obtain patch metadata.

Figure 1: Update Manager architecture as per the Update Manager manual

All configuration data (next to some options in the HK Local Machine registry hive) is saved in a file called vci-integrity.xml located under \Program Files\VMware\Infrastructure\Update Manager. Make sure you use the VI Client plug-in for Update Manager (as discussed later in the article) to change the global configuration settings!

Update Manager: installation and configuration

Ok, let’s look at the installation steps. The first thing to note is that it needs a separate database to function. You have only a few options: SQL Express, SQL Server or Oracle. You could use a SQL Express edition that comes with the installation package, but we suggest avoiding this in a production environment. As a first step you should ask your DBA to create a new database, install the SQL Native client for SQL Server 2005 on the Update Manager server and then create a data source name (DSN) pointing to the SQL database created.

The Update Manager component can be installed on the VirtualCenter server itself or on a separate server. We would suggest the second scenario: you install the Update Manager component on a separate server in a separate domain (DMZ maybe) and connect to the VirtualCenter server. There must be 18Gb+ of free storage or the installer will complain. During installation, select Custom and check VMware Update Manager (figure 2 below)

Note:
To determine the disk and database size needed for your environment, make sure to use the VMware update Manager Sizing Estimator.

Figure 2: Components selection window

The next window is an important one: the credentials are used to connect to VirtualCenter itself and register the extensions used by Update Manager.

Figure 3: Link Update Manager to your VirtualCenter host

Another important step in the installation procedure is determining what ports to use for the Update Manager. If the default settings are kept during the installation, the Update Manager Web server listens on 9084 TCP and the Update Manager SOAP server listens on 8084 TCP. Both are accessed through a reverse proxy that listens on the standard ports 80 (and 443). If you have IIS installed for example (let’s say you combine WSUS and VMware update Manager on the same server), you must change port 80 to something else.

It is also possible to configure the Proxy server and port. The strange thing is that when using the Update Manager plug-in in the Vi Client you can use authentication (fill out a username/password)… Why is it not present in this window?

Figure 4: Port and proxy settings window

As a final step you must configure the location for downloading all patches. We suggest putting all patches on a separate (large enough!) partition.

Note: If you want to change your patches download location after installing Update Manager, refer to the following KB article.

Once the installation finishes, launch your Vi Client, go to Plugins => Manage plugins and click the Download and install… button. After restarting Vi Client, go back to Plugins => Manage plugins and enable the VUM client. Once you click this button, a connection will be established to you Update Manager Server.

Note: If you get an error downloading and installing the VUM client, go to \updateManager\VMware-UMClient.exe on the VirtualCenter media and install it manually

Figure 5: Plug-in manager

A new icon will appear in you VI Client toolbar, called Update Manager

Figure 6: Update Manager in the Vi Client toolbar

Once this is done, you will notice a number of extra Update Manager tabs in the VI Client (for hosts and guests).

Using Update Manager

Update Manager is rather simple to use. You need to create 1 or more baselines, attach these to your ESX hosts, Scan the ESX host and remediate it when not compliant.

Let’s start with creating a baseline for our ESX hosts. Click the Update Manager icon. On the Baselines tab right-click an empty space and select New baseline… Note that you have to specify a baseline target: ESX server itself or VM updates. In this article, we will focus on patching ESX hosts so we select ESX server updates.

Figure 7: Creating a baseline

As you can see on Figure 8 you get a number of options based on product type, severity, language and update vendor. Note the Released date field. ESX update releases are cumulative. A new update release contains all fixes from the previous update release. The after field should thus be set to the release date of the latest version: when you apply ESX 3.5 U2 you should set the after date to the date it was released.

So in our example we select 13 August because that is the release date of ESX 3.5 U2. If you don’t set the after date, Update Manager will show that Update 1 is not present and so the ESX host is NOT compliant. This is a known issue.

Figure 8: Selecting baseline options

The next step is to actually use the baseline: left-click on an ESX host in the VI Client and on the Update Manager tab attach the baseline.

Figure 9: Attaching a specific baseline to an ESX host

Once connected, right-click the ESX host you want to patch and select Scan for Updates in the menu. The scan will complete and will state if it is compliant or not.

Figure 10: Compliant or not, that is the question

Now, right-click the ESX host you want to patch and select Remediate… in the menu. Sit back and relax as the machine is put into maintenance mode and patched (and hope nothing goes wrong during the process…)

What about VirtualCenter and VCB?

Yes, a good question: Upgrading/updating your ESX host is usually not the only thing on your task list. The first step in a large scale upgrade project (such as 3.0 to 3.5 or 3.5 to 4.0) is upgrading the VirtualCenter and the VMware Consolidated Backup (VCB) to the latest version. This is (of course) a manual process. We suggest testing, testing and testing in a design environment.

Conclusion

In part 1 of this article series, we described how to install and use Update Manager, a feature set introduced in VI 2.5. Using update Manager is far easier then running custom scripts or using community tools. The great thing about it is… it works quite well. Stay tuned for our next article on patching virtual machines and other extra tools!

Server Virtualization Performance

Server virtualization has been a hot topic for a few years now. The concept continues to excite IT managers with the possibility of running multiple OSes on one system. But in the midst of the hype, it's easy to overlook how issues such as CPU overhead can seriously impact server performance. Before you commit to server virtualization, the pitfalls and remedies deserve some exploration.

The Pitfalls

Virtualization technology vendors will claim that they can drive I/O capacities up to wire speed, but they do not discuss the amount of CPU power that is needed to do that. Salsburg says, "Workloads that are data-intensive may utilize far more of your CPU power than you expect. Future hypervisors, working with the processor and HBA/NIC vendors, will drive down this CPU overhead, but that is later on their roadmap." High CPU overhead will cause erratic and degraded performance.

The virtualization of the x86 architecture has been accomplished in two ways: full virtualization and paravirtualization. while paravirtualization offers important performance benefits, it also requires modification of the operating system, which may impact application certifications.

Full virtualization, on the other hand, relies on sophisticated but fragile software techniques to trap and virtualize the execution of certain sensitive, ‘non-virtualizable’ instructions in software via binary patching. With this approach, critical instructions are discovered at run-time and replaced with a trap into the VMM to be emulated in software. These techniques incur large performance overhead (as much as 20 to 40%). This becomes a problem in the area of system calls, interrupt virtualization, and frequent access to the privileged resources.

The successor to full and paravirtualization is native virtualization. He says with native virtualization the VMM can efficiently virtualize the x86 instruction set by handling the sensitive, “non-virtualizable” instructions using a classic trap and emulate model in hardware vs. software. “Native virtualization has just become available on the market in the last nine months. While it is a new approach, it offers considerable benefits to users in performance and ease of implementation. It also protects the investment in existing IT infrastructure. This new approach is worthy of consideration for those planning their next steps in server virtualization.”

Problems & Remedies

Two problems that come to mind are the issues of security and management. A typical three-tier security model (with the Web tier isolated from the application tier, which is isolated from the database tier) cannot be deployed on a single consolidated server today using current hypervisors. "If one tier is infected and this brings down the hypervisor, you have not sufficiently isolated one tier from another"

Regarding management, consolidating many OS images on a single server may diminish the operation costs for the hardware but not for the various OS images. "In addition, virtualization will spawn many more OS images, due to the simplicity of setting them up. The hypervisor vendors are working on better management, but their solutions do not today scale up to an enterprise-level management structure"

It's important to consider the applications and the deployment goals to match the appropriate virtualization technology. Whether virtual machine technology (such as VMware, Xen, or Virtual Iron) or OS virtualization (such as SWsoft Virtuozzo). "New issues are created through virtualization, such as virtual machine sprawl due to the ease of deploying a new virtualized server, as compared with setting up a new physical server"

Virtualization is a technology shift, a process change. "Once organizations move forward in their deployment, they often realize that no one virtualization technology is perfect for every need. For that reason, many organizations are deploying virtual machines for test and development because the big advantage to this technology is the ability to load many different operating systems on the same server. In the same organizations, they are using OS virtualization for high I/O and production applications because it enables density of up to hundreds of virtual environments per physical server"

Companies that plan to virtualize their x86 infrastructure need the right tools and expertise to manage this virtual infrastructure. “The best server virtualization solutions have built-in capabilities such as Live Capacity and Live Migration (transparent workload migration) that enable users to optimize virtual server utilization across a shared pool of resources.” With these types of tools, users can take advantage of policy-driven management capabilities that continuously sample performance data from every server and every virtual server to automatically relocate running OSes and applications from one physical server to another (without losing any state). "This streamlines the management of the data centre greatly while also reducing the potential for error"