I’ve talked about how vSphere has been moving towards a “secure by default” stance over the past few years. This can clearly be seen in the new vSphere 6.5 Security Configuration Guide where the number of “hardening” steps are growing smaller with every release. In this blog post we will go over another “secure by default” feature of vSphere 6.5 that provides hypervisor assurance, Secure Boot for ESXi.
What’s New In ESXi 6.5 U3. The ixgben driver adds queue pairing to optimize CPU efficiency. With ESXi 6.5 Update 3 you can track license usage, refresh switch topology and see improvements in the search and Developer Center in the vSphere Client; ESXi 6.5 Update 3 provides legacy support for AMD Zen 2 servers. IMPORTANT: For clusters using VMware vSAN, you must first upgrade the vCenter Server system.Upgrading only ESXi is not supported. Before an upgrade, always verify in the VMware Product Interoperability Matrix compatible upgrade paths from earlier versions of ESXi, vCenter Server and vSAN to the current version.
- May 04, 2017 With ESXi 6.5, we take this capability of the firmware storing digital certificates and validating the boot loader and we build upon that. ESXi is comprised of a number of components. There is the boot loader, the VM Kernel, Secure Boot Verifier and VIBs, or “vSphere Installation Bundles”.
- Feb 18, 2017 vSphere ESXi 6.5 – Unable to take quiesced snapshot ESXi, VMware, vSphere, Windows Server Add comments. This is an issue that effects quite a few people and numerous forum threads can be found on the internet by those searching for the solution.
- If you are installing this VMware ESXi 6.5 as a nested virtual machine, please be sure to Expose Hardware Assisted Virtualization to the Guest OS. This option can be found in the vSphere Web UI, edit the virtual machine settings, virtual hardware and expand CPU – Hardware Virtualization.
One of the coolest things in 6.5, in my opinion, is the adoption of Secure Boot for ESXi. Now, you might say “But my laptop has had Secure Boot since Windows 8, what’s the big deal?”
Well, the “big deal” is that we’ve gone beyond the default behavior of Secure Boot and we now leverage the capabilities of the UEFI firmware to ensure that ESXi not only boots with a signed bootloader validated by the host firmware but that it also ensures that unsigned code won’t run on the hypervisor. Best of all, it’s simple to implement! Let’s dive in!
Secure Boot and UEFI
Vsphere Esxi 6.5 Client
Let’s do a brief overview of UEFI firmware and Secure Boot.
UEFI, or Unified Extensible Firmware Interface, is a replacement for the traditional BIOS firmware that has its roots in the original IBM PC. I would highly recommend reading the Wikipedia overview on UEFI to get a better understanding of all the capabilities it can present. I can also recommend the Ubuntu blog article on how they use UEFI. I’ve consulted both for use in this blog. For the purposes of this article, I’ll focus on how UEFI and Secure Boot relates to ESXi.
In UEFI parlance, Secure Boot is a “protocol” of the UEFI firmware. This capability was designed to ensure that boot loaders are not compromised by validating their digital signature against a digital certificate in the firmware. A typical compromise on your desktop or laptop would be if malware installed a root kit. This would change the digital signature and the UEFI firmware would check and not allow further booting.
UEFI can store whitelisted/valid digital certificates in a signature database (DB) . There is also a blacklist of forbidden certificates (DBX), a Key Exchange Keys (KEK) database and a platform key. These form the basis of a root of trust that begins with the firmware installed on your host.
These digital certificates are used by the UEFI firmware to validate the boot loader. Boot loaders are typically cryptographically signed and their digital signature chains to the certificate in the firmware. The default digital certificate in just about every implementation of UEFI firmware is a x509 Microsoft UEFI Public CA cert. Most UEFI implementations also allow for the installation of additional digital certificates. A typical use for this would be if you were developing a custom boot loader that’s signed against your own certificate. You could install that certificate in the UEFI firmware and UEFI would validate your boot loader against it.
Default certificates are part of the firmware installation from your server vendor, not VMware. When you update your UEFI firmware on your server, the digital certificate(s) are included.
How ESXi builds upon UEFI and Secure Boot
With ESXi 6.5, we take this capability of the firmware storing digital certificates and validating the boot loader and we build upon that.
ESXi is comprised of a number of components. There is the boot loader, the VM Kernel, Secure Boot Verifier and VIBs, or “vSphere Installation Bundles”. Each of these components is cryptographically signed. Let’s step through each of these.
Boot Loader
As mentioned above, the UEFI firmware itself verifies the bootloader’s digital signature to validate bootloader integrity. Normally, with many operating systems, that’s the limit of what happens because the threat of root kits are now mitigated. But not so with ESXi. We go beyond and ensure that all content shipped is cryptographically signed.
The ESXi boot loader is signed with the Microsoft UEFI Public CA cert. This ensures that standard UEFI Secure Boot firmware can validate the VMware boot loader. The boot loader code also contains a VMware public key. This VMware key is used to validate the VM Kernel and a small subset of the system that includes the Secure Boot Verifier, used to validate the VIBs.
VM Kernel
The VM Kernel itself is also cryptographically signed using the VMware private key. The boot loader validates the kernel using the VMware public key it has. The first thing the VM Kernel runs is the Secure Boot Verifier.
Secure Boot Verifier
The Secure Boot Verifier validates every cryptographically signed VIB against the VMware public key. The VMware public key is part of the Secure Boot Verifier codebase. (You can see in the graphic that the VMware Public Key is in two places, the ESXi Boot Loader and the Secure Boot Verifier)
VIB
A VIB is a “package”. It comprises a file archive (TAR g-zipped file), an XML descriptor file and a digital signature file. (Read more here: https://blogs.vmware.com/vsphere/2011/09/whats-in-a-vib.html)
Coles hydra crane parts manual. When ESXi boots, it creates a file system in memory that maps to the contents of the VIBs. If the file never leaves the cryptographically signed “package” then you don’t have to sign every file, just the package.
This means you’re signing an order of magnitude less files, thereby limiting the impact on boot times. And because we have already had that digital signature process in place for years, it was the logical way to extend the Secure Boot capabilities.
The VIBs are signed with the VMware public key and validated with the Secure Boot Verifier.
The boot process
- Host Power On
- UEFI Firmware validates the ESXi Boot Loader against the Microsoft digital certificate in the UEFI firmware
- ESXi Boot Loader validates the kernel against the VMware digital certificate in the Boot Loader
- Kernel runs the Secure Boot Verifier
- Secure Boot Verifier validates each VIB against the VMware digital certificate in the Secure Boot Verifier
- Management apps (DCUI, hostd, etc) now run
Upgrades .vs. Fresh Installs
Because of changes in signing older VIBs, you may encounter some issues if you are upgrading a host from previous ESXi versions to 6.5 and enabling Secure Boot. Also, you may also find out that you have unsigned code running on your older systems. For example, these could possibly be beta drivers for a specific hardware device and usually fall under the “if it ain’t broke, don’t fix it”. The reasons they were never updated are usually lost to folklore. Things like this could be an issue so we’ll go over some steps to help mitigate it.
Personally, I try to treat ESXi servers less like “pets” and more like “cattle” (to use the popular vernacular of the DevOps crowd). I like to build ESXi servers in my lab from scratch every time. Use of Host Profiles can help lessen the impact and there are various other methods for automating and configuring ESXi hosts. Building from scratch ensures everything is “clean” and helps tremendously with troubleshooting issues. Consider adopting this way of thinking if possible. Again, this is just my personal preference.
Possible upgrade issues
UEFI secure boot requires that the original VIB signatures are persisted. Older versions of ESXi do not persist the signatures, but the upgrade process updates the VIB signatures.
If your host was upgraded using the ESXCLI command then your bootloader wasn’t upgraded and doesn’t persist the signatures. King cobra 400 sz unlimited driver. When you enable Secure Boot after the upgrade, an error occurs. You can’t use Secure Boot on these installations and will have to re-install from scratch to gain that support.
If you upgraded using the ISO method then old VIBs may be retained and the Secure Boot process cannot verify the signatures for the old VIB(s) and the boot process will fail. The ISO you use must contain new versions of all installed VIBs that are on the host. This ensures that signatures are updated. You may encounter this if you upgrade a vendor installation with the VMware ISO. If you do this, you will have to reinstall ESXi using a fresh install to enable Secure Boot. Realistic hair sims 4.
Community supported VIBs
Because these VIBs are not signed they are not able to be installed on an ESXi host that has Secure Boot enabled.
Post-Upgrade Secure Boot Check
A script to check your environment after you’ve upgraded is available on ESXi 6.5. Its purpose is to ensure you can enable Secure Boot after you have done the upgrade. One caveat: UEFI secure boot also requires an up-to-date bootloader. This script does not check for an up-to-date bootloader.
Prerequisites
- Verify that the hardware supports UEFI secure boot. You may have to check for a firmware upgrade.
- Verify that all VIBs are signed with an acceptance level of at least PartnerSupported. If you include VIBs at the CommunitySupported level, you cannot use secure boot.
If you have upgraded your host to 6.5 and haven’t tried enabling Secure Boot then you can run a validation script located on the ESXi host. The script is called:
/usr/lib/vmware/secureboot/bin/secureBoot.py -c
The output either includes Secure Boot can be enabled or Secure boot CANNOT be enabled. Rotting christ kata ton daimona eaytoy blogspot.
If Secure Boot cannot be enabled then see “Possible upgrade issues” above. You may have a situation that requires an clean installation. ESXi will continue to run just fine, however you won’t be able to take advantage of Secure Boot for ESXi.
PSOD’s, unsigned VIBs and File Integrity Monitoring (FIM)
PSOD – Purple Screen of Death
If you already have unsigned VIBs on your ESXi host and you enable Secure Boot in the firmware then ESXi will boot into a purple screen and tell you which VIB is unsigned. The error should look similar to this:
To get out of this situation do the following:
- Restart and turn off Secure Boot in the UEFI firmware and boot the host with Secure Boot turned off.
- When booted, log into the host and remove the offending VIB and shutdown.
- Re-enable Secure Boot and restart the host and the system should boot normally.
You can only get into this situation if you have pre-existing unsigned code installed.
File Integrity Management
A customer ask I’ve heard for many years has been the ability to install File Integrity Monitoring (FIM) on ESXi. You can’t do this because ESXi isn’t Linux. It’s structured differently and we don’t allow user-level processes to run. All users run as “root” on VMware and their permissions are controlled with the vSphere API. However, with 6.5 and Secure Boot, you can address File Integrity Monitoring by enabling Secure Boot and collecting SYSLOG output from the ESXi hosts. The combination of these two technologies ensures that only signed code can run and any changes are monitored. Remember, all shell actions are sent out via SYSLOG and can be reported on by your log collection system, like VMware Log Insight.
Another key feature of enabling Secure Boot for ESXi is that you cannot forcibly install unsigned VIBs if Secure Boot is enabled! Commands like the following just won’t work:
The only requirement to leverage this new capability is that BOTH the GuestOS type is ESXi 6.5 (vmkernel65) and the actual OS is running ESXi 6.5. The underlying physical ESXi host can either be ESXi 6.0 or ESXi 6.5. In addition to new virtual hardware defaults, I have also found that the new ESXi 6.5 GuestOS type now uses EFI firmware over the legacy BIOS compared to previous ESXi 6.x/5.x/4.x GuestOS types.
For customers who wish to push their storage IO a bit more for Nested ESXi guests, this is a great addition, especially with lower overhead when using a PVSCSI adapter.
GuestOS Customization
One of the very last capability that has been missing from Nested ESXi is the ability to perform a simple GuestOS customization when cloning or deploying a VM from template running Nested ESXi. Today, you can deploy my Nested ESXi Virtual Appliance which basically provides you with the ability to customize your deployment but would it not be great if this was native in the platform? Well, I am pleased to say this is now possible!
When you go and clone or deploy a VM from template that is a Nested ESXi VM, you will now have the option to select the Customize guest OS option. As you can see from the screenshot below, you can now create a new Customization Spec which is based on the Linux customization spec. The customization only covers networking configuration (IP Address, Netmask, Gateway and Hostname) and only applies it to the first VMkernel interface, all others will be ignored. The thinking here is that once you have your Nested ESXi VM on the network, you can then fully manage and configure it using the vSphere API rather than re-creating the same functionality just for cloning.
To use this new Nested ESXi GuestOS customization, there are two things you will need to do:
- Perform two configuration changes within the Nested ESXi VM which will prepare them for cloning. You can find the configuration changes described in my blog post here
- Ensure BOTH the GuestOS type is ESXi 6.5 (vmkernel65) and the actual OS is running ESXi 6.5. This means that your underlying physical vSphere infrastructure can be running either vSphere 6.0 Update 2 or vSphere 6.5
You can monitor the progress of the guest customization by going to the VM's Monitor->Tasks & Events using the vSphere Web Client or vSphere API if you are doing this programmatically. Below is a screenshot of a successful Nested ESXi guest customization. If there are any errors, you can take a look at /var/log/vmware-imc/toolsDeployPkg.log within the cloned Nested ESXi VM to determine what went wrong.
![Vsphere Esxi 6.5 Vsphere Esxi 6.5](https://www.ntweekly.com/wp-content/uploads/2016/11/112216_0507_14ThingsYou1.png)
I know this will be a very welcome capability for customers who extensively use the guest customization feature or if you just want to quickly clone an existing Nested ESXi VM that you have already configured.
Pre-vSphere 6.5 enablement in vSphere 6.0 Update 2
By now, you probably have figured out what this last enhancement is all about ? It is exactly as it sounds, we are enabling customers to try out ESXi 6.5 by running it as a Nested ESXi VM on your existing vSphere 6.0 environment and specifically the Update 2 release (this includes both vCenter Server as well as ESXi). Although this has always been possible with past releases of vSphere running newer versions, we are now pre-enabling ESXi 6.5 specific Nested ESXi capabilities in the latest release of vSphere 6.0 Update 2. This means when vSphere 6.5 is generally available, you will be able to test drive some of the new Nested ESXi 6.5 capabilities that I had mentioned on your existing vSphere infrastructure. This is pretty darn cool if you ask me!?
Virtual NVMe support
Vsphere Esxi 6.5 Client Download
I had a few folks ask on whether the upcoming Virtual NVMe capability in vSphere 6.5 would be possible with Nested ESXi and the answer is yes. Please have a look at this post here for more details.
For those of you who use Nested ESXi, hopefully you will enjoy these new updates! As always, if you have any feedback or requests, feel free to leave them on my blog and I will be sure to share it with the Engineering team and who knows, it might show up in the future just like some of these updates which have been requested from customers in the past ?