One of the little known or little explored secrets with servers running VMware ESXi 5.x is they can be booted off of USB or SD flash media. In such a scenario, enterprises could dump HBAs or RAID cards and the hard drives or SSDs that are used to boot the servers. Going this route yields a CAPEX benefit for new systems, the cost of flash media for boot is insignificant compared to the drives and storage adapter cards required to spin up a server. There are also gains in OPEX; the implications of moving to USB or SD flash media have a big advantage when considering the power required for a small flash media drive for boot vs. the more common practice of adapters and drives.
One of the little known or little explored secrets with servers running VMware ESXi 5.x is they can be booted off of USB or SD flash media. In such a scenario, enterprises could dump HBAs or RAID cards and the hard drives or SSDs that are used to boot the servers. Going this route yields a CAPEX benefit for new systems, the cost of flash media for boot is insignificant compared to the drives and storage adapter cards required to spin up a server. There are also gains in OPEX; the implications of moving to USB or SD flash media have a big advantage when considering the power required for a small flash media drive for boot vs. the more common practice of adapters and drives.
In this piece we specifically look at the impact that flash media can have in data center operations. VMware actually has a knowledge base article on how to deploy USB or SD flash media as a boot device for ESXi 5.x and most server vendors offer configurations with SD media installed for operations such as this. The VMware KB piece is pretty straight forward and recommends a deployment of a drive no less than 8GB in capacity, so there's room for a scratch partition on the drive. VMware goes on to suggest a 16GB or larger drive is ideal so that the extra flash can be used for wear leveling to prolong the live of the drive. Given the cost delta between 8GB and 16GB, there's little reason to not opt for the 16GB drives. The only other consideration is card quality, which is why we've selected SanDisk for this project. SanDisk consistently proves their leadership position in portable flash media, routinely besting all others in performance and warranty. While throughput is less of a concern here, most servers are using USB 2.0, the heritage is important, as even data centers relying on clustering for virtualized servers appreciate any improvement in component failure rates.
The SanDisk Cruzer Fit tucked into the RD630 motherboard
We're using ten of the SanDisk Cruzer Fit USB Flash Drives for this test. The Cruzer Fit is a low-profile, unobtrusive flash drive that can hide away on internal server USB ports, or sit close to the server if needed to be mounted in front. It is stripped of any embellishments (other than its general ergonomic design) and is as minimalistic as it gets for flash drives. The 16GB version retails for a little more than $10 and scales up to 64GB with a retail of under $60.
How to Install ESXi on Portable Flash Media
VMware offers a step-by-step set of instructions in the KB article on how to accomplish this. We've simplified the steps some, opting for a virtual optical drive rather than burning and booting from optical media. Our process for installation is as follows:
- Download the ESXi 5.x ISO image from VMware that you wish to install and either load it on a USB thumbdrive with a utility such as UNetbootin or mount the ISO remotely through your server's iKVM portal.
- When turning on your server, be prepared to hit the key required to enter the BIOS or to change the boot order so the system loads the VMware ESXi 5.x ISO.
- Let the server load the VMware ESXi software until it prompts you for user input. At this time press F11 to accept the user agreement and then select the device you wish to install ESXi onto. In our case we chose the SanDisk Cruzer Fit as our storage device.
- Select the keyboard layout that best fits your region, and set the root password for your ESXi 5.x host. Next accept you installation settings and let the installer load the software onto your storage device.
- When the installation has finished, the installer will now prompt to restart your server. Use this opportunity to change the boot order of your server to automatically load from the device you installed ESXi onto.
After the system boots, users will be presented with a warning that there is no local datastore and logs are being stored on non-persistent storage. To fix these issues, attach your networked shared storage of choice, such as iSCSI, FC or NFS storage.
Once one or more datastores are attached to the ESXi host, to solve the log warning, go into the advanced settings and assign a location for the system logs. In our case we used our datastore "Storage" for this purpose.
USB Flash Drive as ESXi Boot Drive
In testing we started out by deploying a Lenovo ThinkServer RD630 in a variety of different configurations and testing power consumption at idle via Eaton Advanced ePDU to get baseline stats. These figures were polled after VMware ESXI 5.1 was installed, booted, and sitting idle in the "Balanced" power mode.
We leveraged three HDD and one SSD configurations in RAID1 connected through the RD630's stock LSI 9260-8i RAID card with battery backup. The last configuration was the USB thumbdrive alone with the RAID card removed.
- (2) 1TB Seagate Constellation.2 7K SAS HDDs in RAID1 – 159W
- (2) 600GB Toshiba MBF2600RC 10K SAS HDDs in RAID1 – 156W
- (2) 147GB Toshiba MK01GRRB 15K SAS HDDs in RAID1 – 170W
- (2) 200GB Seagate 600 Pro SATA SSDs in RAID1 – 163W
- (1) 16GB SanDisk Cruzer Fit – 135W
In the five configurations tested, we measured power savings ranging from 21 to 35W, which translates into significant power savings over the span of each year per server. With an average cost of $0.12/kWh, power savings ranged from $22 to just under $37 per year. When those savings are multiplied across many servers, power savings add up. In addition to sheer power savings though, the reduction in components also leads to reduced heat and potential savings in cooling.
Beyond the OPEX savings there are huge savings on the CAPEX side. In just the scenarios tested, the RAID card and base HDDs or SSDs range from $800 to as much as $1,500 when configuring servers from popular tier 1 brands. This means that when deploying a handful of servers, either businesses can take advantage of the savings by spending less, or the same spend can go further. This could mean another server node or two for a cluster, or funds could be allocated to invest further in CPUs and memory to support the virtualized environment.
This plan of attack may not be for everyone, the traditional RAID card or HBA paired with drives will still dominate boot duty for most. At scale though, these expenses add up, not to mention the OPEX benefits of reducing power consumption in the highly virtualized data center. For a $10 per server investment though, it's hard to figure another way a Hamilton could be better spent.
