With VMware vSAN 8 Update 2 GA now, vSAN customers are likely ready to take the leap to the new storage architecture.
This report is sponsored by Western Digital. All views and opinions expressed in this report are based on our unbiased view of the product(s) under consideration.
VMware vSAN 8 heralded a massive update for the company’s flagship HCI product, improving efficiency, scalability, and performance. Since the initial release, vSAN 8 has seen two updates that enhance the story. Critical in vSAN 8 is the Express Storage Architecture (ESA), a new alternative to the Original Storage Architecture (OSA), simplifying the storage layout and many other improvements.
VMware vSAN ESA Cluster
With vSAN 8 Update 2 available, many enterprises are nearing a decision on which vSAN deployment to undertake. As such, we wanted to better understand the performance profile of vSAN 8 ESA paired with modern servers, flash, and networking.
Dell R6625 Servers – Western Digital Ultrastar DC SN655 NVMe SSDs
We started with 32x Ultrastar DC SN655 NVMe SSDs from Western Digital, four Dell PowerEdge R6625 servers, and a 100GbE fabric. But before we dive into the various RAID and drive configuration results, let’s do a quick primer on vSAN 8 ESA and why this architecture change is so significant.
What Does vSAN 8 ESA Bring to the Table?
VMware’s vSAN 8, featuring ESA, marks a significant advancement in storage technology. This new architecture, tailored to exploit the full potential of modern hardware, offers unprecedented efficiency, scalability, and performance. Reflecting the evolution of enterprise storage from traditional spinning media to advanced NAND flash, ESA leverages NVMe interfaces for enhanced capacity and speed at reduced costs. The shift in hardware technology, increased server CPU core counts, and advanced networking capabilities have been a driving force behind the introduction of ESA, ensuring that it meets contemporary business demands for high-performance storage solutions.
ESA in vSAN 8 is distinguished by its performance enhancements and capacity improvements. The adaptive write path in ESA caters to various I/O sizes, optimizing data writing processes, which is particularly beneficial for flash devices. This results in lower latency and consistent performance for virtual machines. In addition, ESA’s new log-structured file system and IO path optimizations significantly reduce write amplification and latency. ESA maximizes device capacities by removing the need for dedicated cache devices, reducing the overall cost per gigabyte. These features underscore ESA’s role in providing efficient, high-capacity storage solutions at an affordable price point.
Another key aspect of ESA is its resource efficiency, delivering robust performance and space efficiency while consuming fewer CPU resources. This efficiency is vital in managing large data volumes cost-effectively. Despite its high performance, ESA does incur storage capacity overheads for metadata, filesystems, and resilience. However, it handles these overheads efficiently to maximize usable capacity. Furthermore, VMware offers flexibility to its customers by allowing them to select between the original storage architecture and ESA, catering to current and future storage needs. This flexibility, combined with ESA’s advanced features, positions VMware’s vSAN 8 as a forward-thinking solution in the storage technology landscape.
Even with this significant change, the fundamental tenants of vSAN and primary use cases don’t change much. VMware vSAN continues to serve as a versatile storage solution tightly integrated with VMware’s vSphere, finding widespread application across various workloads. It excels in virtual desktop infrastructure (VDI) environments, delivering scalable and high-performance storage for numerous virtual desktops. In scenarios like edge and remote office/branch office (ROBO) deployments, vSAN streamlines infrastructure and operations. vSAN also scales way up, providing the underpinnings for software-defined data centers.
vSAN 8 Update 2 Test Platform
The Ultrastar DC SN655 Enterprise PCIe Gen 4.0 dual-port NVMe SSDs represent a significant advancement in solid-state drive technology, offering high performance, increased capacity, and robust reliability tailored for demanding enterprise environments. These drives are particularly well-suited for applications requiring high-speed data access and transfer, making them an ideal choice for integration with VMware vSAN solutions.
Performance
The Western Digital Ultrastar DC SN655 SSDs leverage the PCIe Gen 4.0 interface, providing substantial improvements in bandwidth compared to their Gen 3.0 predecessors. This translates to faster data transfer rates, which is crucial for applications that rely on rapid data access and processing. The dual-port NVMe feature enhances reliability and availability, ensuring uninterrupted data access even during a port failure. This dual-port design is particularly beneficial for mission-critical applications where downtime can have significant implications.
Capacity and Efficiency
With their advanced NAND technology, these SSDs offer high-density storage solutions. The increased storage capacity is vital for enterprises dealing with large volumes of data, such as those in data analytics, cloud computing, and virtualized environments. The Ultrastar DC SN655 series also offers better power efficiency than earlier models, contributing to lower operating costs and a reduced carbon footprint, an essential consideration for eco-conscious enterprises.
Integration with VMware vSAN
VMware vSAN dramatically benefits from integrating Ultrastar DC SN655 SSDs. The high performance and reliability of these SSDs align well with vSAN’s requirements for efficient, resilient, and scalable storage. VMware vSAN environments, known for their demanding, unpredictable performance requirements, can leverage the speed and consistency offered by the Ultrastar DC SN655 to ensure smooth operation and high availability.
Using the Ultrastar DC SN655s with VMware vSAN also simplifies deployment and management, as these drives are designed to integrate seamlessly into a variety of ecosystems. This ease of integration saves time and reduces potential issues, streamlining the management of virtualized storage resources.
The Dell PowerEdge R6625s have much to offer, making them ideal for our VMware vSAN HCI use case. From the compute side, they are dual-socket servers, supporting enough processing power to tackle any workload. In our configuration, we have dual AMD EPYC 9554 64-core CPUs with 128GB of RAM per node. The R6625 also supports up to ten 2.5″ NVMe SSDs, giving them a good-density story, even though our build focused on four and eight SSDs per node.
The PowerEdge R6625 supports many options from a networking perspective, with one OCP 3.0 slot and three PCIe Gen5 slots. This configuration allows us to leverage 100GbE and beyond, with additional slots left available for other devices. Speaking of switching, the Dell PowerSwitch S5232F-ON ties this cluster together. It is a 1U multilayer switch with 32x 100 GbE QSFP28 ports and 2x 10GbE SFP+ ports.
Worth noting here is the work we’ve done in this report is on some components that are either not on the VMware vSAN HCI yet or are not officially supported by Dell as of today. That said, we followed VMware best practices for vSAN throughout this testing and had VMware vSAN engineers review the data.
vSAN 8 Update 2 Performance
In our approach to testing VMware’s vSAN 8 with ESA, we focused on a four and eight SSD per node configuration. With four hosts, this worked out to 16 or 32 SSDs in total.
We leveraged 100Gb Ethernet in the cluster, which theoretically allowed us to focus on the NVMe storage to view where it starts to saturate system resources.
Another variable we measured in this report was the impact of RAID1 and RAID5 on the overall storage performance inside our vSAN cluster. Historically, RAID1 has been a popular storage policy for vSAN deployments. The recommended deployment setting for ESA is “vSAN ESA Default Policy – RAID5,” which offers nearly the same performance with much greater capacity. RAID1 has an immediate 50 percent hit, but RAID5 limits your parity overhead, giving you much more usable capacity.
To measure the performance of our VMware vSAN cluster running ESXi 8 Update 2, we leveraged HCIbench. HCIbench was developed to simplify the testing of HCI clusters, where you need to deploy worker VMs, spawn vdisks, and orchestrate testing across an entire cluster while aggregating all of those results. In our testing, we used the following settings for our four ESA configurations:
HCIBench VM Deployment:
- 16 VMs
- 8 x 50GiB Data Disks per VM
- 16 vCPUs per VM
- 8 GB RAM per VM
- 3600 Second Testing Duration per workload
- 8 Threads per Disk for sequential workloads and 16 threads for random workloads
| Western Digital Ultrastar DC SN655 vSAN ESA DATA | 1024k Sequential Write MB/s | 1024K Sequential Read MB/s | 4K Random Write IOPS | 4K Random Read IOPS | 8K Random 70/30 IOPS |
|---|---|---|---|---|---|
| vSAN ESA 4-SSD Per Node RAID1 | 10,914 | 17,638 | 378,861 | 563,054 | 318,640 |
| vSAN ESA 4-SSD Per Node RAID5 | 10,999 | 17,726 | 476,770 | 524,076 | 301,960 |
| vSAN ESA 8-SSD Per Node RAID1 | 12,702 | 24,128 | 520,632 | 526,504 | 323,674 |
| vSAN ESA 8-SSD Per Node RAID5 | 14,336 | 21,994 | 504,508 | 523,666 | 292,557 |
We split the performance data into two overlapping sections: four vs eight SSDs per node and RAID1 vs RAID5. In our initial comparison, we focused on the point at which performance begins to saturate as more disks are added to the cluster. With four hosts, we looked at 16 SSDs in the cluster compared to 32 when we ramped each node to eight SSDs. The second aspect is how significant of a difference the RAID1 ESA policy makes compared to the recommended RAID5 ESA policy. The benefits of using RAID5 are pretty obvious: users gain a massive amount of available capacity thanks to the data protection policy, paired with the dense Western Digital SSDs.
We noticed a few trends popping up as we looked at the data collected from our testing. If you focus on total bandwidth, the VMware vSAN ESA cluster saw a significant boost in performance by moving from four to eight SSDs per node. We measured about 10.9GB/s on sequential write from both RAID1 and RAID5 configurations with a 4 SSD setup. On sequential read, we measured 17.6-17.7GB/s in RAID1 and RAID5 modes. Shifting towards random workloads, we found that RAID5 had an advantage in the 4 SSD config, measuring 476K IOPS in 4K random write, compared to 379k IOPS in RAID1. On 4K random read, both configurations started to get close to one another, with RAID1 measuring 563k IOPS to 524k IOPS in RAID5. In 8K 70/30, the difference was much closer, with RAID1 having an edge measuring 319k IOPS versus 302k IOPS from RAID5.
Moving to eight SSDs per node, we saw an uplift in performance, but it didn’t double the performance compared to four SSDs. This is where the upper-performance limits of vSAN and the bandwidth limit of 100GbE across our nodes started to appear. Yes, faster network connections are available, but that moves further away from the HCI and typical SMB/SME deployments.
Regarding write bandwidth, RAID5 had the upper hand, measuring 14.3GB/s versus 12.7GB/s from RAID1. RAID1 had the lead on read bandwidth with 24.1GB/s against RAID5 with 22GB/s. With random 4K write transfers, RAID1 and RAID5 were close, although we measured 521K IOPS in R1 against 505K IOPS in R5. As we looked at 4K random read performance, we found ourselves running into a vSAN cluster performance limit where four and eight SSD setups came in very close to one another. Here, with our eight SSD setup, we measured 527k IOPS in RAID1 with 524k IOPS in RAID5. In 8K 70/30, RAID1 ultimately had the advantage, with 324k IOPS versus 293k IOPS in RAID5.
It is important to note that with VMware vSAN ESA, the RAID1 versus RAID5 configuration isn’t a decision set in stone. It is a storage policy applied at a VM vDisk level, meaning both can co-exist simultaneously. So, say you have a database VM where every last bit of I/O matters; give that a RAID1 policy, where everything else is on RAID5. That way, you optimize your space utilization as best as possible.
Final Thoughts
The Western Digital Ultrastar DC SN655 Enterprise PCIe Gen 4.0 dual-port NVMe SSD represents the next level in dense enterprise-grade storage while maintaining the mainstream affordability Western Digital is known for. It comes in the U.3 form factor but is also U.2 backward compatible. This means administrators and data engineers can increase storage density with a familiar form factor in dense 1U servers.
The increase in capacity density also means a more significant improvement in storage resource utilization for use cases, such as increasing the number of virtualized hosts per SSD and consolidating larger application datasets across fewer drives. These big data analytics and AI/ML datasets also benefit from moving to higher capacities by unlocking low latency and higher throughput of the SN655 NVMe SSD—which translates into faster time-to-insights and real-time analytics.
Our performance findings show that the Western Digital Ultrastar DC SN655 NVMe SSDs are well-suited for VMware vSAN 8 ESA deployments, especially regarding SMB, edge, retail, and thousands of other use cases. For companies looking to start with a small deployment and grow as the demands increase, we found that just four Western Digital SSDs per node offered much of the performance in our eight SSD per node configuration, with plenty of room to grow, and the vSAN applications and data need grows. This helps extend IT budgets since organizations don’t need to over-commit on disks to reach performance metrics, and adding SSDs to vSAN couldn’t be any easier.
With VMware vSAN 8 Update 2 GA now, vSAN customers are likely ready to take the leap to the new storage architecture. There are clearly many benefits with ESA, but we’re most satisfied with the simplicity vSAN offers now; select the server nodes and SSDs, and off you go with the same “point and click” ease of use vSAN has always provided. When considering SSDs for vSAN 8, we found the Ultrastar DC SN655 NVMe SSD performs exceedingly well, delivering excellent performance in RAID5, which helps customers maximize their storage footprint in vSAN without sacrificing performance. Even better, the drives are cost-effective, providing even more value for those looking to maximize their investment in VMware vSAN 8.
Ultrastar DC SN655 NVMe SSD Product Page
Engage with StorageReview
Newsletter | YouTube | Podcast iTunes/Spotify | Instagram | Twitter | TikTok | RSS Feed
