Update 10/10/12 – We’ve added a video overview of the current configuration which includes 48 Intel SSD 520’s, Mellanox InfiniBand switching, cards and cables, LSI HBAs and compute via a pair of HP ProLiant Gen 8 servers and a custom built SuperMicro loaded up in a 15U StarTech rack.
————————-
StorageReview has started testing the FlacheSAN2 48-bay 2U storage array from EchoStreams. This platform features 48 densely packed 7mm SSDs, five PCIe 3.0 LSI 9207-8i HBAs, three Mellanox ConnectX-3 56Gb/s dual-port InfiniBand adapters and is built around a custom two-processor Intel motherboard. Follow this log for the ongoing build process and benchmark results.

StorageReview has started testing the FlacheSAN2 48-bay 2U storage array from EchoStreams. This platform features 48 densely packed 7mm SSDs, five PCIe 3.0 LSI 9207-8i HBAs, three Mellanox ConnectX-3 56Gb/s dual-port InfiniBand adapters and is built around a custom two-processor Intel motherboard. Follow this log for the ongoing build process and benchmark results.

Build of Materials

48X Intel SSD 520 180GB
- Sustained sequential read: Up to 550 MB/s
- Sustained sequential write: Up to 520 MB/s
- Random 4K read: 50,000 IOPS
- Random 4k write: 60,000 IOPS
3X Mellanox ConnectX-3 InfiniBand PCIe 3.0 Dual-Port Adapter
- Up to 56Gb/s InfiniBand or 40 Gigabit Ethernet per port
- 1μs MPI ping latency
5X LSI SAS 9207-8i PCIe 3.0 HBA
- LSI SAS2308 6Gb/s SAS Controller
- Fusion-MPT 2.0 architecture can achieve more than 700,000 I/Os per second
EchoStreams FlacheSAN2:
- Capable of supporting 2+ million IOPS
- 2 x Intel Xeon E5-2690
- 64GB DDR3
- 48 x 6G SAS/SATA 2.5”x7mm hot-swap SSD bays + 1 x internal
- 2.5” x 7mm SSD for OS
- Quad GbE ports onboard MB
- 7x PCIe Gen3 X8 LP + 1x PCIe Gen2 X4 LP slots + 1x Intel PCIe Gen3

Status Reports

7/12/2012

We received shipment of the EchoStreams FlacheSAN2 today, and installed it into our Enterprise Test Lab. Considering the performance potential from this all-flash SSD array, it’s impressive that it only takes up 2U of space.

Looking at the front, the FlashSAN2 includes a super-dense pack of 48 7mm SSD bays, split into four groups. Right now each tray is loaded with a 180GB Intel SSD 520. This platform will no doubt start pushing some impressive performance numbers. Each SSD supports 60,000 IOPS 4K burst, while the LSI 9207-8i HBAs support up to 700,000 IOPS, meaning we could get close to 2 million IOPS throughput burst from this current setup or around 24,000MB/s in bandwidth.

On the back you can see the five half-height LSI 9207-8i HBAs, as well as three Mellanox 56Gb/s InfiniBand adapters.

8/10/2012

Mellanox has supplied our lab with the latest InfiniBand infrastructure including the 36-port 56Gb/s SX6036 switch, ConnectX-3 Dual-Port VPI NICs and all the appropriate FDR cabling.

We’re working on our InfiniBand fabric with Mellanox and now have our brand new SX6036 switch managing the data paths between a pair of HP ProLiant DL380p Gen8 servers and the EchoStreams FlacheSAN2.

The next steps are to finalize the Windows Server 2012 configuration and SMB 3.0. We’re working with Microsoft to set up the compute side to optimize the array for the best performance in various configurations.

Update 8/31/2012

As we start to prepare the compute and networking infrastructure for the EchoSteams FlacheSAN2 its important to know what the platform is capable of internally, to get a better understanding of how well it will perform in RAID and over our InfiniBand network. To that end we have been testing the SSD array in a native fashion; accessing each SSD individually outside of RAID. Our initial results listed below utilize 44 SSDs, which we will later be bumping it up to 48 when we finalize our testing process.

In our first workload we use a 128k sequential transfer to show peak bandwidth in and out of the array. In this test we measured just shy of 16.5GB/s read and 17.8GB/s write.

Our next workload measures 8k sequential access, which we measured 1.4 million IOPS read and 1.26 million IOPS write.

Transitioning to random access, we lowered the transfer size to 4K and measured performance with a queue depth scaling from 1 to 64. In this test we came just shy of 2 million IOPS with a measurement of 1.96 million IOPS 4K random read and 1.7 million IOPS 4K write.

Our last group of tests measure mixed workload performance utilizing traditional Database, Workstation, File Server, and Web Server workloads.

Update 8/31/2012

Installed the updated backplanes in the FlacheSAN2 which allowed us to take advantage of all 48 SSDs. With four more Intel SSD 520’s added to the group, we broke past the 2 million IOPS barrier in 4K random read transactions. The top speed with an outstanding I/O figure of 32, we hit 2,045,787 IOPS 4k read and 1,798,432 IOPS 4k write with an outstanding I/O figure of 4.