The EchoStreams GridStreams is a 2U, high-density quad-node server platform designed for high-performance datacenter and Open Computing Alliance cloud service providers. Each of the four nodes features a half-width motherboard with support for single or dual Intel Xeon 2600 processors and room for one or two PCIe expansion slots depending on motherboard configuration. The GridStreams offers a bank of eight 7mm 2.5" drive bays in the middle, designed to be shared two each across the compute nodes. Configured for data center use, the nodes and drive bays serviceable from the front of the chassis, both are hot swappable. Only fans and power are on the back of the chassis, designed to be swapped out in new datacenter environments in which the rack provides both 12VDC and cooling airflow to the server chassis.
The EchoStreams GridStreams is a 2U, high-density quad-node server platform designed for high-performance datacenter and Open Computing Alliance cloud service providers. Each of the four nodes features a half-width motherboard with support for single or dual Intel Xeon 2600 processors and room for one or two PCIe expansion slots depending on motherboard configuration. The GridStreams offers a bank of eight 7mm 2.5" drive bays in the middle, designed to be shared two each across the compute nodes. Configured for data center use, the nodes and drive bays serviceable from the front of the chassis, both are hot swappable. Only fans and power are on the back of the chassis, designed to be swapped out in new datacenter environments in which the rack provides both 12VDC and cooling airflow to the server chassis.
The GridStreams joins the StorageReview Test Lab as a key component for testing high-performance all-flash storage arrays in our new database and software application testing environment. The compute-dense 2U quad-node system will provide the system resources required to effectively stress high-end storage devices in our upcoming application tests. Synthetic benchmarks don’t require (relatively speaking) a lot of system resources to run and can be contained on a single dual-CPU server even when generating 1-2MM IOPS in synthetic workloads. Consuming that amount of I/O from an actual application requires exponentially greater resources, requiring more CPU cores and RAM than what could be contained in a single server. To replicate those loads in the StorageReview Test Lab, we use a variety of systems and configurations, including the four-node EchoStreams GridStreams, to give us the compute-dense resources needed to run the latest application benchmarks.
EchoStreams GridStreams Specifications
- Chassis height: 2U
- Compute Nodes: 4
- Motherboard: Half-width up to 18" deep
- Chipset: Intel C602
- Processor (per node)
- Configured: 2 x Intel E5-2640 (2.5Ghz, 15MB cache, 6-core)
- Supports: 2 x Intel Socket 2011 / Xeon E5-2600 Family
- Memory (per node)
- Configured: 128GB (16 x 8GB)
- Total Slots : 16 (4-channel per CPU, 8 DIMM per CPU)
- Maximum up to 512GB RDIMM/Maximum up to 128GB UDIMM
- Storage (per node)
- Configured: 2 x 100GB Micron RealSSD P400e
- 2 x 2.5" 7mm hard drive or SSD
- 2 x SATA 6.0Gb/s
- I/O Expansion (per node)
- Configured: Mellanox ConnectX-3 QDR InfiniBand 56Gb/s NIC
- 1 x PCIe x16 (Gen3 x16 Link)
- Power Supply (per node): 400W open-frame PSU
- Chassis Cooling: 4 x hot-swap 80mm fans
- Front Panel (per node)
- Power switch with LED
- Locate switch with LED
- Reset switch
- System warning LED
- On-board Connectivity (per node)
- 2 x Intel Server-Class GbE LAN
- 1 x Management LAN with iKVM support
- Optional on-board single-port Mellanox ConnectX-3 QDR InfiniBand
Hardware Options
The EchoStreams GridStreams Quad-Node Server is based on commodity hardware, meaning that hardware options can depending on buyers demands. The system StorageReview was supplied with offers a variety of connectivity options, from onboard SAS and SATA to Ethernet and optional on-board InfiniBand. Each node supports single and dual-processor configurations, currently EchoStreams supports Intel Xeon E5-2600 processors.
The way local storage is provisioned inside the GridStreams chassis is by node; each node is connected via socket connector to two of the eight 7mm SSD/HDD bays on the front. Each node is by definition independent and can be quickly disconnected and pulled out in the event of needed service. Integrated RAID is also an option, which would allow a RAID1 configuration over two hard drives or SSDs to cope with a drive failure. In our lab we utilize two Micron P400e SSDs for each compute node in a multi-OS environment without redundancy. One SSD boots the node with CentOS 6.3 and the other Windows Server 2012.
To meet our Lab’s unique needs, we supplied our own Mellanox ConnectX-3 InfiniBand NICs to occupy the single PCIe 3.0 slot in each node. InfiniBand was specifically chosen for its high-throughput and low latency attributes that are required in our newest application benchmarks. If the customer requests it, each motherboard can also be equipped with an on-board Mellanox ConnectX-3 QDR interface, freeing up the single PCIe slot for other uses. This gives each system a great deal of flexibility and allows the end-user to make the most out of each node.
Monitoring and Management
Each node includes a dedicated Ethernet port for on-board management through an iKVM interface. The iKVM supports console redirection and remote drive hosting, allowing administrators to setup or diagnose systems remotely, without having to connect to the system directly. The GridStreams iKVM interface was used extensively in the initial setup of each node to quickly get the platform online in the test lab.
The included management software also supports remote power control and gives the user up to the second thermal, power and cooling information. At a moment’s glance the user can check PSU voltage leads, fan speeds as well as motherboard and CPU temperatures. Each of these sensors can also log, providing an opportunity to monitor how conditions vary from idle to heavy load to ensure the systems are properly equipped for their environment.
Design and Build
The unique design of the EchoStreams GridStreams positions the motherboard and I/O connections at the front of the chassis instead of the rear, giving greater flexibility in datacenters designed for this layout. Even though the StorageReview lab has a traditional rack-mount environment, we were still able to integrate the GridStreams with cables routed through slots in one of our Eaton S-series racks. For on-board network connectivity, the nodes include two Intel 1GbE ports as well as another 1GbE port dedicated to iKVM management. There is also a cutout on the front access panel for a single QSFP InfiniBand port, which is found on motherboards equipped with this optional add-on.
One of the more unique design aspects of the GridStreams server is the hot-swap drive unit on the front of the chassis. While many compute nodes have dual storage bays embedded within the node, EchoStreams took a different path. In the GridStreams chassis, each node has direct access to two bays, generally configured in a RAID1 for redundant system drives. Our configuration utilizes Micron’s 100GB RealSSD P400e SSD for both CentOS 6.3 and Windows Server 2012 boot drives.
The EchoStreams GridStreams is designed to be easily serviced in a production environment, with each node mounted to a tray that can be easily slid out to remove the motherboard and PCIe interface in one quick motion. Each tray interfaces with chassis using quick-release power and data connections. This design lets users set up new hardware or service individual nodes outside of the rack environment in easily manageable chunks, without hauling the entire system out of the rack.
EchoStreams provides sliding rails with the GridStreams server utilizing quick-connect fittings to easily mount the rails in a rack and slide the system into position. While the rails allowed some extension after the server was mounted in our rack environment, they don’t fully extend to service the entire chassis without removing it completely. Given the front-mount node access and rear-mount fan access, the only time a user would need to remove the entire chassis from the rack would be in the event of a power supply failure.
Cooling
The EchoStreams GridStreams offers a unique front to back cooling design that incorporates four 80mm fans located in the rear of the chassis. These four fans provide the sole cooling and airflow for the GridStreams; drawing cool air across the PCIe card and SSDs or hard drives first, processors second, and power supplies last. The fan speed is controlled by each node, allowing for throttling profiles or locking the fans to full speed. For our on-going lab usage we keep fans at 100% for all of our tests. In an Open Compute rack that supplies both power and cooling, the fans can be removed from the chassis to allow the rack to provide the cooling for the GridStreams chassis. Each of the four fans is field replaceable to cope with long term maintenance requirements. To remove the failed fan unit, loosen two holding screws with finger-tight knobs and pull the unit from the rear of the chassis. Each fan has a quick-release power connection, so there’s no need to mess around with wires when removing or inserting a new cooling fan.
In our tests with an average lab temperature of 25.8C each node idled with a CPU temperature of 42C/45C (front/rear) which then increased to 68C/90C under load. Allowing the lab to warm up to the upper operating temperature limits of the EchoStreams GridStreams, we measured each node at 47C/51C idle and 74C/90C under load with an ambient rack temperature of 31C.
Power
The GridStreams can be powered off either 100-240V sources or common 12VDC in Open Compute datacenters equipped with low-voltage Rack Power Centers. Our system features four high-efficiency 400-watt 100-240v power supplies, which are non-redundant across nodes. This means that each node has its own power connection and if one power supply fails, the power supply feeding that particular node goes down. Customers opting for a greater level of efficiency or power redundancy can turn to one of EchoStreams 12VDC Rack Power Centers that support multiple GridStreams servers. The RPC offers automatically-scaling internal N+1 power nodes which adapt to demand, powering up or down to keep the system loaded in a state that maximizes efficiency.
In our tests with a node configured using a single 8GB DDR3 RDIMM, two Intel Xeon E5-2640 CPUs and a Mellanox ConnectX-3 InfiniBand NIC, we measured an idle power consumption of 98 watts which increased to 232 watts under full CPU load with our Eaton Advanced ePDU. At idle the power factor measured 0.93, increasing to 0.97 under load.
Conclusion
The EchoStreams GridStreams quad-node 2U compute platform joins the StorageReview lab at a critical time as we scale into larger application-based testing which requires increasingly versatile compute solutions to drive best case application performance. The GridStreams addresses not just performance issues of course, but also density and specialized requirements for Open Computing Alliance members. The serviceability of the enclosure and blades, along with the front-mounted I/O ports make the GridStreams an ideal solution for cloud service providers, who have few options available to them from the traditional server providers. Additionally, EchoStreams plans to continue to evolve the platform, eventually removing fans and power supplies, which are merging into smart racks that handle these duties in large data centers.
Our purposes are slightly less ambitious than those of a large data center, but on a more modest scale, being able to simulate real-world compute and storage environments is critical to proper testing and evaluation. To that end the GridStreams solution becomes an invaluable portion of the lab, as we continue to scale up our scope of testing as it relates to storage and the ecosystem at large.
