ZutaCore’s two-phase cooling on a Supermicro server with AMD EPYC, delivers improved cooling, efficiency, and record-breaking performance.
As part of our ongoing evaluation of advanced cooling technologies for high-performance servers, we tested the ZutaCore® two-phase direct-to-chip (DTC) cooling solution on a Supermicro server with dual AMD EPYC Bergamo CPUs. This testing occurred at the Centersquare Data center in Reading, Berkshire, UK, in collaboration with integrator Boston Limited.
The Need for Efficient Cooling Solutions
As CPUs continue to grow in core count and power consumption, traditional air-cooling solutions struggle to keep pace with the thermal demands of cutting-edge processors. This is particularly evident in high-density compute nodes, where air cooling methods fail to effectively manage the significant heat produced by high-performance multi-core CPUs, such as AMD’s 128-core EPYC Bergamo.
As these powerful processors generate substantial thermal output, the limitations of air cooling become more pronounced. Even the most advanced heatsinks face challenges in maintaining optimal thermal performance without adding noise or drawing excess power.
This project centers around upgrading a dual-socket Supermicro chassis from traditional air cooling to ZutaCore’s two-phase DTC solution. We tested the system pre-conversion, observed significant cooling efficiency limitations, and benchmarked the server after the DTC installation.
The Hardware Setup
The test system featured a 1U Supermicro chassis with dual AMD EPYC Bergamo CPUs, each boasting 128 cores. With 256 cores under the hood and 768GB of RAM, this server is designed to handle dense workloads, especially in AI and data-intensive environments. However, even with large traditional air-cooling heatsinks, the system struggled to keep temperatures in check during stress testing, resulting in thermal throttling.
We opted for ZutaCore’s two-phase liquid cooling system to improve cooling and enhance performance. This innovative system utilizes a heat transfer fluid that transitions from liquid to vapor within the cold plates directly attached to the CPUs. This phase change allows for much higher thermal efficiency than traditional liquid or air coolers, as the latent heat from the fluid vaporization helps draw heat away from the CPU cores more effectively.
The Conversion Process
Preparing the Server
The conversion process began by disassembling the stock air-cooling components. Removing the traditional heatsinks revealed the small, dense heat spreaders on the Bergamo CPUs, clearly undersized for managing the thermal load of these power-hungry chips. During the initial tests, we noticed the CPUs reaching high temperatures and struggling with sustained performance under stress.
Installing the ZutaCore Cold Plates
The next step was introducing the heart of the two-phase cooling system: ZutaCore’s custom-designed cold plates. These cold plates are fitted with internal evaporators that allow the heat transfer fluid to absorb heat and evaporate as it passes through the system. The vaporized fluid is then returned to the condenser, cooled, and reintroduced into the system to continue the cycle.
One of the more fascinating features of this system is its mechanical self-regulation. Each evaporator’s float mechanism adjusts the heat transfer fluid flow based on the CPU’s thermal load. A closed-loop system like this ensures that each CPU receives the necessary amount of fluid without manual intervention.
The cold plates were carefully seated on the CPUs, ensuring consistent pressure and optimal thermal contact. The tubing that carries the heat transfer fluid was connected to the cold plates, and we ran a series of pressure tests to ensure there were no leaks before proceeding with the complete installation.
Integration with the Server’s Power and Thermal Management Systems
The next challenge was integrating the cooling system into the broader data center infrastructure. ZutaCore’s DTC system integrates with the server’s existing thermal management systems and offers enhanced control through ZutaCore’s cooling management software. This software allows for real-time monitoring of key metrics, such as refrigerant pressure, temperature at various points in the system, and CPU junction temperatures.
The software connects to the server’s IPMI (Intelligent Platform Management Interface), allowing seamless monitoring and control of cooling parameters. We were particularly impressed by the detail provided, including fan speeds, vapor temperatures, and pump duty cycles. This granular visibility into the cooling process enables fine-tuning system performance to match the workload.
After plumbing the system into the heat rejection unit with the quick disconnect fittings and quickly purging the atmosphere gases from the loop using the included service kit, the system was ready to power on.
Performance Highlights
Pre-Conversion Results
As noted above, before switching to ZutaCore’s two-phase system, we ran a series of benchmarks with stock air-cooling. Using workloads designed to stress all 256 cores, the system quickly reached thermal limits, causing the CPUs to throttle performance to stay within safe operating temperatures. Sustained heavy loads resulted in CPU temperatures hovering near 85°C, with noticeable dips in clock speeds.
Post-Conversion Results
After installing the DTC solution, we reran those benchmarks, and the results were dramatic. The CPU temperatures, under full load, dropped significantly, with peak temperatures remaining under 65°C. More importantly, thermal throttling was eliminated. The two-phase system kept the Bergamo CPUs running at higher sustained clock speeds throughout the test.
Unsurprisingly, the system’s power consumption dropped, as the fans no longer needed to spin up to maximum RPM to compensate for high temperatures. The quieter operation was immediately noticeable, even within a noisy data center environment.
Since this exercise was intended for hands-on experience with the ZutaCore solution, all measurements were qualitative. It is worth noting that at the time of filming and writing, the ZutaCore-cooled Bergamo system achieved a few world records in the y-cruncher BBP categories and verified over at HWBot.org.
Power and Cooling Efficiency
One of the standout advantages of ZutaCore’s system is the potential for energy savings. The reduced reliance on air cooling lowers the need for high-RPM fans and air conditioning, translating into real-world energy cost savings. Furthermore, the system allows the server to operate in higher ambient temperatures without risk of overheating. This feature is invaluable for data centers looking to optimize their PUE (Power Usage Effectiveness) by raising the temperature of their facilities.
Long-Term Implications for Data Centers
ZutaCore’s two-phase cooling solution effectively manages high thermal loads and provides long-term benefits for data centers. Its enhanced energy efficiency and lower cooling costs could make it crucial for high-density computing environments.
For data centers adopting next-generation processors and GPUs, such as AMD’s EPYC and NVIDIA’s H100, the ZutaCore cooling solution presents a scalable option to manage the increasing heat output without requiring substantial changes to existing infrastructure.
Additionally, the two-phase cooling method is reasonably environmentally friendly. ZutaCore’s heat transfer fluids are non-toxic and low GWP (Global Warming Potential). The use of dielectric fluids ensures that, even in the event of a leak, there is no risk of electrical shorting or equipment damage.
Conclusion
The conversion of the Supermicro dual AMD EPYC Bergamo server to ZutaCore’s two-phase direct-to-chip cooling solution demonstrated a clear improvement in thermal performance, noise reduction, and energy efficiency. As data centers continue to scale up and push the limits of server performance, advanced cooling technologies like ZutaCore offer a promising path forward for managing the growing thermal demands of modern compute environments.
If you’re operating high-density workloads and pushing your hardware to its limits, investing in advanced cooling solutions like ZutaCore’s could improve performance and save on operational costs in the long run.
Big shout out to Boston Limited for supplying the hardware and Centersquare for providing the data center!
Engage with StorageReview
Newsletter | YouTube | Podcast iTunes/Spotify | Instagram | Twitter | TikTok | RSS Feed
