Preface

If there is any technology in the current data center field that is both hot and cold, liquid cooling is definitely on the list. The so-called cold refers to the high heat dissipation efficiency of liquid cooling, which has a good cooling effect on facilities with large power density. The so-called hot refers to the hot development momentum of liquid cooling, and the recognition and attention in the industry are getting higher and higher. Since the 1960s, liquid cooling has been used in large computers, but because the heat load is not high, the cost of air cooling is lower and safer, the liquid cooling industry has not been much developed. With the development of mobile Internet, data centers need to deal with increasing business complexity, power density rapid growth, data centers are becoming more and more "hot", under the pressure of green and efficient development, liquid cooling technology strong return.

Second, demand analysis

1. 5G

Today, the Internet has become the core driving force of the fourth industrial revolution, with the development of this industrial revolution, all aspects of human life are changing, especially in the data traffic demand continues to surge, network load significantly increased. 5G is the key to solving network communication problems and opening the door to the Internet of everything that the world is looking forward to. As one of the important infrastructure supporting 5G, the base station undertakes the heavy task of processing and transmitting massive data, which will generate a lot of waste heat. According to statistics, the base station itself occupies 80% of the total energy consumption of the communication network transmission. With the deepening of 5G and the landing of application scenarios, these base station heat dissipation problems can be solved by liquid cooling. At present, Nokia has deployed the world's first liquid cooled base station system in a building in Finland, and recycled base station waste heat for building heating, effectively reducing energy consumption.

2. Edge computing

A rule of thumb for data center construction is that a 10.5kW server rack requires 1200CFM of cooling air per minute, which is enough to fill an 11-square-meter, 3-meter-high machine room space per minute, and it takes a lot of energy just to move this air. Liquid cooling systems can provide higher cooling performance than air cooling systems, but they do not need to consume as much space as air cooling, and it requires much less volume in the machine room. In other words, optimally designed liquid cooling solutions will enable data centers to deliver more computing power in a smaller space. Edge data centers are usually designed to be smaller in order to be close to the business end, and to accommodate more high-density hardware facilities in order to meet the business needs, liquid cooling can meet the small space and high performance characteristics expected by edge data centers, and has the ability to become the ideal cooling solution for edge data centers in the future.

3. Artificial Intelligence

From general-purpose cpus to Gpus (graphics processing units), FPgas (field programmable gate arrays) and ASics (application-specific integrated circuits), the emergence of a new generation of AI processors enables powerful machine learning analysis programs to run smoothly, providing the power support for modern AI. However, the amount of computing and iteration needs of artificial intelligence scenarios are very high, and devices such as servers usually use cluster deployment of AI accelerators to improve computing power, so the power of a single server is getting higher and higher, and the cooling capacity of data center infrastructure is becoming more and more strained. In Google's case, its TPU pod computers, designed for AI business, consume eight times more power in the upgraded generation (from 2.0 to 3.0), so it has to use liquid cooling in the data center. In the future, as more and more enterprises and organizations begin to use machine learning to provide AI solutions as a service, the demand for device heat dissipation will drive the implementation and landing of liquid cooling faster.

It is concerned about the good prospects of liquid cooling for new data center business, the Data center Department of the Cloud Computing and Big Data Institute of the China Academy of Information and Communications Technology began to carry out various work such as data center liquid cooling technology research, standard development, testing and verification since 2016, and achieved phased results such as standards, books, white papers, awards and so on.

Third, liquid cooling technology analysis

Liquid cooling technology takes away most of the heat through the liquid circulation medium, the air demand of a single server is reduced, and the overall air supply demand of the machine room is also reduced, which can achieve the effect of reducing the consumption of the heat dissipation system, so as to promote the green process of the data center. At present, data center liquid cooling mainly has two mainstream technology routes: indirect liquid cooling and direct liquid cooling. Indirect liquid cooling is mainly cold plate, and the technology maturity is high. Direct liquid cooling is mainly immersed, and the heat dissipation efficiency is high. Direct liquid cooling has also emerged in the spray and other new forms, is still in the early stage of development, public display of research and practice results are few, has a certain development potential.

1, indirect liquid cooling - cold plate

Cold plate liquid cooling has low requirements for modification and adaptation of heating devices, high technical maturity, and the fastest application progress. The cold plate liquid cooling system is composed of a heat exchange cold plate, a liquid division unit, a heat exchange unit, a circulation pipeline and a coolant, which is a way of transferring the heat of the heating device to the cooling liquid closed in the circulation pipeline through a heat exchange cold plate (usually a closed chamber composed of high thermal conductivity metals such as copper and aluminum), which can be divided into series and parallel according to the different connection methods of the pipeline. In the series mode, the cooling liquid first enters a cold plate cavity to dissipate heat and then continues to flow into the next cold plate cavity. The pipeline connection is simple, but the CPU in different parts will have temperature differences. In the parallel mode, the coolant is shunt before entering the different cavities, and then enters the cavity separately, which is more efficient in heat dissipation but more complex in the piping system. Because the heating device in the cold plate liquid cooling does not need to be in direct contact with the liquid, the transformation and adaptation requirements of the heating device itself are small, so the maturity of the cold plate liquid cooling is relatively high. At present, Baidu, Tencent, Meituan and other Internet companies have begun to carry out technical research and test verification of cold plate liquid cooling, which has formed a strong driving role in the cold plate liquid cooling industry.

2, direct liquid cooling - immersion type

Submerged heat dissipation has the most obvious advantages, which can greatly improve energy efficiency, and is more suitable for new data centers. Submerged liquid cooling is a cooling technology that uses liquid as a heat transfer medium, completely immerses the heating device in the liquid, and directly contacts the heating device with the liquid and carries out heat exchange. According to whether there is a phase change in the coolant during the heat exchange process, it can be divided into single-phase liquid cooling and phase-change liquid cooling. The difference is whether the liquid as a heat transfer medium only changes in temperature during the heat transfer process, or there is a phase transition. The submerged liquid cooling system is generally divided into four parts: the submerged chamber subsystem, the cooling subsystem, the outdoor cold source system and the monitoring system. Due to the special system architecture, the coolant can be in direct contact with the heating device, so the immersed heat dissipation efficiency is high, can support higher power density IT deployment, can greatly improve the energy efficiency (PUE). Although the cooling effect is better, due to the need to directly contact the coolant and heating devices, the action of data center equipment transformation is larger, so the immersion type is more suitable for new data centers. At present, Alibaba has carried out the large-scale deployment of submerged liquid cooling in its Beijing Winter Olympics Cloud data center, opening the official commercial use of submerged liquid cooling.

4. Changes and impacts

Revolutionize the overall architecture of the data center. From avoiding liquid contact with equipment in the equipment room to actively introducing and using liquids, the use of liquid cooling technology has changed the infrastructure of the data center. From the floor to the ceiling, from the line to the pipeline, from the external structure of the cabinet to the internal deployment, liquid cooling will bring a completely different architecture of the data center.

Revolutionize the way data centers dissipate heat. In the past, data center heat dissipation was mainly air-cooled, and heat dissipation accounted for most of the energy consumption. After the liquid cooling gradually replaced the air cooling heat dissipation, the heat dissipation efficiency was significantly improved. The application proves that PUE<1.1 can be obtained in most areas after using liquid cooling technology.

Refactor the design and deployment of IT equipment and key components. Server-based IT devices, including key components such as cpus, Gpus, storage, and network cards, will begin to contact liquids directly or indirectly in order to better heat dissipation (for example, the core temperature of some cpus at full load can be reduced by 20 ° C compared with air cooling), and their design and deployment will inevitably change in order to be compatible with liquid cooling systems.

Disrupting the monitoring and operation model of the data center. After the liquid cooling technology enters the equipment room, the monitoring and control system of the data center will focus on it, such as the deployment of leakage monitoring, temperature monitoring, pressure monitoring, and so on. In addition, the operation and maintenance ideas of the data center will also change, and liquid cooling supporting facilities such as liquids, pipelines, and interfaces will bring new requirements to the operation and maintenance.

Improve energy efficiency evaluation indicators and methods. At present, the main indicator to evaluate the energy efficiency of data centers is PUE, which is the ratio of the total energy consumption of data centers to the energy consumption of IT equipment. At this stage, the data center liquid cooling deployment is still in the transition stage, that is, the same data center may use air cooling and liquid cooling systems at the same time, in order to fully understand the performance of the two parts of the system, it is necessary to measure energy efficiency separately, so the measurement indicators of PUE can be further improved.

V. Outlook

In the data center, limited by the room design and supporting facilities, when the power density of a single stand is close to 20kW, the air cooling system has reached its economic and effective cooling limit. Liquid cooling technology is an open and flexible cooling solution that supports workloads such as technical computing, grid deployment and data analysis, and can effectively cope with high power density scenarios such as high performance computing (HPC), large-scale cloud data centers, edge computing, artificial intelligence, and so on. It has obvious advantages in reducing energy consumption and TCO in data centers. It has broad market prospects.

However, the advantages and potential of liquid cooling have not formed a strong business-driven effect, from the signal released by the world's Internet giants, they have great concerns about the standardization and standardization of liquid cooling systems and spare parts products, and are reluctant to face the trouble of equipment procurement and later maintenance. Therefore, while strengthening liquid cooling technology and product innovation, we should also focus on standardization and adaptation issues.

At present, the Data center Department of the Cloud Computing and Big Data Institute of the China Academy of Information and Communications Technology has promoted the China Communications Standardization Association (CCSA) and the Open Data Center Standards Promotion Committee (ODCC) to do a lot of work in this regard, and cooperated with Alibaba, Tencent, Baidu, Meituan, China Telecom, China Mobile, etc., and introduced six liquid cooling group standards. Six liquid cooling industry standards have been established, and the development of the liquid cooling industry has been effectively regulated through these standard documents. In addition, two books on liquid cooling have been published, detailing the past, present and future of liquid cooling technology, providing an important reference for the use of liquid cooling technology in the industry. In addition to technical achievements, the liquid cooling project declared by ODCC also won the highest award for leading scientific and technological achievements of the 2019 "Number Expo" - "Black Technology" award, which greatly enhanced the industry influence of liquid cooling solutions. If there is more direct standard guidance and policy support for data center liquid cooling at the national level in the future, then liquid cooling technology will have a stronger impetus to improve, and will more significantly promote the green and energy-saving development of data centers.

Source: CAICT, China Academy of Information and Communications Technology