en.Wedoany.com Reported - Two research teams from South Korea and Japan have proposed side-stacking solutions for DRAM chips at the IEEE Symposium on VLSI Technology and Circuits, aiming to overcome the overheating and bandwidth bottlenecks faced by High Bandwidth Memory (HBM).

Current HBM used in data center GPUs consists of multiple layers of DRAM chips vertically stacked on a substrate, transmitting data and power through Through-Silicon Vias (TSVs). The thermal conductivity of the gap-fill material between chips is significantly lower than that of the silicon substrate, hindering heat transfer to the package heat sink. As the number of stacked layers increases, not only does the heating problem worsen, but the area occupied by TSVs also encroaches on memory cell space, exacerbating the conflict between storage capacity and bandwidth.

To address these limitations, the South Korean research team took a different approach. The lab of Kwon Ji-min at the Ulsan National Institute of Science and Technology (UNIST), in collaboration with the team of Kim Sung-joo at Hanbat National University, proposed a solution named V-Die. This approach stacks DRAM chips vertically on their sides and introduces microfluidic cooling channels between the chips. Simulations show that with direct liquid cooling (DLC) technology, the maximum temperature of the V-Die stack drops to approximately 45°C, far below the typical peak of over 80°C for HBM4. By eliminating TSVs and the base chip, the entire sidewall of each chip facing the interposer can be used for I/O routing, allowing for up to four times the number of connection points compared to HBM4. Simulations based on the JEDEC HBM4 specification indicate that the V-Die architecture achieves a peak bandwidth 4.01 times higher than HBM4, with a 37.2% reduction in read latency. In real AI workload tests, simulations of a GPT-3 scale (175B parameters) large language model running on 8 GPU compute nodes showed that the V-Die system can process 540 tokens per second, compared to 296 tokens per second for an HBM4 system of equivalent capacity, representing a 1.82x improvement in decoding throughput. The latency for processing the first token was reduced by approximately 32% (24 milliseconds). The research team predicts an overall 82% speed improvement for V-Die compared to HBM4. The team is currently developing a prototype to verify thermal and electrical characteristics.

The Japanese team focused on solving the integration challenges of side-stacking. Researchers from the University of Tokyo, Tohoku University, and Riken demonstrated the MOSAIC solution. They abandoned traditional electrical connections, instead fabricating rectangular inductive coils approximately 80 micrometers by 240 micrometers on one side of the memory chip, while placing corresponding coils vertically on the substrate to transmit data signals via magnetic field induction. Since the coils do not need to overlap completely, this method relaxes the requirements for chip thickness uniformity. Power connections are placed on the side of the memory cube. The MOSAIC design is intended to be mounted on top of a GPU, with each cube integrating 98 chips to provide 294 GB of storage capacity, double that of an HBM4 module of equivalent volume. Although it does not use liquid cooling, heat can be dissipated upward through silicon fins, keeping the peak temperature within 81.3°C. The team noted that if the DRAM chip thickness is reduced from the conventional value to 100 micrometers, the same volume could integrate 294 chips, achieving a capacity of 882 GB.

James Myers, Project Director at imec, pointed out that side-stacking solutions face practical integration challenges. Even a few micrometers of thickness variation between DRAM chips can lead to misalignment with substrate pads after multiple layers are stacked. Both solutions were presented at the IEEE Symposium on VLSI Technology and Circuits last month.