NVIDIA Hopper H200 GPU Continues To Dominate In Latest MLPerf 4.0 Results: Up To 3x Gain In GenAI With TensorRT-LLM

27.03.2024 - 15:33 / wccftech.com / Hassan Mujtaba

NVIDIA continues to push the AI envelope with its strong TensorRT-LLM suite, boosting the H200 GPUs to new heights in the latest MLPerf v4.0 results.

Blackwell Is Here But NVIDIA Continues Pushing Hopper H100 & H200 AI GPUs With New TensorRT-LLM Optimizations For Up To 3x Gain In MLPerf v4.0

Generative AI or GenAI is an emerging market and all hardware manufacturers are trying to grab their slice of the cake. But despite their best efforts, it's NVIDIA that has so far taken the bulk of the share and there's no stopping the green giant as it has showcased some utterly strong benchmarks and records within the MLPerf v4.0 inference results.

Related Story Qualcomm, Intel, & Google Join Hands To Come For NVIDIA, Plans On Dethroning CUDA Through oneAPI

Fine-tuning on TensorRT-LLM has been ongoing ever since the AI Software suite was released last year. We saw a major increase in performance with the previous MLPerf v3.1 results & now with MLPerf v4.0, NVIDIA is supercharging Hopper's performance. Why inference matters is because it accounts for 40% of the data center revenue (generated last year). Inference workloads range from LLMs (Large Language Models), Visual Content, and Recommenders. As these models increase in size, there comes more complexity and the need to have both strong hardware and software.

That's why TensorRT-LLM is there as a state-of-the-art inference compiler that is co-designed with the NVIDIA GPU architectures. Some features of TensorRT-LLMs include:

• In-Flight Sequence Batching (Optimizes GPU Utilization)
• KV Cache Management (Higher GPU Memory Utilization)
• Generalized Attention (XQA Kernel)
• Multi-GPU Multi-Node (Tensor & Pipeline Parallel)
• FP8 Quantization (Higher Perf & Fit Larger Models)

Using the latest TensorRT-LLM optimizations, NVIDIA has managed to squeeze in an additional 2.9x performance for its Hopper GPUs (such as the H100) in MLPerf v4.0 versus MLPerf v3.1. In today's benchmark results, NVIDIA has set new performance records in MLPerf Llama 2 (70 Billion) with up to 31,712 tokens generated per second on the H200 (Preview) and 21,806 tokens generated per second on the H100.

It should be mentioned that the H200 GPU was benchmarked about a month ago which is why its mentioned in the preview state but NVIDIA has stated that they are now shipping these GPUs to customers.

The NVIDIA H200 GPU manages to offer an additional 45% performance gain in Llama 2 versus the H100 GPUs thanks to its higher memory configuration of 141 GB HBM3E and faster bandwidth of up to 4.8 TB/s. Meanwhile, the H200 is a behemoth against Intel's Gaudi 2, the only other competitor solution submitted within the MLPerf v4.0 benchmarks while the H100 also sits in at a massive 2.7x gain.

In addition to these,