GPU Performance Deep Learning Benchmarks - BERT, GNMT, & More

When selecting hardware for AI and machine learning workloads, comprehensive performance data across real-world scenarios is essential for making informed decisions. We will benchmark a variety of NVIDIA GPUs using deep learning benchmarks. Here is our system configuration:

While these workloads are not one-to-one with your particular workload, they can serve as a good understanding of relative performance across multiple GPUs.

This blog will continue to grow as we test more and more GPUs. Current GPUs tested:

• NVIDIA RTX PRO 6000 Blackwell Max-Q 96GB GDDR7 
• NVIDIA RTX 6000 Ada 48GB GDDR6
• NVIDIA L40S 48G GDDR6

Testing Environment and Methodology

Our testing utilized a deep learning benchmark suite representing real-world AI workloads:

• BERT Base/Large: Natural language processing and transformer models
• GNMT (Google Neural Machine Translation): Sequence-to-sequence translation
• ResNet50: Image classification model for computer vision applications
• TransformerXL Base/Large: Advanced natural language processing for sequence modeling and context understanding in text
• Tacotron2: Text-to-speech synthesis model
• WaveGlow: High-quality audio generation
• SSD (Single Shot MultiBox Detector): Object detection for computer vision
• Neural Content Filtering (NCF): Recommendation system

These performance numbers are not to be taken at face value. Instead, they should be used as a reference point for performance comparison in similar workloads. We selected a diverse set of workloads, including vision, transformer, and classification models.

Benchmark Notes

• GPU Scalability is relatively strong with some performance overhead. We are seeing a near 1:1 ratio of Speedup to GPU count. However, in workloads like Tacotron2, TransformerXL Large, and NCF, the GPU scalability is impacted. This is likely due to their smaller batch size type workloads that need to pull from memory often.
• NVIDIA Blackwell performs well beyond the NVIDIA Ada, where in some cases, double the performance (ResNet50, WaveGlow, SSD, and NCF). With increased memory, models and data can reside in memory, which reduces memory bandwidth bottlenecks
• RTX 6000 Ada and L40S show very similar performance since they are the same GPU architecture and die. We saw a few anomalies between these two GPUs, with both of them trading performance positions in some workloads.
• Tacotron2 had a DNR for the multi-GPU NVIDIA RTX PRO 6000 Blackwell runs due to an error on our end. We will fix those in due time. This is not a performance issue.

Deep Learning GPU Benchmarks on FP32

Deep Learning GPU Benchmarks on FP16

Multi-GPU Scaling Analysis - Scaling Efficiency (vs Single GPU)

Next, we will look at the scaling efficiency compared to single-GPU performance. We took an average of the 3 GPUs to calculate the scalability, which will tell us which workloads benefit from multiple GPUs.

FP16 Speedup vs FP32 Precision Performance Comparison

Here we check to see which workloads can benefit from the FP16 speedup. We calculated the average FP16 score divided by FP32 score. In this case, WaveGlow, Tacotron2, SSD, and NCF do not see meaningful benefits.

Performance Insights from Our Benchmark Results

Development Teams

• Mixed Precision Acceleration: Up to 2.04x speedup with FP16 for BERT Base, allowing twice as many experiments in the same timeframe
• Exceptional Multi-GPU Scaling: Additional GPUs will have some performance overhead, but each benchmark has near-perfect scaling for multi-GPU configurations.
• Memory Efficiency: 96GB VRAM on the NVIDIA RTX PRO 6000 Blackwell supports larger batch sizes in BERT and TransformerXL Large models

Production Deployments

• Workload Versatility: Strong performance across all 10 benchmarked models from NLP to computer vision
• Linear Scaling: 3.94x scaling with 4x GPUs for BERT Base demonstrates excellent performance-to-cost ratio
• Cost-Optimized Performance: All GPUs deliver amazing scaling efficiency for production AI workloads

Research Organizations

• Domain-Specific Excellence: Top performance in vision (ResNet50, SSD), language (BERT, TransformerXL), and audio (Tacotron2, WaveGlow) models
• Precision Flexibility: Choose between FP16 (1.70-2.04x speedup) and FP32 based on accuracy requirements. Check our average speedup to see if your type of workload would scale effectively.
• Hardware Optimization: These benchmarks can help guide you to choose the right GPU architecture based on your deep learning workload needs, optimizing ROI for your AI infrastructure investment

Conclusion

Our extensive benchmarking analysis reveals the exceptional performance of modern GPUs across a diverse range of deep learning workloads. The data demonstrates near-linear scaling for multi-GPU configurations, with 2-GPU setups achieving up to 1.99x performance and 4-GPU systems delivering 3.94x speedups.

FP16 precision offers excellent acceleration in most workloads, especially transformer-based models (BERT and TransformerXL). For multi-GPU scalability, transformer-based models like show excellent multi-GPU efficiency, while models like WaveGlow, NCF, Tacotron2, and SSD scale less efficiently.

These insights provide valuable guidance for organizations designing AI infrastructure to match their specific workload requirements, whether for research experimentation, model development, or production deployment.

Need guidance with configuration and integration for your computing infrastructure? Exxact is a leading solution integrator helping to deliver custom-configurable workstations, servers, and full rack clusters. Contact us for more information!