GPU Performance Deep Learning Benchmarks - BERT, GNMT, & More

Update 8/28/25: Added NVIDIA RTX PRO 6000 Blackwell Server Edition

Update 4/2/26: Reformatted Tables and Added Additional Context

GPU Performance Deep Learning Benchmarks: Comprehensive Performance Analysis

When selecting hardware for AI and machine learning workloads, comprehensive performance data across real-world scenarios is essential for making informed decisions. We will benchmark a variety of NVIDIA GPUs using deep learning benchmarks. Here is our system configuration:

While these workloads are not one-to-one with your particular workload, they can serve as a good understanding of relative performance across multiple GPUs.

This blog will continue to grow as we test more and more GPUs. Current GPUs tested:

Key Findings:

• Blackwell Delivers Up to 2x the Performance of Ada. Across our benchmark suite, the NVIDIA RTX PRO 6000 Blackwell Server Edition delivers an average of 1.89x the throughput of the L40S on a single GPU — with some workloads like SSD object detection reaching 2.46x.
• Near-Linear Multi-GPU Scaling. Adding GPUs pays off: 4-GPU configurations deliver an average 3.9x speedup, and 8-GPU systems achieve approximately 8x, making multi-GPU servers a strong investment for production and research workloads.
• Mixed Precision (FP16) Doubles Throughput for Transformer Models. BERT Base sees up to 2.06x acceleration with FP16, while vision and audio workloads like WaveGlow and Tacotron2 see minimal FP16 benefit — an important consideration when planning your pipeline.
• 96GB VRAM Changes the Game. The Blackwell GPUs' 96GB GDDR7 memory allows larger batch sizes and entire models to reside in memory, eliminating memory bandwidth bottlenecks that constrain 48GB cards.

Testing Environment and Methodology

Our testing utilized a deep learning benchmark suite representing real-world AI workloads:

These benchmarks are not one-to-one representations of your specific workload. They serve as reference points for understanding relative performance across GPU configurations in similar classes of work.

Benchmark Notes

• GPU Scalability is relatively strong with some performance overhead. We are seeing a near 1:1 ratio of Speedup to GPU count. However, in workloads like Tacotron2, TransformerXL Large, and NCF, the GPU scalability is impacted. This is likely due to their smaller batch size type workloads that need to pull from memory often.
• NVIDIA Blackwell performs well beyond the NVIDIA Ada, where in some cases, double the performance (ResNet50, WaveGlow, SSD, and NCF). With increased memory, models and data can reside in memory, which reduces memory bandwidth bottlenecks
• RTX 6000 Ada and L40S show very similar performance since they are the same GPU architecture and die. We saw a few anomalies between these two GPUs, with both of them trading performance positions in some workloads.
• NVIDIA RTX PRO 6000 Blackwell Server Edition has slightly better performance than the NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition due to the higher power draw.

Generational Leap to Blackwell

On average, a single RTX PRO 6000 Blackwell Server Edition delivers nearly twice the throughput of an L40S. For workloads like SSD object detection, a single Blackwell card does what previously required two Ada-generation GPUs. The RTX 6000 Ada and L40S perform within ~12% of each other, sharing the same Ada Lovelace architecture; your choice should come down to price, availability, and form factor

A note on NCF: Neural Content Filtering is an interesting outlier where the RTX 6000 Ada outperforms both Blackwell GPUs at 1.32x vs. ~1.21x. NCF is a memory-bound, small-batch workload. RTX 6000 Ada has higher clock speeds and each subsequent sample (in our test) wouldn’t saturate the full 96GB in the Blackwell GPUs. If recommendation systems are your primary workload, factor this into your decision.

Multi-GPU Scaling Analysis - Scaling Efficiency (vs Single GPU)

Next, we will look at the scaling efficiency compared to single-GPU performance. Does adding GPUs deliver proportional performance gains? If 4 GPUs cost 4x as much, do you get 4x the throughput? We took an average of the 3 GPUs to calculate the scalability, which will tell us which workloads benefit from multiple GPUs.

Performance Insights from Our Benchmark Results

• LLM Training & Fine-tuning
  • Recommended GPU: RTX PRO 6000 Blackwell Server
  • 96GB VRAM fits larger models in memory; 1.9x throughput vs. Ada
• NLP / Transformer Research
  • Recommended GPU: RTX PRO 6000 Blackwell (Server or Max-Q)
  • BERT and TransformerXL see the largest generational gains; FP16 provides up to 2x additional acceleration
• Computer Vision (Training)
  • Recommended GPU: RTX PRO 6000 Blackwell Server
  • 1.87x on ResNet50 and 2.46x on SSD vs. L40S; excellent multi-GPU scaling for large datasets
• Audio / Speech Synthesis
  • Recommended GPU: RTX PRO 6000 Blackwell Max-Q
  • Strong Tacotron2/WaveGlow performance; FP16 provides no benefit, so FP32 accuracy comes free; Max-Q saves power in workstation setups
• Recommendation Systems
  • Recommended GPU: RTX 6000 Ada or L40S
  • NCF shows Ada outperforming Blackwell; memory-bound workloads don't fully leverage Blackwell's compute advantage
• Budget-Conscious Research
  • Recommended GPU: L40S 48GB
  • Within ~12% of RTX 6000 Ada; strong multi-GPU scaling; good performance-per-dollar entry point
• Production Inference at Scale
  • Recommended GPU: RTX PRO 6000 Blackwell Server
  • Maximum single-GPU throughput reduces the number of cards needed; 96GB VRAM supports larger batch sizes for higher utilization

Conclusion

Our extensive benchmarking analysis reveals the exceptional performance of modern GPUs across a diverse range of deep learning workloads. The data demonstrates near-linear scaling for multi-GPU configurations, with 2-GPU setups achieving up to 1.99x performance and 4-GPU systems delivering 3.94x speedups.

FP16 precision offers excellent acceleration in most workloads, especially transformer-based models (BERT and TransformerXL). For multi-GPU scalability, transformer-based models like show excellent multi-GPU efficiency, while models like WaveGlow, NCF, Tacotron2, and SSD scale less efficiently.

These insights provide valuable guidance for organizations designing AI infrastructure to match their specific workload requirements, whether for research experimentation, model development, or production deployment.

Need guidance with configuration and integration for your computing infrastructure? Exxact is a leading solution integrator helping to deliver custom-configurable workstations, servers, and full rack clusters. Contact us for more information!