Setting expectations for flagship smartphones in 2023


Just last week saw the announcement of the Qualcomm Snapdragon 8 Gen 2 at the company’s Technology Summit in Hawaii. Qualcomm’s latest chipset packages upgraded specs and over TSMC’s manufacturing process, which, if the 8 Plus Gen 1 is anything to go by, should deliver some efficiency gains. Moreover, while the company has been reluctant to provide in-depth technical details in some aspects (including neglecting to mention the Adreno or Kryo version name), we can still run a series of popular benchmarks on the Snapdragon 8 Gen 2 reference device. These benchmarks help lay the groundwork for performance projections for the next flagship in 2023, giving us something to look forward to.


About this articleQualcomm sponsored my colleague, Rich Woods, to attend the Snapdragon Tech Summit in Maui, Hawaii. The company paid for the flight and hotel. However, Qualcomm did not have any input regarding the content of this article.

How we evaluated the performance of the Snapdragon 8 Gen 2

On Qualcomm’s Snapdragon 8 Gen 2 reference device, we ran a single comprehensive benchmark (AnTuTu), a CPU-centric benchmark (Geekbench), a GPU-focused benchmark (GFXBench), and MLPerf benchmarks. Each criterion was run three times, and we took the average of the three results. Qualcomm enabled the “UI Perf Mode” option by default which we left enabled. It’s effectively trying to force benchmark apps to run on Prime cores to get a slightly higher score on certain benchmarks, so please keep that in mind when looking at these scores. It’s also worth noting that once we get a commercial device with the Qualcomm Snapdragon 8 Gen 2, we’ll be rerunning these benchmarks.

Qualcomm has provided us with a range of expected benchmark scores based on its own testing. We used this as a reference only, and a table is available at the bottom of this article containing the test scores that Qualcomm expected the reference device to achieve.

Snapdragon 8 Gen 2 benchmarks overview

  • AnTuTu: This is a comprehensive standard. AnTuTu tests CPU, GPU, and memory performance, while including both abstract tests and simulating the relevant user experience (for example, a subtest involving scrolling through a ListView). The end result is weighted according to the designer’s considerations.
  • GeekBench: This is a CPU-focused test that uses several computational workloads, including encoding, compression (text and images), rendering, physical simulation, computer vision, ray tracing, speech recognition, and convolutional neural network inference on images. Point division gives specific metrics. The end result is weighted according to the designer’s considerations, with a large focus on correct performance (65%), then float performance (30%), and finally encryption (5%).
  • GFXBench: It aims to simulate the rendering of video game graphics using the latest APIs, which includes a lot of on-screen effects and high-quality textures. Newer tests use Vulkan, while older tests use OpenGL ES 3.1. The outputs are frames during the test and frames per second (the other number divided by the length of the test, basically) rather than a weighted score.
    • Aztec ruins: These tests are the most computationally difficult offered by GFXBench. Right now, the best mobile chipsets can’t handle 30fps. Specifically, the test offers high polygon count geometry, hardware tessellation, high-resolution textures, global illumination and plenty of shadow maps, copious particle effects, as well as field effects and depth of bloom. Most of these technologies will emphasize the shading computing capabilities of the processor.
    • Manhattan S 3.0 / 3.1: This test remains relevant since modern games have already reached the suggested graphical resolution and implement the same types of technologies. It features a complex geometry that uses multiple rendering objectives, reflections (cubic maps), mesh rendering, and many delayed light sources, as well as blooms and depth of field in the post-processing lane.
  • MLPerf Mobile: MLPerf Mobile is an open source benchmark for mobile AI performance testing. It was created by MLCommons, an open, not-for-profit engineering consortium, “to provide transparency and equal opportunity for comparing machine learning systems, software, and solutions.” The first iteration of MLPerf Mobile provides a benchmark for performing inference for a handful of computer vision and natural language processing tasks. For more information, refer to this paper titled “MLPerf Mobile Inference Benchmark: Why Mobile AI Benchmarking Is Hard and What to Do About It.”
    • Image rating: This test involves label inference to apply to an input image. Typical use cases include image searches or text extraction. The reference model used is MobileNetEdgeTPU with 4M parameters, the dataset is ImageNet 2012 (224 × 224), and the quality target is 98% of FP32 (76.19% Top-1).
    • Image cropping: This test involves dividing the input image into tagged objects. Typical use cases include autonomous driving or remote sensing. The reference model used is DeepLab v3+ with 2M parameters, the dataset is ADE20K (512 × 512), and the quality target is 93% of FP32 (0.244 mA).
    • object detection: This test involves drawing bounding boxes around objects and providing a label to those objects. Typical use cases include camera input such as hazard detection or traffic analysis while driving. The reference model is SSD-MobileNet v2 with 17M parameters, the dataset is COCO 2017 (300 x 300), and the quality target is 97% of FP32 (54.8% mIoU).
    • Language processing: This test involves answering questions colloquially. Typical use cases include online search engines. The reference model is MobileBERT with 25M parameters, the dataset is mini Squad (Stanford Questions Answer Dataset) v1.1 dev, and the quality target is 93% of FP32 (93.98%F1).

benchmark results

Antoto

Antutu-Scores-Snapdragon-8-Gen-2


As in previous years, we’re seeing nearly a 10% improvement in our AnTuTu score with this year’s Snapdragon 8 Gen 2. And that’s a big enough improvement that already indicates from the start that the Snapdragon 8 Gen 2 is the most powerful chipset out of every other Qualcomm chip to date. It’s not quite on par with the 35% faster CPU performance, but with AnTuTu being such a comprehensive benchmark it doesn’t necessarily mean it will fully reverse any CPU gains.

Geekbench5

Geekbench-5-Scores-Snapdragon-8-Gen-2-1


Geekbench, however, he is CPU-centric performance measure. We see roughly a 30% gain in multi-core performance, which seems to be on track toward the 35% improvement Qualcomm is announcing. Benchmarks won’t always reflect these gains that Qualcomm measures, but that’s because of the difference in scaling. Each tool has a different method that it uses when calculating scores and testing chipsets, and the Geekbench method may not necessarily reveal those improvements that Qualcomm will have made. The 30% improvement reflected in the year-over-year improvement is still impressive.

GFXBench

GFXBench-Snapdragon-8-Gen-2


Qualcomm hasn’t revealed much about the Adreno GPU in the Snapdragon 8 Gen 2, so we don’t have much to say about the GPU other than its performance gains. We don’t know the base number, we don’t know the iteration, and we don’t even have the version number. This is a change that happened with the Snapdragon 8 Gen 1, and it’s disappointing when you compare GPUs. It is much easier to explain the differences in the context of version numbers than to name a specific chip each time.

However, the results show an overall improvement in graphics performance, regardless of GFXBench’s test of the T-Rex. This test is a low-intensity test, so I wouldn’t put a lot of stock into it aside from the fact that it has a lower frame rate. It could just be an optimization, and the other, more intensive tests have much better results. In GFXBench’s Manhattan test, which uses the OpenGL ES 3.1 API and renders an off-screen scene in 1080p, the Snapdragon 8 Gen 1 had a frame rate of 179fps. In contrast, the Snapdragon 8 Gen 2 hit 222 fps.

In GFXBench’s Aztec Ruins test, which uses the Vulkan graphics API and renders an off-screen scene in 1080p, the Snapdragon 8 Gen 1 had a frame rate of 49fps. In comparison, the Snapdragon 8 Gen 2 pulled in 65fps. Graphics performance has obviously improved, and some of those gains are significant. That’s a 44% improvement on the Aztec Ruins Vulkan test, and a 24% improvement on the Manhattan test.

Some great Android games just require a lot of GPU power, but the improved GPU performance is useful for more than just gaming.

MLPerf

MLPerf - Snapdragon 8 Gen 2-Watermarked


Qualcomm has been particularly keen on detail regarding AI improvements, and that’s always been the case. We don’t have any numbers for TOPS (trillion operations per second), although the company did give us information on some tangible improvements, such as a 435% increase in AI performance and 65% better performance per watt. The results above show how the Snapdragon 8 Gen 2 performs in AI, and you can compare it to other devices tested by MLCommons.

Conclusion and expected results

The table Qualcomm provided us with expected benchmark scores below, which you can see is mostly in line with the results we achieved above.

standard Issuance method expected result range
System Geekbench ST v5.4.4 At a rate of 3 repetitions ~ 1485 – 1495
System Geekbench MT v5.4.4 At a rate of 3 repetitions ~5050 – 5200
System AnTuTu Version 9.3.0 First round: ~1.27 – 1.28m Average 3 reps: ~1.26m
System PCMark Version 3.0.4061 At a rate of 3 repetitions ~18.5 – 18.9k
Browser (Chrome v95.0.4638.74 64-bit) Jet v2.0 At a rate of 3 repetitions ~167 – 170
browser Speedometer v2.0 At a rate of 3 repetitions ~144 – 146
browser WebXPRT Version 3.0 At a rate of 3 repetitions ~219 – 220
graphics GFXBench Manhattan 3.0 Offscreen (1080p) (FPS) v5.0 At a rate of 3 repetitions ~ 329 – 332 fps
graphics GFXBench T-Rex – Off Screen (1080p) (FPS) v5.0 At a rate of 3 repetitions ~ 481 – 484 fps
graphics GFXBench Manhattan 3.1 Offscreen (1080p) (FPS) v5.0 At a rate of 3 repetitions ~ 224 – 226 fps
graphics GFXBench Car Chase Offscreen (1080p) ES3.1 (FPS) v5.0 At a rate of 3 repetitions ~129-130fps
graphics GFXBench Aztec Ruins Vulkan (High Level) Offscreen (1440p) (FPS) v5.0 At a rate of 3 repetitions ~60fps
graphics GFXBench Aztec Ruins OpenGL (High Level) Offscreen (1080p) (FPS) v5.0 At a rate of 3 repetitions ~ 178 – 179 fps
graphics 3DMark Wild Life Unlimited v2.2.4786 At a rate of 3 repetitions 82
graphics 3DMark Wild Life Extreme Unlimited v2.2.4786 At a rate of 3 repetitions 23
Amnesty International MLPerf v2.1 Image rating: 3915 – 3920 Object detection: 1765 – 1800 V2.0 Image segmentation: 945 – 950 Language comprehension: 185 Image rating (offline): 4980 – 5020

Qualcomm says the first devices powered by the Snapdragon 8 Gen 2 will be here by the end of 2022. We’ll be watching how the Snapdragon 8 Gen 2 performs when compared to the likes of the MediaTek Dimensity 9200. If you’re upgrading from a device at least a couple of years older, the improvements are likely to be Note, though, that the huge gains in AI performance will likely go unnoticed by most people. Rarely do companies harness the full potential of AI when it comes to Qualcomm chips, and it will likely be the same again here.

Qualcomm has confirmed that the following companies will launch devices powered by Snapdragon 8 Gen 2 technology: Redmagic, Honor, ZTE, Xiaomi, Meizu, Vivo, Sony, Redmi, OPPO, nubia, Motorola, OnePlus, Sharp, Asus, and iQOO. We look forward to experimenting with these chips in a more controlled environment in commercial hardware in the future.

Leave a Reply

%d bloggers like this: