The Raw Benchmark Numbers

Section By Andrei Frumusanu

Before we go into more details, we're going to have a look at how much of a difference this behavior contributes to benchmarking scores. The key is in the differences between having Huawei/Honor's benchmark detection mode on and off. We are using our mobile GPU test suite which includes of Futuremark’s 3DMark and Kishonti’s GFXBench.

The analysis right now is being limited to the P20’s and the new Honor Play, as I don’t have yet newer stock firmwares on my Mate 10s. It is likely that the Mate 10 will exhibit similar behaviour - Ian also confirmed that he's seeing cheating behaviour on his Honor 10. This points to most (if not all) Kirin 970 devices released this year as being affected.

Without further ado, here’s some of the differences identified between running the same benchmarks while being detected by the firmware (cheating) and the default performance that applies to any non-whitelisted application (True Performance). The non-whitelisted application is a version provided to us from the benchmark manufacturer which is undetectable, and not publicly available (otherwise it would be easy to spot). 

3DMark Sling Shot 3.1 Extreme Unlimited - Graphics - Peak 

3DMark Sling Shot 3.1 Extreme Unlimited - Physics - Peak 

GFXBench Aztec High Off-screen VK - Peak 

GFXBench Aztec Normal Off-screen VK - Peak 

GFXBench Manhattan 3.1 Off-screen - Peak 

GFXBench T-Rex Off-screen - Peak

We see a stark difference between the resulting scores – with our internal versions of the benchmark performing significantly worse than the publicly available versions. We can see that all three smartphones perform almost identical in the higher power mode, as they all share the same SoC. This contrasts significantly with the real performance of the phones, which is anything but identical as the three phones have diferent thermal limits as a result of their different chassis/cooling designs. Consequently, the P20 Pro, being the largest and most expensive, has better thermals in the 'regular' benchmarking mode.

Raising Power and Thermal Limits

What is happening here with Huawei is a bit unusual in regards to how we’re used to seeing vendors cheat in benchmarks. In the past we’ve seen vendors actually raise the SoC frequencies, or locking them to their maximum states, raising performance beyond what’s usually available to generic applications.

What Huawei instead is doing is boosting benchmark scores by coming at it from the other direction – the benchmarking applications are the only use-cases where the SoC actually performs to its advertised speeds. Meanwhile every other real-world application is throttled to a significant degree below that state due to the thermal limitations of the hardware. What we end up seeing with unthrottled performance is perhaps the 'true' form of an unconstrained SoC, although this is completely academic when compared to what users actually expereience.

To demonstrate the power behaviour between the two different throttling modes, I measured the power on the newest Honor Play. Here I’m showcasing total device power at fixed screen brightness; for GFXBench the 3D phase of the benchmark is measured for power, while for 3DMark I’m including the totality of the benchmark run from start to finish (because it has different phases).

Honor Play Device Power - Default vs Cheating

The differences here are astounding, as we see that in the 'true performance' state, the chip is already reaching 3.5-4.4W. These are the kind of power figures you would want a smartphone to limit itself to in 3D workloads. By contrast, using the 'cheating' variants of the benchmarks completely explodes the power budget. We see power figures above 6W, and T-Rex reaching an insane 8.5W. On a 3D battery test, these figures very quickly trigger an 'overheating' notification on the device, showing that the thermal limits must be beyond what the software is expecting.

This means that the 'true performance' figures aren’t actually stable - they strongly depend on the device’s temperature (this being typical for most phones). Huawei/Honor are not actually blocking the GPU from reaching its peak frequency state: instead, the default behavior is a very harsh thermal throttling mechanism in place that will try to maintain significantly lower SoC temperature levels and overall power consumption.

The net result is that that in the phones' normal mode, peak power consumption during these tests can reach the same figures posted by the unthrottled variants. But the numbers very quickly fall back in a drastic manner. Here the device thottles down to 2.2W in some cases, reducing performance quite a lot.

Benchmarking Bananas: A Recap Getting the Real Data: Kirin 970 GPU Performance Overview
POST A COMMENT

84 Comments

View All Comments

  • Andrei Frumusanu - Tuesday, September 4, 2018 - link

    As noted, the Aztec labels have been fixed. Reply
  • eastcoast_pete - Tuesday, September 4, 2018 - link

    @Andrei and Ian: I also wonder how the apparent superiority of highly customized GPUs in Apple and QC SoCs reflects a conundrum that ARM faces with their Mali designs: Basically, Mali GPUs have to work in configurations ranging from as few as 1-2 to as many as 20 (or 24) units and with widely varying CPU cores (dual-core A53 or 55 at the low end to now A76 or mongoose octacores). In contrast, Apple GPUs and QC's Adrenos appear to be a lot more "tailored" to the SoCs they end up in, which, together with optimized drivers, probably gives them a leg up. Andrei, as you are truly an seasoned expert in this field, I wonder if you could comment on this, maybe even in some detail in a dedicated article in the future? Reply
  • mekpro - Wednesday, September 5, 2018 - link

    Qualcomm also need to ship smaller version of Adreno to their lower-end SoC like Snapdragon 625 or so. While Anandtech haven't cover the efficiency of lower-end Snapdragon SoC, my experience confirmed that Snapdragon 625 had better much better GPU efficiency than Kirin 960 (lower temp at the same frame rate) Reply
  • vladx - Tuesday, September 4, 2018 - link

    What about Honor 9, is it affected after upgrading to EMUI 8? Reply
  • nfriedly - Tuesday, September 4, 2018 - link

    > So Honor is trying to promote the Honor Play as a gaming-centric phone, making bold marketing claims about its performance and experience. This is a quite courageous marketing strategy given the fact that the SoC powering the phone is currently the worst of its generation when it comes to gaming.

    Do you mean "courageous" in the Apple sense?
    Reply
  • V900 - Tuesday, September 4, 2018 - link

    I’m shocked and stunned, nay SHOCKED AND STUNNED that Chinese Smartphone/SOC vendors have been caught cheating in benchmarks.

    Such a surprising development from a country that’s known for being the high watermark in ethical and honest business practices.

    (Industrial espionage and stealing stealth fighters through hacking not withstanding.)
    Reply
  • Allan_Hundeboll - Tuesday, September 4, 2018 - link

    I know Oneplus used to do something similar but they did it for benchmarks and games, witch is'nt cheating I my book because end users do get the performance shown in benchmarks.
    Is oneplus still doing this? The oneplus 6 seems to benchmark faster than other sd845 based phones, so this would kind of explain how.
    Reply
  • zodiacfml - Tuesday, September 4, 2018 - link

    Is this click bait or something?
    One cannot simply add more power/speed to a device without throttling or shutting down. If one would run gaming benchmark for more than 10 minutes then it levels out. It is a different story with browser benchmarks where burst speeds for a few seconds is valuable.

    It would have been cheating if a phone starts dimming or turning off the display when there is a benchmark run.

    I wouldn't put much more value on battery efficiency when running games or benchmarks because it will be difficult to regulate. Everyone will have differing opinions about it.
    When you turbo/boost (any chip), it is the least efficient anyway.
    Reply
  • unixfg - Wednesday, September 5, 2018 - link

    No, not Huawei.

    https://www.engadget.com/2018/08/20/huawei-caught-...
    Reply
  • mekpro - Wednesday, September 5, 2018 - link

    I always need extra cooling in order to make PUBG playable on My poor Mate 9.
    Sometimes I use wet tissue to paste on the back of the chassis, sometimes I just put Ice and let it melt to cool the phone down. (which usually take less than 3 minutes to melt all). Yes I know this phone is not water resistance certified but come on! its just a hot potatoes that need to be cooled and I don't expect flagship phone to have this behavior. The ironic is I had buy Xiaomi Redmi Note 4X with Snapdragon 625 for 150$ and this devices play PUBG much better than Huawei's Flagship.

    After read this article, I choose to not believe Huawei's marketing anymore, I don't believe that Kirin 980 can close the gap in GPU performance with Snapdragon 845, let alone acceptable performance.
    Reply

Log in

Don't have an account? Sign up now