The Raw Benchmark Numbers

Section By Andrei Frumusanu

Before we go into more details, we're going to have a look at how much of a difference this behavior contributes to benchmarking scores. The key is in the differences between having Huawei/Honor's benchmark detection mode on and off. We are using our mobile GPU test suite which includes of Futuremark’s 3DMark and Kishonti’s GFXBench.

The analysis right now is being limited to the P20’s and the new Honor Play, as I don’t have yet newer stock firmwares on my Mate 10s. It is likely that the Mate 10 will exhibit similar behaviour - Ian also confirmed that he's seeing cheating behaviour on his Honor 10. This points to most (if not all) Kirin 970 devices released this year as being affected.

Without further ado, here’s some of the differences identified between running the same benchmarks while being detected by the firmware (cheating) and the default performance that applies to any non-whitelisted application (True Performance). The non-whitelisted application is a version provided to us from the benchmark manufacturer which is undetectable, and not publicly available (otherwise it would be easy to spot). 

3DMark Sling Shot 3.1 Extreme Unlimited - Graphics - Peak 

3DMark Sling Shot 3.1 Extreme Unlimited - Physics - Peak 

GFXBench Aztec High Off-screen VK - Peak 

GFXBench Aztec Normal Off-screen VK - Peak 

GFXBench Manhattan 3.1 Off-screen - Peak 

GFXBench T-Rex Off-screen - Peak

We see a stark difference between the resulting scores – with our internal versions of the benchmark performing significantly worse than the publicly available versions. We can see that all three smartphones perform almost identical in the higher power mode, as they all share the same SoC. This contrasts significantly with the real performance of the phones, which is anything but identical as the three phones have diferent thermal limits as a result of their different chassis/cooling designs. Consequently, the P20 Pro, being the largest and most expensive, has better thermals in the 'regular' benchmarking mode.

Raising Power and Thermal Limits

What is happening here with Huawei is a bit unusual in regards to how we’re used to seeing vendors cheat in benchmarks. In the past we’ve seen vendors actually raise the SoC frequencies, or locking them to their maximum states, raising performance beyond what’s usually available to generic applications.

What Huawei instead is doing is boosting benchmark scores by coming at it from the other direction – the benchmarking applications are the only use-cases where the SoC actually performs to its advertised speeds. Meanwhile every other real-world application is throttled to a significant degree below that state due to the thermal limitations of the hardware. What we end up seeing with unthrottled performance is perhaps the 'true' form of an unconstrained SoC, although this is completely academic when compared to what users actually expereience.

To demonstrate the power behaviour between the two different throttling modes, I measured the power on the newest Honor Play. Here I’m showcasing total device power at fixed screen brightness; for GFXBench the 3D phase of the benchmark is measured for power, while for 3DMark I’m including the totality of the benchmark run from start to finish (because it has different phases).

Honor Play Device Power - Default vs Cheating

The differences here are astounding, as we see that in the 'true performance' state, the chip is already reaching 3.5-4.4W. These are the kind of power figures you would want a smartphone to limit itself to in 3D workloads. By contrast, using the 'cheating' variants of the benchmarks completely explodes the power budget. We see power figures above 6W, and T-Rex reaching an insane 8.5W. On a 3D battery test, these figures very quickly trigger an 'overheating' notification on the device, showing that the thermal limits must be beyond what the software is expecting.

This means that the 'true performance' figures aren’t actually stable - they strongly depend on the device’s temperature (this being typical for most phones). Huawei/Honor are not actually blocking the GPU from reaching its peak frequency state: instead, the default behavior is a very harsh thermal throttling mechanism in place that will try to maintain significantly lower SoC temperature levels and overall power consumption.

The net result is that that in the phones' normal mode, peak power consumption during these tests can reach the same figures posted by the unthrottled variants. But the numbers very quickly fall back in a drastic manner. Here the device thottles down to 2.2W in some cases, reducing performance quite a lot.

Benchmarking Bananas: A Recap Getting the Real Data: Kirin 970 GPU Performance Overview
Comments Locked


View All Comments

  • goatfajitas - Tuesday, September 4, 2018 - link

    I think it speaks directly to the subject matter. Far too many companies, tech sites, and users hang far too much on bench-marking. Does 1% better "geekmarks" actually make a difference in usage?
  • cfenton - Tuesday, September 4, 2018 - link

    But it's not 1%, or even close to that. The OnePlus 6 is 3x faster than the P20 Pro in most of the sustained performance benchmarks.

    I think it's important for users to know that the Kirin 970 has a significantly weaker GPU than the S845, especially when Huawei is marketing some of these phones as 'gaming phones'.
  • goatfajitas - Tuesday, September 4, 2018 - link

    Now you are comparing 2 different class phones (flagship OP6 vs mid-range P20) and CPU from 2 different years (2017's 970 vs 2018's S845). In most areas like vs. like is very close, and there is alot more complexity to it than just that. It's all way over-hyped and over-emphasized
  • R0H1T - Tuesday, September 4, 2018 - link

    No he's right, current QC flagship SoC vs reigning Hisilicon Flasghip. In case you din't know SD845 is available in phones priced as low as $300 lower than P20 in fact.
  • R0H1T - Tuesday, September 4, 2018 - link

    >reply meant for goatfajitas
  • goatfajitas - Tuesday, September 4, 2018 - link

    Fair enough, but the Kirin 980 is out in a month. It is last years product and when it came out it compared to the S835. Both on yearly cadence appx 6 months apart.
  • Cicerone - Friday, September 7, 2018 - link

    A phone price is not standing from it's SOC. Display, cameras quality, type of storage and software is accounted.
  • cfenton - Tuesday, September 4, 2018 - link

    The P20 Pro came out this year and actually cost more at launch than the OP6 (~700-800 euro vs ~500-600 euro). The P20 Pro also came out after the GS9, which has the S845. Even the P20 launched at 649 euro. If you'd prefer to compare phones that came out at the same time, then the GS9 is still over 2x faster. If you want to compare the Kirin 970 to the S835, since both came out in 2017, then the 835 is still 2x as fast.
  • goatfajitas - Tuesday, September 4, 2018 - link

    From a GPU perspective you aren't incorrect... If you are buying a phone as a gaming device you might want to look at GPU benchmarks more, but you do however seem to be overly obsessed with bench-marking... You are also not comparing like for like. The P20 is not Huawei's flagship, the Mate is. Oneplus is a crappy budget brand with massive quality issues and the price shows it. IF you want to buy one go for it.
  • cfenton - Tuesday, September 4, 2018 - link

    The P20 Pro is priced like a flagship and has their top-end SOC. Huawei just has two high-end models for some reason.

    Again, even if you insist on comparing only flagships from big companies, Huawei comes out looking bad. The Pixel 2 and Galaxy S8 are both significantly faster than the Mate 10.

    I'm not obsessed with benchmarks, but I think they have value. We aren't talking about modern SSD benchmarks where one brand reads at 3100mbps and the other at 3200mbps. That's a small difference. This is very different. We're talking about at least double the performance.

Log in

Don't have an account? Sign up now