Mea-Culpa: It Should Have Been Caught Earlier

Section By Andrei Frumusanu

As stated on the previous page, I had initially had seen the effects of this behaviour back in January when I was reviewing the Kirin 970 in the Mate 10. The numbers I originally obtained showed worse-than-expected performance of the Mate 10, which was being beaten by the Mate 9. When we discussed the issue with Huawei, they attributed it to a firmware bug, and pushed me a newer build which resolved the performance issues. At the time, Huawei never discussed what that 'bug' was, and I didn't push the issue as performance bugs do happen.

For the Kirin 970 SoC review, I went through my testing and published the article. Later on, in the P20 reviews, I observed the same lower performance again. As Huawei had told me before it was a firmware issue, I had also attributed the bad performance to a similar issue, and expected Huawei to 'fix' the P20 in due course.

Looking back in hindsight, it is pretty obvious there’s been some less than honest communications with Huawei. The newly detected performance issues were not actually issues – they were actually the real representation of the SoC's performance. As the results were somewhat lower, and Huawei was saying that they were highly competetive, I never would have expected these numbers as genuine.

It's worth noting here that I naturally test with our custom benchmark versions, as they enable us to get other data from the tests than just a simple FPS value. It never crossed my mind to test the public versions of the benchmarks to check for any discrepancy in behaviour. Suffice to say, this will change in our testing in the future, with numbers verified on both versions.

Analyzing the New Competitive Landscape

With all that being said, our past published results for Kirin 970 devices were mostly correct - we had used a variant of the benchmark that wasn’t detected by Huawei’s firmware. There is one exception however, as we weren't using a custom version of 3DMark at the time. I’ve now re-tested 3DMark, and updated the corresponding figures in past reviews to reflect the correct peak and sustained performance figures.

As far as I could tell in my testing, the cheating behaviour has only been introduced in this year’s devices. Phones such as the Mate 9 and P10 were not affected. If I’m to be more precise, it seems that only EMUI 8.0 and newer devices are affected. Based on our discussions with Huawei, we were told that this was purely a software implementation, which also corroborates our findings.

Here is the competitive landscape across our whole mobile GPU performance suite, with updated figures where applicable. We are also including new figures for the Honor Play, and the new introduction of the GFXBench 5.0 Aztec tests across all of our recent devices:

3DMark Sling Shot 3.1 Extreme Unlimited - Graphics 

3DMark Sling Shot 3.1 Extreme Unlimited - Physics 

GFXBench Aztec Ruins - High - Vulkan/Metal - Off-screen GFXBench Aztec Ruins - Normal - Vulkan/Metal - Off-screen 

GFXBench Manhattan 3.1 Off-screen 

GFXBench T-Rex 2.7 Off-screen

Overall, the graphs are very much self-explanatory. The Kirin 960 and Kirin 970 are lacking in both performance and efficiency compared almost every device in our small test here. This is something Huawei is hoping to address with the Kirin 980, and features such as GPU Turbo.

Raw Benchmark Numbers The Reality of Silicon And Market Pressure
POST A COMMENT

84 Comments

View All Comments

  • arpit - Saturday, September 8, 2018 - link

    Benchmarking these phone after few times will degrade its SOC due to overheating. Idk why Huawei/Honor has resorted to using this cheat tricks. Reply
  • LemmingOverlord - Tuesday, September 11, 2018 - link

    I think the real questions Andrei and Ian should be asking is whether the devices are detecting not only benchmarking software but also popular software (games included) that people use for extended periods. If that is the case, the device's behavior will shorten its lifespan over a shorter amount of time. Reply
  • ashraf mahdy - Friday, February 8, 2019 - link

    I know this thread is old but I wanted to share my story as well, so yesterday EMUI 9 updated arrived for my Honor Play in my country. bringing the "performance mode" toggle from the battery settings. anyone watching "Gary Explains" on youtube knows that Kirin 980 can have increased performance turning on "performance mode" in the battery settings.
    before updating i ran a few antutu benchmarks, my scores were 206-210,000. my highest was 212,000. using antutu sensors my Hottest CPU & battery were 41-43C
    right after updating I was getting 165-170,000. and my lowest score was 153,000. which is basically SD660 & Kirin 710 Performance, but my phone was not getting warm at all, like max CPU & Battery was 32C after 3 tests. at this moment I was like "HUAWEI THOSE LYIN CHEATIN BASTARDS" then I remembered Performance Mode, Turned it on, and BOOM, ran 3 Tests, was getting around 206-210,000. with my LOWEST Score being 198,000 after 4 looping tests and my SoC & battery were peaked at 41C. and after letting the phone cooldown to around 19C I ran a final test and got 216,000 with SoC & battery at like 36C.
    my point is as follows.
    SoC sustained performance vs peak can vary by as much as 20,000 on antutu, it all depends on the phone's cooling ability. the Mate 10 Pro has thermal paste on the Kirin 970. the honor Play does not. (check Jerryrigeverything for teardown of M10Pro).
    "performance mode" is not "cheat mode". the kirin 970 may not be as efficient as SD835 but it does manage to go neck and neck. it's like desktop GPUs. the RX 580 vs 1060.
    yes in phones batteries are the main culprit. but that peak power draw only happens when gaming, and let's face it, no one games for 1-2 hours on their phones, it's more like 2-3 games of PUBG mobile or a few fifa soccer matches. 30mins or so.
    so Huawei are shady af yes, Qualcomm's SD is better yes. but kirin is respectable as well.
    Reply
  • jessicacoles - Saturday, December 12, 2020 - link

    https://www.shampooadvice.com/ Reply

Log in

Don't have an account? Sign up now