How to solve the inconsistent Lighthouse performance score problem?

One of the biggest problems for developers who want to improve and maintain their website's speed and performance is to get a performance measurement tool and environment that gives them a consistent and trustable score. The developers express this problem often, if you face the below problem read on

"the performance score is different each time, How do i even debug, if there is no consistency? How do I solve? "

"The only problem is the results keep changing, It changes whenever I refresh the page, after one second. It never remains the same although I've made NO updates on my website. Why? "

"After some work, Lighthouse (embedded in Chrome) gives me a Server Response time of 200ms, my page is noticeably faster. But Pagespeed insights still gives me results in the 1.5s range. Which one should I trust? "

Even Google acknowledges the lighthouse score variability problem here.

"Lighthouse performance scores will change due to inherent variability in web and network technologies, even if there hasn't been a code change. "

Tools and Platforms - What do they do?

Lighthouse

Lighthouse is a tool by Google which helps with different aspects of your web page like performance, SEO, accessibility, progressive web app support and more. It has become the most popular tool by developers to measure web page performance through the Chrome browser.

It can be used to measure the performance score of your website, but it will be specific to the hardware you opened the browser and website. So the score you see on your desktop could be drastically different from the one on a mobile device that is used by the visitor of your website.

Pagespeed Insights

Pagespeed Insights is a platform for performance measurement by Google that gives you both Lighthouse score and real world data from Chrome User Experience Report(CrUX) dataset.

You don't have any option to set any of the test configurations like browser version, lighthouse version, OS, network speed, CPU Memory power. It runs a Lighthouse test on a headless Chrome browser in one of their data centers that is closer to you. For eg: I'm in India and the tests triggered by me are run in 'Asia'. These restrictions make the Lighthouse score on this platform less valuable for your specific scenario.

The platform also gives you the real world measurement of Core Web Vital(CWV) metrics from CrUX dataset. But you will get data only if you have sufficient traffic, otherwise your website data will not be available.

Core Web Vitals

Core Web Vitals are a set of user experience focussed performance metrics that Google collects from real Chrome users visiting your website. Lighthouse score is a lab measurement(synthetic data) of your website performance and Core Web Vitals is the real world measurement(RUM) of your website's performance.

Google uses this data as a factor in the search results ranking of your website. Improving your Core Web Vitals will improve your websites performance for your users and in turn your ranking in search results. But it is a lagging indicator or measurement of your performance improvement efforts as it will take at least 28 days to reflect.

Performance test variables

There are multiple variables to be taken into account while measuring your website performance. Some of them are

• Browser
• Browser version
• Device/Desktop hardware - CPU and Memory(RAM),
• Network latency and speed(bandwidth)
• Location(users location)
• Performance measurement tool version(eg: Lighthouse version)
• Web server load
• Local Network load(Devices and traffic on your LAN/wifi)

Reliable Performance Testing

SpeedLab performance tests eliminate most of the effects of these variable factors. We do it by providing a test environment that is consistent and removes any kind of simulation.

CPU and Memory

It is important to test performance of your websites on mobile devices as mobile internet usage continues to grow at a faster rate than desktop internet usage. Lighthouse tests on your local and on pagespeed insights platform simulate CPU throttling. They run the tests on VMs in Google data centers and they simulate the CPU performance of mobile devices.

The Lighthouse test on a local computer and PageSpeed use CPU simulation for Moto G4 devices. This model is 6 years old and discontinued, very few people use this model. In general, CPU simulation has many limitations. It is done by introducing time delays on a test run on unthrottled CPU post the test run. Simulation is only for the test process and doesn't take into account the background processes running on a CPU, the age related deterioration in performance, heat related deterioration, etc.

In SpeedLab you can run Lighthouse tests on real mobile devices, you can choose a device that has similar configuration to your users devices. By using real devices SpeedLab eliminates the need for CPU and Memory simulation. If you have made any improvements to your website and want to measure the performance before rolling it out to your users, you can test them on SpeedLab. Do remember that page speed insights only simulates Moto G4.

Network Latency and Bandwidth

Lighthouse by default gives you simulated network throttling, i.e. the actual test is run on an unthrottled network and then Lighthouse introduces delays in the network times based on the chosen network speed.

In SpeedLab, we have i mplemented packet level throttling across all our devices and desktops. So, we have eliminated another simulation by implementing real throttling. This may not be the same as a user experiencing a slower network in the real world, but it is the closest possible option today in a lab environment with no software simulation.

Location

In Lighthouse tests run on your local computer or PageSpeed Insights, you don't have any option to test on a different location that may be closer to your users. SpeedLab gives you multiple real location options across the globe for Lighthouse tests. The performance scores differ significantly based on the location of the test. You have to test closest to your users.

If you test on a different location than the majority of your users, then your Lighthouse metrics will not match with the Core Web Vitals metrics. It will be impossible to predict the impact of your optimizations, you will have to wait at least 28 days to see the results. SpeedLab solves this problem by giving real locations to test from.

External Factors and 5 Run Median Score

There are factors that are beyond our control or it may not be realistically possible to control them. Some of these factors are web server load, internal browser tasks, hops in network routes, etc. All these factors impact the score.

SpeedLab reduces the impact of external factors by running 5 measurements for each test and selects the median score. When we repeat the test multiple times, the impact of external factors is reduced. As per a study by Google, the median score from 5 runs is twice as stable as a score from a single run.

Browser and Browser Versions

SpeedLab provides the latest versions of all major browsers. These are real browsers on real devices and desktops, not simulated versions. You can choose to measure the performance on the browser versions that is commonly used by your website users.

Reduced Variability

SpeedLab has focussed on giving a stable environment and the tools to measure performance. We believe we have eliminated and reduced most of the factors that contribute towards score variance. We also continue to identify new opportunities to help developers and QA in improving and maintaining the website performance. I would encourage you to test the performance of your website in a stable environment with real devices for free.