The post Is (website) performance improved after improving percentile 90? first appeared on Deployment Every Day.

DOMContentLoaded

According to docs when browser fires this event it means

• DOM has been completely parsed
• <script defer src="…"> and <script type="module"> has been loaded
• async script, images, frames might not be loaded yet

load event

When browser fires load event it means DOM content is fully loaded. Browser has finished

• executing all asynchronous resources from the DOM tree
• loading images, frames etc.
• it comes always after DOMContentLoaded

Both events are NOT about REST calls. And both are not relevant to the user browsing the website.

What do web apps automation tools like playwright and selenium do by default?

Take a look at following lines of code measuring navigating to a page and waiting for a button to be visible.

using playwright

val millisBefore = System.currentTimeMillis()
page.navigate("https://github.com")
page.getByText("Try Now").waitFor()
val latency = System.currentTimeMillis() - millisBefore

or doing the same with selenium

val millisBefore = System.currentTimeMillis()
driver.navigate().to("https://github.com")
WebDriverWait(driver, 30, 100).until(ExpectedConditions.visibilityOfElementLocated(By.linkText("Sign up for GitHub")))
val latency = System.currentTimeMillis() - millisBefore

When you navigate to a page, by default those automation tools wait for load event. So the line with locator is not executed until then.

You might think Pff, so what. It’s just a technical detail

It is indeed a technical detail but it impacts performance benchmark dramatically.

Waiting for DOMContentLoaded VS load VS NOT waiting for any event. Fight!

Let’s measure how quickly user can see github call to action button.

Baseline benchmark – wait for load event

Take 1000 samples. Wait for load event, then wait for github call to action button

// only most important code shown here. Processing measurements is out of the scope
page.navigate("https://github.com")
page.getByText("Sign up for GitHub").first().waitFor(WaitForOptions().setTimeout(2000.0))

It looks like this

How to interpret the video of web app measurement?

Playwright video recording is imperfect. Video frame is often not aligned with what was seen in web browser. I confirmed that with modifying the measured page after measurement end.

During recording web app, right after measurement end timer is added to the page. This way it doesn’t interfere with measurement and helps to see when measurement has actually finished on the video.

const measuerementEndMillis = ..; // depends on the measuerement
const timerDiv = document.createElement('div');
timerDiv.style.position = 'fixed';
timerDiv.style.top = '0';
timerDiv.style.left = '0';
timerDiv.style.opacity = '0.8';
timerDiv.style.fontSize = '14px';
timerDiv.style.backgroundColor = 'yellow';
timerDiv.style.zIndex = '10000';
document.body.appendChild(timerDiv);

setInterval(function() {
  const now = performance.now();
  timerDiv.textContent = now.toFixed(0) + ' ms, ' + (now - measurementEndMillis).toFixed(0) + ' ms';
}, 20);

When measuring very fast page like github.com landing page, you’re often going to see a blank page and right after that frames recorded some time after measurement end (signup button visible). Frames are lost. Based on that you should treat every moment of playwright video as possibly inaccurate.

Having said that, let’s move on.

Experiment benchmark 1 – no waiting

Take 1000 samples. Don’t wait for any event, wait for github call to action button

import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT

// from playwright docs WaitUntilState.COMMIT:
// consider operation to be finished when network response is received and the document started loading
page.navigate("https://github.com", NavigateOptions().setWaitUntil(COMMIT))
page.locator("#summary-val").first().waitFor()

In selenium you have to opt out from waiting for DOMCompleteEvent this way

val options = new ChromeOptions()
options.setPageLoadStrategy(PageLoadStrategy.NONE)    
val driver = new ChromeDriver(options)
driver.navigate(..).to()

Experiment benchmark 2 – wait for DOMContentLoaded event

Take 1000 samples. Wait for DOMContentLoaded, wait wait for github call to action button

import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.DOMCONTENTLOADED

// from playwright docs WaitUntilState.DOMCONTENTLOADED:
// consider operation to be finished when the DOMContentLoaded event is fired.
page.navigate("https://github.com", NavigateOptions().setWaitUntil(DOMCONTENTLOADED))
page.locator("#summary-val").first().waitFor()

In selenium you have to opt out from waiting for DOMContentLoaded event this way

val options = new ChromeOptions()
options.setPageLoadStrategy(PageLoadStrategy.EAGER)    
val driver = new ChromeDriver(options)
driver.navigate(..).to()

And now let’s compare the results

P95 comparison

Percentiles do not answer the question how distribution compares? Down the line you’re going to see Hodges-Lehman estimator

What hypotheses are we verifying here?

• load event
  • (baseline) null hypothesis: waiting for load event has completion has no performance impact
  • (experiment 1) alternative hypothesis: waiting for loadevent has impact on measured latencies
• DOMContentLoaded event
  • (baseline) null hypothesis: waiting for DOMContentLoad event has has no impact on measured latencies
  • (experiment 2) alternative hypothesis: waiting for DOMContentLoadevent has impact on measured latencies

Waiting for load event VS not waiting for any event

Title of this blog post says there is 530% performance degradation when you rely on browser automation defaults. So where is it? When you reverse baseline VS experiment, there it is

Null hypothesis is rejected, waiting for load event comes with bigger latencies

Waiting for DOMContentLoaded VS not waiting for any event

Null hypothesis is rejected, waiting for DOMContentLoaded increases measured latencies

Waiting for DOMContentLoaded VS waiting for load event

Having 2 experiments we can compare them too. Since load event comes always after DOMContentLoaded it’s expected to see bigger latencies which is reflected in measurements

Based on measurements, what are undesired effects of waiting for DOMContentLoad or load event?

• To understand better following part take a look at difference between accuracy and precision

• The undesired effects
• Measuring something neither reflecting user experience nor relevant to user experience. User sees a button and clicks a button regardless of events in the browser
  • No user ever thought “I politely wish this site had dom completion faster“. User thinks “Why the f^%@ is this button not working yet?!!??“
• (much) longer benchmarks
  • for ones that aim for measurement count, e.g. we want to have 200 samples
• Less precise benchmarks
  • for ones that aim for duration, e.g. running for 10 minutes
  • because you will get less samples
• Less accurate benchmarks
  • because you’re not measuring what user experiencies
  • As a result system under benchmark might actually meet some acceptance criteria, while measurements show it’s not
• Disturbs calculating correlations and figuring out frontend latency causation

Side effects of not waiting for DOMContentLoaded or loaded event

User can click a visible button, that does not react on clicks yet. Which actually is a reality

However that complicates measurements. Frameworks do not check if a button has event listeners. It often requires polling for a desired button state. e.g. await condition click and check if expected message is visible

Summary

I’ve taken thousands of measurements of a few different sites, not only github. Systems behave the same. Waiting for browser events makes measured latencies inaccurate. It might be 10% away from real user experience, it might be 530%.

It’s non-zero, reproducible and deterministic. So if you already made an effort to make synthetic performance benchmarks, I’d recommend putting an effort also in increasing their accuracy.

Consider changing default options for browser automation

For playwright and selenium (and whatever browser automation tools there are) you should opt out from waiting for any browser events if you’re using those tools for performance measurements.

The post How to make performance benchmark 530% worse by relying on web app automation defaults? first appeared on Deployment Every Day.

First things first. What’s the goal of a UI measurement?

Who consumes UI? Humans from planet Earth, of course. They get annoyed, frustrated and want to throw away your website out of the window when it’s too slow. They also think very neat things about your product when they are waiting too long.

So the ultimate goal of measuring UI is to know if/what parts of the system should be improved in order to keep users happy.

What question should performance pass / fail criteria answer?

It should answer the question: “Are users happy using my product?“.

Why happy user is that important? And what are consequences of having frustrated users because of poor performance?

You’d better keep users happy by keeping your site fast. Not only you users suffer. Your business suffers. The longer you’re making users to wait, the more

• bounce rate goes up
• you’re losing traffic from organic search as search engine prefers faster sites
  • It’s a real deal. I already spoke to people having this particular problem. Google changed the rules (web vitals) and whole team needed to work on performance to bring back lost organic traffic
• and for systems that companies use internally daily
  • users will start to look for alternative solutions that frustrate them less
  • or have a grumpy-cat face all day

When system passes acceptance criteria it means particular part of your system is fast enough. It means there aren’t too many frustrated users. Product does not need to be fixed in this area.

When system does not pass acceptance criteria it means it’s too slow. Which means there are too many frustrated users. Product needs to be fixed in this area.

What’s the perfect threshold for UI pass / fail acceptance criteria?

0 ms so no one waits for anything Jokes aside, let’s see what research says about latencies.

What research says about latency and user frustration

• https://hpbn.co/primer-on-web-performance/#speed-performance-and-human-perception
• Latency longer than 1s causes likely mental context switch. And bounce rate goes up
• And that might be our frustration point

Even if a particular part of your UI is far away from latency 1s, it might be a good, ambitious goal that you can track. Don’t lower the expectations just because system will not pass the criteria. You can always measure the gap to passing the criteria and progress on closing the gap.

Measuring user happiness and frustration is nothing new and it works well

There already are well thought concepts. For example

APDEX

• https://docs.newrelic.com/docs/apm/new-relic-apm/apdex/apdex-measure-user-satisfaction/
• User satisfaction based on T, below which user is satisfied
  • Satisfied, latency <0, T]
  • Tolerated, latency <T, 4T]
  • Frustrated, latency <4T, infinity) or error
• Score [0, 1]
  • 0 – all frustrated
  • 1 – all satisfied
  • (number of requests within satisfied range + (number of requests within tolerated range) / 2) / total number of requests

APDEX makes sense. However I will explain percentile rank as even simpler and as powerful concept.

You might have also heard about percentiles, so let’s compare percentiles and percentile rank in practical terms.

Percentiles. Usually P90, P95 or P99

What’s the idea?

Sort the measured latencies array ascending. P90 is index in this array at 90th %.

P90 = latencies[latencies.size * 0.9]

P95 = latencies[latencies.size * 0.95]

etc

Using high percentiles (P90-P99.9) you can find out the slowest latencies user expierence.

Calculating percentiles requires whole population (latencies sorted ascending). It’s not applicable for live monitoring of systems as you have to store every single measured latency.

Percentile rank

Statistical meaning for PR(n)

• How many % of other values are lower than n
• Llike on exam: how many % of other students you’ve beaten with your score

It doesn’t require whole population to calculate, you can count it even on stream. Just increment counter of the passed threshold(s).

Implementation is simple

val badCount = 0
val goodCount = 0
val threshold = 1000
latencies.forEach {
 if (it < threshold) goodCount++ else badCount++
}
val percentileRankPercent = goodCount * 100.0 / (badCount + goodCount)

Why not just use percentiles and P90?

P90 means what's the value at 90%th index of sorted array?

PR(1s) means how many latencies are below 1s?. Not below or equal and that makes the difference.

P90 = 1s doesn’t mean PR(1s) = 90%

Take a look at the example. Given following, sorted array of latencies [s], 10 elements for simplicity

• 0.25, 0.3, 0.6, 0.6, 0.7, 0.8, 0.85, 0.9, 1.0, 1.4
  • P90 = 1.0
  • PR(1) = 80%
  • PR(1.1) = 90%
  • PR(1.5) = 100%

Let’s change the example. Latencies got worse, but P90 stays the same

• 0.25, 0.3, 0.6, 0.6, 0.7, 1.0, 1.0, 1.0, 1.0, 1.4
  • P90 = 1.0
  • PR(1) = 50%
• As you can see, a single percentile does not say how many samples

Let’s change the example. Latencies got even worse, P90 stays the same

• 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.4
  • P90 = 1.0
  • PR(1) = 0%

As you can see, relying on a single percentile brings a problem. For P90 anything between [0, 90%] of latencies might be as bad as P90.

Percentiles are not designed to answer question how good/bad this latency is?

Let’s construct the pass / fail acceptance criteria based on percentile rank

Based on previously mentioned research and 1s as a user frustration point, let’s construct the acceptance criteria.

PR(1s) is expected to be at least 99%

• Which means “At least 99% of website interactions user should experience latency below 1s“
• Which also means “User can be frustrated by max 1% of website interactions“

What if after a benchmark, criteria is NOT met?

e.g. PR(1s) = 80% and is expected to be at least 99%

It means 19% of the responses are too slow. 19% of interactions needs to be fixed.

How to measure improvement if you start with PR(1s) = 0%

You might be in a situation that all latencies are >= 1s. 100% frustrated user. You want to improve the product by chaning something.

Before change

• PR(1s) = 0%
• All measured latencies are be above 2s

After change, new commits added to the product

• PR(1s) = 0%
• All of measured latencies are above 1.5s. It is an improvement, but percentile rank hasn’t changed.

And you can’t just say “this improvement doesn’t matter, user is still frustrated because criteria measuring user’s frustration is till unmet“. Some improvements require a lot of iterative work and you need a devloop in which you notice the improvement. Even if it’s still far away from having happy user.

To compare if there is an improvement you could use comparison benchmark with Hodges-Lehmann estimator.

• That requires access to every single latency from baseline and experiment

You could also use a simple measure which I call ‘wasted user time’ which you want to keep at 0.

val thresholdMillis = 1000
val allowedDurationMillis = latencies.size * thresholdMillis
val totalDurationMillis = latencies.sumOf { it.toMillis() }
val excessiveDurationMillis = totalDurationMillis - allowedDurationMillis
val wastedUserTimePercent = excessiveDurationMillis * 100.0 / totalDurationMillis

This way even if percentile rank hasn’t changed you can still measure the improvement.

And that’s the exact methodology I’m using to measure webpages performance and answer questions “how to improve the system?”

The post How to use percentile rank to measure performance first appeared on Deployment Every Day.