The post Is (website) performance improved after improving percentile 90? first appeared on Deployment Every Day.

Following tools are used to verify website performance and get fix suggestions

• https://pagespeed.web.dev/
  • tool that says what potential performance problems a website has
• and my own performance-explainer which aims to explain
  • what exactly needs to fixed
  • what the ROI (return of investment) of each fix is – to focus on fixes giving the biggest improvement

I’m going to use those tools to see

• What the user perceived performance is
• How to fix performance

Let’s start with website 1

PageSpeed insights website 1

Let’s see what a popular tool says about performance of website https://donorperfect.com. That’s just a sample website that was sent to me during early phase of developing performance explainer to check what is says about performance of this website.

It says overall performance is poor. FCP, which is our focus in this blog post, is scored medium.

Let’s take a look at FCP improvement suggestions from PageSpeed insights.

Problem 1: There are render blocking resources slowing down FCP

It says potential saving is 1,050 ms. That would be a huge FCP boost and much better use experience when fixes applied!

So as a developer you might think: if I or my team puts an effort to fix blocking resources, performance will be fixed. Right? More on that later

Problem 2: big images

Ok. Looks easy. If images are compressed more, those should be sent faster and should speed up FCP. Right?

At this point a developer trying to improve a website might think:

• It’s clear how to fix performance
• And a there might be a shy thought When I apply the fix, how do I know it really improved FCP?
  • PageSpeed insights does not aswer this question before the fix.

Is the performance tool job done? Well, if the goal is to actually improve performance, that’s merely a beginning of the job. After the fix developer needs to know if performance was really fixed. That requires measuring the system again. The tool needs to be a part of development cycle.

Hold on. Let’s use much more time effective approach. performance explainer in action

First: take many samples and see what the performance really is.

Let’s apply acceptance criteria: For 99% of visits, FCP should be below 1s.

Is it achieved?

Unfortunately not. Within 150 samples, FCP below 1s was never experienced.

Let’s take a look at FCP across all samples on a percentile chart. What’s the latency user experiences?

Now you can see the big picture what’s going on at this website. 2.3s as suggested by PageSpeed insights never happened. Maybe PageSpeed insights analyzed samples when that indeed happened?

Let’s apply fixes suggested by PageSpeed insights with a no-code experiments and see if FCP is fixed.

performance-explainer can make a no code experiments in the real browser to answer question: “how much boosting particular requests impacts the measurement?“. In this case – How much will it improve FCP?

So let’s pick requests too boost.

Boost render blocking resources suggested by PageSpeed to simulate the fix without any coding

What if I speed up those requests to a possible maximum? For example, if original request in the browser takes 400ms, it’s possible to cut down the latency in experiment to ~50ms. This way you can see what would happen if you changed the production system.

Ok, go!

A few minutes later the result is ready. 400 baseline measurements (original website) compared to 400 experiment measurements (selected resourced served to browser almost instantly).

What’s the result? Does implementing PageSpeed insights suggestions would fix FCP? No. Developers would work on something that doesn’t improve it

If you speed up those requests by 75-88% those will have no impact on FCP. In production it would mean either actually sending them faster or compressing. Neither would help fixing FCP.

Depending on the product, company and the developers team – fixing performance might take anything from a few hours to a few weeks. So in case of a false signal, a lot of time and money could be wasted.

Another PageSpeed insights hint is to fix images size. Let’s make a no-code experiment and boost sending images

Wha’t the result of boosting images that are supposed to improve FCP?

Note images are boosted almost 100%. That means those are almost instantly available for the browser yet it would have no impact on FCP.

Devs would again work on something not improving FCP!

So what will actually improve FCP?

This particular website makes ~120 requests. Which one impacts FCP?

It’s possible to tell with a static analysis. But which one has impact strong enough to make it meaningful and worth fixing?

It’s impossible to tell without statistical analysis based on taking samples from real system. Which means using browser, just like user does.

Some of requests are not even ever started before FCP across all samples – that means there are requests very likely to be irrelevant for measurement. But you can’t tell that doing 1 or a few checks. If you take hundreds of samples, you can start making conclusions how tested system really behaves.

“PageSpeed shows suggestions that do not help fixing FCP. What to do now? “

Request boos exploration for the rescue!

I’ve worked really hard to implement request boost exploration in performance explainer.

The methodology is the following:

• For every single request browser makes at given website
• In the real browser simulate request boost up from 10 to 100% (incrementally)
• Take hundreds of samples and apply statistics to see what’s the noise and what’s the true impact
  • It’s not using production system, everything happens in the browsers with requests fulfilled with controlled latencies
• Tell if it’s worth speeding up or not

Instead of guessing what to do, let the tooling figure out what’s worth fixing. It might take some time, but the results are accurate.

The lower the measurement latency impact, the better.

Are those easy fixes? Myabe it’s just a matter of deferring the js resource or making it async. I don’t know. Latency fixes are often difficult. That’s something that now developers can figure out.

Huge part of the job is already done – knowing what to improve. The more first request from above table is sped up, the better FCP is. And that’s where the biggest ROI for improving FCP is. You could also speed up main site, but the ROI there is smaller.

Main focus of this blog post is FCP, but let’s take at LCP (Largest Contentful Paint) also. How to improve it?

LCP has a different bottleneck. It’s an image, with maximum possible improvement opportunity 65%. What improves LCP might not improve FCP (as in this case). Because it ‘s a different goal.

Note that there is better LCP when image is sped up 80% than 99%. Why? It sounds counterintuitive.

Because there are request races in the browser.

// R1, R2, R3 - browser requests
// R1 is most likely to be a LCP bottleneck

R1 --------------------------------------------
R2 ------------------------------
R3 -------------------------------------

// after enough R1 boost LCP has no further improvements
// now R3 is most likely to be a LCP bottleneck
R1 ----------------------
R2 ------------------------------
R3 -------------------------------------

Systems don’t behave deterministic, that’s why hundreds of samples need to be taken. There is a lot of going on in the browser. If you cut off latency of a single request enough, you’re going to see the improvement and after that new bottleneck might be uncovered.

That’s a lo of knowledge, I’m glad you’re still here

Now let’s move on to

PageSpeed insights: website 2

I feel at the moment of writing my wordpress blog you’re just at is not fast. So let’s just measure it.

What does PageSpeed insight say?

It says 1.8s for FCP, 6.2s for LCP. It could be better.

However does it mean every user experiences FCP 1.8s? Or maybe that’s the worst case? Or the best?

How about fixes?

It points initial server response time could be better as it’s 1 140 ms

• Is it where most improvement can be made? I don’t know but it’s the biggest latency shown on the list.
• Other fixes show potential [kb] savings.
  • What does it mean in terms of latency? How much will it improve FCP? Is it worth investing my time to apply fixes there?

I value my time. I need data telling me what should be improved and in what order, not only what’s wrong without even knowing if it will help.

What does performance explainer FCP benchmark say?

It says user never experiences the desired FCP below or equal 1s.

It also says TTFB (Time To First Byte) is where 72% of time user is waiting for FCP and that it needs to be sped up 54.55% to meet acceptance criteria and have 99% latencies below 1s.

FCP takes from 1326 ms to 4807 ms across 150 measurements. Ouch. I want to give my readers a better experience.

FCP at 1.8s level shown in PageSpeed didn’t look that bad.

So let’s figure out ROIs using request boost exploration in performance explainer

The only fix worth investing is fixing TTFB. Not resources mentioned by PageSpeed insights.

I’m no wordpress expert, but I thought “How about caching the whole response to spend less time on producing HTML? There must be some plugins for caching“

So I installed WP Fastest Cache plugin, turned it on with default settings hoping to improve TTFB which is a bottleneck for FCP. Immediately I had questions in my mind.

• Did it improve performance?
• How would I know it?
• Would refreshing page in the browser give the feeling it’s better?
• What if it gave the feeling it’s better, but if I refreshed the page 10 more times it would actually give feeling it’s worse?

Let the data speak. And what does it say after taking samples and comparing it to samples before installing cache plugin?

FCP did not improve at all. Without measuring it after installing plugin and comparing to baseline (no plugin installed) I’d be pretty sure performance is better. While my readers would feel no difference at all.

Additional plugin would only add complexity to the system. And I would think it’s improving the situation. I removed the plugin immediately.

Now I have more ideas for speeding up TTFB.

• better hosting
• different caching plugin
• different skin
• … and there might be 10 more ideas

However that’s a typical trial and error fixing methodology when no data is available for showing bottleneck in more details.

It’s easy to produce ideas. It’s expensive to implement them. And it’s risky because it might fix nothing.

When you think a fix helped (while it didn’t), unnecessary complexity grows over time. It contributes to a growing production systems maintenance cost.

Since I know the bottleneck is PHP backend, what would really help is to have profiler samples. Or maybe some plugin showing what’s slow on wordpress backend if there is such one.

It would show where time is spent when request is made from the browser. In another words – data allowing understand better the root cause would appear.

If I knew PHP backend bottleneck, there would be no guessing what to fix. As a result

• much quicker fix delivery
• less time spent
• no unjustifiable complexity added for solutions that were supposed to fix performance, while fixing nothing

PageSpeed isights VS performance explainer summary

PageSpeed insights

• It shows improvement ideas. Those look good and worth spending time on
  • But in reality those fixes might not fix performance at all

performance explainer

• It’s showing the whole picture and how often user experiences good and bad performance.
  • Thanks to to hundreds of samples taken
• it does not present good sounding fixes ideas. It presents the ones that actually impact performance

Do you want to know if your website (on premise product too) is fast enough and having it fixed if needed?

Reach me at mikolaj.grzaslewicz@gmail.com if you want to have performance of your website fixed via consultations. My goal is to have performance explainer available as a service to support more customers.

The post No More Guesswork: Validating what improves First Contentful Paint upfront with No-Code Experiments first appeared on Deployment Every Day.

For measurements I’m going to use 2 different approaches to see which one is better for performance benchmarks and has better accuracy.

• first using playwright Page.waitForCondition
• and the second using custom condition polling

Custom condition polling implementation

// part of performance-explainer project I'm developing to measure things

package com.performance.explainer.api.waiter

import com.performance.explainer.spi.waiter.ConditionPolling
import java.time.Duration
import java.time.Instant
import java.util.function.BooleanSupplier


class ConditionPolling : SpiConditionPolling {

    override fun await(
        retryWait: Duration,
        timeout: Duration,
        condition: BooleanSupplier
    ): Boolean {
        val start = Instant.now()
        while (true) {
            val satisfied = condition.asBoolean
            val elapsed = Duration.between(start, Instant.now())
            if (satisfied) {
                return true
            }
            if (elapsed.plus(retryWait) > timeout) {
                return false
            }
            Thread.sleep(retryWait)
        }
    }

}

Website 1: https://www.theathletesfoot.nl/

Measurement ends when button starts working. Starts working means, it’s reacting on clicks. At this particular website it means validation error shows up.

That unfortunately cannot be checked with locator and .click() as button is not disabled and it’s clickable before it has event listeners attached. That could be fixed with waiting for loaded event, but that’s a bad idea for performance benchmark as explained with details in this blog post. So below code is NOT working for checking if button is working. Take a look at the explanation in source code

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
    page.navigate(
        "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
        NavigateOptions().setWaitUntil(COMMIT)
    )
    val button = page.locator("#add-to-cart")
    button.waitFor(WaitForOptions().setTimeout(3000.0))
    // plawright waits for a clickable button.
    // Clickable means it's visible and not disabled
    // This button is visible and not disabled before all resources are loaded.
    // Because of that, it does not actually measure user experience (button is working) 
    button.click()
}
// measured latency is duration of above

baseline 1 – wait for working button, use custom polling

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer

Consumer<Page> { page ->
   val conditionPolling = NonBlockingConditionPolling.default()
   page.navigate(
       "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.locator("#add-to-cart")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   val success = conditionPolling.await(
       retryWait = ofMillis(10),
       timeout = ofMillis(5000)
   ) {
       button.click()
       val formError = page.locator(".js-select-size-and-color-error")
       formError.count() == 1
   }
   if (!success) {
       throw TimeoutError("Waiting for working button timed out")
   }
}
// measured latency is duration of above

experiment 1 – wait for working button, use Page.waitForCondition

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.locator("#add-to-cart")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   page.waitForCondition({
       button.click()
       val formError = page.locator(".js-select-size-and-color-error")
       formError.count() == 1
   }, Page.WaitForConditionOptions().setTimeout(5000.0))
}
// measured latency is duration of above

Website 2: https://www.blogmarketingacademy.com/

baseline 2 – wait for working button, use custom polling

import com.microsoft.playwright.*
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.getByText("Subscribe Weekly")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   val condition = NonBlockingConditionPolling.default()
   val isButtonWorking = condition.await(retryWait = ofMillis(10), timeout = ofMillis(3000)) {
       button.click()
       page.getByText("This field is required").count() == 1
   }
   if (!isButtonWorking) {
       throw TimeoutError("Waiting for working button timed out")
   }
}
// measured latency is duration of above

experiment 2 – wait for working button, use Page.waitForCondition

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.getByText("Subscribe Weekly")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   page.waitForCondition({
       button.click()
       page.getByText("This field is required").count() == 1
   }, Page.WaitForConditionOptions().setTimeout(3000.0))
}
// measured latency is duration of above

Custom polling VS Page.waitForCondition results

Each baseline and experiment has 200 samples taken. Hodges-Lehmann estimator is used for comparing results

baseline 1 VS experiment 1

baseline 2 VS experiment 2

Let’s increase custom condition polling retryWait and compare results again

Results should be closer to Page.waitForCondition, right? Right?

Intuitively with increased wait before another condition poll, results should get less accurate. And this way get closer to less accurate playwright’s Page.waitForCondition. Right?

Nope, in reality increasing retryWait up to 40ms makes no difference.

Custom polling with retryWait 10 ms VS longer retryWait

10 ms VS 20 ms – no difference

10 ms VS 40 ms – no difference

10 ms VS 80 ms – 2.644% worse

So there is difference noticeable in measurements when increasing polling retry wait from 10 ms to 80 ms. Sweet spot is 40 ms

Why is there a difference between custom condition polling and Page.waitForCondition?

Let’s take a look at playwright’s implementation for Page.waitForCondition

// com.microsoft.playwright.Page#waitForCondition(java.util.function.BooleanSupplier)
 @Override
 public void waitForCondition(BooleanSupplier predicate, WaitForConditionOptions options) {
   List<Waitable<Void>> waitables = new ArrayList<>();
   waitables.add(createWaitForCloseHelper());
   waitables.add(createWaitableTimeout(options == null ? null : options.timeout));
   waitables.add(new WaitablePredicate<>(predicate));
   runUntil(() -> {}, new WaitableRace<>(waitables));
 }

 // com.microsoft.playwright.impl.ChannelOwner#runUntil
 <T> T runUntil(Runnable code, Waitable<T> waitable) {
   try {
     code.run();
     while (!waitable.isDone()) {
       connection.processOneMessage();
     }
     return waitable.get();
   } finally {
     waitable.dispose();
   }
 }

 // com.microsoft.playwright.impl.Connection#processOneMessage
 void processOneMessage() {
   JsonObject message = transport.poll(Duration.ofMillis(10));
   if (message == null) {
     return;
   }
   Gson gson = gson();
   Message messageObj = gson.fromJson(message, Message.class);
   dispatch(messageObj);
 }

Under the hood in java implementation, playwright uses condition polling every 10 ms. The same interval which was used in custom polling. So why it has worse accuracy than custom condition polling? Source code at this level does not answer the question and that might be a question to playwright dev team.

How about Page.locator VS Page.isVisible polling?

After I saw it makes a difference when polling is used for more complex scenarios I thought “let’s also checked if it there is difference for plain locators like Page.getByText.

baseline

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
   page.getByText("Sign up for GitHub").first().waitFor(WaitForOptions().setTimeout(2000.0))
}

experiment

import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.time.Duration.ofSeconds
import java.util.function.Consumer

Consumer<Page> { page ->
   val conditionPolling = NonBlockingConditionPolling.default()
   page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
   val isButtonVisible = conditionPolling.await(retryWait = ofMillis(40), timeout = ofSeconds(2)) {
       page.isVisible("text=Sign up for GitHub")
   }
   if (!isButtonVisible) {
       throw TimeoutError("Button not visible")
   }
}

Summary

I started with checking if playwright Page.waitForCondition replaced by custom condition polling every 10ms makes a difference. And it’s faster which means you get more accurate results with this approach.

Then I also measured 20ms, 40ms, 80ms custom condition polling interval. Up to 40ms there is no difference in measured latencies and it’s a sweet spot for retries between condition check.

Conclusion

If you want higher accuracy of web app performance measurement, use custom condition polling 40ms instead of playwright Page.waitForCondition

The post Want to have better accuracy of performance benchmark? Stop using playwright Page.waitForCondition first appeared on Deployment Every Day.

DOMContentLoaded

According to docs when browser fires this event it means

• DOM has been completely parsed
• <script defer src="…"> and <script type="module"> has been loaded
• async script, images, frames might not be loaded yet

load event

When browser fires load event it means DOM content is fully loaded. Browser has finished

• executing all asynchronous resources from the DOM tree
• loading images, frames etc.
• it comes always after DOMContentLoaded

Both events are NOT about REST calls. And both are not relevant to the user browsing the website.

What do web apps automation tools like playwright and selenium do by default?

Take a look at following lines of code measuring navigating to a page and waiting for a button to be visible.

using playwright

val millisBefore = System.currentTimeMillis()
page.navigate("https://github.com")
page.getByText("Try Now").waitFor()
val latency = System.currentTimeMillis() - millisBefore

or doing the same with selenium

val millisBefore = System.currentTimeMillis()
driver.navigate().to("https://github.com")
WebDriverWait(driver, 30, 100).until(ExpectedConditions.visibilityOfElementLocated(By.linkText("Sign up for GitHub")))
val latency = System.currentTimeMillis() - millisBefore

When you navigate to a page, by default those automation tools wait for load event. So the line with locator is not executed until then.

You might think Pff, so what. It’s just a technical detail

It is indeed a technical detail but it impacts performance benchmark dramatically.

Waiting for DOMContentLoaded VS load VS NOT waiting for any event. Fight!

Let’s measure how quickly user can see github call to action button.

Baseline benchmark – wait for load event

Take 1000 samples. Wait for load event, then wait for github call to action button

// only most important code shown here. Processing measurements is out of the scope
page.navigate("https://github.com")
page.getByText("Sign up for GitHub").first().waitFor(WaitForOptions().setTimeout(2000.0))

It looks like this

How to interpret the video of web app measurement?

Playwright video recording is imperfect. Video frame is often not aligned with what was seen in web browser. I confirmed that with modifying the measured page after measurement end.

During recording web app, right after measurement end timer is added to the page. This way it doesn’t interfere with measurement and helps to see when measurement has actually finished on the video.

const measuerementEndMillis = ..; // depends on the measuerement
const timerDiv = document.createElement('div');
timerDiv.style.position = 'fixed';
timerDiv.style.top = '0';
timerDiv.style.left = '0';
timerDiv.style.opacity = '0.8';
timerDiv.style.fontSize = '14px';
timerDiv.style.backgroundColor = 'yellow';
timerDiv.style.zIndex = '10000';
document.body.appendChild(timerDiv);

setInterval(function() {
  const now = performance.now();
  timerDiv.textContent = now.toFixed(0) + ' ms, ' + (now - measurementEndMillis).toFixed(0) + ' ms';
}, 20);

When measuring very fast page like github.com landing page, you’re often going to see a blank page and right after that frames recorded some time after measurement end (signup button visible). Frames are lost. Based on that you should treat every moment of playwright video as possibly inaccurate.

Having said that, let’s move on.

Experiment benchmark 1 – no waiting

Take 1000 samples. Don’t wait for any event, wait for github call to action button

import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT

// from playwright docs WaitUntilState.COMMIT:
// consider operation to be finished when network response is received and the document started loading
page.navigate("https://github.com", NavigateOptions().setWaitUntil(COMMIT))
page.locator("#summary-val").first().waitFor()

In selenium you have to opt out from waiting for DOMCompleteEvent this way

val options = new ChromeOptions()
options.setPageLoadStrategy(PageLoadStrategy.NONE)    
val driver = new ChromeDriver(options)
driver.navigate(..).to()

Experiment benchmark 2 – wait for DOMContentLoaded event

Take 1000 samples. Wait for DOMContentLoaded, wait wait for github call to action button

import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.DOMCONTENTLOADED

// from playwright docs WaitUntilState.DOMCONTENTLOADED:
// consider operation to be finished when the DOMContentLoaded event is fired.
page.navigate("https://github.com", NavigateOptions().setWaitUntil(DOMCONTENTLOADED))
page.locator("#summary-val").first().waitFor()

In selenium you have to opt out from waiting for DOMContentLoaded event this way

val options = new ChromeOptions()
options.setPageLoadStrategy(PageLoadStrategy.EAGER)    
val driver = new ChromeDriver(options)
driver.navigate(..).to()

And now let’s compare the results

P95 comparison

Percentiles do not answer the question how distribution compares? Down the line you’re going to see Hodges-Lehman estimator

What hypotheses are we verifying here?

• load event
  • (baseline) null hypothesis: waiting for load event has completion has no performance impact
  • (experiment 1) alternative hypothesis: waiting for loadevent has impact on measured latencies
• DOMContentLoaded event
  • (baseline) null hypothesis: waiting for DOMContentLoad event has has no impact on measured latencies
  • (experiment 2) alternative hypothesis: waiting for DOMContentLoadevent has impact on measured latencies

Waiting for load event VS not waiting for any event

Title of this blog post says there is 530% performance degradation when you rely on browser automation defaults. So where is it? When you reverse baseline VS experiment, there it is

Null hypothesis is rejected, waiting for load event comes with bigger latencies

Waiting for DOMContentLoaded VS not waiting for any event

Null hypothesis is rejected, waiting for DOMContentLoaded increases measured latencies

Waiting for DOMContentLoaded VS waiting for load event

Having 2 experiments we can compare them too. Since load event comes always after DOMContentLoaded it’s expected to see bigger latencies which is reflected in measurements

Based on measurements, what are undesired effects of waiting for DOMContentLoad or load event?

• To understand better following part take a look at difference between accuracy and precision

• The undesired effects
• Measuring something neither reflecting user experience nor relevant to user experience. User sees a button and clicks a button regardless of events in the browser
  • No user ever thought “I politely wish this site had dom completion faster“. User thinks “Why the f^%@ is this button not working yet?!!??“
• (much) longer benchmarks
  • for ones that aim for measurement count, e.g. we want to have 200 samples
• Less precise benchmarks
  • for ones that aim for duration, e.g. running for 10 minutes
  • because you will get less samples
• Less accurate benchmarks
  • because you’re not measuring what user experiencies
  • As a result system under benchmark might actually meet some acceptance criteria, while measurements show it’s not
• Disturbs calculating correlations and figuring out frontend latency causation

Side effects of not waiting for DOMContentLoaded or loaded event

User can click a visible button, that does not react on clicks yet. Which actually is a reality

However that complicates measurements. Frameworks do not check if a button has event listeners. It often requires polling for a desired button state. e.g. await condition click and check if expected message is visible

Summary

I’ve taken thousands of measurements of a few different sites, not only github. Systems behave the same. Waiting for browser events makes measured latencies inaccurate. It might be 10% away from real user experience, it might be 530%.

It’s non-zero, reproducible and deterministic. So if you already made an effort to make synthetic performance benchmarks, I’d recommend putting an effort also in increasing their accuracy.

Consider changing default options for browser automation

For playwright and selenium (and whatever browser automation tools there are) you should opt out from waiting for any browser events if you’re using those tools for performance measurements.

The post How to make performance benchmark 530% worse by relying on web app automation defaults? first appeared on Deployment Every Day.

First things first. What’s the goal of a UI measurement?

Who consumes UI? Humans from planet Earth, of course. They get annoyed, frustrated and want to throw away your website out of the window when it’s too slow. They also think very neat things about your product when they are waiting too long.

So the ultimate goal of measuring UI is to know if/what parts of the system should be improved in order to keep users happy.

What question should performance pass / fail criteria answer?

It should answer the question: “Are users happy using my product?“.

Why happy user is that important? And what are consequences of having frustrated users because of poor performance?

You’d better keep users happy by keeping your site fast. Not only you users suffer. Your business suffers. The longer you’re making users to wait, the more

• bounce rate goes up
• you’re losing traffic from organic search as search engine prefers faster sites
  • It’s a real deal. I already spoke to people having this particular problem. Google changed the rules (web vitals) and whole team needed to work on performance to bring back lost organic traffic
• and for systems that companies use internally daily
  • users will start to look for alternative solutions that frustrate them less
  • or have a grumpy-cat face all day

When system passes acceptance criteria it means particular part of your system is fast enough. It means there aren’t too many frustrated users. Product does not need to be fixed in this area.

When system does not pass acceptance criteria it means it’s too slow. Which means there are too many frustrated users. Product needs to be fixed in this area.

What’s the perfect threshold for UI pass / fail acceptance criteria?

0 ms so no one waits for anything Jokes aside, let’s see what research says about latencies.

What research says about latency and user frustration

• https://hpbn.co/primer-on-web-performance/#speed-performance-and-human-perception
• Latency longer than 1s causes likely mental context switch. And bounce rate goes up
• And that might be our frustration point

Even if a particular part of your UI is far away from latency 1s, it might be a good, ambitious goal that you can track. Don’t lower the expectations just because system will not pass the criteria. You can always measure the gap to passing the criteria and progress on closing the gap.

Measuring user happiness and frustration is nothing new and it works well

There already are well thought concepts. For example

APDEX

• https://docs.newrelic.com/docs/apm/new-relic-apm/apdex/apdex-measure-user-satisfaction/
• User satisfaction based on T, below which user is satisfied
  • Satisfied, latency <0, T]
  • Tolerated, latency <T, 4T]
  • Frustrated, latency <4T, infinity) or error
• Score [0, 1]
  • 0 – all frustrated
  • 1 – all satisfied
  • (number of requests within satisfied range + (number of requests within tolerated range) / 2) / total number of requests

APDEX makes sense. However I will explain percentile rank as even simpler and as powerful concept.

You might have also heard about percentiles, so let’s compare percentiles and percentile rank in practical terms.

Percentiles. Usually P90, P95 or P99

What’s the idea?

Sort the measured latencies array ascending. P90 is index in this array at 90th %.

P90 = latencies[latencies.size * 0.9]

P95 = latencies[latencies.size * 0.95]

etc

Using high percentiles (P90-P99.9) you can find out the slowest latencies user expierence.

Calculating percentiles requires whole population (latencies sorted ascending). It’s not applicable for live monitoring of systems as you have to store every single measured latency.

Percentile rank

Statistical meaning for PR(n)

• How many % of other values are lower than n
• Llike on exam: how many % of other students you’ve beaten with your score

It doesn’t require whole population to calculate, you can count it even on stream. Just increment counter of the passed threshold(s).

Implementation is simple

val badCount = 0
val goodCount = 0
val threshold = 1000
latencies.forEach {
 if (it < threshold) goodCount++ else badCount++
}
val percentileRankPercent = goodCount * 100.0 / (badCount + goodCount)

Why not just use percentiles and P90?

P90 means what's the value at 90%th index of sorted array?

PR(1s) means how many latencies are below 1s?. Not below or equal and that makes the difference.

P90 = 1s doesn’t mean PR(1s) = 90%

Take a look at the example. Given following, sorted array of latencies [s], 10 elements for simplicity

• 0.25, 0.3, 0.6, 0.6, 0.7, 0.8, 0.85, 0.9, 1.0, 1.4
  • P90 = 1.0
  • PR(1) = 80%
  • PR(1.1) = 90%
  • PR(1.5) = 100%

Let’s change the example. Latencies got worse, but P90 stays the same

• 0.25, 0.3, 0.6, 0.6, 0.7, 1.0, 1.0, 1.0, 1.0, 1.4
  • P90 = 1.0
  • PR(1) = 50%
• As you can see, a single percentile does not say how many samples

Let’s change the example. Latencies got even worse, P90 stays the same

• 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.4
  • P90 = 1.0
  • PR(1) = 0%

As you can see, relying on a single percentile brings a problem. For P90 anything between [0, 90%] of latencies might be as bad as P90.

Percentiles are not designed to answer question how good/bad this latency is?

Let’s construct the pass / fail acceptance criteria based on percentile rank

Based on previously mentioned research and 1s as a user frustration point, let’s construct the acceptance criteria.

PR(1s) is expected to be at least 99%

• Which means “At least 99% of website interactions user should experience latency below 1s“
• Which also means “User can be frustrated by max 1% of website interactions“

What if after a benchmark, criteria is NOT met?

e.g. PR(1s) = 80% and is expected to be at least 99%

It means 19% of the responses are too slow. 19% of interactions needs to be fixed.

How to measure improvement if you start with PR(1s) = 0%

You might be in a situation that all latencies are >= 1s. 100% frustrated user. You want to improve the product by chaning something.

Before change

• PR(1s) = 0%
• All measured latencies are be above 2s

After change, new commits added to the product

• PR(1s) = 0%
• All of measured latencies are above 1.5s. It is an improvement, but percentile rank hasn’t changed.

And you can’t just say “this improvement doesn’t matter, user is still frustrated because criteria measuring user’s frustration is till unmet“. Some improvements require a lot of iterative work and you need a devloop in which you notice the improvement. Even if it’s still far away from having happy user.

To compare if there is an improvement you could use comparison benchmark with Hodges-Lehmann estimator.

• That requires access to every single latency from baseline and experiment

You could also use a simple measure which I call ‘wasted user time’ which you want to keep at 0.

val thresholdMillis = 1000
val allowedDurationMillis = latencies.size * thresholdMillis
val totalDurationMillis = latencies.sumOf { it.toMillis() }
val excessiveDurationMillis = totalDurationMillis - allowedDurationMillis
val wastedUserTimePercent = excessiveDurationMillis * 100.0 / totalDurationMillis

This way even if percentile rank hasn’t changed you can still measure the improvement.

And that’s the exact methodology I’m using to measure webpages performance and answer questions “how to improve the system?”

The post How to use percentile rank to measure performance first appeared on Deployment Every Day.

I was trying to check what can be improved during working on a small project. As usual, I run intellij profiler from time to time to see the hottest methods. And that was part of my development workflow aside from TDD.

2x speedup – remove String.format() usage (technical optimization)

git commit

Remove usage of String.format("Move index %s from %s to %s", indexFrom, listFrom, listTo);

It’s not surprising it was slow, but discovering that is not obvious. Imagine you’re introduced to a new, big project – there is no way you would eyeball the code and say this is a bottleneck (hot method)

10x speedup – replace streams with for each (technical optimization).

git commit

Replace

return calculators.stream().mapToInt(c -> c.calculate(listA, listB)).sum();

With

  int sum = 0;
  for (var calculator : calculators) {
      sum += calculator.calculate(listA, listB);
  }
  return sum;

Bottleneck (hot method) found

2x speedup – add sum caching (technical optimization).

git commit

Replace

  private int sum(List<Integer> list) {
      int result = 0;
      for (var i : list) {
          result += i;
      }
      return result;
      return Math.abs(listA.sum() - listB.sum());
  }

With a sum caching java.util.List decorator

  public class SumCachingList implements SummingList {
      private final List<Integer> decorated;

      public SumCachingList(List<Integer> decorated) {
          this.decorated = decorated;
      }

      private int sum;
      private boolean sumCalculated = false;

      @Override
      public int sum() {
          if (!sumCalculated) {
              calculateSum();
              sumCalculated = true;
          }
          return sum;
      }

      private void calculateSum() {
          for (var i : decorated) {
              sum += i;
          }
      }

      @Override
      public boolean add(Integer integer) {
          sum = sum() + integer;
          return decorated.add(integer);
      }

      @Override
      public boolean remove(Object o) {
          var removed = decorated.remove(o);
          if (removed) {
              sum = sum() - (Integer) o;
          }
          return removed;
      }

     @Override
     public boolean removeIf(Predicate<? super Integer> filter) {
         var anyRemoved = decorated.removeIf(filter);
         if (anyRemoved) {
             sumCalculated = false;
         }
         return anyRemoved;
     }

      // ...
  }

After above optimizations flamegraph looks like that

Big picture optimisation VS technical optimisation

Ok, now you see that you can discover bottlenecks using async profiler recording. Above example is rather an easy one. For huge systems with tons of legacy code which is hard to maintain and change, it might be really hard.

Even discovering slow parts of big systems is hard, but that’s a topic for another blog post.

Technical optimizations are often easier to find.

Technical optimization

You don’t necessarily have to understand what the application is doing. You can read straight from the flamegraph that adding a cache or optimizing loops is going to help.

Big picture optimisation

You need to understand what your application is doing. You can optimize by organizing processing in a different way, e.g. discovering that you don’t have to fetch and process some data to in order to display particular piece of frontend.

Run integration tests with profiler regularly

That should be part of your development process. Like running integration tests.

In a perfect case, scenario of integration test should be as similar as possible to production scenario. This way you’d discover potential performance improvements or issues before production deployment.

Know the difference between async profiler sampling modes: Wall Clock and CPU usage only

Wall Clock (Total Time)

If you’re interested in real latency of your system, including

IO, e.g.

• connection polling
• DB transaction locks
• reads/writes

synchronization, e.g.

• waiting for critical section access
• waiting for tasks in a thread pool

use Wall Clock sampling in async-profiler. This mode will collect events from threads in SLEEPING state also.

Beware it affects performance of measured process more than measuring only ACTIVE threads. Why? Because measured JVM has more threads to iterate over every interval.

CPU only

In other words, without Wall Clock mode, measuring sampling only JVM threads in ACTIVE state

In the examples above you can see hot method httpClient.newCall(request).execute()

• Time spent measured with CPU Time is 120 ms
• Time spent measured with Wall Clock (Total Time) is 9170 ms

Would you optimize CPU time in this case? Probably not. You’d first focus on IO as it’s the bottleneck.

Subject to optimize needs to be chosen case by case. In order to have this choice, you need have both CPU Time and Total Time in flamegraph – and it’s available only when using wall clock mode sampling.

Resources

• Evaluating the Accuracy of Java Profilers
• Safepoint bias problem
  • https://jpbempel.github.io/2022/06/22/debug-non-safepoints.html
  • https://psy-lob-saw.blogspot.com/2015/12/safepoints.html
• https://github.com/mgrzaslewicz/set-equalizer

The post Example of 40x speedup using async-profiler recording analysis (JFR file) first appeared on Deployment Every Day.