For measurements I’m going to use 2 different approaches to see which one is better for performance benchmarks and has better accuracy.

• first using playwright Page.waitForCondition
• and the second using custom condition polling

Custom condition polling implementation

// part of performance-explainer project I'm developing to measure things

package com.performance.explainer.api.waiter

import com.performance.explainer.spi.waiter.ConditionPolling
import java.time.Duration
import java.time.Instant
import java.util.function.BooleanSupplier


class ConditionPolling : SpiConditionPolling {

    override fun await(
        retryWait: Duration,
        timeout: Duration,
        condition: BooleanSupplier
    ): Boolean {
        val start = Instant.now()
        while (true) {
            val satisfied = condition.asBoolean
            val elapsed = Duration.between(start, Instant.now())
            if (satisfied) {
                return true
            }
            if (elapsed.plus(retryWait) > timeout) {
                return false
            }
            Thread.sleep(retryWait)
        }
    }

}

Website 1: https://www.theathletesfoot.nl/

Measurement ends when button starts working. Starts working means, it’s reacting on clicks. At this particular website it means validation error shows up.

That unfortunately cannot be checked with locator and .click() as button is not disabled and it’s clickable before it has event listeners attached. That could be fixed with waiting for loaded event, but that’s a bad idea for performance benchmark as explained with details in this blog post. So below code is NOT working for checking if button is working. Take a look at the explanation in source code

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
    page.navigate(
        "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
        NavigateOptions().setWaitUntil(COMMIT)
    )
    val button = page.locator("#add-to-cart")
    button.waitFor(WaitForOptions().setTimeout(3000.0))
    // plawright waits for a clickable button.
    // Clickable means it's visible and not disabled
    // This button is visible and not disabled before all resources are loaded.
    // Because of that, it does not actually measure user experience (button is working) 
    button.click()
}
// measured latency is duration of above

baseline 1 – wait for working button, use custom polling

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer

Consumer<Page> { page ->
   val conditionPolling = NonBlockingConditionPolling.default()
   page.navigate(
       "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.locator("#add-to-cart")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   val success = conditionPolling.await(
       retryWait = ofMillis(10),
       timeout = ofMillis(5000)
   ) {
       button.click()
       val formError = page.locator(".js-select-size-and-color-error")
       formError.count() == 1
   }
   if (!success) {
       throw TimeoutError("Waiting for working button timed out")
   }
}
// measured latency is duration of above

experiment 1 – wait for working button, use Page.waitForCondition

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.locator("#add-to-cart")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   page.waitForCondition({
       button.click()
       val formError = page.locator(".js-select-size-and-color-error")
       formError.count() == 1
   }, Page.WaitForConditionOptions().setTimeout(5000.0))
}
// measured latency is duration of above

Website 2: https://www.blogmarketingacademy.com/

baseline 2 – wait for working button, use custom polling

import com.microsoft.playwright.*
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.getByText("Subscribe Weekly")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   val condition = NonBlockingConditionPolling.default()
   val isButtonWorking = condition.await(retryWait = ofMillis(10), timeout = ofMillis(3000)) {
       button.click()
       page.getByText("This field is required").count() == 1
   }
   if (!isButtonWorking) {
       throw TimeoutError("Waiting for working button timed out")
   }
}
// measured latency is duration of above

experiment 2 – wait for working button, use Page.waitForCondition

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.getByText("Subscribe Weekly")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   page.waitForCondition({
       button.click()
       page.getByText("This field is required").count() == 1
   }, Page.WaitForConditionOptions().setTimeout(3000.0))
}
// measured latency is duration of above

Custom polling VS Page.waitForCondition results

Each baseline and experiment has 200 samples taken. Hodges-Lehmann estimator is used for comparing results

baseline 1 VS experiment 1

baseline 2 VS experiment 2

Let’s increase custom condition polling retryWait and compare results again

Results should be closer to Page.waitForCondition, right? Right?

Intuitively with increased wait before another condition poll, results should get less accurate. And this way get closer to less accurate playwright’s Page.waitForCondition. Right?

Nope, in reality increasing retryWait up to 40ms makes no difference.

Custom polling with retryWait 10 ms VS longer retryWait

10 ms VS 20 ms – no difference

10 ms VS 40 ms – no difference

10 ms VS 80 ms – 2.644% worse

So there is difference noticeable in measurements when increasing polling retry wait from 10 ms to 80 ms. Sweet spot is 40 ms

Why is there a difference between custom condition polling and Page.waitForCondition?

Let’s take a look at playwright’s implementation for Page.waitForCondition

// com.microsoft.playwright.Page#waitForCondition(java.util.function.BooleanSupplier)
 @Override
 public void waitForCondition(BooleanSupplier predicate, WaitForConditionOptions options) {
   List<Waitable<Void>> waitables = new ArrayList<>();
   waitables.add(createWaitForCloseHelper());
   waitables.add(createWaitableTimeout(options == null ? null : options.timeout));
   waitables.add(new WaitablePredicate<>(predicate));
   runUntil(() -> {}, new WaitableRace<>(waitables));
 }

 // com.microsoft.playwright.impl.ChannelOwner#runUntil
 <T> T runUntil(Runnable code, Waitable<T> waitable) {
   try {
     code.run();
     while (!waitable.isDone()) {
       connection.processOneMessage();
     }
     return waitable.get();
   } finally {
     waitable.dispose();
   }
 }

 // com.microsoft.playwright.impl.Connection#processOneMessage
 void processOneMessage() {
   JsonObject message = transport.poll(Duration.ofMillis(10));
   if (message == null) {
     return;
   }
   Gson gson = gson();
   Message messageObj = gson.fromJson(message, Message.class);
   dispatch(messageObj);
 }

Under the hood in java implementation, playwright uses condition polling every 10 ms. The same interval which was used in custom polling. So why it has worse accuracy than custom condition polling? Source code at this level does not answer the question and that might be a question to playwright dev team.

How about Page.locator VS Page.isVisible polling?

After I saw it makes a difference when polling is used for more complex scenarios I thought “let’s also checked if it there is difference for plain locators like Page.getByText.

baseline

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
   page.getByText("Sign up for GitHub").first().waitFor(WaitForOptions().setTimeout(2000.0))
}

experiment

import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.time.Duration.ofSeconds
import java.util.function.Consumer

Consumer<Page> { page ->
   val conditionPolling = NonBlockingConditionPolling.default()
   page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
   val isButtonVisible = conditionPolling.await(retryWait = ofMillis(40), timeout = ofSeconds(2)) {
       page.isVisible("text=Sign up for GitHub")
   }
   if (!isButtonVisible) {
       throw TimeoutError("Button not visible")
   }
}

Summary

I started with checking if playwright Page.waitForCondition replaced by custom condition polling every 10ms makes a difference. And it’s faster which means you get more accurate results with this approach.

Then I also measured 20ms, 40ms, 80ms custom condition polling interval. Up to 40ms there is no difference in measured latencies and it’s a sweet spot for retries between condition check.

Conclusion

If you want higher accuracy of web app performance measurement, use custom condition polling 40ms instead of playwright Page.waitForCondition

The post Want to have better accuracy of performance benchmark? Stop using playwright Page.waitForCondition first appeared on Deployment Every Day.

DOMContentLoaded

According to docs when browser fires this event it means

• DOM has been completely parsed
• <script defer src="…"> and <script type="module"> has been loaded
• async script, images, frames might not be loaded yet

load event

When browser fires load event it means DOM content is fully loaded. Browser has finished

• executing all asynchronous resources from the DOM tree
• loading images, frames etc.
• it comes always after DOMContentLoaded

Both events are NOT about REST calls. And both are not relevant to the user browsing the website.

What do web apps automation tools like playwright and selenium do by default?

Take a look at following lines of code measuring navigating to a page and waiting for a button to be visible.

using playwright

val millisBefore = System.currentTimeMillis()
page.navigate("https://github.com")
page.getByText("Try Now").waitFor()
val latency = System.currentTimeMillis() - millisBefore

or doing the same with selenium

val millisBefore = System.currentTimeMillis()
driver.navigate().to("https://github.com")
WebDriverWait(driver, 30, 100).until(ExpectedConditions.visibilityOfElementLocated(By.linkText("Sign up for GitHub")))
val latency = System.currentTimeMillis() - millisBefore

When you navigate to a page, by default those automation tools wait for load event. So the line with locator is not executed until then.

You might think Pff, so what. It’s just a technical detail

It is indeed a technical detail but it impacts performance benchmark dramatically.

Waiting for DOMContentLoaded VS load VS NOT waiting for any event. Fight!

Let’s measure how quickly user can see github call to action button.

Baseline benchmark – wait for load event

Take 1000 samples. Wait for load event, then wait for github call to action button

// only most important code shown here. Processing measurements is out of the scope
page.navigate("https://github.com")
page.getByText("Sign up for GitHub").first().waitFor(WaitForOptions().setTimeout(2000.0))

It looks like this

How to interpret the video of web app measurement?

Playwright video recording is imperfect. Video frame is often not aligned with what was seen in web browser. I confirmed that with modifying the measured page after measurement end.

During recording web app, right after measurement end timer is added to the page. This way it doesn’t interfere with measurement and helps to see when measurement has actually finished on the video.

const measuerementEndMillis = ..; // depends on the measuerement
const timerDiv = document.createElement('div');
timerDiv.style.position = 'fixed';
timerDiv.style.top = '0';
timerDiv.style.left = '0';
timerDiv.style.opacity = '0.8';
timerDiv.style.fontSize = '14px';
timerDiv.style.backgroundColor = 'yellow';
timerDiv.style.zIndex = '10000';
document.body.appendChild(timerDiv);

setInterval(function() {
  const now = performance.now();
  timerDiv.textContent = now.toFixed(0) + ' ms, ' + (now - measurementEndMillis).toFixed(0) + ' ms';
}, 20);

When measuring very fast page like github.com landing page, you’re often going to see a blank page and right after that frames recorded some time after measurement end (signup button visible). Frames are lost. Based on that you should treat every moment of playwright video as possibly inaccurate.

Having said that, let’s move on.

Experiment benchmark 1 – no waiting

Take 1000 samples. Don’t wait for any event, wait for github call to action button

import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT

// from playwright docs WaitUntilState.COMMIT:
// consider operation to be finished when network response is received and the document started loading
page.navigate("https://github.com", NavigateOptions().setWaitUntil(COMMIT))
page.locator("#summary-val").first().waitFor()

In selenium you have to opt out from waiting for DOMCompleteEvent this way

val options = new ChromeOptions()
options.setPageLoadStrategy(PageLoadStrategy.NONE)    
val driver = new ChromeDriver(options)
driver.navigate(..).to()

Experiment benchmark 2 – wait for DOMContentLoaded event

Take 1000 samples. Wait for DOMContentLoaded, wait wait for github call to action button

import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.DOMCONTENTLOADED

// from playwright docs WaitUntilState.DOMCONTENTLOADED:
// consider operation to be finished when the DOMContentLoaded event is fired.
page.navigate("https://github.com", NavigateOptions().setWaitUntil(DOMCONTENTLOADED))
page.locator("#summary-val").first().waitFor()

In selenium you have to opt out from waiting for DOMContentLoaded event this way

val options = new ChromeOptions()
options.setPageLoadStrategy(PageLoadStrategy.EAGER)    
val driver = new ChromeDriver(options)
driver.navigate(..).to()

And now let’s compare the results

P95 comparison

Percentiles do not answer the question how distribution compares? Down the line you’re going to see Hodges-Lehman estimator

What hypotheses are we verifying here?

• load event
  • (baseline) null hypothesis: waiting for load event has completion has no performance impact
  • (experiment 1) alternative hypothesis: waiting for loadevent has impact on measured latencies
• DOMContentLoaded event
  • (baseline) null hypothesis: waiting for DOMContentLoad event has has no impact on measured latencies
  • (experiment 2) alternative hypothesis: waiting for DOMContentLoadevent has impact on measured latencies

Waiting for load event VS not waiting for any event

Title of this blog post says there is 530% performance degradation when you rely on browser automation defaults. So where is it? When you reverse baseline VS experiment, there it is

Null hypothesis is rejected, waiting for load event comes with bigger latencies

Waiting for DOMContentLoaded VS not waiting for any event

Null hypothesis is rejected, waiting for DOMContentLoaded increases measured latencies

Waiting for DOMContentLoaded VS waiting for load event

Having 2 experiments we can compare them too. Since load event comes always after DOMContentLoaded it’s expected to see bigger latencies which is reflected in measurements

Based on measurements, what are undesired effects of waiting for DOMContentLoad or load event?

• To understand better following part take a look at difference between accuracy and precision

• The undesired effects
• Measuring something neither reflecting user experience nor relevant to user experience. User sees a button and clicks a button regardless of events in the browser
  • No user ever thought “I politely wish this site had dom completion faster“. User thinks “Why the f^%@ is this button not working yet?!!??“
• (much) longer benchmarks
  • for ones that aim for measurement count, e.g. we want to have 200 samples
• Less precise benchmarks
  • for ones that aim for duration, e.g. running for 10 minutes
  • because you will get less samples
• Less accurate benchmarks
  • because you’re not measuring what user experiencies
  • As a result system under benchmark might actually meet some acceptance criteria, while measurements show it’s not
• Disturbs calculating correlations and figuring out frontend latency causation

Side effects of not waiting for DOMContentLoaded or loaded event

User can click a visible button, that does not react on clicks yet. Which actually is a reality

However that complicates measurements. Frameworks do not check if a button has event listeners. It often requires polling for a desired button state. e.g. await condition click and check if expected message is visible

Summary

I’ve taken thousands of measurements of a few different sites, not only github. Systems behave the same. Waiting for browser events makes measured latencies inaccurate. It might be 10% away from real user experience, it might be 530%.

It’s non-zero, reproducible and deterministic. So if you already made an effort to make synthetic performance benchmarks, I’d recommend putting an effort also in increasing their accuracy.

Consider changing default options for browser automation

For playwright and selenium (and whatever browser automation tools there are) you should opt out from waiting for any browser events if you’re using those tools for performance measurements.

The post How to make performance benchmark 530% worse by relying on web app automation defaults? first appeared on Deployment Every Day.