plawright - Deployment Every Day http://deploymenteveryday.com JVM (java/kotlin) performance, programming, frequent deployments, devops Mon, 13 May 2024 22:11:25 +0000 en-US hourly 1 https://wordpress.org/?v=7.1.3 Want to have better accuracy of performance benchmark? Stop using playwright Page.waitForCondition http://deploymenteveryday.com/want-to-have-better-accuracy-of-performance-benchmark-stop-using-playwright-page-waitforcondition/?utm_source=rss&utm_medium=rss&utm_campaign=want-to-have-better-accuracy-of-performance-benchmark-stop-using-playwright-page-waitforcondition http://deploymenteveryday.com/want-to-have-better-accuracy-of-performance-benchmark-stop-using-playwright-page-waitforcondition/#respond Mon, 13 May 2024 09:21:53 +0000 https://deploymenteveryday.com/?p=168 Let’s compare accuracy of Page.waitForCondition with custom condition polling For measurements I’m going to use 2 different approaches to see which one is better for performance benchmarks and has better accuracy. Custom condition polling implementation Website 1: https://www.theathletesfoot.nl/ Measurement ends when button starts working. Starts working means, it’s reacting on clicks. At this particular website […]

The post Want to have better accuracy of performance benchmark? Stop using playwright Page.waitForCondition first appeared on Deployment Every Day.

]]> Let’s compare accuracy of Page.waitForCondition with custom condition polling

For measurements I’m going to use 2 different approaches to see which one is better for performance benchmarks and has better accuracy.

  • first using playwright Page.waitForCondition
  • and the second using custom condition polling

Custom condition polling implementation

Kotlin
// part of performance-explainer project I'm developing to measure things

package com.performance.explainer.api.waiter

import com.performance.explainer.spi.waiter.ConditionPolling
import java.time.Duration
import java.time.Instant
import java.util.function.BooleanSupplier


class ConditionPolling : SpiConditionPolling {

    override fun await(
        retryWait: Duration,
        timeout: Duration,
        condition: BooleanSupplier
    ): Boolean {
        val start = Instant.now()
        while (true) {
            val satisfied = condition.asBoolean
            val elapsed = Duration.between(start, Instant.now())
            if (satisfied) {
                return true
            }
            if (elapsed.plus(retryWait) > timeout) {
                return false
            }
            Thread.sleep(retryWait)
        }
    }

}
NonBlockingConditionPolling

Website 1: https://www.theathletesfoot.nl/

Measurement ends when button starts working. Starts working means, it’s reacting on clicks. At this particular website it means validation error shows up.

That unfortunately cannot be checked with locator and .click() as button is not disabled and it’s clickable before it has event listeners attached. That could be fixed with waiting for loaded event, but that’s a bad idea for performance benchmark as explained with details in this blog post. So below code is NOT working for checking if button is working. Take a look at the explanation in source code

Kotlin
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
    page.navigate(
        "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
        NavigateOptions().setWaitUntil(COMMIT)
    )
    val button = page.locator("#add-to-cart")
    button.waitFor(WaitForOptions().setTimeout(3000.0))
    // plawright waits for a clickable button.
    // Clickable means it's visible and not disabled
    // This button is visible and not disabled before all resources are loaded.
    // Because of that, it does not actually measure user experience (button is working) 
    button.click()
}
// measured latency is duration of above
Broken measurement
Measurement target – wait for working button adding item to the shop cart

baseline 1 – wait for working button, use custom polling

Kotlin
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer

Consumer<Page> { page ->
   val conditionPolling = NonBlockingConditionPolling.default()
   page.navigate(
       "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.locator("#add-to-cart")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   val success = conditionPolling.await(
       retryWait = ofMillis(10),
       timeout = ofMillis(5000)
   ) {
       button.click()
       val formError = page.locator(".js-select-size-and-color-error")
       formError.count() == 1
   }
   if (!success) {
       throw TimeoutError("Waiting for working button timed out")
   }
}
// measured latency is duration of above
baseline 1 – wait for working button, use custom polling

experiment 1 – wait for working button, use Page.waitForCondition

Kotlin
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.locator("#add-to-cart")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   page.waitForCondition({
       button.click()
       val formError = page.locator(".js-select-size-and-color-error")
       formError.count() == 1
   }, Page.WaitForConditionOptions().setTimeout(5000.0))
}
// measured latency is duration of above
experiment 1 – wait for working button, use Page.waitForCondition

Website 2: https://www.blogmarketingacademy.com/

Measurement target – wait for working subscribe button
Measurement target – wait for working subscribe button

baseline 2 – wait for working button, use custom polling

Kotlin
import com.microsoft.playwright.*
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.getByText("Subscribe Weekly")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   val condition = NonBlockingConditionPolling.default()
   val isButtonWorking = condition.await(retryWait = ofMillis(10), timeout = ofMillis(3000)) {
       button.click()
       page.getByText("This field is required").count() == 1
   }
   if (!isButtonWorking) {
       throw TimeoutError("Waiting for working button timed out")
   }
}
// measured latency is duration of above
baseline 2 – wait for working button, use custom polling

experiment 2 – wait for working button, use Page.waitForCondition

Kotlin
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.getByText("Subscribe Weekly")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   page.waitForCondition({
       button.click()
       page.getByText("This field is required").count() == 1
   }, Page.WaitForConditionOptions().setTimeout(3000.0))
}
// measured latency is duration of above
experiment 2 – wait for working button, use Page.waitForCondition

Custom polling VS Page.waitForCondition results

Each baseline and experiment has 200 samples taken. Hodges-Lehmann estimator is used for comparing results

baseline 1 VS experiment 1

Page.waitForCondition 7.958% worse than custom polling

Page.waitForCondition 14.182% worse than custom condition polling when TTF (time to first byte, irrelevant for experiment) deducted from measured latencies

baseline 2 VS experiment 2

Page.waitForCondition 2.665% worse than custom condition polling. Additionally timeouts went up from 3.5% to 6%
Page.waitForCondition17.427% worse than custom condition polling when TTF (time to first byte, irrelevant for experiment) deducted from measured latencies

Let’s increase custom condition polling retryWait and compare results again

Results should be closer to Page.waitForCondition, right? Right?

Intuitively with increased wait before another condition poll, results should get less accurate. And this way get closer to less accurate playwright’s Page.waitForCondition. Right?

Nope, in reality increasing retryWait up to 40ms makes no difference.

Custom polling with retryWait 10 ms VS longer retryWait

10 ms VS 20 ms – no difference

10 ms VS 40 ms – no difference

10 ms VS 80 ms – 2.644% worse

So there is difference noticeable in measurements when increasing polling retry wait from 10 ms to 80 ms. Sweet spot is 40 ms

Why is there a difference between custom condition polling and Page.waitForCondition?

Let’s take a look at playwright’s implementation for Page.waitForCondition

Kotlin
// com.microsoft.playwright.Page#waitForCondition(java.util.function.BooleanSupplier)
 @Override
 public void waitForCondition(BooleanSupplier predicate, WaitForConditionOptions options) {
   List<Waitable<Void>> waitables = new ArrayList<>();
   waitables.add(createWaitForCloseHelper());
   waitables.add(createWaitableTimeout(options == null ? null : options.timeout));
   waitables.add(new WaitablePredicate<>(predicate));
   runUntil(() -> {}, new WaitableRace<>(waitables));
 }

 // com.microsoft.playwright.impl.ChannelOwner#runUntil
 <T> T runUntil(Runnable code, Waitable<T> waitable) {
   try {
     code.run();
     while (!waitable.isDone()) {
       connection.processOneMessage();
     }
     return waitable.get();
   } finally {
     waitable.dispose();
   }
 }

 // com.microsoft.playwright.impl.Connection#processOneMessage
 void processOneMessage() {
   JsonObject message = transport.poll(Duration.ofMillis(10));
   if (message == null) {
     return;
   }
   Gson gson = gson();
   Message messageObj = gson.fromJson(message, Message.class);
   dispatch(messageObj);
 }

Under the hood in java implementation, playwright uses condition polling every 10 ms. The same interval which was used in custom polling. So why it has worse accuracy than custom condition polling? Source code at this level does not answer the question and that might be a question to playwright dev team.

How about Page.locator VS Page.isVisible polling?

After I saw it makes a difference when polling is used for more complex scenarios I thought “let’s also checked if it there is difference for plain locators like Page.getByText.

baseline

Kotlin
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
   page.getByText("Sign up for GitHub").first().waitFor(WaitForOptions().setTimeout(2000.0))
}

experiment

Kotlin
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.time.Duration.ofSeconds
import java.util.function.Consumer

Consumer<Page> { page ->
   val conditionPolling = NonBlockingConditionPolling.default()
   page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
   val isButtonVisible = conditionPolling.await(retryWait = ofMillis(40), timeout = ofSeconds(2)) {
       page.isVisible("text=Sign up for GitHub")
   }
   if (!isButtonVisible) {
       throw TimeoutError("Button not visible")
   }
}
Uff, no difference between using Page.getByText vs custom condition polling

Summary

I started with checking if playwright Page.waitForCondition replaced by custom condition polling every 10ms makes a difference. And it’s faster which means you get more accurate results with this approach.

Then I also measured 20ms, 40ms, 80ms custom condition polling interval. Up to 40ms there is no difference in measured latencies and it’s a sweet spot for retries between condition check.

Conclusion

If you want higher accuracy of web app performance measurement, use custom condition polling 40ms instead of playwright Page.waitForCondition

The post Want to have better accuracy of performance benchmark? Stop using playwright Page.waitForCondition first appeared on Deployment Every Day.

]]> http://deploymenteveryday.com/want-to-have-better-accuracy-of-performance-benchmark-stop-using-playwright-page-waitforcondition/feed/ 0