Let’s compare accuracy of Page.waitForCondition with custom condition polling
For measurements I’m going to use 2 different approaches to see which one is better for performance benchmarks and has better accuracy.
- first using playwright
Page.waitForCondition - and the second using custom
condition polling
Custom condition polling implementation
// part of performance-explainer project I'm developing to measure things
package com.performance.explainer.api.waiter
import com.performance.explainer.spi.waiter.ConditionPolling
import java.time.Duration
import java.time.Instant
import java.util.function.BooleanSupplier
class ConditionPolling : SpiConditionPolling {
override fun await(
retryWait: Duration,
timeout: Duration,
condition: BooleanSupplier
): Boolean {
val start = Instant.now()
while (true) {
val satisfied = condition.asBoolean
val elapsed = Duration.between(start, Instant.now())
if (satisfied) {
return true
}
if (elapsed.plus(retryWait) > timeout) {
return false
}
Thread.sleep(retryWait)
}
}
}Website 1: https://www.theathletesfoot.nl/
Measurement ends when button starts working. Starts working means, it’s reacting on clicks. At this particular website it means validation error shows up.
That unfortunately cannot be checked with locator and .click() as button is not disabled and it’s clickable before it has event listeners attached. That could be fixed with waiting for loaded event, but that’s a bad idea for performance benchmark as explained with details in this blog post. So below code is NOT working for checking if button is working. Take a look at the explanation in source code
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer
Consumer<Page> { page ->
page.navigate(
"https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
NavigateOptions().setWaitUntil(COMMIT)
)
val button = page.locator("#add-to-cart")
button.waitFor(WaitForOptions().setTimeout(3000.0))
// plawright waits for a clickable button.
// Clickable means it's visible and not disabled
// This button is visible and not disabled before all resources are loaded.
// Because of that, it does not actually measure user experience (button is working)
button.click()
}
// measured latency is duration of above
baseline 1 – wait for working button, use custom polling
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer
Consumer<Page> { page ->
val conditionPolling = NonBlockingConditionPolling.default()
page.navigate(
"https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
NavigateOptions().setWaitUntil(COMMIT)
)
val button = page.locator("#add-to-cart")
button.waitFor(WaitForOptions().setTimeout(3000.0))
val success = conditionPolling.await(
retryWait = ofMillis(10),
timeout = ofMillis(5000)
) {
button.click()
val formError = page.locator(".js-select-size-and-color-error")
formError.count() == 1
}
if (!success) {
throw TimeoutError("Waiting for working button timed out")
}
}
// measured latency is duration of aboveexperiment 1 – wait for working button, use Page.waitForCondition
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer
Consumer<Page> { page ->
page.navigate(
"https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
NavigateOptions().setWaitUntil(COMMIT)
)
val button = page.locator("#add-to-cart")
button.waitFor(WaitForOptions().setTimeout(3000.0))
page.waitForCondition({
button.click()
val formError = page.locator(".js-select-size-and-color-error")
formError.count() == 1
}, Page.WaitForConditionOptions().setTimeout(5000.0))
}
// measured latency is duration of above
Website 2: https://www.blogmarketingacademy.com/
baseline 2 – wait for working button, use custom polling
import com.microsoft.playwright.*
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer
Consumer<Page> { page ->
page.navigate(
"https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
NavigateOptions().setWaitUntil(COMMIT)
)
val button = page.getByText("Subscribe Weekly")
button.waitFor(WaitForOptions().setTimeout(3000.0))
val condition = NonBlockingConditionPolling.default()
val isButtonWorking = condition.await(retryWait = ofMillis(10), timeout = ofMillis(3000)) {
button.click()
page.getByText("This field is required").count() == 1
}
if (!isButtonWorking) {
throw TimeoutError("Waiting for working button timed out")
}
}
// measured latency is duration of aboveexperiment 2 – wait for working button, use Page.waitForCondition
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer
Consumer<Page> { page ->
page.navigate(
"https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
NavigateOptions().setWaitUntil(COMMIT)
)
val button = page.getByText("Subscribe Weekly")
button.waitFor(WaitForOptions().setTimeout(3000.0))
page.waitForCondition({
button.click()
page.getByText("This field is required").count() == 1
}, Page.WaitForConditionOptions().setTimeout(3000.0))
}
// measured latency is duration of aboveCustom polling VS Page.waitForCondition results
Each baseline and experiment has 200 samples taken. Hodges-Lehmann estimator is used for comparing results
baseline 1 VS experiment 1
Page.waitForCondition 7.958% worse than custom polling
Page.waitForCondition 14.182% worse than custom condition polling when TTF (time to first byte, irrelevant for experiment) deducted from measured latenciesbaseline 2 VS experiment 2
Page.waitForCondition 2.665% worse than custom condition polling. Additionally timeouts went up from 3.5% to 6%
Page.waitForCondition17.427% worse than custom condition polling when TTF (time to first byte, irrelevant for experiment) deducted from measured latenciesLet’s increase custom condition polling retryWait and compare results again
Results should be closer to Page.waitForCondition, right? Right?
Intuitively with increased wait before another condition poll, results should get less accurate. And this way get closer to less accurate playwright’s Page.waitForCondition. Right?
Nope, in reality increasing retryWait up to 40ms makes no difference.
Custom polling with retryWait 10 ms VS longer retryWait
10 ms VS 20 ms – no difference
10 ms VS 40 ms – no difference
10 ms VS 80 ms – 2.644% worse
So there is difference noticeable in measurements when increasing polling retry wait from 10 ms to 80 ms. Sweet spot is 40 ms
Why is there a difference between custom condition polling and Page.waitForCondition?
Let’s take a look at playwright’s implementation for Page.waitForCondition
// com.microsoft.playwright.Page#waitForCondition(java.util.function.BooleanSupplier)
@Override
public void waitForCondition(BooleanSupplier predicate, WaitForConditionOptions options) {
List<Waitable<Void>> waitables = new ArrayList<>();
waitables.add(createWaitForCloseHelper());
waitables.add(createWaitableTimeout(options == null ? null : options.timeout));
waitables.add(new WaitablePredicate<>(predicate));
runUntil(() -> {}, new WaitableRace<>(waitables));
}
// com.microsoft.playwright.impl.ChannelOwner#runUntil
<T> T runUntil(Runnable code, Waitable<T> waitable) {
try {
code.run();
while (!waitable.isDone()) {
connection.processOneMessage();
}
return waitable.get();
} finally {
waitable.dispose();
}
}
// com.microsoft.playwright.impl.Connection#processOneMessage
void processOneMessage() {
JsonObject message = transport.poll(Duration.ofMillis(10));
if (message == null) {
return;
}
Gson gson = gson();
Message messageObj = gson.fromJson(message, Message.class);
dispatch(messageObj);
}Under the hood in java implementation, playwright uses condition polling every 10 ms. The same interval which was used in custom polling. So why it has worse accuracy than custom condition polling? Source code at this level does not answer the question and that might be a question to playwright dev team.
How about Page.locator VS Page.isVisible polling?
After I saw it makes a difference when polling is used for more complex scenarios I thought “let’s also checked if it there is difference for plain locators like Page.getByText.
baseline
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer
Consumer<Page> { page ->
page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
page.getByText("Sign up for GitHub").first().waitFor(WaitForOptions().setTimeout(2000.0))
}experiment
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.time.Duration.ofSeconds
import java.util.function.Consumer
Consumer<Page> { page ->
val conditionPolling = NonBlockingConditionPolling.default()
page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
val isButtonVisible = conditionPolling.await(retryWait = ofMillis(40), timeout = ofSeconds(2)) {
page.isVisible("text=Sign up for GitHub")
}
if (!isButtonVisible) {
throw TimeoutError("Button not visible")
}
}
Page.getByText vs custom condition pollingSummary
I started with checking if playwright Page.waitForCondition replaced by custom condition polling every 10ms makes a difference. And it’s faster which means you get more accurate results with this approach.
Then I also measured 20ms, 40ms, 80ms custom condition polling interval. Up to 40ms there is no difference in measured latencies and it’s a sweet spot for retries between condition check.
Conclusion
If you want higher accuracy of web app performance measurement, use custom condition polling 40ms instead of playwright Page.waitForCondition
