The post How To Find Performance Bottleneck For Any Website first appeared on Deployment Every Day.

The post Is (website) performance improved after improving percentile 90? first appeared on Deployment Every Day.

Following tools are used to verify website performance and get fix suggestions

• https://pagespeed.web.dev/
  • tool that says what potential performance problems a website has
• and my own performance-explainer which aims to explain
  • what exactly needs to fixed
  • what the ROI (return of investment) of each fix is – to focus on fixes giving the biggest improvement

I’m going to use those tools to see

• What the user perceived performance is
• How to fix performance

Let’s start with website 1

PageSpeed insights website 1

Let’s see what a popular tool says about performance of website https://donorperfect.com. That’s just a sample website that was sent to me during early phase of developing performance explainer to check what is says about performance of this website.

It says overall performance is poor. FCP, which is our focus in this blog post, is scored medium.

Let’s take a look at FCP improvement suggestions from PageSpeed insights.

Problem 1: There are render blocking resources slowing down FCP

It says potential saving is 1,050 ms. That would be a huge FCP boost and much better use experience when fixes applied!

So as a developer you might think: if I or my team puts an effort to fix blocking resources, performance will be fixed. Right? More on that later

Problem 2: big images

Ok. Looks easy. If images are compressed more, those should be sent faster and should speed up FCP. Right?

At this point a developer trying to improve a website might think:

• It’s clear how to fix performance
• And a there might be a shy thought When I apply the fix, how do I know it really improved FCP?
  • PageSpeed insights does not aswer this question before the fix.

Is the performance tool job done? Well, if the goal is to actually improve performance, that’s merely a beginning of the job. After the fix developer needs to know if performance was really fixed. That requires measuring the system again. The tool needs to be a part of development cycle.

Hold on. Let’s use much more time effective approach. performance explainer in action

First: take many samples and see what the performance really is.

Let’s apply acceptance criteria: For 99% of visits, FCP should be below 1s.

Is it achieved?

Unfortunately not. Within 150 samples, FCP below 1s was never experienced.

Let’s take a look at FCP across all samples on a percentile chart. What’s the latency user experiences?

Now you can see the big picture what’s going on at this website. 2.3s as suggested by PageSpeed insights never happened. Maybe PageSpeed insights analyzed samples when that indeed happened?

Let’s apply fixes suggested by PageSpeed insights with a no-code experiments and see if FCP is fixed.

performance-explainer can make a no code experiments in the real browser to answer question: “how much boosting particular requests impacts the measurement?“. In this case – How much will it improve FCP?

So let’s pick requests too boost.

Boost render blocking resources suggested by PageSpeed to simulate the fix without any coding

What if I speed up those requests to a possible maximum? For example, if original request in the browser takes 400ms, it’s possible to cut down the latency in experiment to ~50ms. This way you can see what would happen if you changed the production system.

Ok, go!

A few minutes later the result is ready. 400 baseline measurements (original website) compared to 400 experiment measurements (selected resourced served to browser almost instantly).

What’s the result? Does implementing PageSpeed insights suggestions would fix FCP? No. Developers would work on something that doesn’t improve it

If you speed up those requests by 75-88% those will have no impact on FCP. In production it would mean either actually sending them faster or compressing. Neither would help fixing FCP.

Depending on the product, company and the developers team – fixing performance might take anything from a few hours to a few weeks. So in case of a false signal, a lot of time and money could be wasted.

Another PageSpeed insights hint is to fix images size. Let’s make a no-code experiment and boost sending images

Wha’t the result of boosting images that are supposed to improve FCP?

Note images are boosted almost 100%. That means those are almost instantly available for the browser yet it would have no impact on FCP.

Devs would again work on something not improving FCP!

So what will actually improve FCP?

This particular website makes ~120 requests. Which one impacts FCP?

It’s possible to tell with a static analysis. But which one has impact strong enough to make it meaningful and worth fixing?

It’s impossible to tell without statistical analysis based on taking samples from real system. Which means using browser, just like user does.

Some of requests are not even ever started before FCP across all samples – that means there are requests very likely to be irrelevant for measurement. But you can’t tell that doing 1 or a few checks. If you take hundreds of samples, you can start making conclusions how tested system really behaves.

“PageSpeed shows suggestions that do not help fixing FCP. What to do now? “

Request boos exploration for the rescue!

I’ve worked really hard to implement request boost exploration in performance explainer.

The methodology is the following:

• For every single request browser makes at given website
• In the real browser simulate request boost up from 10 to 100% (incrementally)
• Take hundreds of samples and apply statistics to see what’s the noise and what’s the true impact
  • It’s not using production system, everything happens in the browsers with requests fulfilled with controlled latencies
• Tell if it’s worth speeding up or not

Instead of guessing what to do, let the tooling figure out what’s worth fixing. It might take some time, but the results are accurate.

The lower the measurement latency impact, the better.

Are those easy fixes? Myabe it’s just a matter of deferring the js resource or making it async. I don’t know. Latency fixes are often difficult. That’s something that now developers can figure out.

Huge part of the job is already done – knowing what to improve. The more first request from above table is sped up, the better FCP is. And that’s where the biggest ROI for improving FCP is. You could also speed up main site, but the ROI there is smaller.

Main focus of this blog post is FCP, but let’s take at LCP (Largest Contentful Paint) also. How to improve it?

LCP has a different bottleneck. It’s an image, with maximum possible improvement opportunity 65%. What improves LCP might not improve FCP (as in this case). Because it ‘s a different goal.

Note that there is better LCP when image is sped up 80% than 99%. Why? It sounds counterintuitive.

Because there are request races in the browser.

// R1, R2, R3 - browser requests
// R1 is most likely to be a LCP bottleneck

R1 --------------------------------------------
R2 ------------------------------
R3 -------------------------------------

// after enough R1 boost LCP has no further improvements
// now R3 is most likely to be a LCP bottleneck
R1 ----------------------
R2 ------------------------------
R3 -------------------------------------

Systems don’t behave deterministic, that’s why hundreds of samples need to be taken. There is a lot of going on in the browser. If you cut off latency of a single request enough, you’re going to see the improvement and after that new bottleneck might be uncovered.

That’s a lo of knowledge, I’m glad you’re still here

Now let’s move on to

PageSpeed insights: website 2

I feel at the moment of writing my wordpress blog you’re just at is not fast. So let’s just measure it.

What does PageSpeed insight say?

It says 1.8s for FCP, 6.2s for LCP. It could be better.

However does it mean every user experiences FCP 1.8s? Or maybe that’s the worst case? Or the best?

How about fixes?

It points initial server response time could be better as it’s 1 140 ms

• Is it where most improvement can be made? I don’t know but it’s the biggest latency shown on the list.
• Other fixes show potential [kb] savings.
  • What does it mean in terms of latency? How much will it improve FCP? Is it worth investing my time to apply fixes there?

I value my time. I need data telling me what should be improved and in what order, not only what’s wrong without even knowing if it will help.

What does performance explainer FCP benchmark say?

It says user never experiences the desired FCP below or equal 1s.

It also says TTFB (Time To First Byte) is where 72% of time user is waiting for FCP and that it needs to be sped up 54.55% to meet acceptance criteria and have 99% latencies below 1s.

FCP takes from 1326 ms to 4807 ms across 150 measurements. Ouch. I want to give my readers a better experience.

FCP at 1.8s level shown in PageSpeed didn’t look that bad.

So let’s figure out ROIs using request boost exploration in performance explainer

The only fix worth investing is fixing TTFB. Not resources mentioned by PageSpeed insights.

I’m no wordpress expert, but I thought “How about caching the whole response to spend less time on producing HTML? There must be some plugins for caching“

So I installed WP Fastest Cache plugin, turned it on with default settings hoping to improve TTFB which is a bottleneck for FCP. Immediately I had questions in my mind.

• Did it improve performance?
• How would I know it?
• Would refreshing page in the browser give the feeling it’s better?
• What if it gave the feeling it’s better, but if I refreshed the page 10 more times it would actually give feeling it’s worse?

Let the data speak. And what does it say after taking samples and comparing it to samples before installing cache plugin?

FCP did not improve at all. Without measuring it after installing plugin and comparing to baseline (no plugin installed) I’d be pretty sure performance is better. While my readers would feel no difference at all.

Additional plugin would only add complexity to the system. And I would think it’s improving the situation. I removed the plugin immediately.

Now I have more ideas for speeding up TTFB.

• better hosting
• different caching plugin
• different skin
• … and there might be 10 more ideas

However that’s a typical trial and error fixing methodology when no data is available for showing bottleneck in more details.

It’s easy to produce ideas. It’s expensive to implement them. And it’s risky because it might fix nothing.

When you think a fix helped (while it didn’t), unnecessary complexity grows over time. It contributes to a growing production systems maintenance cost.

Since I know the bottleneck is PHP backend, what would really help is to have profiler samples. Or maybe some plugin showing what’s slow on wordpress backend if there is such one.

It would show where time is spent when request is made from the browser. In another words – data allowing understand better the root cause would appear.

If I knew PHP backend bottleneck, there would be no guessing what to fix. As a result

• much quicker fix delivery
• less time spent
• no unjustifiable complexity added for solutions that were supposed to fix performance, while fixing nothing

PageSpeed isights VS performance explainer summary

PageSpeed insights

• It shows improvement ideas. Those look good and worth spending time on
  • But in reality those fixes might not fix performance at all

performance explainer

• It’s showing the whole picture and how often user experiences good and bad performance.
  • Thanks to to hundreds of samples taken
• it does not present good sounding fixes ideas. It presents the ones that actually impact performance

Do you want to know if your website (on premise product too) is fast enough and having it fixed if needed?

Reach me at mikolaj.grzaslewicz@gmail.com if you want to have performance of your website fixed via consultations. My goal is to have performance explainer available as a service to support more customers.

The post No More Guesswork: Validating what improves First Contentful Paint upfront with No-Code Experiments first appeared on Deployment Every Day.

For measurements I’m going to use 2 different approaches to see which one is better for performance benchmarks and has better accuracy.

• first using playwright Page.waitForCondition
• and the second using custom condition polling

Custom condition polling implementation

// part of performance-explainer project I'm developing to measure things

package com.performance.explainer.api.waiter

import com.performance.explainer.spi.waiter.ConditionPolling
import java.time.Duration
import java.time.Instant
import java.util.function.BooleanSupplier


class ConditionPolling : SpiConditionPolling {

    override fun await(
        retryWait: Duration,
        timeout: Duration,
        condition: BooleanSupplier
    ): Boolean {
        val start = Instant.now()
        while (true) {
            val satisfied = condition.asBoolean
            val elapsed = Duration.between(start, Instant.now())
            if (satisfied) {
                return true
            }
            if (elapsed.plus(retryWait) > timeout) {
                return false
            }
            Thread.sleep(retryWait)
        }
    }

}

Website 1: https://www.theathletesfoot.nl/

Measurement ends when button starts working. Starts working means, it’s reacting on clicks. At this particular website it means validation error shows up.

That unfortunately cannot be checked with locator and .click() as button is not disabled and it’s clickable before it has event listeners attached. That could be fixed with waiting for loaded event, but that’s a bad idea for performance benchmark as explained with details in this blog post. So below code is NOT working for checking if button is working. Take a look at the explanation in source code

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
    page.navigate(
        "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
        NavigateOptions().setWaitUntil(COMMIT)
    )
    val button = page.locator("#add-to-cart")
    button.waitFor(WaitForOptions().setTimeout(3000.0))
    // plawright waits for a clickable button.
    // Clickable means it's visible and not disabled
    // This button is visible and not disabled before all resources are loaded.
    // Because of that, it does not actually measure user experience (button is working) 
    button.click()
}
// measured latency is duration of above

baseline 1 – wait for working button, use custom polling

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer

Consumer<Page> { page ->
   val conditionPolling = NonBlockingConditionPolling.default()
   page.navigate(
       "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.locator("#add-to-cart")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   val success = conditionPolling.await(
       retryWait = ofMillis(10),
       timeout = ofMillis(5000)
   ) {
       button.click()
       val formError = page.locator(".js-select-size-and-color-error")
       formError.count() == 1
   }
   if (!success) {
       throw TimeoutError("Waiting for working button timed out")
   }
}
// measured latency is duration of above

experiment 1 – wait for working button, use Page.waitForCondition

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.theathletesfoot.nl/producten/dames/air-max-90-sneakers/iic.nike.fz5163.133.html",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.locator("#add-to-cart")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   page.waitForCondition({
       button.click()
       val formError = page.locator(".js-select-size-and-color-error")
       formError.count() == 1
   }, Page.WaitForConditionOptions().setTimeout(5000.0))
}
// measured latency is duration of above

Website 2: https://www.blogmarketingacademy.com/

baseline 2 – wait for working button, use custom polling

import com.microsoft.playwright.*
import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.getByText("Subscribe Weekly")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   val condition = NonBlockingConditionPolling.default()
   val isButtonWorking = condition.await(retryWait = ofMillis(10), timeout = ofMillis(3000)) {
       button.click()
       page.getByText("This field is required").count() == 1
   }
   if (!isButtonWorking) {
       throw TimeoutError("Waiting for working button timed out")
   }
}
// measured latency is duration of above

experiment 2 – wait for working button, use Page.waitForCondition

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate(
       "https://www.blogmarketingacademy.com/membership-site-service-credit-system/",
       NavigateOptions().setWaitUntil(COMMIT)
   )
   val button = page.getByText("Subscribe Weekly")
   button.waitFor(WaitForOptions().setTimeout(3000.0))
   page.waitForCondition({
       button.click()
       page.getByText("This field is required").count() == 1
   }, Page.WaitForConditionOptions().setTimeout(3000.0))
}
// measured latency is duration of above

Custom polling VS Page.waitForCondition results

Each baseline and experiment has 200 samples taken. Hodges-Lehmann estimator is used for comparing results

baseline 1 VS experiment 1

baseline 2 VS experiment 2

Let’s increase custom condition polling retryWait and compare results again

Results should be closer to Page.waitForCondition, right? Right?

Intuitively with increased wait before another condition poll, results should get less accurate. And this way get closer to less accurate playwright’s Page.waitForCondition. Right?

Nope, in reality increasing retryWait up to 40ms makes no difference.

Custom polling with retryWait 10 ms VS longer retryWait

10 ms VS 20 ms – no difference

10 ms VS 40 ms – no difference

10 ms VS 80 ms – 2.644% worse

So there is difference noticeable in measurements when increasing polling retry wait from 10 ms to 80 ms. Sweet spot is 40 ms

Why is there a difference between custom condition polling and Page.waitForCondition?

Let’s take a look at playwright’s implementation for Page.waitForCondition

// com.microsoft.playwright.Page#waitForCondition(java.util.function.BooleanSupplier)
 @Override
 public void waitForCondition(BooleanSupplier predicate, WaitForConditionOptions options) {
   List<Waitable<Void>> waitables = new ArrayList<>();
   waitables.add(createWaitForCloseHelper());
   waitables.add(createWaitableTimeout(options == null ? null : options.timeout));
   waitables.add(new WaitablePredicate<>(predicate));
   runUntil(() -> {}, new WaitableRace<>(waitables));
 }

 // com.microsoft.playwright.impl.ChannelOwner#runUntil
 <T> T runUntil(Runnable code, Waitable<T> waitable) {
   try {
     code.run();
     while (!waitable.isDone()) {
       connection.processOneMessage();
     }
     return waitable.get();
   } finally {
     waitable.dispose();
   }
 }

 // com.microsoft.playwright.impl.Connection#processOneMessage
 void processOneMessage() {
   JsonObject message = transport.poll(Duration.ofMillis(10));
   if (message == null) {
     return;
   }
   Gson gson = gson();
   Message messageObj = gson.fromJson(message, Message.class);
   dispatch(messageObj);
 }

Under the hood in java implementation, playwright uses condition polling every 10 ms. The same interval which was used in custom polling. So why it has worse accuracy than custom condition polling? Source code at this level does not answer the question and that might be a question to playwright dev team.

How about Page.locator VS Page.isVisible polling?

After I saw it makes a difference when polling is used for more complex scenarios I thought “let’s also checked if it there is difference for plain locators like Page.getByText.

baseline

import com.microsoft.playwright.Locator.WaitForOptions
import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import java.util.function.Consumer

Consumer<Page> { page ->
   page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
   page.getByText("Sign up for GitHub").first().waitFor(WaitForOptions().setTimeout(2000.0))
}

experiment

import com.microsoft.playwright.Page
import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.TimeoutError
import com.microsoft.playwright.options.WaitUntilState.COMMIT
import com.performance.explainer.api.waiter.NonBlockingConditionPolling
import java.time.Duration.ofMillis
import java.time.Duration.ofSeconds
import java.util.function.Consumer

Consumer<Page> { page ->
   val conditionPolling = NonBlockingConditionPolling.default()
   page.navigate("https://github.com/", NavigateOptions().setWaitUntil(COMMIT))
   val isButtonVisible = conditionPolling.await(retryWait = ofMillis(40), timeout = ofSeconds(2)) {
       page.isVisible("text=Sign up for GitHub")
   }
   if (!isButtonVisible) {
       throw TimeoutError("Button not visible")
   }
}

Summary

I started with checking if playwright Page.waitForCondition replaced by custom condition polling every 10ms makes a difference. And it’s faster which means you get more accurate results with this approach.

Then I also measured 20ms, 40ms, 80ms custom condition polling interval. Up to 40ms there is no difference in measured latencies and it’s a sweet spot for retries between condition check.

Conclusion

If you want higher accuracy of web app performance measurement, use custom condition polling 40ms instead of playwright Page.waitForCondition

The post Want to have better accuracy of performance benchmark? Stop using playwright Page.waitForCondition first appeared on Deployment Every Day.

DOMContentLoaded

According to docs when browser fires this event it means

• DOM has been completely parsed
• <script defer src="…"> and <script type="module"> has been loaded
• async script, images, frames might not be loaded yet

load event

When browser fires load event it means DOM content is fully loaded. Browser has finished

• executing all asynchronous resources from the DOM tree
• loading images, frames etc.
• it comes always after DOMContentLoaded

Both events are NOT about REST calls. And both are not relevant to the user browsing the website.

What do web apps automation tools like playwright and selenium do by default?

Take a look at following lines of code measuring navigating to a page and waiting for a button to be visible.

using playwright

val millisBefore = System.currentTimeMillis()
page.navigate("https://github.com")
page.getByText("Try Now").waitFor()
val latency = System.currentTimeMillis() - millisBefore

or doing the same with selenium

val millisBefore = System.currentTimeMillis()
driver.navigate().to("https://github.com")
WebDriverWait(driver, 30, 100).until(ExpectedConditions.visibilityOfElementLocated(By.linkText("Sign up for GitHub")))
val latency = System.currentTimeMillis() - millisBefore

When you navigate to a page, by default those automation tools wait for load event. So the line with locator is not executed until then.

You might think Pff, so what. It’s just a technical detail

It is indeed a technical detail but it impacts performance benchmark dramatically.

Waiting for DOMContentLoaded VS load VS NOT waiting for any event. Fight!

Let’s measure how quickly user can see github call to action button.

Baseline benchmark – wait for load event

Take 1000 samples. Wait for load event, then wait for github call to action button

// only most important code shown here. Processing measurements is out of the scope
page.navigate("https://github.com")
page.getByText("Sign up for GitHub").first().waitFor(WaitForOptions().setTimeout(2000.0))

It looks like this

How to interpret the video of web app measurement?

Playwright video recording is imperfect. Video frame is often not aligned with what was seen in web browser. I confirmed that with modifying the measured page after measurement end.

During recording web app, right after measurement end timer is added to the page. This way it doesn’t interfere with measurement and helps to see when measurement has actually finished on the video.

const measuerementEndMillis = ..; // depends on the measuerement
const timerDiv = document.createElement('div');
timerDiv.style.position = 'fixed';
timerDiv.style.top = '0';
timerDiv.style.left = '0';
timerDiv.style.opacity = '0.8';
timerDiv.style.fontSize = '14px';
timerDiv.style.backgroundColor = 'yellow';
timerDiv.style.zIndex = '10000';
document.body.appendChild(timerDiv);

setInterval(function() {
  const now = performance.now();
  timerDiv.textContent = now.toFixed(0) + ' ms, ' + (now - measurementEndMillis).toFixed(0) + ' ms';
}, 20);

When measuring very fast page like github.com landing page, you’re often going to see a blank page and right after that frames recorded some time after measurement end (signup button visible). Frames are lost. Based on that you should treat every moment of playwright video as possibly inaccurate.

Having said that, let’s move on.

Experiment benchmark 1 – no waiting

Take 1000 samples. Don’t wait for any event, wait for github call to action button

import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.COMMIT

// from playwright docs WaitUntilState.COMMIT:
// consider operation to be finished when network response is received and the document started loading
page.navigate("https://github.com", NavigateOptions().setWaitUntil(COMMIT))
page.locator("#summary-val").first().waitFor()

In selenium you have to opt out from waiting for DOMCompleteEvent this way

val options = new ChromeOptions()
options.setPageLoadStrategy(PageLoadStrategy.NONE)    
val driver = new ChromeDriver(options)
driver.navigate(..).to()

Experiment benchmark 2 – wait for DOMContentLoaded event

Take 1000 samples. Wait for DOMContentLoaded, wait wait for github call to action button

import com.microsoft.playwright.Page.NavigateOptions
import com.microsoft.playwright.options.WaitUntilState.DOMCONTENTLOADED

// from playwright docs WaitUntilState.DOMCONTENTLOADED:
// consider operation to be finished when the DOMContentLoaded event is fired.
page.navigate("https://github.com", NavigateOptions().setWaitUntil(DOMCONTENTLOADED))
page.locator("#summary-val").first().waitFor()

In selenium you have to opt out from waiting for DOMContentLoaded event this way

val options = new ChromeOptions()
options.setPageLoadStrategy(PageLoadStrategy.EAGER)    
val driver = new ChromeDriver(options)
driver.navigate(..).to()

And now let’s compare the results

P95 comparison

Percentiles do not answer the question how distribution compares? Down the line you’re going to see Hodges-Lehman estimator

What hypotheses are we verifying here?

• load event
  • (baseline) null hypothesis: waiting for load event has completion has no performance impact
  • (experiment 1) alternative hypothesis: waiting for loadevent has impact on measured latencies
• DOMContentLoaded event
  • (baseline) null hypothesis: waiting for DOMContentLoad event has has no impact on measured latencies
  • (experiment 2) alternative hypothesis: waiting for DOMContentLoadevent has impact on measured latencies

Waiting for load event VS not waiting for any event

Title of this blog post says there is 530% performance degradation when you rely on browser automation defaults. So where is it? When you reverse baseline VS experiment, there it is

Null hypothesis is rejected, waiting for load event comes with bigger latencies

Waiting for DOMContentLoaded VS not waiting for any event

Null hypothesis is rejected, waiting for DOMContentLoaded increases measured latencies

Waiting for DOMContentLoaded VS waiting for load event

Having 2 experiments we can compare them too. Since load event comes always after DOMContentLoaded it’s expected to see bigger latencies which is reflected in measurements

Based on measurements, what are undesired effects of waiting for DOMContentLoad or load event?

• To understand better following part take a look at difference between accuracy and precision

• The undesired effects
• Measuring something neither reflecting user experience nor relevant to user experience. User sees a button and clicks a button regardless of events in the browser
  • No user ever thought “I politely wish this site had dom completion faster“. User thinks “Why the f^%@ is this button not working yet?!!??“
• (much) longer benchmarks
  • for ones that aim for measurement count, e.g. we want to have 200 samples
• Less precise benchmarks
  • for ones that aim for duration, e.g. running for 10 minutes
  • because you will get less samples
• Less accurate benchmarks
  • because you’re not measuring what user experiencies
  • As a result system under benchmark might actually meet some acceptance criteria, while measurements show it’s not
• Disturbs calculating correlations and figuring out frontend latency causation

Side effects of not waiting for DOMContentLoaded or loaded event

User can click a visible button, that does not react on clicks yet. Which actually is a reality

However that complicates measurements. Frameworks do not check if a button has event listeners. It often requires polling for a desired button state. e.g. await condition click and check if expected message is visible

Summary

I’ve taken thousands of measurements of a few different sites, not only github. Systems behave the same. Waiting for browser events makes measured latencies inaccurate. It might be 10% away from real user experience, it might be 530%.

It’s non-zero, reproducible and deterministic. So if you already made an effort to make synthetic performance benchmarks, I’d recommend putting an effort also in increasing their accuracy.

Consider changing default options for browser automation

For playwright and selenium (and whatever browser automation tools there are) you should opt out from waiting for any browser events if you’re using those tools for performance measurements.

The post How to make performance benchmark 530% worse by relying on web app automation defaults? first appeared on Deployment Every Day.

First things first. What’s the goal of a UI measurement?

Who consumes UI? Humans from planet Earth, of course. They get annoyed, frustrated and want to throw away your website out of the window when it’s too slow. They also think very neat things about your product when they are waiting too long.

So the ultimate goal of measuring UI is to know if/what parts of the system should be improved in order to keep users happy.

What question should performance pass / fail criteria answer?

It should answer the question: “Are users happy using my product?“.

Why happy user is that important? And what are consequences of having frustrated users because of poor performance?

You’d better keep users happy by keeping your site fast. Not only you users suffer. Your business suffers. The longer you’re making users to wait, the more

• bounce rate goes up
• you’re losing traffic from organic search as search engine prefers faster sites
  • It’s a real deal. I already spoke to people having this particular problem. Google changed the rules (web vitals) and whole team needed to work on performance to bring back lost organic traffic
• and for systems that companies use internally daily
  • users will start to look for alternative solutions that frustrate them less
  • or have a grumpy-cat face all day

When system passes acceptance criteria it means particular part of your system is fast enough. It means there aren’t too many frustrated users. Product does not need to be fixed in this area.

When system does not pass acceptance criteria it means it’s too slow. Which means there are too many frustrated users. Product needs to be fixed in this area.

What’s the perfect threshold for UI pass / fail acceptance criteria?

0 ms so no one waits for anything Jokes aside, let’s see what research says about latencies.

What research says about latency and user frustration

• https://hpbn.co/primer-on-web-performance/#speed-performance-and-human-perception
• Latency longer than 1s causes likely mental context switch. And bounce rate goes up
• And that might be our frustration point

Even if a particular part of your UI is far away from latency 1s, it might be a good, ambitious goal that you can track. Don’t lower the expectations just because system will not pass the criteria. You can always measure the gap to passing the criteria and progress on closing the gap.

Measuring user happiness and frustration is nothing new and it works well

There already are well thought concepts. For example

APDEX

• https://docs.newrelic.com/docs/apm/new-relic-apm/apdex/apdex-measure-user-satisfaction/
• User satisfaction based on T, below which user is satisfied
  • Satisfied, latency <0, T]
  • Tolerated, latency <T, 4T]
  • Frustrated, latency <4T, infinity) or error
• Score [0, 1]
  • 0 – all frustrated
  • 1 – all satisfied
  • (number of requests within satisfied range + (number of requests within tolerated range) / 2) / total number of requests

APDEX makes sense. However I will explain percentile rank as even simpler and as powerful concept.

You might have also heard about percentiles, so let’s compare percentiles and percentile rank in practical terms.

Percentiles. Usually P90, P95 or P99

What’s the idea?

Sort the measured latencies array ascending. P90 is index in this array at 90th %.

P90 = latencies[latencies.size * 0.9]

P95 = latencies[latencies.size * 0.95]

etc

Using high percentiles (P90-P99.9) you can find out the slowest latencies user expierence.

Calculating percentiles requires whole population (latencies sorted ascending). It’s not applicable for live monitoring of systems as you have to store every single measured latency.

Percentile rank

Statistical meaning for PR(n)

• How many % of other values are lower than n
• Llike on exam: how many % of other students you’ve beaten with your score

It doesn’t require whole population to calculate, you can count it even on stream. Just increment counter of the passed threshold(s).

Implementation is simple

val badCount = 0
val goodCount = 0
val threshold = 1000
latencies.forEach {
 if (it < threshold) goodCount++ else badCount++
}
val percentileRankPercent = goodCount * 100.0 / (badCount + goodCount)

Why not just use percentiles and P90?

P90 means what's the value at 90%th index of sorted array?

PR(1s) means how many latencies are below 1s?. Not below or equal and that makes the difference.

P90 = 1s doesn’t mean PR(1s) = 90%

Take a look at the example. Given following, sorted array of latencies [s], 10 elements for simplicity

• 0.25, 0.3, 0.6, 0.6, 0.7, 0.8, 0.85, 0.9, 1.0, 1.4
  • P90 = 1.0
  • PR(1) = 80%
  • PR(1.1) = 90%
  • PR(1.5) = 100%

Let’s change the example. Latencies got worse, but P90 stays the same

• 0.25, 0.3, 0.6, 0.6, 0.7, 1.0, 1.0, 1.0, 1.0, 1.4
  • P90 = 1.0
  • PR(1) = 50%
• As you can see, a single percentile does not say how many samples

Let’s change the example. Latencies got even worse, P90 stays the same

• 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.4
  • P90 = 1.0
  • PR(1) = 0%

As you can see, relying on a single percentile brings a problem. For P90 anything between [0, 90%] of latencies might be as bad as P90.

Percentiles are not designed to answer question how good/bad this latency is?

Let’s construct the pass / fail acceptance criteria based on percentile rank

Based on previously mentioned research and 1s as a user frustration point, let’s construct the acceptance criteria.

PR(1s) is expected to be at least 99%

• Which means “At least 99% of website interactions user should experience latency below 1s“
• Which also means “User can be frustrated by max 1% of website interactions“

What if after a benchmark, criteria is NOT met?

e.g. PR(1s) = 80% and is expected to be at least 99%

It means 19% of the responses are too slow. 19% of interactions needs to be fixed.

How to measure improvement if you start with PR(1s) = 0%

You might be in a situation that all latencies are >= 1s. 100% frustrated user. You want to improve the product by chaning something.

Before change

• PR(1s) = 0%
• All measured latencies are be above 2s

After change, new commits added to the product

• PR(1s) = 0%
• All of measured latencies are above 1.5s. It is an improvement, but percentile rank hasn’t changed.

And you can’t just say “this improvement doesn’t matter, user is still frustrated because criteria measuring user’s frustration is till unmet“. Some improvements require a lot of iterative work and you need a devloop in which you notice the improvement. Even if it’s still far away from having happy user.

To compare if there is an improvement you could use comparison benchmark with Hodges-Lehmann estimator.

• That requires access to every single latency from baseline and experiment

You could also use a simple measure which I call ‘wasted user time’ which you want to keep at 0.

val thresholdMillis = 1000
val allowedDurationMillis = latencies.size * thresholdMillis
val totalDurationMillis = latencies.sumOf { it.toMillis() }
val excessiveDurationMillis = totalDurationMillis - allowedDurationMillis
val wastedUserTimePercent = excessiveDurationMillis * 100.0 / totalDurationMillis

This way even if percentile rank hasn’t changed you can still measure the improvement.

And that’s the exact methodology I’m using to measure webpages performance and answer questions “how to improve the system?”

The post How to use percentile rank to measure performance first appeared on Deployment Every Day.

I was trying to check what can be improved during working on a small project. As usual, I run intellij profiler from time to time to see the hottest methods. And that was part of my development workflow aside from TDD.

2x speedup – remove String.format() usage (technical optimization)

git commit

Remove usage of String.format("Move index %s from %s to %s", indexFrom, listFrom, listTo);

It’s not surprising it was slow, but discovering that is not obvious. Imagine you’re introduced to a new, big project – there is no way you would eyeball the code and say this is a bottleneck (hot method)

10x speedup – replace streams with for each (technical optimization).

git commit

Replace

return calculators.stream().mapToInt(c -> c.calculate(listA, listB)).sum();

With

  int sum = 0;
  for (var calculator : calculators) {
      sum += calculator.calculate(listA, listB);
  }
  return sum;

Bottleneck (hot method) found

2x speedup – add sum caching (technical optimization).

git commit

Replace

  private int sum(List<Integer> list) {
      int result = 0;
      for (var i : list) {
          result += i;
      }
      return result;
      return Math.abs(listA.sum() - listB.sum());
  }

With a sum caching java.util.List decorator

  public class SumCachingList implements SummingList {
      private final List<Integer> decorated;

      public SumCachingList(List<Integer> decorated) {
          this.decorated = decorated;
      }

      private int sum;
      private boolean sumCalculated = false;

      @Override
      public int sum() {
          if (!sumCalculated) {
              calculateSum();
              sumCalculated = true;
          }
          return sum;
      }

      private void calculateSum() {
          for (var i : decorated) {
              sum += i;
          }
      }

      @Override
      public boolean add(Integer integer) {
          sum = sum() + integer;
          return decorated.add(integer);
      }

      @Override
      public boolean remove(Object o) {
          var removed = decorated.remove(o);
          if (removed) {
              sum = sum() - (Integer) o;
          }
          return removed;
      }

     @Override
     public boolean removeIf(Predicate<? super Integer> filter) {
         var anyRemoved = decorated.removeIf(filter);
         if (anyRemoved) {
             sumCalculated = false;
         }
         return anyRemoved;
     }

      // ...
  }

After above optimizations flamegraph looks like that

Big picture optimisation VS technical optimisation

Ok, now you see that you can discover bottlenecks using async profiler recording. Above example is rather an easy one. For huge systems with tons of legacy code which is hard to maintain and change, it might be really hard.

Even discovering slow parts of big systems is hard, but that’s a topic for another blog post.

Technical optimizations are often easier to find.

Technical optimization

You don’t necessarily have to understand what the application is doing. You can read straight from the flamegraph that adding a cache or optimizing loops is going to help.

Big picture optimisation

You need to understand what your application is doing. You can optimize by organizing processing in a different way, e.g. discovering that you don’t have to fetch and process some data to in order to display particular piece of frontend.

Run integration tests with profiler regularly

That should be part of your development process. Like running integration tests.

In a perfect case, scenario of integration test should be as similar as possible to production scenario. This way you’d discover potential performance improvements or issues before production deployment.

Know the difference between async profiler sampling modes: Wall Clock and CPU usage only

Wall Clock (Total Time)

If you’re interested in real latency of your system, including

IO, e.g.

• connection polling
• DB transaction locks
• reads/writes

synchronization, e.g.

• waiting for critical section access
• waiting for tasks in a thread pool

use Wall Clock sampling in async-profiler. This mode will collect events from threads in SLEEPING state also.

Beware it affects performance of measured process more than measuring only ACTIVE threads. Why? Because measured JVM has more threads to iterate over every interval.

CPU only

In other words, without Wall Clock mode, measuring sampling only JVM threads in ACTIVE state

In the examples above you can see hot method httpClient.newCall(request).execute()

• Time spent measured with CPU Time is 120 ms
• Time spent measured with Wall Clock (Total Time) is 9170 ms

Would you optimize CPU time in this case? Probably not. You’d first focus on IO as it’s the bottleneck.

Subject to optimize needs to be chosen case by case. In order to have this choice, you need have both CPU Time and Total Time in flamegraph – and it’s available only when using wall clock mode sampling.

Resources

• Evaluating the Accuracy of Java Profilers
• Safepoint bias problem
  • https://jpbempel.github.io/2022/06/22/debug-non-safepoints.html
  • https://psy-lob-saw.blogspot.com/2015/12/safepoints.html
• https://github.com/mgrzaslewicz/set-equalizer

The post Example of 40x speedup using async-profiler recording analysis (JFR file) first appeared on Deployment Every Day.

• If you also love automating everything and hate doing repeatable job more than a few times, consider automating provisioning your workstation machine.
• You regularly or occasionally use many apps from apt and snap repositories.
• You’re using some tools that are not accessible via apt and snap repositories. Those tools get updates and I really wanted to consume them regularly. Manual updates were a no-go anymore as there are too many of them.
• In my case over time, the list got big enough to make it impossible to move easily to another workstation.
• Moreover, I wanted to keep portable dotfiles between machines, have a backup, easy restore and history of changes.
  What does my workstations setup look like?
  • stationary workstation 2x ubuntu on 2 different drives.
    • primary That’s where I spend most of my working time
    • fallback in case primary fails for whatever reason
  • laptop 1x ubuntu
    • I grab it when going to the office
• As you can see, there are 3 ubuntu installations. I want to switch between workstations at any time without losing updates to software, its configuration and my dotfiles.

Here goes the full list of software I’m using regularly or occasionally

• It might have evolve from the time of writing this blog post, but you get the idea. Just imagine updating those manually or providing configuration – from the scratch on a different workstation.
  I’m an experienced java/kotlin developer hence many tools are related to backend programming and debugging.
  • apt
    • bat # alternative to cat
    • hexyl # hex viewer
    • crudini # to manipulate ini files from command line easily
    • fd-find # alternative to find with some improvements
    • imagemagic
      • I generate PDFs from images from time to time, it requires tool and conf. I don’t want to remember about providing special config entries every time I change machine in /etc/ImageMagick-6/policy.xml
    • mc
    • nmap
    • oathtool # 2FA code, e.g. oathtool -b –totp ‘secret code’ | xclip -sel clip
    • paprefs # to have virtual device capable of streaming via multiple physical devices
    • python3
    • python3-pip
    • pipx
    • snap
    • speech-dispatcher # for speaking exit code in terminal of last executed command
    • tmux
    • tmuxinator
    • unrar
    • unzip
    • vagrant
    • virtualbox
    • yarn
    • zbar-tools # to scan QR codes from image or camera
  • snap packages
    • foobar2000
    • intellij-idea-ultimate
    • postman
    • signal-desktop
    • spotify
    • sublime-text
    • zoom-client
      • and its configuration to open meetings from browser links
    • yq
  • a few python packages
  • git
    • git-global-config
    • lazygit
    • custom aliases 1, 2
  • knowledge
    • cheat and cheatsheets
    • loqseq
  • zsh
    • oh-my-zsh
    • powerlevel10k
    • aliases
  • java
    • eclipse-mat
    • jdk v8, v9, v11, v14, v17, v19, v21
    • jdk-tools
    • jenv # to use JDK version set per project
      • and its zsh conf
    • mvnvm (maven wrapper)
  • docker
    • docker-registry
    • lazydocker
  • javascript
    • nvm (node wrapper) and its zsh conf
  • browser
    • brave
    • chromium
    • edge
    • firefox
    • tor
  • various
    • broot
    • delta
    • imagemagick
    • konsave
    • lsd
    • mcfly and its zsh conf
    • viddy
      zoom-client
      zoxide # smarter cd alternative
  • dotfiles
    • home-dir-as-git-repository
      • tmux
      • tmuxinator sessions for multiple projects I’m working on
        • my own
        • company I’m working for (Atlassian at the time of writing)
      • project-specific bash/zsh aliases
      • various apps settings
  • vim
    • config
    • vim-plug
    • neovim

To sum up: what’s the problem to solve?

I want to run single command to have my machine set up or updated in a few minutes.
I don’t want to be afraid of upgrading or reinstalling system. I don’t want to miss anything when changing machine I’m working on. And I want to have fun and joy of just running provision command to setup my workstation

• install all the software I need. And I need dozens of tools
• update software versions regardless of download source
• restore dotfiles

Benefits of automated workstation provisioning

• You don’t have to care about configuring manually your laptop or any other machine running ubuntu.
• You just run provision, go for a coffee and after coming back – everything is configured exactly your way
• You can throw away your machine and have the same configuration within minutes
  • It took many hours to get to this point in my case, but it’s totally worth it.
• It trains your mindset that you can apply to other projects: your machine setup is a project and list of git commits
• You know exactly what toolset you have. When you forget something, just take a look into the repository

Side effects of automated workstation provisioning

Whenever you make a change to configuration, you have to make a change in the code. That’s a approach shift

Thanks to that, manual changes do not get lost as you’re relying on automation instead

• Manual changes are extremely easy to forget about
• Manual changes can be overwritten by accident. Can means it will happen
• You’re going to forget about your own decisions in a few months from now. In git history, you can keep the why for changes

Let’s switch to solution mode: how to implement WaC?

You don’t need anything fancy for provisioning, however bunch of bash scripts is IMHO a no-go for a maintainable project. If you want a robust solution, it needs to have timeouts, retries, self verification of changes and good reporting. There is already an out-of-the-box solution for that.

ansible for provisioning workstation and keeping it up to date

Way better than tons of bash scripts. Maintainable and easy to read.

HOME dir as a git repository

Did you know you can use multiple git repositories in the same folder?

So in fact you can use your HOME dir as many git repositories

This way you can use dotfiles for as many projects as you want or need

e.g. GIT_DIR=.git_my_project git pull

• and this way you can keep your project-specific aliases in a dedicated repository

tmux + tmuxinator

• I just type tmuxinator autocoin, tmuxinator jira, tmuxinator jpt etc. and have all the right set of console tabs open. Even automatic ssh connection in a separate console tab is waiting for me

<% REPO_BASE = "~/repos/autocoin" %>
<% DATA_BASE = "~/repos/autocoin/data" %>
project_name: autocoin
project_root: <%= REPO_BASE %>
windows:
  - mediator:
      root: <%= REPO_BASE %>/autocoin-exchange-mediator
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-exchange-mediator/
        - <%= REPO_BASE %>/autocoin-exchange-mediator/scripts
        - <%= DATA_BASE %>/autocoin-exchange-mediator
  - binance-bot:
      root: <%= REPO_BASE %>/autocoin-binance-bot
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-binance-bot
        - <%= REPO_BASE %>/autocoin-binance-bot
        - ~/.trading-bot/file-repository
  - gateway:
      root: <%= REPO_BASE %>/exchange-gateway
      layout: tiled
      panes:
        - <%= REPO_BASE %>/exchange-gateway
        - <%= REPO_BASE %>/exchange-gateway/scripts
  - arbitrage-monitor:
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-arbitrage-monitor
        - <%= REPO_BASE %>/autocoin-arbitrage-monitor/scripts
        - <%= DATA_BASE %>/autocoin-arbitrage-monitor
  - auth:
      layout: tiled
      root: <%= REPO_BASE %>/autocoin-auth-service
      panes:
        - <%= REPO_BASE %>/autocoin-auth-service
        - <%= REPO_BASE %>/autocoin-auth-service/scripts
        - <%= DATA_BASE %>/autocoin-auth-service
  - frontend:
      root: <%= REPO_BASE %>/autocoin-frontend
      layout: even-horizontal
      panes:
        - <%= REPO_BASE %>/autocoin-frontend
        - <%= REPO_BASE %>/autocoin-frontend
  - xchange: <%= REPO_BASE %>/XChange
  - prod:
      layout: tiled
      panes:
        - ssh-autocoin-prod
        - ssh-autocoin-prod
        - ssh-autocoin-prod
        - ssh-autocoin-prod
  - balance-monitor:
      root: <%= REPO_BASE %>/autocoin-balance-monitor
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-balance-monitor
        - <%= REPO_BASE %>/autocoin-balance-monitor/scripts
        - <%= DATA_BASE %>/autocoin-balance-monitor
  - strategy-executor:
      root: <%= REPO_BASE %>/autocoin-strategy-executor
      layout: even-horizontal
      panes:
        - <%= REPO_BASE %>/autocoin-strategy-executor
        - <%= REPO_BASE %>/autocoin-strategy-executor/scripts
  - puppet-deprecated:
      root: <%= REPO_BASE %>/autocoin-puppet
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-puppet
        - <%= REPO_BASE %>/autocoin-infrastructure/machine-developer
  - infrastructure-provision:
      root: <%= REPO_BASE %>/autocoin-infrastructure-provision
      layout: even-horizontal
      panes:
        - <%= REPO_BASE %>/autocoin-infrastructure-provision
        - <%= REPO_BASE %>/autocoin-infrastructure-provision
  - db: <%= REPO_BASE %>/autocoin-db
  - metrics-client: <%= REPO_BASE %>/autocoin-metrics-client
  - telegraf-influx-grafana: <%= REPO_BASE %>/autocoin-tig-monitoring
  - autocoin-data: <%= REPO_BASE %>/data

Lessons learned

ansible

• It does its job quite good for linear tasks. However I would not recommend using it for more advanced problems like running things in parallel to speed up provisioning within a single machine. YML is not meant for advanced programming
• I haven’t used ansible before starting ubuntu-workstation-provision project. Ansible documentation is quite impressive and has good examples. It’s easy to learn.

Bash installation scripts

• Some tools on github have a installation script e.g. wget <link> | bash. It’s a bit risky as the script might do anything or get hacked, but I accept the risk in favor of convenience.
  On the other hand I applied similar approach to provide easy dotfiles installation for any project and that’s the code I’m in control of – so no risk. Example here so you can create something similar on your own

Fail fast with fresh virtualbox machine

Running provisioning on a fresh virtualbox machine uncovers some problems and I run provisioning of such machines from time to time. Here’s an example how to use ansible with vagrant

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/noble64"
config.vm.provision "ansible" do |ansible|
  ansible.playbook = "/repos/ubuntu-workstation-provision/playbooks/provision.yml"
  #ansible.skip_tags = "graphical"
end
end

Things to avoid

Try not to parse 3rd party HTML to get software versions if there is no API providing latest version

• HTML is not an API and your automation will break at any time
  • You’re going to have more maintenance work
  • Example regarding zoom installation
    • Initially I used zoom-client installation via snap as there is no zoom in apt repositories
    • However it was failing to open zoom meeting via clicking on a link in browser
    • So I moved to installing .deb package downloaded from official zoom site
    • Version was buried in the HTML code so I parsed it with regex
    • I finally had working zoom meeting links
    • And one day HTML on zoom site was changed, parsing latest version failed
    • So I switched back to installing zoom via snap and zoom links were broken again
    • And I provided proper KDE configuration to make zoom links work as expected

Yet another experience strengthening belief that automation allowing to throw away and recreate things wins over keeping state and list of steps

• Approach with manual steps that you run or delegate to other developers is
  • not scalable
  • not maintainable
  • hard to reproduce by others (and ==you== in the near future)
  • error prone
    • sooner or later you or other developer will make a mistake. Period. I’ve seen it dozens of times in many projects. It also relates to installing software and configuring it.
• Where else can you see the power of throwing away and recreating easily? A few examples
  • Docker images
    • They have a recipe that creates an image. If you rely on manually created accessible in some repository, butone with no recipe (no Dockerfile) you will get into trouble
  • Test datasets
    • In the performance team we missed AWS bucket deadline and lost our test datasets. No one new how those were created and people who created them were no longer working in the company. We were fucked for a while and had to create automation to recreate them anyway.
  • Browser tabs
    • Why bother keeping all open for your project?
      It’s easy to lose it and have fear of closing the tabs. Just use session manager to save the session, reopen it at any time and have the state you want again. Then you won’t have to keep dozens of browser tabs open
  • Performance testing
    • Create a process every developer in the company can run easily, in a perfect world with a single command. Now compare that to list of 20 steps to run manually listed on confluence page
  • Example from workspace-provision project
    • Manual eclipse-mat download takes 2 minutes if just unzipped and run. However during manual installation, I’ve discovered with trial and error that actual steps to have it working goes as follow:
      • 1) download eclipse-mat
      • 2) unzip it
      • 3) move it to /opt directory
      • 4) set -Xmx4g in MemoryAnalyzer.ini. Default heap size is too low and I will have to do it anyway after download
        • Why? I already had it consuming 100% CPU and taking ages to analyze heap dump. GC was running all the time
      • 5) set it up to use java version 17
        • Why? It’s the minimum required by eclipse-mat. I have multiple JVMs so there is no best default for the whole system. It just didn’t want to start because of too low default java version
      • 6) create symlink /user/local/bin/mat to run it easily from command line or krunner
    • It’s very likely I will forget step 4 and 5 when installing newer version or trying to run it on another ubuntu (including other developer’s machine)
    • Having it automated means I can send someone a command to run
      • git clone https://github.com/mgrzaslewicz/ubuntu-workstation-provision.git
      • TAGS=eclipse-mat ./provision
      • and it’s
        • scalable
          • n other developers can run it without asking me for anything
        • maintainable
          • fix needed? No problem, just change the code
        • easy to reproduce by others
          • no brainer, just copy paste
        • error proof
          • well, you can always make an error while copying and pasting

Automated windows provisioning?

• Nowadays I use windows to play games only, but I used to use it as a workstation too so I had some attempts for provisioning automation.
• Take a look at chocolatey. It’s a windows package manager so it’s easy to install most of needed software automatically. It can be used with ansible too. And that’s the simple provisioning script using chocolatey I used to use.

Resources summary

• set of alternative/better/fresh unix tools: github.com/ibraheemdev/modern-unix
• WaC implementation: github.com/mgrzaslewicz/ubuntu-workstation-provision
  • it’s using also HOME dir as git repo

What does your WaC look like?

Have you already tried implementing your own workstation as a code? What have you learned, what worked well, what were you struggling with?

The post How To Provision Your Ubuntu Workstation – Workstation As a Code (WaC) first appeared on Deployment Every Day.

• People want easy, 1-click money
• There is high demand for software that might help achieve that. But such software is impossible to create. There is no easy money. In order to make money, you need to put effort into whatever software you use.
• It’s hard to build you own product

Have you ever wondered what effort it requires to build a cryptocurrency arbitrage monitor (scanner)?

Can you make money from executing cryptocurrency arbitrage opportunities?

At the time of writing (Jan 2023) autocoin-trader.com serves 2 main functionalities:

• Find and display arbitrage opportunities with list of steps to execute them
• Calculates your cryptocurrency portfolio balance based on different sources: wallets, coins, exchanges

It got 30+ paid subscriptions, but almost no recurring ones. 

Before I got to this point it was a long journey full of motivation ups and downs.

I’m a highly experienced backend java+kotlin developer and started working commercially in 2007. Angular based frontend is a no-brainer to me…

And at the end of the day it seems good technical skills are not what makes a successful product/saas at all.
Ready for a journey? Let’s go

2017 Initial idea: Why not make a cryptocurrency trading bot and earn easy money?

I had a brilliant idea. Why not make a bot that will buy and sell cryptocurrencies and make money for me on autopilot? So I would just press enter and wait for money to multiply? Sounds sexy!

I would just put effort into creating and maintaining the bot, money would multiply itself.

It just needs some buy and sell strategy. I started with a really simple one. Sell when price grows 5%, buy when price falls 5%.

I did some backtesting to see how the strategy behaves with historical data. As far as I remember it had some advantages compared to just holding ETH. At least in the simulation.

When I started executing the strategy at bittrex exchange with ETH, I realized in the short term it’s losing money doing a few trades daily. I run it also with GLM (golem network) cryptocurrency and results were similar.

After a while it occurred to me that more advanced price analysis is required. I researched what technical analysis libraries there are. I started using ta4j as the most interesting one. ta4j is a technical analysis library written in java. I’ve even ported the ta4j library to kotlin, it’s on github. 

I spent a while implementing basic technical analysis signals. 

I soon realized creating a bot that really earns money on autopilot would require:

• Providing a working trading strategy with set of rules
• Or using machine learning

I’m not a trader and I don’t know trading strategies. I’m not experienced in using machine learning either.

Creating software that does the calculations and makes trades is far more attractive to me than learning technical analysis itself. I imagined people providing their own set of technical analysis signals and buying or selling based on them. Software would ‘just’ monitor the market. And when market conditions satisfy the signal rules, buy or sell.

This vision that people could run their own trading strategies made me excited.

So I stopped thinking of working on my own trading strategy and shifted towards creating an engine to run the strategies.

To sum up, I earned zero money at this point. On the other side I gained crucial experience around cryptocurrency exchanges which is:

• Connecting to exchanges and using their APIs
• Implementing technical analysis at a basic level which produces signals for buying/selling
• And that was a good code base and experience for moving on to the next step

Next step: turn the bot into half automated cryptocurrency trading bot as a service

A friend of mine was trading daily and he wanted a tool that would help him manage many exchange accounts at the same time. He had a few people that trusted him and he was also doing trades on their accounts. As you can imagine, managing 10 accounts at the same time is a lot of work. And the speed of execution matters a lot. Doing that manually is time consuming. 

After speaking to him I thought automating what he does would be exciting. And it would later scale up as probably more traders need it.

So what did he need?

• Managing many accounts from one place via API keys
• Viewing open orders from many exchanges that he would cancel at any time
• Simple trading strategies that he could run in parallel. E.g. sell n% of ETH every time it grows up X% and at the same time BTC is above Y Z price

I will make software as a service for one man, more people will need it for sure!

With that mindset and a few months of hard work I had a UI and backend services able to run many simple trading strategies on multiple exchange accounts at the same time. Plus all the monitoring to know the state of the system from devops perspective.

I tried to broadcast my system on reddit

• https://www.reddit.com/r/BitcoinMarkets/comments/dak7z7/automated_trader_does_it_do_what_you_need/
• https://www.reddit.com/r/BitcoinMarkets/comments/dbh1bd/what_do_you_hatelove_automated_trading_software/

People were skeptical to say the least and posts gained very little attention.

I gave up focusing on marketing. Motivation dropped down. I stopped working on the software for a few months.

2020: “I’ve put so much effort into my product and I’m not making any money from it. Should I give it up?”

No money earned for such a long time, motivation drops down

After 3 years of work at https://autocoin-trader.com/ as a side project there was

• trading automation operating at many exchanges
• managing open orders at exchanges
• displaying summary of exchanges balances. Kind of ugly, but working

I also added an arbitrage monitoring microservice working within my infrastructure. Source code is on github. The UI was not intuitive at the beginning.

Zero coins earned working on the project so far. I just felt tired maintaining the project and I did not want to put any effort into the project for some time.

I could either:

1. Abandon the project and move on to some other side project
2. Figure out how to use what I already have, make it more attractive and valuable

So I decided to give 2) a chance.

2021: Less is more – why I removed most of functionalities

Initially many features in the product

I had many functionalities in the system:

• Trading bot
• Showing exchange balances
• Tracking and canceling open orders
• Arbitrage monitor (arbitrage scanner)

And none of the above were great.

I bought a ‘from developer to founder’, Bogusz Pękalski’s course. It changed my mindset and I knew what steps to take to increase the probability of creating a SaaS.

Back then I had ~3000 users that signed up to autocoin-trader from this single video. 

That’s ~2h of marketing as a total effort. That’s almost zero marketing and for such a little effort, the result was huge.

The demand for an arbitrage monitor became quite obvious.

I tried to manually use some arbitrage opportunities and I found it hard to use. Spreads at 2021 were quite small. Benefit/effort ratio was too low for me.

In autocoin architecture, there is exchange-mediator which is a unified API to all exchanges. Whenever I made breaking changes in the mediator, I had to maintain functionality no one was really using – like managing open orders and trading bot.

Remove features to speed up making rest of them much better

I decided to turn off 2 functionalities and leave 2 of them only. This way it would be much easier to polish them, starting with arbitrage monitoring.

After the decision that’s what was left to be improved and maintained:

• Trading bot
• Showing exchange balances
• Tracking and canceling open orders
• Arbitrage monitor (arbitrage scanner)

Let’s focus on making ONE feature great and see what happens

Strong market demand

Using arbitrage opportunities manually has low chances of being successfully used by people. You have to compete with bots and other people.

However I could not deny there was and there still is a strong market demand.

I’ve created a video which is ~2h of marketing work. 14k views

And that resulted in 4k+ registered users

I decided to give it a go anyway because of a market demand.

Apart from the email list, there was already an audience I could chat with on telegram group

I decided to deliver what people say they want.

Anytime there was:

• Question on a telegram asking how to use or understand the product
• Future request

I’ve been putting everything into the backlog of improvements and bug fixes. Before going straight to the implementation I sometimes waited to see if more people wanted the same feature or improvement.

Some of the user questions and demands are in the README.md of open source part of arbitrage-monitor service.

When arbitrage-monitor was good enough shape (objectively not very good yet) I’ve splitted product into 2 plans:

• Free plan
• Pro plan for $10 USDT or other coin

At the beginning I’ve asked people to try the Pro plan for free to get more feedback and improve the product even more.

Within the following months arbitrage-monitor SaaS went from kind of good to much better.

Being honest about expectations

I’ve communicated clearly from the very beginning and many times on telegram that

• there is no easy money
• you have to put effort and time
• Executing arbitrage opportunity has its risks

https://t.me/c/1189042043/2139

https://t.me/c/1189042043/2284

the execution steps for arbitrage opportunities shows the risks listed

Honest expectation shown for Pro plan payment explanation

Great experience learned the hard way

I’ve gained great experience in building a product. I’ve also learned for the second time how not to do it. It wasn’t my first attempt to build a SaaS, but that’s a different story.

Just don’t be like me. Don’t start with a product and then try to sell it.

Treat building a product like an end to end test. Figure out what’s the lazy and low effort to test it. This way you will discover the probability of success.

I’ve asked questions about new features not related to arbitrage on the telegram group. No one was answering them so I knew if I deliver, those features will not make those people excited.

I delivered some of them anyway as I’m using them, e.g. tracking wallets.

2022: It’s time to finally add the landing page

5 years without landing page

I had no landing page for 5 years. And without it 4k+ people registered on the mailing list.

It worked, because people knew what to expect after watching a video on youtube and went straight to the registration form.

I’ve added a landing page because people were asking for a landing page with a summary of what the product offers.

2022 Nov: Unexpected (but not surprising) side effect of being experienced in implementing automated trading strategies – being asked to make flash crash-ready trading bot

In November 2022 I had a message from a crypto investor and later a call.

In short:

Crypto investor: “I need a trading bot that will buy as much BTC as possible in case a flash crash on binance happens. The problem is Binance does not allow to place buy limit orders with price lower than 1/ 5 of current price. It needs to reposition buy orders to really low prices as the price goes down. I need the bot as soon as possible”

Me: “Sounds exciting. I can do it within a few days thx to already many pieces ready in my system”

Crypto investor reached me because he knew I’ve been working on such software for a long time and I was his natural choice.

So we agreed upfront on the price plus percent of bitcoins that bot will buy (in case a flash crash  happens and bot does its job). Flash crash is a deep, short price drop. It might last a few minutes or even less. If it happens, it’s a chance to multiply bitcoins on autopilot.

I exceptionally sacrificed my healthy habits like exercising and sleeping well for some time.

After 5 days of crazy work after hours it was delivered.

As of 13/12/2022 flash crash on Binance didn’t happen yet

Key lessons learned after working hard on SaaS which which lead to no recurring customers

• During building the product it’s crucial to have users you can easily communicate with
  • The hard work is to get the users that you will build the product for. That will allow you to find out the user needs before putting a huge amount of work up front. I saw similar articles on India hackers mentioning the same argument. And even though I already read about it I made the same mistakes. Don’t be like me
  • When you have users to communicate with, they will tell you what they want. You will just have to deliver it, without guessing what might sell
• You’re going to have motivational ups and downs. When you love the process of creating the product, it keeps you running as it’s disconnected from the end result which might not be what you want
• If I were to start another project from the scratch
  • I’d consider starting without MPV, with landing page only – to get users
  • Building a really low effort MVP and then landing page is a good option as well. However, due to my engineering skills and mind I’m afraid I’d focus too much on technical stuff. So personally I’d start without MVP just not to get lost in solving technical problems 
  • This way you can avoid a huge amount of work that will lead to no happy customers at the end. Plus no happy solopreneur
• People are eager to use tools that will give them easy money
  • Cryptocurrency arbitrage, in theory, is simple money. It attracts new users easily
  • To my knowledge, there is no publicly accessible software that will bring anyone easy money
  • And if there is one advertising as such, it is likely to be a scam like a pyramid scheme
• Don’t assume that users that signed up for X want X’
  • In my case X is tracking arbitrage opportunities, X’ is tracking cryptocurrency portfolio. Kind of similar, and users should want X’ right? (slap in the face)
  • I had ~5k users on email list and ~200 users on telegram that signed up for tracking arbitrage opportunities. It sounds like a great market channel to offer different functionalities.
• • 
  • I assumed they might be willing to track their portfolio. That’s the functionality I need so I’ve implemented it and broadcasted many times on telegram group. No responses
• Finding the right problem to solve is crucial Articles like the one below 
• Next time, validate your ideas better. This blog post has good steps for that
  • https://www.indiehackers.com/post/instant-landing-page-killing-a-startup-idea-quickly-2099c29a0a ->
  • https://kyleplatt.com/instant-landing-page-coming-up-with-and-quickly-killing-the-idea/

Next steps after learning from mistakes

I’ve put a huge effort into building a side project. 4 years have passed and it has no recurring customers. It was and it still really is fun to develop the project. However, it feels much better to develop something that people are willing to pay for.

At this point I have to consider what to do next. From a financial perspective I don’t need to work on any side project. Working at Atlassian as senior Java software engineer at the time of writing this blog post satisfies me financially and professionally.

However, regardless of my primary income stream, I feel a strong urge to

• build a second income stream
• satisfy the need of impacting people’s lives positively
  • solving their problems with my products, content or service is an answer to that

That’s buried deep inside me and I can’t fight it.

Possible next steps ideas

1. I’ve already built a lot of software using cryptocurrency exchanges. It might be given new direction
   • Expose it as unified exchange gateway – API for other developers
   • Come back to the initial idea and improve it – half automated trading bot like greenhopper so people can run their own trading strategies
2. Drop the niche completely and focus on something else – find better problem to solve
3. Try freelancing. It might uncover some problems that might be addressed as a SaaS
4. Redesign and cleanup the core functionality (exchange gateway). Make it an open source library.
   • It’s in progress: https://github.com/mgrzaslewicz/exchange-gateway

What are your thoughts? Leave a comment below

The post Not Giving Up After 4 years, 30 One Time Customers And SaaS Failure first appeared on Deployment Every Day.

• You’re looking for a platform that will find cryptocurrency arbitrage opportunities between centralized or decentralized exchanges (a cryptocurrency arbitrage opportunities scanner). And you’re looking for an answer – can you make money from arbitrage opportunities?
• You’re a programmer and you’re thinking of creating your own arbitrage bot

Does chasing arbitrage opportunities manually still make sense?

TLDR: Not for making profit. Easy money attracts many people making it totally not easy

Earning money with arbitrage as an idea is quite simple. Let’s take a look at the example.

Imagine potato sellers offer different prices in two cities. Obviously they don’t have an unlimited amount of potatoes to sell. Each seller has 200 kg.

• In city A potatoes cost $100
• In city B potatoes cost $110
• That gives a $10 price difference which is 10%
• Let’s call it a spread

If you

• Buy 200 kg of potatoes in city A for $100
  So you spend 200 * $100 = $20 000
• Drive to city B
• Sell 200 kg of potatoes in city B for $110
  So now you have 200 * $110 = $22 000
• You’ve got a profit
• The profit is 10% of $20 000 is $2000 (minus transport cost)

Let’s assume there is a hypothetical, unrealistic arbitrage opportunity for ETH/BTC currency pair

Using ethereum and bitcoin just to have some familiar names.

• At exchanges A the price of ETH/BTC is 100 BTC for 200 ETH
• at exchange B the price of ETH/BTC is 110 BTC for 200 ETH

That means there is a 10% spread between two exchanges and that’s an arbitrage opportunity

Is it as easy as potatoes to make money with? In theory, yes  

• Buy 200 ETH at exchange A for 100 BTC
  So you spend 200 * 100 = 20 000 BTC
• Transfer ETH to exchange B
• Sell 200 ETH at exchange B for 110 BTC
  So now you have 200 * 110 = 22 000 BTC
• You’re going to have 2 000 BTC more which is pure profit.

In theory, that’s easy money assuming you found the opportunity.

Easy money attracts many people so it becomes a race of many people to use the same opportunity. However, each exchange has a limited amount of cryptocurrency.

Who gets the profit? The fastest ones.

If you’re trying to manually buy, transfer and sell you have to know that there are hundreds (thousands?) of programmers who have automated that.

What you’re doing within seconds, they are doing with milliseconds.

If you want to compete and try to earn something, you’d better create a bot and not do it by hand. And even that is very difficult.

Market is flooded with bots doing the arbitrage. You have no chances competing with machines and earn coins

How do I know? 

You can observe it yourself. Because spreads usually keep around 1%. It’s just not possible without bots intervention to keep such low spreads between exchanges with huge volumes. 

I’ve been working on an arbitrage scanner for a long time. 

I hoped people could use it and earn something. I had only one man reported earning 800 euro.

Does it make sense to make your own automated cryptocurrency arbitrage bot?

Probably not. Competition is really high. I’ve tried to create an automated one.
I’ve also contacted programmers who have been working on arbitrage bots for a long time.

If you follow this path and create your own one, you’re probably wasting your time if your main goal is to earn money.

Sign up for free at https://autocoin-trader.com so you can

• watch live arbitrage opportunities
• track your cryptocurrency portfolio

If you want to use an arbitrage scanner strictly for making money you have to know it’s really difficult. If you dedicate your time and effort, you might get something in return.

What are your thoughts? Leave a comment below

The post How much money (if any) can you make with cryptocurrency arbitrage? first appeared on Deployment Every Day.