Why would you want to have fully automated provisioning of your ubuntu workstation(s)?

• If you also love automating everything and hate doing repeatable job more than a few times, consider automating provisioning your workstation machine.
• You regularly or occasionally use many apps from apt and snap repositories.
• You’re using some tools that are not accessible via apt and snap repositories. Those tools get updates and I really wanted to consume them regularly. Manual updates were a no-go anymore as there are too many of them.
• In my case over time, the list got big enough to make it impossible to move easily to another workstation.
• Moreover, I wanted to keep portable dotfiles between machines, have a backup, easy restore and history of changes.
  What does my workstations setup look like?
  • stationary workstation 2x ubuntu on 2 different drives.
    • primary That’s where I spend most of my working time
    • fallback in case primary fails for whatever reason
  • laptop 1x ubuntu
    • I grab it when going to the office
• As you can see, there are 3 ubuntu installations. I want to switch between workstations at any time without losing updates to software, its configuration and my dotfiles.

Here goes the full list of software I’m using regularly or occasionally

• It might have evolve from the time of writing this blog post, but you get the idea. Just imagine updating those manually or providing configuration – from the scratch on a different workstation.
  I’m an experienced java/kotlin developer hence many tools are related to backend programming and debugging.
  • apt
    • bat # alternative to cat
    • hexyl # hex viewer
    • crudini # to manipulate ini files from command line easily
    • fd-find # alternative to find with some improvements
    • imagemagic
      • I generate PDFs from images from time to time, it requires tool and conf. I don’t want to remember about providing special config entries every time I change machine in /etc/ImageMagick-6/policy.xml
    • mc
    • nmap
    • oathtool # 2FA code, e.g. oathtool -b –totp ‘secret code’ | xclip -sel clip
    • paprefs # to have virtual device capable of streaming via multiple physical devices
    • python3
    • python3-pip
    • pipx
    • snap
    • speech-dispatcher # for speaking exit code in terminal of last executed command
    • tmux
    • tmuxinator
    • unrar
    • unzip
    • vagrant
    • virtualbox
    • yarn
    • zbar-tools # to scan QR codes from image or camera
  • snap packages
    • foobar2000
    • intellij-idea-ultimate
    • postman
    • signal-desktop
    • spotify
    • sublime-text
    • zoom-client
      • and its configuration to open meetings from browser links
    • yq
  • a few python packages
  • git
    • git-global-config
    • lazygit
    • custom aliases 1, 2
  • knowledge
    • cheat and cheatsheets
    • loqseq
  • zsh
    • oh-my-zsh
    • powerlevel10k
    • aliases
  • java
    • eclipse-mat
    • jdk v8, v9, v11, v14, v17, v19, v21
    • jdk-tools
    • jenv # to use JDK version set per project
      • and its zsh conf
    • mvnvm (maven wrapper)
  • docker
    • docker-registry
    • lazydocker
  • javascript
    • nvm (node wrapper) and its zsh conf
  • browser
    • brave
    • chromium
    • edge
    • firefox
    • tor
  • various
    • broot
    • delta
    • imagemagick
    • konsave
    • lsd
    • mcfly and its zsh conf
    • viddy
      zoom-client
      zoxide # smarter cd alternative
  • dotfiles
    • home-dir-as-git-repository
      • tmux
      • tmuxinator sessions for multiple projects I’m working on
        • my own
        • company I’m working for (Atlassian at the time of writing)
      • project-specific bash/zsh aliases
      • various apps settings
  • vim
    • config
    • vim-plug
    • neovim

To sum up: what’s the problem to solve?

I want to run single command to have my machine set up or updated in a few minutes.
I don’t want to be afraid of upgrading or reinstalling system. I don’t want to miss anything when changing machine I’m working on. And I want to have fun and joy of just running provision command to setup my workstation

• install all the software I need. And I need dozens of tools
• update software versions regardless of download source
• restore dotfiles

Benefits of automated workstation provisioning

• You don’t have to care about configuring manually your laptop or any other machine running ubuntu.
• You just run provision, go for a coffee and after coming back – everything is configured exactly your way
• You can throw away your machine and have the same configuration within minutes
  • It took many hours to get to this point in my case, but it’s totally worth it.
• It trains your mindset that you can apply to other projects: your machine setup is a project and list of git commits
• You know exactly what toolset you have. When you forget something, just take a look into the repository

Side effects of automated workstation provisioning

Whenever you make a change to configuration, you have to make a change in the code. That’s a approach shift

Thanks to that, manual changes do not get lost as you’re relying on automation instead 🙂

• Manual changes are extremely easy to forget about
• Manual changes can be overwritten by accident. Can means it will happen
• You’re going to forget about your own decisions in a few months from now. In git history, you can keep the why for changes

Let’s switch to solution mode: how to implement WaC?

You don’t need anything fancy for provisioning, however bunch of bash scripts is IMHO a no-go for a maintainable project. If you want a robust solution, it needs to have timeouts, retries, self verification of changes and good reporting. There is already an out-of-the-box solution for that.

ansible for provisioning workstation and keeping it up to date

Way better than tons of bash scripts. Maintainable and easy to read.

HOME dir as a git repository

Did you know you can use multiple git repositories in the same folder?

So in fact you can use your HOME dir as many git repositories

This way you can use dotfiles for as many projects as you want or need

e.g. GIT_DIR=.git_my_project git pull

• and this way you can keep your project-specific aliases in a dedicated repository

tmux + tmuxinator

• I just type tmuxinator autocoin, tmuxinator jira, tmuxinator jpt etc. and have all the right set of console tabs open. Even automatic ssh connection in a separate console tab is waiting for me

<% REPO_BASE = "~/repos/autocoin" %>
<% DATA_BASE = "~/repos/autocoin/data" %>
project_name: autocoin
project_root: <%= REPO_BASE %>
windows:
  - mediator:
      root: <%= REPO_BASE %>/autocoin-exchange-mediator
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-exchange-mediator/
        - <%= REPO_BASE %>/autocoin-exchange-mediator/scripts
        - <%= DATA_BASE %>/autocoin-exchange-mediator
  - binance-bot:
      root: <%= REPO_BASE %>/autocoin-binance-bot
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-binance-bot
        - <%= REPO_BASE %>/autocoin-binance-bot
        - ~/.trading-bot/file-repository
  - gateway:
      root: <%= REPO_BASE %>/exchange-gateway
      layout: tiled
      panes:
        - <%= REPO_BASE %>/exchange-gateway
        - <%= REPO_BASE %>/exchange-gateway/scripts
  - arbitrage-monitor:
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-arbitrage-monitor
        - <%= REPO_BASE %>/autocoin-arbitrage-monitor/scripts
        - <%= DATA_BASE %>/autocoin-arbitrage-monitor
  - auth:
      layout: tiled
      root: <%= REPO_BASE %>/autocoin-auth-service
      panes:
        - <%= REPO_BASE %>/autocoin-auth-service
        - <%= REPO_BASE %>/autocoin-auth-service/scripts
        - <%= DATA_BASE %>/autocoin-auth-service
  - frontend:
      root: <%= REPO_BASE %>/autocoin-frontend
      layout: even-horizontal
      panes:
        - <%= REPO_BASE %>/autocoin-frontend
        - <%= REPO_BASE %>/autocoin-frontend
  - xchange: <%= REPO_BASE %>/XChange
  - prod:
      layout: tiled
      panes:
        - ssh-autocoin-prod
        - ssh-autocoin-prod
        - ssh-autocoin-prod
        - ssh-autocoin-prod
  - balance-monitor:
      root: <%= REPO_BASE %>/autocoin-balance-monitor
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-balance-monitor
        - <%= REPO_BASE %>/autocoin-balance-monitor/scripts
        - <%= DATA_BASE %>/autocoin-balance-monitor
  - strategy-executor:
      root: <%= REPO_BASE %>/autocoin-strategy-executor
      layout: even-horizontal
      panes:
        - <%= REPO_BASE %>/autocoin-strategy-executor
        - <%= REPO_BASE %>/autocoin-strategy-executor/scripts
  - puppet-deprecated:
      root: <%= REPO_BASE %>/autocoin-puppet
      layout: tiled
      panes:
        - <%= REPO_BASE %>/autocoin-puppet
        - <%= REPO_BASE %>/autocoin-infrastructure/machine-developer
  - infrastructure-provision:
      root: <%= REPO_BASE %>/autocoin-infrastructure-provision
      layout: even-horizontal
      panes:
        - <%= REPO_BASE %>/autocoin-infrastructure-provision
        - <%= REPO_BASE %>/autocoin-infrastructure-provision
  - db: <%= REPO_BASE %>/autocoin-db
  - metrics-client: <%= REPO_BASE %>/autocoin-metrics-client
  - telegraf-influx-grafana: <%= REPO_BASE %>/autocoin-tig-monitoring
  - autocoin-data: <%= REPO_BASE %>/data

Lessons learned

ansible

• It does its job quite good for linear tasks. However I would not recommend using it for more advanced problems like running things in parallel to speed up provisioning within a single machine. YML is not meant for advanced programming
• I haven’t used ansible before starting ubuntu-workstation-provision project. Ansible documentation is quite impressive and has good examples. It’s easy to learn.

Bash installation scripts

• Some tools on github have a installation script e.g. wget <link> | bash. It’s a bit risky as the script might do anything or get hacked, but I accept the risk in favor of convenience.
  On the other hand I applied similar approach to provide easy dotfiles installation for any project and that’s the code I’m in control of – so no risk. Example here so you can create something similar on your own

Fail fast with fresh virtualbox machine

Running provisioning on a fresh virtualbox machine uncovers some problems and I run provisioning of such machines from time to time. Here’s an example how to use ansible with vagrant

Vagrant.configure("2") do |config|
config.vm.box = "ubuntu/noble64"
config.vm.provision "ansible" do |ansible|
  ansible.playbook = "/repos/ubuntu-workstation-provision/playbooks/provision.yml"
  #ansible.skip_tags = "graphical"
end
end

Things to avoid

Try not to parse 3rd party HTML to get software versions if there is no API providing latest version

• HTML is not an API and your automation will break at any time
  • You’re going to have more maintenance work
  • Example regarding zoom installation
    • Initially I used zoom-client installation via snap as there is no zoom in apt repositories
    • However it was failing to open zoom meeting via clicking on a link in browser 🥹
    • So I moved to installing .deb package downloaded from official zoom site
    • Version was buried in the HTML code so I parsed it with regex ⛔️
    • I finally had working zoom meeting links 😀
    • And one day HTML on zoom site was changed, parsing latest version failed 🥹
    • So I switched back to installing zoom via snap and zoom links were broken again 🥹
    • And I provided proper KDE configuration to make zoom links work as expected 😀

Yet another experience strengthening belief that automation allowing to throw away and recreate things wins over keeping state and list of steps

• Approach with manual steps that you run or delegate to other developers is
  • not scalable
  • not maintainable
  • hard to reproduce by others (and ==you== in the near future)
  • error prone
    • sooner or later you or other developer will make a mistake. Period. I’ve seen it dozens of times in many projects. It also relates to installing software and configuring it.
• Where else can you see the power of throwing away and recreating easily? A few examples
  • Docker images
    • They have a recipe that creates an image. If you rely on manually created accessible in some repository, butone with no recipe (no Dockerfile) you will get into trouble
  • Test datasets
    • In the performance team we missed AWS bucket deadline and lost our test datasets. No one new how those were created and people who created them were no longer working in the company. We were fucked for a while and had to create automation to recreate them anyway.
  • Browser tabs
    • Why bother keeping all open for your project?
      It’s easy to lose it and have fear of closing the tabs. Just use session manager to save the session, reopen it at any time and have the state you want again. Then you won’t have to keep dozens of browser tabs open
  • Performance testing
    • Create a process every developer in the company can run easily, in a perfect world with a single command. Now compare that to list of 20 steps to run manually listed on confluence page🤦🏻‍♂️
  • Example from workspace-provision project
    • Manual eclipse-mat download takes 2 minutes if just unzipped and run. However during manual installation, I’ve discovered with trial and error that actual steps to have it working goes as follow:
      • 1) download eclipse-mat
      • 2) unzip it
      • 3) move it to /opt directory
      • 4) set -Xmx4g in MemoryAnalyzer.ini. Default heap size is too low and I will have to do it anyway after download
        • Why? I already had it consuming 100% CPU and taking ages to analyze heap dump. GC was running all the time
      • 5) set it up to use java version 17
        • Why? It’s the minimum required by eclipse-mat. I have multiple JVMs so there is no best default for the whole system. It just didn’t want to start because of too low default java version
      • 6) create symlink /user/local/bin/mat to run it easily from command line or krunner
    • It’s very likely I will forget step 4 and 5 when installing newer version or trying to run it on another ubuntu (including other developer’s machine)
    • Having it automated means I can send someone a command to run
      • git clone https://github.com/mgrzaslewicz/ubuntu-workstation-provision.git
      • TAGS=eclipse-mat ./provision
      • and it’s
        • scalable
          • n other developers can run it without asking me for anything
        • maintainable
          • fix needed? No problem, just change the code
        • easy to reproduce by others
          • no brainer, just copy paste
        • error proof
          • well, you can always make an error while copying and pasting 🥹

Automated windows provisioning?

• Nowadays I use windows to play games only, but I used to use it as a workstation too so I had some attempts for provisioning automation.
• Take a look at chocolatey. It’s a windows package manager so it’s easy to install most of needed software automatically. It can be used with ansible too. And that’s the simple provisioning script using chocolatey I used to use.

Resources summary

• set of alternative/better/fresh unix tools: github.com/ibraheemdev/modern-unix
• WaC implementation: github.com/mgrzaslewicz/ubuntu-workstation-provision
  • it’s using also HOME dir as git repo

What does your WaC look like?

Have you already tried implementing your own workstation as a code? What have you learned, what worked well, what were you struggling with?