David Runger : Blog

Easy PostgreSQL major version upgrades for a Docker Compose hobby app

2026-01-22T00:00:00-06:00

Upgrading PostgreSQL major versions in Docker doesn’t have to be scary. For small hobby apps, a simple dump-and-restore process can be the safest and most understandable method.

Below is a step-by-step runbook that I use when upgrading PostgreSQL to a new major version for my hobby app, which is managed using Docker Compose. The runbook is a sequence of commands that I paste into an SSH session one at a time, checking results after every step.

This approach has worked well for my small app, and this post walks through the workflow and explains why each step exists.

Caveats

This upgrade process works well for my small hobby app, but it does come with tradeoffs.

It requires some downtime (hopefully just a few minutes), involves manually running commands directly on the server (somewhat violating strict “infrastructure as code” practices), and assumes a relatively small database (mine is currently ~122 MB). For much larger databases or stricter uptime requirements, this approach is likely too slow and too manual.

But if you’re in a similar position — running a personal app and looking for a straightforward, low-risk way to do Postgres major upgrades — this process might be useful to you.

Assumptions

This workflow assumes:

Postgres runs via Docker Compose
Your database data lives in a named Docker volume
You’re OK with downtime during dump + restore
You can SSH into the server and run commands interactively

You’ll need to adapt service names, database names, and volume names to your specific setup.

High-level strategy

This upgrade process will:

Verify the current database state
Take a logical backup
Start a brand-new Postgres major version with a fresh data directory (new volume)
Restore the backup to the new Postgres database
Verify correctness
Delete the old data only after everything looks good

The old database remains untouched until the very end, which makes rollback trivial.

Check application compatibility first

Before starting the upgrade process, verify that your application is compatible with the new Postgres major version. Review the Postgres release notes for breaking changes or deprecated features that might affect your app and run your test suite against the new version.

Step-by-step upgrade runbook (Postgres 17 to 18 example)

Each step below is intended to be copy/pasted individually on your server machine, with you verifying output before moving on.

0. Make sure your deployment config is ready

Before touching production, make sure that a PR to update your Postgres image and volume name (with the exact same changes seen in the sed commands below) is ready to merge. In the steps below, we will apply changes temporarily to upgrade, then revert to keep the repo clean, then merge a PR to make them permanent.

# Make sure that PR to update docker-compose.yml to Postgres 18 is ready to merge.

1. Check current state

git status
git show

docker compose exec postgres psql -U app_user app_production -c 'SELECT VERSION();'

docker compose exec postgres psql -U app_user app_production -c \
  'SELECT COUNT(*) FROM users; SELECT * FROM orders ORDER BY created_at DESC LIMIT 1;'

This gives you a concrete baseline for later verification.

2. Create an off-server backup (optional, but recommended)

If you have an existing backup pipeline (e.g. S3 snapshots), now is the time to run it:

bin/backup-to-s3.sh

This gives you an escape hatch, even if the server melts down mid-upgrade.

3. Update Docker config to the new Postgres version and volume

Edit docker-compose.yml so that:

The Postgres image moves from postgres:17.x to postgres:18.x
The data volume moves from postgres-data-v17 to postgres-data-v18

Changing the volume name ensures the new Postgres version starts with a completely empty directory, avoiding errors like “The data directory was initialized by PostgreSQL version 17, which is not compatible with this version 18.0”.

For example:

sed -i'' 's/postgres:17.6-alpine/postgres:18.0-alpine/g' docker-compose.yml
sed -i'' 's/postgres-data-v17:/postgres-data-v18:/g' docker-compose.yml

Then review:

git status
git diff

Nothing should be changed except the Postgres image tag and volume name in docker-compose.yml.

4. Pre-pull the new Postgres image

This reduces downtime later (since we won’t waste time downloading the new Postgres Docker image while our app is down):

docker compose pull postgres
docker images postgres

5. Stop services that talk to the database or serve the web app

docker compose stop web worker nginx

At this point, nothing should be writing to Postgres.

6. Create a logical backup from the old database

We use pg_dumpall to capture everything, including global objects (such as roles and permissions). This creates a portable SQL script that works across Postgres major versions.

⚠️ Permission note: pg_dumpall requires superuser privileges for complete dumps. To check if your app user is a superuser, run this command (substituting your Postgres app user name for app_user):

docker compose exec postgres psql -U app_user -c \
  'SELECT rolsuper FROM pg_roles WHERE rolname = current_user;'

If this returns true/t, then you can proceed.

docker compose exec postgres pg_dumpall -U app_user > backup.sql
head -5 backup.sql | grep -q "PostgreSQL" && echo "✓ Backup format looks correct"
ls -lh backup.sql

Confirm the file size looks roughly correct before continuing.

7. Shut down the old Postgres container

docker compose down postgres
docker ps

You should no longer see the Postgres container running.

8. Start Postgres on the new major version

docker compose up --detach postgres

Then verify:

docker compose exec postgres psql -U app_user -c 'SELECT VERSION();'

You should now see PostgreSQL 18.x.

9. Restore the backup

docker compose exec --no-TTY postgres psql -U app_user < backup.sql

This puts the data into the new Postgres database.

10. Verify the data

docker compose exec postgres psql -U app_user app_production -c \
  'SELECT COUNT(*) FROM users; SELECT * FROM orders ORDER BY created_at DESC LIMIT 1;'

You should see the same results as before the upgrade.

11. Restart application services

docker compose up -d web worker nginx

Now verify that your web app loads and that database writes succeed (e.g. fill out a form to create a test record).

12. Clean up

Once you’re fully confident:

rm backup.sql
git checkout docker-compose.yml
docker volume rm app_postgres-data-v17

13. Merge PR that commits the above changes

Merge and deploy the PR that you prepared in Step 0, committing to version control the same changes that you temporarily made above.

14. Post-deploy sanity check

After the PR has deployed to your server, make sure that the Postgres version is still the new major version and confirm that the data is still what you expect.

docker compose exec postgres psql -U app_user app_production -c 'SELECT VERSION();'
docker compose exec postgres psql -U app_user app_production -c \
  'SELECT COUNT(*) FROM users; SELECT * FROM orders ORDER BY created_at DESC LIMIT 1;'

At this point, the upgrade is complete.

Rolling back

Rollback is possible at any point before executing docker volume rm app_postgres-data-v17 in Step 12.

If anything fails before deleting the old volume:

docker compose down
git checkout docker-compose.yml
docker compose up -d postgres
docker compose up -d web worker nginx

You’re instantly back to the original database.

When this approach is a bad fit

You probably want to use pg_upgrade or do a replication-based cutover if your database is very large and/or absolutely minimizing downtime is important.

But, for a small hobby app, this approach is pretty safe, easy, and simple for an app whose Postgres database is managed with Docker Compose, which can make Postgres major version upgrades relatively complicated.

Takeaways

If you want Postgres major upgrades that are:

Low-risk
Easy to roll back
Easy to reason about
Easy to repeat every few years

… then this pattern has worked well for me. It’s not the fastest possible approach, but it’s pretty safe and pretty easy.

I hope that this might be helpful to you!

Appendix: a complete, real example script

I use this exact process to upgrade Postgres major versions in production for the davidrunger.com database. You can see the current version of the playbook here (and also my docker-compose.yml).

The playbook includes:

My service names
My deployment flow
My verification queries
My volume naming scheme

Seeing that concrete example for my specific app might help you to understand how you might need to adjust the commands above to fit your own app.

Automatically compiling Crystal for development tooling

2024-08-08T00:00:00-05:00

The importance of good tooling

Having good tooling can be an important part of the software development process. Good tooling can help a developer to operate more quickly/efficiently, and also to get into a flow state. If one’s tooling can take care of some of the more mundane aspects of software development, then more of that developer’s time, energy, and uninterrupted focus can be spent on the higher-value aspects of software delivery.

Custom tooling

Lots of great tooling is available off the shelf, and it’s usually best to leverage the work done by others, when it’s possible to do so, rather than reinventing the wheel.

However, there are some tools one will want that are unique to one’s idiosyncratic workflow, or, for whatever other reason, off-the-shelf tooling might not be available to meet a given need. In these cases, happily, as software developers, we can write our own tooling! I often do.

A lot of the tools that I create for myself are programs that I execute from a terminal, or which are invoked indirectly by some other command that I execute in a terminal.

For example, I often want to pull updates from a GitHub repository down to my local machine, and then rebase my branch onto that updated version of the main branch. I do this by executing in my terminal a command that I’ve written called gform (which stands for “git fetch origin and rebase with main”).

In addition to updating my branch with the latest version of the main branch, this gform command also executes another program that I’ve written, called install-packages-in-background, which looks at the project’s dependency lock files (the Gemfile.lock, pnpm-lock.yaml, etc) and checks whether the relevant package installation command (e.g. bundle install or pnpm install) has ever been executed on my machine for the current version of the dependency lock file. If not, then my install-packages-in-background script will execute the relevant package installation command (in the background).

Which language to use?

Which computer language should install-packages-in-background be written in? Since it’s just a command that I am running on my local machine, I could write it in pretty much any language that I can install on my machine.

Consideration: startup time

As a primarily Ruby and JavaScript developer, I always have those languages installed on my development machine, so those are options. However, one downside of these languages is that they have a somewhat noticeable startup time. Running about the most minimal Ruby program imaginable takes over 130 milliseconds on my machine:

❯ time ruby -e 'puts("Hi!")'
Hi!
ruby -e 'puts("Hi!")'  0.09s user 0.04s system 99% cpu 0.132 total

Node is even a little bit slower, taking about 200 ms to start up:

❯ time node -e 'console.log("Hi!")'
Hi!
node -e 'console.log("Hi!")'  0.18s user 0.07s system 122% cpu 0.205 total

A few hundred milliseconds might not seem like a lot, but it adds up, and the time spent waiting for a relatively slow Ruby or Node program to execute risks interfering with one’s development flow state.

Bash: the fast solution?

Another (and typically much faster-executing) option is to use a shell scripting language, such as bash. And, indeed, I do write the vast majority of my little development tooling scripts in bash. Bash starts up much more quickly than Ruby or Node:

❯ time bash -c 'echo "Hi!"'
Hi!
bash -c 'echo "Hi!"'  0.00s user 0.00s system 89% cpu 0.006 total

Just six milliseconds!

Bash: not fun for complicated logic

However, while bash can do a lot, I don’t find it very pleasant to work with when it comes to performing any sort of semi-complicated logic.

Bash: no package system to lean on

Additionally, the bash scripting language doesn’t really have a package management framework, so we don’t really have the ability to easily leverage the work of others. Our bash scripts must be pretty self contained and do everything themselves. Bash scripts can call out to other programs, but there aren’t generally libraries that we can load into our script to provide useful functionality.

What about Lua?

Another language that I experimented with recently is Lua. It has a fast startup time that is comparable to bash’s (just a few milliseconds) and a package management system (LuaRocks). I was hopeful that I’d find the language/syntax more pleasant and natural to work with than bash.

However, after trying Lua, I found it’s standard library to be quite limited. As a result, I had to manually implement some basic functionality (such as merging two dictionaries) that I had hoped/expected would be built into the language.

Could we use a compiled language?

All of the languages that I have named so far are interpreted languages (as opposed to compiled ones). I never thought that a compiled programming language would work well for writing the sort of developer tooling programs that I’ve been discussing, for the simple reason that I’d have to remember and bother with recompiling the program into a binary whenever I make a change to the program, and I’d also need to make that compiled binary available as an executable program on my PATH (so that they can be invoked easily from the command line or from other programs).

In contrast, when writing a program in an interpreted language, all that I have to do is to put the program’s source code in a location on my PATH. Then, whenever I edit the program source code, that new version of the program is immediately “live” on my system, without any additional steps on my part. The relative hassle of a compiled language didn’t seem worthwhile to me.

The big idea: automatically compiled programs

However, that all changed, when ChatGPT and I collaborated on a way to automatically compile and make available on my PATH the executable binary output of compiled development tooling programs. This opened up a whole new world of compiled programming language options that I could now consider for writing development tooling programs, without the hassle that had made me shy away from compiled languages for this purpose up until now.

My go-to compiled language: Crystal

The compiled language that I decided to try out first as a language in which to write development tooling is Crystal. Crystal is a compiled language with a syntax that is very similar to Ruby (with the main difference being that Crystal sometimes requires type annotations). Crystal programs are extremely fast, often comparable to (or even faster than) a raw C implementation. Crystal programs also use much less memory than an equivalent Ruby program would.

Crystal also has extremely helpful and well formatted error messages, and the documentation is useful and easy to read.

Overall, I find it a pleasure to work with, and I feel lucky that there exists a language with the ease and beauty of Ruby and yet also with the speed of C and light memory usage.

These concepts apply to any compiled language

The focus of this blog post is not about Crystal, though. What I want to focus on is the framework that ChatGPT and I came up with to automatically compile my Crystal development tooling programs and make them available on my PATH. Indeed, there is very little that is Crystal-specific in what I’m about to share, and I think that this framework/process could easily be adapted to almost any other compiled language, such as Rust or Go.

Example: my `unique-union` program

To illustrate concretely, let’s look at one particular program that I have written in Crystal, called unique-union. This program takes two arguments, and prints the set of words in those arguments, deduplicated and sorted. Example:

❯ unique-union "wave hello bye" "hello ocean wave"
bye
hello
ocean
wave

(This might seem kind of pointless, but it’s useful within one of my other tools.)

That functionality is implemented in Crystal as such:

#!/usr/bin/env crystal

# file location: ~/code/dotfiles/crystal-programs/unique-union.cr

def unique_union(set1 : String, set2 : String) : Array(String)
  set1_words = set1.split
  set2_words = set2.split

  (set1_words | set2_words).sort
end

if ARGV.size == 2
  unique_union(ARGV[0], ARGV[1]).each { |item| puts(item) }
else
  puts "Usage: unique-union \"first list of words\" \"second list of words\""
  exit(1)
end

Setting up automatic Crystal compilation

What we want is some way to automatically convert that Crystal source code into an executable binary.

Part 1: `symlink-crystal-programs`

Part of the trick comes from adding this line to my ~/.zshrc file:

# Set up (in the background) symlinks for programs written in Crystal
{ ( symlink-crystal-programs >&3 & ) } 3>&1

Most of that syntax is just to make the command execute without printing any output, and to do so in the background (so that it doesn’t slow down my shell startup time). The key point is that, whenever I open a new terminal tab, symlink-crystal-programs will execute, which is the following bash program.

It iterates over all of the Crystal source code files in my crystal-programs source directory, and, for each program, creates a symlink that is available in my PATH and which points to another bash program (run-crystal-program) that will automatically compile the Crystal source code into a binary, as needed.

The end result will look like this:

❯ tree ~/bin/crystal-symlinks
/home/david/bin/crystal-symlinks
├── install-packages-in-background -> /home/david/code/dotfiles/bin/run-crystal-program
├── open-pr-in-browser -> /home/david/code/dotfiles/bin/run-crystal-program
├── runger-config -> /home/david/code/dotfiles/bin/run-crystal-program
└── unique-union -> /home/david/code/dotfiles/bin/run-crystal-program

Here’s the script that does it:

#!/usr/bin/env bash
# file name (available on PATH): symlink-crystal-programs

crystal_programs_source_code_directory="$HOME/code/dotfiles/crystal-programs"
crystal_executable_symlinks_directory="$HOME/bin/crystal-symlinks"

# Delete the symlinks directory, to ensure that there aren't any dangling programs left there.
rm -rf "$crystal_executable_symlinks_directory"

# Recreate the symlinks directory.
mkdir -p "$crystal_executable_symlinks_directory"

for crystal_program_source_file in "$crystal_programs_source_code_directory"/*.cr ; do
  symlink_name=$(basename "$crystal_program_source_file" .cr)
  ln -sf ~/code/dotfiles/bin/run-crystal-program "$crystal_executable_symlinks_directory/$symlink_name"
done

The symlinks drop the .cr extension. That way, although I write the source code in a file called unique-union.cr, I will be able to invoke the compiled program from my command line (or other programs) via simply unique-union.

Part 2: `run-crystal-program`

The final piece of the puzzle is the run-crystal-program bash script referenced as the target of the symlinks above. That script does two main things:

compiles (or recompiles) the relevant Crystal source code into an executable binary, if no binary has yet been compiled, or if the source code or the run-crystal-program script itself has been modified more recently than the binary was compiled
executes the compiled binary, forwarding along any arguments

#!/usr/bin/env bash
# file location: ~/code/dotfiles/bin/run-crystal-program

crystal_compiled_binaries_directory="$HOME/bin/crystal-binaries"

script_name=$(basename "$0")
source_file="$HOME/code/dotfiles/crystal-programs/$script_name.cr"
binary_file="$crystal_compiled_binaries_directory/$script_name"

# Check if the binary doesn't exist, the source file has changed, or this
# compilation script has changed.
if [ ! -f "$binary_file" ] || \
    [ "$source_file" -nt "$binary_file" ] || \
    [ "$(realpath "$0")" -nt "$binary_file" ] ; then
  # Create the compiled binaries directory.
  mkdir -p "$crystal_compiled_binaries_directory"

  # Add shards directory for dotfiles to the CRYSTAL_PATH.
  # More details: https://github.com/davidrunger/dotfiles/commit/d73a9df .
  export CRYSTAL_PATH="$HOME/.shards/dotfiles:$CRYSTAL_PATH"

  # Compile the binary.
  echo "Compiling $source_file ..." >&2
  if ! crystal build --warnings=none "$source_file" -o "$binary_file" ; then
    echo "There was an error compiling $source_file ." >&2
    exit 1
  fi
fi

# Execute the compiled binary, passing along any provided arguments.
"$binary_file" "$@"

Review and overview

This all might seem a little bit complicated, and it is. There are three different directories involved, each with a different purpose:

~/code/dotfiles/crystal-programs/: This is where the Crystal source code lives, e.g. ~/code/dotfiles/crystal-programs/unique-union.cr.
~/bin/crystal-symlinks/: This directory (which is on my PATH) contains symlinks, one for each Crystal program. These symlinks all point to the run-crystal-program script.
~/bin/crystal-binaries/: This is where the actual compiled binaries created from the Crystal source code are stored. These compiled binaries will be invoked by run-crystal-program.

So, when I invoke unique-union from my command line, that refers to the ~/bin/crystal-symlinks/unique-union file (since ~/bin/crystal-symlinks is on my path). That file is actually just a symlink to the shell script ~/code/dotfiles/bin/run-crystal-program, so that is what actually executes when invoking unique-union. run-crystal-program ensures that there is an up-to-date compiled binary located at ~/bin/crystal-binaries/unique-union, and then executes that binary, passing along any arguments.

Downsides

Overall, after ironing out a kink or two, this system seems to work pretty smoothly, but there are some downsides.

One downside is that, after I change any Crystal source file, then the next time that I invoke that command, there is a significant delay (a few seconds), as run-crystal-program compiles an up-to-date version of the executable binary. However, thereafter, run-crystal-program simply invokes the already-compiled binary, and so the compiled program executes quickly.

Another downside is that, even when an up-to-date binary has already been compiled, this system does waste a little bit of time executing the run-crystal-program bash script, which probably adds somewhere on the order of 8 milliseconds or so to the overall execution time. It would be faster if, instead of going through the run-crystal-program bash script, calling unique-union would directly invoke the compiled Crystal binary. However, then I’d lose the benefit of automatic compilation and the assurance that I’m always running a binary that has been compiled using the latest version of the Crystal source code.

Summary: I’m happy 🙂

For me, the benefits of this framework outweigh these downsides, and I really enjoy being able to write some of my development tooling programs using Crystal, and then executing the resulting, automatically compiled, quick, and low-memory compiled binaries.

Flaky specs due to ActionCable leakage

2024-07-25T00:00:00-05:00

A spec setup at risk of flaking

If the following are all true of your application, then your tests might flake (i.e. fail randomly).

You run tests in parallel (locally and/or in CI)
Your application uses ActionCable with the Redis adapter
Your parallel tests use the same Redis instance (even if different Redis database numbers)
You don’t have a channel_prefix for the test environment in your config/cable.yml, or the channel_prefix doesn’t have any dynamic interpolation

Why this setup causes flakiness

This test setup causes flakiness because Redis’s publish/subscribe messaging system is global to a Redis instance. This creates a risk of unintentional interaction between your specs. If Spec A triggers an ActionCable update to be broadcast, then that ActionCable update might be received in some other, simultaneously executing Spec B. If Spec B is a system spec or feature spec, for example, then this ActionCable update might cause a change in the data in the browser, and this unanticipated change to the browser state might cause the spec to fail.

In a case like this, uncovering the cause of the flakiness can be really tricky, because — no matter how hard you look through the code of Spec B — there is nothing there that gives a hint about why the browser UI sometimes seems to update in a seemingly random way that causes the spec to fail. This is because the thing that’s causing the unexpected state change actually originates in the code of an entirely different spec. What’s more, it’s probably pretty random which specs happen to execute at the same time in any given test run. This makes it challenging to reproduce the flakiness/failures with any reliability, adding to the difficulty of investigating the cause of the flakiness.

The solution: use a `channel_prefix` that is unique to each test process

Fortunately, ActionCable includes a configuration option — channel_prefix — that can prevent this problem, by effectively keeping the ActionCable broadcasts issued by one Rails/RSpec process from being received by tests that are executing in another Rails/RSpec process.

In my case, the solution looked like changing the test section of my config/cable.yml from this:

test:
  <<: *default
  channel_prefix: david_runger_test
  url: redis://localhost:6379

to this:

test:
  <<: *default
  channel_prefix: david_runger_test<%= ENV['DB_SUFFIX'] %>
  url: redis://localhost:6379

Note the addition of the <%= ENV['DB_SUFFIX'] %> ERB interpolation tag at the end of the channel_prefix. In my CI test setup, each of the RSpec processes (some of which might execute simultaneously) is provided with a different DB_SUFFIX environment variable (such as _unit, _api, _feature, etc.). I’m leveraging that environment variable to ensure that the channel_prefix is distinct for each test process, meaning that their ActionCable broadcasts will remain isolated from each other, even when executing in parallel.

Here’s some relevant Rails documentation about this feature: Redis Adapter.

Note: Using distinct Redis database numbers will not work

I initially tried to fix this problem in a different (but conceptually similar) way, by using a different Redis database number for each RSpec process. However, that didn’t work, because Redis’s publish/subscribe functionality (the basis for the ActionCable functionality) is global to all of the databases of a given Redis instance.

Enforce Git hooks in a Rails initializer

2024-07-02T00:00:00-05:00

Git hooks

Git has a cool feature called Git hooks:

Git Hooks are scripts that Git can execute automatically when certain events occur, such as before or after a commit, push, or merge.

For example, I have a pre-push Git hook script that I want to be run before I push any code up to GitHub for my website’s repository. This script performs various checks, such as using Gitleaks to scan the diff for any secrets (like an API access token) and to abort the push if any such secrets are found. The pre-push hook also runs various linters.

As long as everyone who works on the repository configures their local Git setup to use the repository’s Git hooks, then Git will run that pre-push check and verify that it passes before Git will actually push any code to GitHub.

But, how to enforce that Git hooks are used?

The problem is that, although we can put Git hook scripts in a repo, we cannot enforce that developers actually use them. The nature of git hooks is that one must “opt in” to use them in the configuration of each local copy of a repository, and it’s easy to fail to do so. For example, I recently set up a Linux boot on my MacBook. I had previously set up the git hooks for the version of my repository in my Mac operating system, but I didn’t think to also set up the git hooks after I then cloned the repository to my fresh Linux OS.

Frequently, a repository’s README.md or setup documentation might say something like, “We encourage you to set up this repository’s Git hooks, by executing [a certain command],” but developers might ignore or accidentally overlook that instruction, or at some later point they might lose their git configuration and not think to reconfigure the git hooks.

Idea: enforce Git hooks configuration in a Rails initializer

One solution to this conundrum recently occurred to me: put a check in a Rails initializer to verify that the project’s git hooks have indeed been configured. If not, then refuse to boot the app.

I should note that, unfortunately, this is not a 100% foolproof way to ensure that developers configure their git setup to use a project’s git hooks. A developer could theoretically modify the project’s source code without ever successfully booting up the app. But, realistically speaking, I think that’s quite unlikely, at least for a developer who is going to do any substantial amount of work on an application. They’re going to need to boot up the app sooner or later.

The solution presented below is nominally Rails-specific (in that it leverages the fact that Rails will automatically load during the boot process any .rb file in the config/initializers/ directory), but it should also be very possible to translate this general approach to other web frameworks (or other types of applications), by hooking into the local development boot process in a similar way.

The initializer

Here’s what my relatively simple initializer looks like (with some minor tweaks):

# File: config/initializers/githooks_check.rb
if (
  Rails.env.local? &&
    ENV['SKIP_GITHOOKS_CHECK'].blank? &&
    ENV['CI'].blank? &&
    `git config core.hooksPath`.strip != 'bin/githooks'
)
  $stderr.puts(<<~ERROR)
    You have not configured the git hooks for this repo! To do so, run:
        git config core.hooksPath bin/githooks
    Or, if you must, you can put SKIP_GITHOOKS_CHECK=1 in your .env file.
  ERROR

  exit(1)
end

(You can see the actual source code here.)

So, if one tries to boot up the Rails app (e.g. rails server) without having locally set up the repository’s Git hooks, then an error message like this will be printed to stderr, and the app will abort the boot process:

❯ bin/rails server
=> Booting Puma
=> Rails 7.1.3.4 application starting in development
=> Run `bin/rails server --help` for more startup options
You have not configured the git hooks for this repo! To do so, run:
    git config core.hooksPath bin/githooks
Or, if you must, you can put SKIP_GITHOOKS_CHECK=1 in your .env file.
Exiting

Breaking it down

Here’s an explanation of each part of the initializer:

Rails.env.local? checks that we are booting in either the development or test environment (i.e. not production, where Git hooks won’t be configured).
ENV['SKIP_GITHOOKS_CHECK'].blank? gives developers a way to skip this check, if that is necessary for some reason, by setting this environment variable.
ENV['CI'].blank? checks that the app is not booting in our Continuous Integration (CI) environment (GitHub Actions), because, like production, our Git hooks aren’t configured there.

If those conditions are all true, then we perform the key part of the check:

`git config core.hooksPath`.strip != 'bin/githooks': This checks if the local git repository has not been configured to use the project’s bin/githooks/ directory as the Git hooks path, and, if it hasn’t, then we enter the body of the if condition.

Within the if block, we print a warning stating that the developer doesn’t have the repository’s Git hooks configured, and providing a command to configure the git hooks. Finally, we abort the boot process, via exit(1). The developer won’t be able to boot the app successfully until they make some sort of change (such as, ideally, configuring the git hooks on their machine, using the provided command).

Configuring the Git hooks

Remedying that situation is as simple as executing the provided command at the top level of the project directory:

❯ git config core.hooksPath bin/githooks

That tells Git to look in the repository’s bin/githooks/ directory for relevant scripts before and after performing key actions. For example, Git will look for (and run) a script called pre-push before pushing, or a pre-push script before committing.

Having configured the Git hooks, the rails server will now boot successfully!:

❯ bin/rails server
=> Booting Puma
=> Rails 7.1.3.4 application starting in development
   [ ... ]
* Listening on http://127.0.0.1:3000
* Listening on http://[::1]:3000
Use Ctrl-C to stop

And, when the developer goes to push their code changes up to GitHub, then all of the repository’s intended pre-push checks will execute, guarding against secrets being accidentally committed to the source code, and helping to keep code quality high by running several of the project’s linting tools.

Using VS Code as a Rails app:update merge tool

2024-06-26T00:00:00-05:00

Background: the `rails app:update` command

As part of bumping the major or minor version of a Rails app, one should execute the rails app:update command (docs). This command is intended to help maintainers of existing Rails applications to update their application configuration files as appropriate to accommodate changes that have been made to Rails’s configuration files and changes of the configuration defaults in the newer version of Rails.

This command will compare, one by one in an interactive console session, the content of the configuration files (such as config/application.rb, config/environments/production.rb, etc.) that would be generated for a brand new Rails app versus the current actual content of those same files in the existing application that is being upgraded.

The developer performing this upgrade will want, in many cases, to keep some parts of their existing application configuration, including some departures from the Rails default configuration that was provided by the initial rails new command that generated the app. In other words, most application configurations will contain some adjustments versus the out-of-the-box Rails defaults, and many of the reasons for those adjustments will still be relevant and appropriate, even after upgrading to the newer version of Rails.

However, for some aspects of a Rails application’s configuration, a developer might want to make some additions, changes, or deletions, to reflect newly available configuration settings that have been added in the new version of Rails, a change that has been made to the default value of a certain configuration, or some other change in the framework.

Using a merge tool will make this task easier

Comparing and understanding the differences between the new default configuration files that Rails would generate for a brand new application versus the current state of the configuration files for an existing application (generated by an earlier version of Rails) is not an easy task, nor is resolving those differences to create a final version of the configuration files that will be used by the app going forward, synthesizing both the old and the new, but using a merge tool as part of this process can be a big help.

Running `rails app:update`

When you execute the rails app:update command, then for each configuration file where there is a difference between what Rails would generate for a new application versus the app’s existing configuration, the tool will note that there is a conflict, and it will wait for input about what to do:

❯ bin/rails app:update
    conflict  config/boot.rb
Overwrite /home/david/code/david_runger/config/boot.rb? (enter "h" for help) [Ynaqdhm]

Entering h for help tells us what the options ([Ynaqdhm]) stand for:

Y - yes, overwrite
n - no, do not overwrite
a - all, overwrite this and all others
q - quit, abort
h - help, show this help
d - diff, show the differences between the old and the new
m - merge, run merge tool

Either totally overwriting the existing configuration file with the default contents for a brand new Rails app (Y) or simply ignoring the new configuration changes and just keeping the existing config (n) would be easy options, but doing that risks, respectively, losing some existing configuration that we want to preserve or missing out on relevant new changes in the Rails framework.

Although it’s more work than those simple options, in my opinion, the best option by far is the last one: m - merge, run merge tool. However, if we choose this, we’ll get an error message (and then will be re-prompted to choose again):

Overwrite /home/david/code/david_runger/config/boot.rb? (enter "h" for help) [Ynaqdhm] m
Please specify merge tool to `THOR_MERGE` env.
Overwrite /home/david/code/david_runger/config/boot.rb? (enter "h" for help) [Ynaqdhm]

So, let’s exit the app:update interactive session by hitting Ctrl-C, and, as the error message suggests, we’ll provide a THOR_MERGE environment variable, which will tell the app:update command which program to use as a merge tool to reconcile the diff between our app’s current configuration file and the version of the configuration file that would be generated for a fresh app by the new version of Rails to which we are upgrading.

Since I already use VS Code as my primary code editor (including using its merge editor to resolve git conflicts), I already have it installed and running, and I’m familiar with how it works. So, I like to use VS Code as the merge tool for rails app:update. However, it’s not immediately obvious how to do this. The next section explains how, by using a relatively simple bash script as a sort of “adapter”.

How to use VS Code as an `app:update` merge tool

Background: `code --merge`

The code --help output includes this:

❯ code --help

[…]

-m --merge <path1> <path2> <base> <result> Perform a three-way merge by providing paths for two modified versions of a file, the common origin of both modified versions and the output file to save merge results.

Background: `rails app:update` and `THOR_MERGE`

For each configuration file that needs to be updated, the rails app:update command will provide two file paths to the THOR_MERGE tool:

The first argument will be the path of a temporary file that will contain the version of the configuration file as Rails would generate that file for a brand new application.
The second argument will be the path of that configuration file in our existing app.

Putting it together

So, the THOR_MERGE program that we use will receive from the app:update tool only two files, but, as seen in the code --help output above, VS Code’s --merge functionality needs to be provided with four files, which it calls <path1>, <path2>, <base>, and <result>.

The trick is that we will create copies of one of the files provided by app:update, and we’ll use these copies as the additional file arguments required by code --merge. To accomplish this, we can write a bash script, which we’ll provide as the THOR_MERGE tool. You can put this bash script anywhere. You might or might not want to commit the script to your repository. I’ll create the file at bin/app_update_thor_merge_tool in my Rails app, and we also need to make it executable:

touch bin/app_update_thor_merge_tool
chmod +x bin/app_update_thor_merge_tool

Then, paste this into the bin/app_update_thor_merge_tool file (or wherever else you are putting the script):

#!/usr/bin/env bash

RAILS_DEFAULT_CONFIG=$1
APP_CURRENT_CONFIG=$2

mkdir -p /tmp/rails-app-update
PATH2=$(mktemp "/tmp/rails-app-update/current_XXXX.rb")
BASE=$(mktemp "/tmp/rails-app-update/base_XXXX.rb")

cp "$APP_CURRENT_CONFIG" "$PATH2"
cp "$APP_CURRENT_CONFIG" "$BASE"

code --merge "$RAILS_DEFAULT_CONFIG" "$PATH2" "$BASE" "$APP_CURRENT_CONFIG" --wait
rm "$RAILS_DEFAULT_CONFIG"

Then, run rails app:update again, this time providing the above script as the THOR_MERGE tool.

Note: It’s important to provide the THOR_MERGE tool as an absolute path.

THOR_MERGE=$(pwd)/bin/app_update_thor_merge_tool bin/rails app:update

Now, if we press m (for merge, run merge tool) when prompted with a config file that needs attention, VS Code will open its three-way merge view:

The left pane shows the content of the config file as Rails would generate it for a brand new application.

The right pane shows the content of the config file as it currently exists in our repository.

The center pane is what will become the final result, which it is your job to craft, using the contents of the left and right panes of the merge view as a starting point and guide.

Read through any relevant Rails documentation about the changes and do whatever else you need to do to figure out how to resolve the discrepancies between your existing configuration and the configuration that Rails would generate for a new app. Then, once you have the middle pane into the desired final state, save the file, and close the tab in VS Code. In your terminal, the rails app:update command will then move on to the next file, for which you can again hit m to open the file in the VS Code three-way merge view. Repeat until you have worked through all of the configuration files that need attention.

Congratulations! You have now completed an important step in upgrading your Rails application.

Quick Tip: Use gem.wtf to go to the GitHub repo of a gem

2023-02-13T00:00:00-06:00

gem.wtf is a handy little website/tool that I use probably at least once a week. Its purpose is quite simple: enter gem.wtf/some_ruby_gem into your browser’s URL bar, and it will redirect you to that gem’s source code repository (i.e. usually their GitHub repo).

Here are some examples:

This is a really quick and convenient way to get from the name of a gem to its GitHub repo, which can be handy…

to quickly see how many GitHub stars a gem has that you’re thinking about using
to view a gem’s README documentation
to search/browse through the GitHub Issues/PRs/Discussions, if you’re running into a bug/problem
to clone the git repository, so that you can dig through the code history and get some additional context about how and why the library works the way it does
and more!

Of course, you can also find the gem’s GitHub repo through Google or RubyGems, but this saves you a click and takes you straight there.

Thanks to the creator of gem.wtf, Zeke Sikelianos. You can check out gem.wtf’s source code here (it’s actually a simple Node express app).

Bonus tip: There’s a similar tool to get to the GitHub repo for a JavaScript npm package: ghub.io. For example, ghub.io/lodash takes you to the GitHub repo for the lodash npm package.

Mocking external requests in Rails feature tests

2023-01-30T00:00:00-06:00

Without any introduction, allow me to share some code that I want to talk about, which exists within a Rails feature test:

around do |spec|
  page.driver.browser.intercept do |request, &continue|
    continue.call(request) do |response|
      if request.url.start_with?('https://accounts.google.com/o/oauth2/auth?')
        response.code = 200
        response.body = 'This is Google OAuth.'
      end
    end
  end

  spec.run
  # ...
end

The purpose of that code is to mock Google’s response to a request made by a Chrome browser that’s being driven through an OAuth authorization flow by Capybara in an automated Rails test suite. I want to share it here because, when I was searching how to mock a browser request made within a feature test, my Googling failed to bring up many relevant or helpful code examples.

If that’s all that you’re looking for, then maybe copy-paste that code and stop reading here. 🙂 One additional note, though: for this to work, you’ll need to add the selenium-devtools gem to the test group of your Gemfile, if it’s not there already.

If you’re interested in some additional background context, though, read on.

Why, though? Some background.

The code above comes from a spec seen here. That spec verifies the OAuth flow for logging in to my website using Google OAuth. That test was added in this commit.

Content Security Policy (CSP)

I had previously introduced a bug when modifying my app’s Content Security Policy (CSP). Setting up an effective CSP can be a little annoying, and there’s a risk of introducing bugs when doing it (as happened to me), but the upside is that a well-crafted CSP can provide a significant amount of protection against various potential security vulnerabilities in a web app.

OAuth

When initiating an OAuth authorization flow (e.g. by clicking a “Sign in with Google” button), a user’s browser will make a request to my app’s server, and my app’s server will respond with a redirect to a Google URL. However, in some browsers (as discussed here), if the application has a form-action CSP directive that doesn’t specifically authorize a redirect to the Google domain in question, the browser will refuse to follow the redirect, thus halting/breaking the OAuth flow and preventing the user from logging in.

I had inadvertently broken my app in that way when setting up a CSP for my application. After realizing that this bug was happening and figuring out how to fix it, I also wanted to write a test to reproduce the bug and ensure that particular bug would never be reintroduced.

Tests should be able to run (and pass) without an Internet connection

A relatively straightforward way to write such a regression test for the aforementioned bug would be to write a feature test that visits my app’s login page (hosted by a test server), clicks the “Sign in with Google” button, and then checks that the browser follows the redirect to a Google login page, and that the page has some content like “Google will share your name, email address, language preference, and profile picture with davidrunger.com”.

However, that approach relies on the test browser making a real, live external request over the Internet to a Google web server in order for the test to pass. This would violate one of my guiding principles of testing: tests should be able to run (and pass) without an Internet connection. There are a few reasons for this, the biggest of which are probably test speed and test reliability.

WebMock for the test browser

When writing unit tests in Ruby, external HTTP requests that are made by the Ruby application can be mocked using the most excellent webmock gem. WebMock can mock requests that are made from the Ruby app in question to an external server. However, it doesn’t allow us to mock a request from a Chrome browser that’s being driven in a feature test to an external server (in this example, a Google server for OAuth).

What we basically need is the equivalent of WebMock for a feature test. And that’s just what the code at the top of this article does.

Below is that code within the larger context of the regression test in question. I’ll make some additional comments about this test beneath the code.

RSpec.describe 'Logging in as a User via Google auth' do
  context 'when OmniAuth test mode is disabled' do
    around do |spec|
      original_omni_auth_test_mode = OmniAuth.config.test_mode
      OmniAuth.config.test_mode = false

      spec.run

      OmniAuth.config.test_mode = original_omni_auth_test_mode
    end

    context 'when Google responds with "This is Google OAuth."' do
      let(:mocked_google_response) { 'This is Google OAuth.' }

      around do |spec|
        page.driver.browser.intercept do |request, &continue|
          continue.call(request) do |response|
            if request.url.start_with?(
              'https://accounts.google.com/o/oauth2/auth?',
            )
              response.code = 200
              response.body = mocked_google_response
            end
          end
        end

        spec.run

        # try to do some cleanup (though I'm not sure how useful this is)
        page.driver.browser.devtools.callbacks.clear
        page.driver.browser.devtools.fetch.disable
      end

      it "renders Google's response" do
        visit(login_path)
        expect(page).to have_css('button.google-login')

        click_button(class: 'google-login')

        # The point of all of this: verify that the browser
        # indeed followed the redirect to Google.
        expect(page).to have_text(mocked_google_response)
      end
    end
  end
end

First, within an RSpec around block, we disable OmniAuth test mode in this test. This is necessary because, when the OmniAuth test mode is enabled, OmniAuth will not actually redirect the browser to an external OAuth server. So, in order to test that the browser does follow a redirect to an external Google OAuth server, we need to temporarily ensure that the OmniAuth test mode is disabled. We then restore OmniAuth.config.test_mode to its original value (presumably true) after the test has completed.

Next is the code that mocks the request to accounts.google.com, providing a specified response code (200) and content (“This is Google OAuth.”). I wrote some code at the end of that around block to try to do some cleanup to restore the test browser etc to its original state (though I’m not sure if it really helps anything).

Finally, is the test example itself, which visits the login page, clicks on the Google login button, and verifies, by checking for the mocked_google_response content, that the browser was indeed willing (even given my application’s Content Security Policy) to follow a redirect to the external Google URL.