Developing Soapbox Legacy with Nix¶
Soapbox Legacy has a long list of dependencies including both system packages and packages from the Ruby and Node package managers. The Nix package manager can make it easy to create reproducible development environments with all the dependencies and isolate them from your other projects and the rest of your system.
Nix vs Nixpkgs vs NixOS¶
The Nix package manager can be installed on any Linux distro as well as macOS, and used independently of any other package managers on the system. Its package descriptions are written in the Nix language.
Nixpkgs is the central repository of package definitions written in the Nix language.
NixOS is the operating system based on Nix and Nixpkgs.
I use NixOS, but the instructions below for developing Soapbox Legacy with Nix should work on other Linux distros. macOS is uncharted territory, so let me know what you find there.
To contain or not to contain¶
The typical way to develop software with Nix is to use the command
nix-shell
to create a shell environment within which the software's
dependencies are present. This is analogous to Ruby's rbenv
or
Python's virtualenv
except that it can pull in dependencies from
multiple other package managers with a single command, and can make
system packages such as ffmpeg
available to only the project under
development instead of installing them globally. Services such as
Redis, however, need to be provided outside of the Nix shell.
It's also possible to run Nix and nix-shell
in a container such as
a Docker image, or a NixOS
container if you are running NixOS. The advantage of this is that the
required services, Redis and PostgreSQL, can be run inside the
container instead of being made globally available on your system.
Containerizing your development environment also allows you to work
with multiple instances of it at once without worrying about conflicts
in Redis and PostgreSQL, or to run development servers of both
Mastodon and Soapbox Legacy at the same time without port number conflicts.
You will also be able to clean up all databases and packages used by
Soapbox Legacy development on your machine by removing the container and
running nix-collect-garbage
.
Be safe¶
The default configuration brings the development server up on port 3000. Don't make that port available to random strangers on the internet or on your favorite coffee shop's wireless network.
Setting up your development environment¶
TLDR¶
- Clone the project.
- Configure the Redis and PostgreSQL services on your system or in a container.
- Use
update.sh
to create Nix descriptions of dependencies. - Use
nix-shell
to create a development environment. - Initialize the development database.
- Use
goreman
to start the four Soapbox Legacy processes.
Cloning the project¶
Clone the project into your user directory with this command:
$ git clone https://gitlab.com/soapbox-pub/soapbox-legacy.git
Runtime services: Redis and PostgreSQL¶
The Soapbox Legacy server relies upon the Redis and PostgreSQL services. If
you are running the Nix package manager on an operating system other
than NixOS (including the Nix Docker image mentioned above, which is
running Alpine Linux), refer to that operating system's documentation
and developer
to set up Redis and PostgreSQL.
Enabling services in configuration.nix
¶
If you are running NixOS and intend to develop in a shell instead of a
container, add the following to your configuration.nix
to enable and
configure Redis and PostgreSQL, filling in your own user account name.
services.redis.enable = true;
services.postgresql = {
enable = true;
ensureUsers = [
{
name = "replace this with your user name";
ensurePermissions = {
"DATABASE soapbox_development" = "ALL PRIVILEGES";
"DATABASE soapbox_test" = "ALL PRIVILEGES";
};
}
];
ensureDatabases = [
"soapbox_development"
"soapbox_test"
];
};
Follow this with:
$ sudo nixos-rebuild switch
to activate the configuration and start the services.
Setting up a NixOS container with Redis and PostgreSQL¶
If you intend to develop in a container, you can use this declarative
container specification which can be added to your configuration.nix
and built with nixos-rebuild switch
. Edit it to add your own ssh
key.
This container uses a bind mount to access your Soapbox Legacy source tree.
Edit the hostPath
below to contain the directory that you cloned the
project into. Bind mounting allows you to edit code from outside the
container and run it inside the container. In order for this to work,
appropriate file permissions need to be set. One way to do that is to
make the uid's of the users inside and outside of the container match.
If you want to use multiple containers to run multiple instances of
the Soapbox Legacy server, they should not share the same source directory,
since webpack
stores its artifacts in the source directory. A
workaround might be to use git worktree
.
Container networking has several features that may be configured, such
as routing to the Internet and port forwarding (see the NixOS
Manual for more information). I've
chosen to give the container below its own address on a private
virtual network interface. NixOS automatically gives declarative
containers such as this one an entry in /etc/hosts
so you don't have
to remember its IP address.
containers.sbdevel = {
bindMounts = {
"/home/sbdevel/soapbox" = {
hostPath = "/path/to/soapbox/git/repo";
isReadOnly = false;
};
};
autoStart = true; # Optional, otherwise use 'nixos-container start sbdevel'.
# Set these addresses as you like for container and host to communicate.
hostAddress = "192.168.100.2";
localAddress = "192.168.101.2";
privateNetwork = true;
config = { config, pkgs, ... }: {
environment.systemPackages = [
# any packages you would like available at the command line
];
users.extraUsers.sbdevel = {
isNormalUser = true;
uid = 1000; # Set UID and GUID as necessary to make bind mount permissions work
description = "soapbox developer";
createHome = true;
openssh.authorizedKeys.keyFiles = [ /path/to/your/public/ssh/key ];
};
services.openssh.enable = true;
services.redis.enable = true;
services.postgresql = {
enable = true;
ensureUsers = [
{
name = "sbdevel";
ensurePermissions = {
"DATABASE soapbox_development" = "ALL PRIVILEGES";
"DATABASE soapbox_test" = "ALL PRIVILEGES";
};
}
];
ensureDatabases = [
"soapbox_development"
"soapbox_test"
];
};
networking.firewall.allowedTCPPorts = [ 3000 3035 4000 ];
# Stop cross-site complaints from the browser by using the
# same hostname inside and outside of the container.
networking.hosts = {
"127.0.2.1" = [ "sbdevel.containers" ];
};
};
}
Build the container, and give it a copy of the NixOS channel, as described in the NixOS manual:
$ sudo nixos-rebuild switch
$ sudo nixos-container run sbdevel -- nix-channel --update
Then you can access the container's user account's command line with:
$ ssh sbdevel@sbdevel.containers
The source tree will be available, and writable, at ~/soapbox
.
So that the Soapbox Legacy development server knows its own hostname, and
makes emails available at /letter_opener instead of trying to launch a
browser within the container, it's helpful to put the following in
~sbdevel/.bashrc
:
export LOCAL_DOMAIN=sbdevel.containers
export REMOTE_DEV=true
Using Nix to fetch dependencies¶
In order to use Nix to fetch Ruby and Node dependencies, two external
tools called bundix
and yarn2nix
are used to convert the
Gemfile
, Gemfile.lock
, package.json
and yarn.lock
into Nix
format. The shell script update.sh
can do this for you, from the
root of the soapbox
source tree, inside or outside of a container:
[~/soapbox]$ nix run -f develop.nix updateScript -c update.sh
This may have to download a lot of packages the first time you run it.
update.sh
also updates Gemfile.lock
and yarn.lock
if any changes
have been made to the Gemfile
or package.json
.
Creating a development shell¶
Once the dependency files are up to date, create a shell (in the container, if you plan to run Soapbox Legacy there) with:
[~/soapbox]$ nix-shell develop.nix
This might have to download a lot of packages the first time you run
it. It will create a shell containing the Ruby bundle; other packages
needed by Soapbox Legacy such as ffmpeg
; yarn
, nodejs
, and goreman
;
and will recreate the node_modules
subdirectory with all packages
called for by yarn.lock
. (After the recreation a yarn install
will print messages like it is installing everything, but it will run
very quickly and end up with the same package versions.)
When you update or change a Ruby or Node dependency, or check out a
git revision with different dependencies, exit from this shell, run
update.sh
again, and then create a new nix-shell
.
bundle exec
problem¶
I don't know why, but the bundle
executable created by nix-shell
fails with infinite recursion. By bisecting the Gemfile
, I found that
the problem is associated with http_parser.rb
and goes away if you
revert it to 0.6 from the git revision given in the
Gemfile
. However, that git revision is there to fix a bug
(tootsuite/mastodon #7467) and was meant to be an interim fix until
the update of the http
gem to v5, but we are stuck on v3 because we
depend on a couple of other undermaintained gems, goldfinger
and
ostatus2
which have not yet been updated to accept http
v4. These
are problems we have inherited from Mastodon.
Running things with bundle exec
is not necessary however, because
within the nix-shell
environment, all the bundled executables are on
the PATH
.
This problem also affects foreman
but there is a Go clone of it in
nixpkgs
called goreman
which does not have that problem. It is
included in the Nix shell created by develop.nix
.
Loading the database schema¶
On non-NixOS systems¶
Set up the development database with:
[nix-shell:~/soapbox]$ rails db:setup
On NixOS systems or in NixOS containers¶
Include these settings in .env
at the root of the Soapbox Legacy source
tree to make local peer authentication work:
DB_HOST=/run/postgresql
DB_PORT=5432
Since the database was already created by configuration.nix
or the
container configuration, you can't use rails db:setup
. Use these
commands instead to load the database schema and seed the username and
password for the admin@localhost:3000
account (localhost
in that
address will be replaced by $LOCAL_DOMAIN
if you have set it).
[nix-shell:~/soapbox]$ rails db:schema:load
[nix-shell:~/soapbox]$ rails db:seed
Starting the development server¶
Because of the bundle exec
bug, Soapbox Legacy's Procfile.dev
won't work
as is, so use Procfile-nix.dev
instead, which omits bundle exec
.
From inside your nix-shell
at the root of the soapbox source tree,
start the development server with:
[nix-shell:~/soapbox]$ rails assets:precompile
[nix-shell:~/soapbox]$ goreman -f Procfile-nix.dev start
If you are running this in your own account the Soapbox Legacy development
server will be available at http://127.0.0.1:3000
. If you are using
the declarative container configuration given above, use the address
http://sbdevel.containers:3000
.
Running tests¶
Use these commands to prepare the directory and databases for running
tests, from the soapbox
directory:
[nix-shell:~/soapbox]$ RAILS_ENV=test rails db:schema:load
[nix-shell:~/soapbox]$ RAILS_ENV=test rails assets:precompile
And this command to run the tests:
[nix-shell:~/soapbox]$ rspec
Everything else¶
At this point, all the development tasks described after the foreman
start
in developer
should just work within the Nix shell.
Remember to omit bundle exec
.