Developing Soapbox Legacy with Nix¶

Soapbox Legacy has a long list of dependencies including both system packages and packages from the Ruby and Node package managers. The Nix package manager can make it easy to create reproducible development environments with all the dependencies and isolate them from your other projects and the rest of your system.

Nix vs Nixpkgs vs NixOS¶

The Nix package manager can be installed on any Linux distro as well as macOS, and used independently of any other package managers on the system. Its package descriptions are written in the Nix language.

Nixpkgs is the central repository of package definitions written in the Nix language.

NixOS is the operating system based on Nix and Nixpkgs.

I use NixOS, but the instructions below for developing Soapbox Legacy with Nix should work on other Linux distros. macOS is uncharted territory, so let me know what you find there.

To contain or not to contain¶

The typical way to develop software with Nix is to use the command nix-shell to create a shell environment within which the software's dependencies are present. This is analogous to Ruby's rbenv or Python's virtualenv except that it can pull in dependencies from multiple other package managers with a single command, and can make system packages such as ffmpeg available to only the project under development instead of installing them globally. Services such as Redis, however, need to be provided outside of the Nix shell.

It's also possible to run Nix and nix-shell in a container such as a Docker image, or a NixOS container if you are running NixOS. The advantage of this is that the required services, Redis and PostgreSQL, can be run inside the container instead of being made globally available on your system.

Containerizing your development environment also allows you to work with multiple instances of it at once without worrying about conflicts in Redis and PostgreSQL, or to run development servers of both Mastodon and Soapbox Legacy at the same time without port number conflicts. You will also be able to clean up all databases and packages used by Soapbox Legacy development on your machine by removing the container and running nix-collect-garbage.

Be safe¶

The default configuration brings the development server up on port 3000. Don't make that port available to random strangers on the internet or on your favorite coffee shop's wireless network.

Setting up your development environment¶

TLDR¶

Clone the project.
Configure the Redis and PostgreSQL services on your system or in a container.
Use update.sh to create Nix descriptions of dependencies.
Use nix-shell to create a development environment.
Initialize the development database.
Use goreman to start the four Soapbox Legacy processes.

Cloning the project¶

Clone the project into your user directory with this command:

$ git clone https://gitlab.com/soapbox-pub/soapbox-legacy.git

Runtime services: Redis and PostgreSQL¶

The Soapbox Legacy server relies upon the Redis and PostgreSQL services. If you are running the Nix package manager on an operating system other than NixOS (including the Nix Docker image mentioned above, which is running Alpine Linux), refer to that operating system's documentation and developer to set up Redis and PostgreSQL.

Enabling services in `configuration.nix`¶

If you are running NixOS and intend to develop in a shell instead of a container, add the following to your configuration.nix to enable and configure Redis and PostgreSQL, filling in your own user account name.

  services.redis.enable = true;
  services.postgresql = {
    enable = true;
    ensureUsers = [
      {
        name = "replace this with your user name";
        ensurePermissions = {
          "DATABASE soapbox_development" = "ALL PRIVILEGES";
          "DATABASE soapbox_test" = "ALL PRIVILEGES";
        };
      }
    ];
    ensureDatabases = [
      "soapbox_development"
      "soapbox_test"
    ];
  };

Follow this with:

$ sudo nixos-rebuild switch

to activate the configuration and start the services.

Setting up a NixOS container with Redis and PostgreSQL¶

If you intend to develop in a container, you can use this declarative container specification which can be added to your configuration.nix and built with nixos-rebuild switch. Edit it to add your own ssh key.

This container uses a bind mount to access your Soapbox Legacy source tree. Edit the hostPath below to contain the directory that you cloned the project into. Bind mounting allows you to edit code from outside the container and run it inside the container. In order for this to work, appropriate file permissions need to be set. One way to do that is to make the uid's of the users inside and outside of the container match.

If you want to use multiple containers to run multiple instances of the Soapbox Legacy server, they should not share the same source directory, since webpack stores its artifacts in the source directory. A workaround might be to use git worktree.

Container networking has several features that may be configured, such as routing to the Internet and port forwarding (see the NixOS Manual for more information). I've chosen to give the container below its own address on a private virtual network interface. NixOS automatically gives declarative containers such as this one an entry in /etc/hosts so you don't have to remember its IP address.

containers.sbdevel = {
  bindMounts = {
    "/home/sbdevel/soapbox" = {
      hostPath = "/path/to/soapbox/git/repo";
      isReadOnly = false;
    };
  };
  autoStart = true;  # Optional, otherwise use 'nixos-container start sbdevel'.

  # Set these addresses as you like for container and host to communicate.
  hostAddress = "192.168.100.2";
  localAddress = "192.168.101.2";
  privateNetwork = true;

  config = { config, pkgs, ... }: {
    environment.systemPackages = [
      # any packages you would like available at the command line
    ];
    users.extraUsers.sbdevel = {
      isNormalUser = true;
      uid = 1000;  # Set UID and GUID as necessary to make bind mount permissions work
      description = "soapbox developer";
      createHome = true;
      openssh.authorizedKeys.keyFiles = [ /path/to/your/public/ssh/key ];
    };
    services.openssh.enable = true;
    services.redis.enable = true;
    services.postgresql = {
      enable = true;
      ensureUsers = [
        {
          name = "sbdevel";
          ensurePermissions = {
            "DATABASE soapbox_development" = "ALL PRIVILEGES";
            "DATABASE soapbox_test" = "ALL PRIVILEGES";
          };
        }
      ];
      ensureDatabases = [
        "soapbox_development"
        "soapbox_test"
      ];
    };
    networking.firewall.allowedTCPPorts = [ 3000 3035 4000 ];

    # Stop cross-site complaints from the browser by using the
    # same hostname inside and outside of the container.
    networking.hosts = {
      "127.0.2.1" = [ "sbdevel.containers" ];
    };
  };
}

Build the container, and give it a copy of the NixOS channel, as described in the NixOS manual:

$ sudo nixos-rebuild switch
$ sudo nixos-container run sbdevel -- nix-channel --update

Then you can access the container's user account's command line with:

$ ssh sbdevel@sbdevel.containers

The source tree will be available, and writable, at ~/soapbox.

So that the Soapbox Legacy development server knows its own hostname, and makes emails available at /letter_opener instead of trying to launch a browser within the container, it's helpful to put the following in ~sbdevel/.bashrc:

export LOCAL_DOMAIN=sbdevel.containers
export REMOTE_DEV=true

Using Nix to fetch dependencies¶

In order to use Nix to fetch Ruby and Node dependencies, two external tools called bundix and yarn2nix are used to convert the Gemfile, Gemfile.lock, package.json and yarn.lock into Nix format. The shell script update.sh can do this for you, from the root of the soapbox source tree, inside or outside of a container:

[~/soapbox]$ nix run -f develop.nix updateScript -c update.sh

This may have to download a lot of packages the first time you run it. update.sh also updates Gemfile.lock and yarn.lock if any changes have been made to the Gemfile or package.json.

Creating a development shell¶

Once the dependency files are up to date, create a shell (in the container, if you plan to run Soapbox Legacy there) with:

[~/soapbox]$ nix-shell develop.nix

This might have to download a lot of packages the first time you run it. It will create a shell containing the Ruby bundle; other packages needed by Soapbox Legacy such as ffmpeg; yarn, nodejs, and goreman; and will recreate the node_modules subdirectory with all packages called for by yarn.lock. (After the recreation a yarn install will print messages like it is installing everything, but it will run very quickly and end up with the same package versions.)

When you update or change a Ruby or Node dependency, or check out a git revision with different dependencies, exit from this shell, run update.sh again, and then create a new nix-shell.

`bundle exec` problem¶

I don't know why, but the bundle executable created by nix-shell fails with infinite recursion. By bisecting the Gemfile, I found that the problem is associated with http_parser.rb and goes away if you revert it to 0.6 from the git revision given in the Gemfile. However, that git revision is there to fix a bug (tootsuite/mastodon #7467) and was meant to be an interim fix until the update of the http gem to v5, but we are stuck on v3 because we depend on a couple of other undermaintained gems, goldfinger and ostatus2 which have not yet been updated to accept http v4. These are problems we have inherited from Mastodon.

Running things with bundle exec is not necessary however, because within the nix-shell environment, all the bundled executables are on the PATH.

This problem also affects foreman but there is a Go clone of it in nixpkgs called goreman which does not have that problem. It is included in the Nix shell created by develop.nix.

Loading the database schema¶

On non-NixOS systems¶

Set up the development database with:

[nix-shell:~/soapbox]$ rails db:setup

On NixOS systems or in NixOS containers¶

Include these settings in .env at the root of the Soapbox Legacy source tree to make local peer authentication work:

DB_HOST=/run/postgresql
DB_PORT=5432

Since the database was already created by configuration.nix or the container configuration, you can't use rails db:setup. Use these commands instead to load the database schema and seed the username and password for the admin@localhost:3000 account (localhost in that address will be replaced by $LOCAL_DOMAIN if you have set it).

[nix-shell:~/soapbox]$ rails db:schema:load
[nix-shell:~/soapbox]$ rails db:seed

Starting the development server¶

Because of the bundle exec bug, Soapbox Legacy's Procfile.dev won't work as is, so use Procfile-nix.dev instead, which omits bundle exec. From inside your nix-shell at the root of the soapbox source tree, start the development server with:

[nix-shell:~/soapbox]$ rails assets:precompile
[nix-shell:~/soapbox]$ goreman -f Procfile-nix.dev start

If you are running this in your own account the Soapbox Legacy development server will be available at http://127.0.0.1:3000. If you are using the declarative container configuration given above, use the address http://sbdevel.containers:3000.

Running tests¶

Use these commands to prepare the directory and databases for running tests, from the soapbox directory:

[nix-shell:~/soapbox]$ RAILS_ENV=test rails db:schema:load
[nix-shell:~/soapbox]$ RAILS_ENV=test rails assets:precompile

And this command to run the tests:

[nix-shell:~/soapbox]$ rspec

Everything else¶

At this point, all the development tasks described after the foreman start in developer should just work within the Nix shell. Remember to omit bundle exec.