Skip to content

Soapbox Legacy Developer Setup

Developers wanting to work on Soapbox Legacy source code and make changes to the system must configure a host for use with Soapbox Legacy's development environment.

The instructions in this file do not create a production-grade host that is secure and can scale. Instead, these instructions deliver a working environment tuned for making changes to Soapbox Legacy and for quickly iterating on those changes to get stuff done.

Host OS

Soapbox Legacy development has been tested on Ubuntu 18.04LTS. As we continue to migrate further away from Soapbox Legacy's code, at least MacOS will be supported as a choice.

We are unlikely to support Windows as a host OS because no part of our software development infrastructure is based on Windows. We will, however, be happy to review and accept your pull requests adding Windows support for development and even production hosting if you think that's not too nutty.

Superuser Host Access Rights Required

This document describes commands intended to be run in a terminal. It also describes changes needed in some components' configuration files. Some of these actions must be performed with the user account you will use for day-to-day development. And, some of the commands need to be performed as the superuser (root) or a user with equivalent administrative privileges.

When superuser permissions are required,

Extend Ubuntu repositories when using Ubuntu 18.04.1LTS or later

Starting with Ubuntu 18.04.1 LTS, Canonical removed the multiverse and restricted repositories from the sources.list in /etc/apt/. It is now necessary to add those repositories manually , otherwise the installation of the following dependencies will fail.

sudo add-apt-repository multiverse
sudo add-apt-repository restricted
sudo apt update

System Dependencies

The following software components and libraries are required by Soapbox Legacy.

  • ImageMagick - Soapbox Legacy uses imagemagick for image related operations
  • FFMPEG - Soapbox Legacy uses ffmpeg for conversion of GIFs to MP4s
  • libprotobuf-dev and protobuf-compiler - Soapbox Legacy uses these for language detection
  • nginx - nginx is our frontend web server
  • Redis - Soapbox Legacy uses redis for its in-memory data structure store
  • postgresql - Soapbox Legacy uses PostgreSQL as its SQL database
  • Node.js - Node is used for Soapbox Legacy's streaming API and other platform services
  • Yarn - Yarn is a Node.js package manager
  • gcc, g++, etc. - these are needed for the compilation of Ruby using ruby-build and to build Node.js extensions

Dependency Installation

All dependencies should be installed as the system superuser (root). Either use the sudo command as required, or by first switching to the superuser using the following command:

sudo -i

If you become root, please be sure to switch back to your regular user account when instructed to do so later.

Install system components

apt-get install -y imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git git-flow g++ libprotobuf-dev protobuf-compiler pkg-config gcc autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm5 libgdbm-dev nginx redis-server redis-tools postgresql postgresql-contrib certbot libidn11-dev libicu-dev

Install Node.js 10.15.3 LTS

Node.js is required for running the Soapbox Legacy Streaming API server and for other system management tasks.

# Install nvm to manage Node.js versions
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash

# Install the Node.js runtime
nvm install 10.15.3

# Install Yarn
npm install -g yarn

Create User Account

Soapbox Legacy requires a standard non-root user account for day-to-day operations and work. This can be your own account or (if following this document for the first time) the soapbox user.

Creating a soapbox user is simple and can make following the rest of this guide very simple.

adduser --disabled-password --quiet soapbox

PostgreSQL Database Creation

Soapbox Legacy requires access to a PostgreSQL instance.

Create a user for a PostgreSQL instance:

# Launch psql as the postgres user
sudo -u postgres psql

# In the following prompt
CREATE USER soapbox CREATEDB;
\q

Note that we do not set up a password of any kind, this is because we will be using ident authentication. This allows local users to access the database without a password.

Switch back to your account

If you became the root user to install system dependencies, please relinquish superuser privileges and return to your user account.

exit

Configure your working environment

The public-facing Web service soapbox-web is currently built using Ruby On Rails. A developer workstation user account, therefore, must configure rbenv and ruby-build as follows:

git clone https://github.com/rbenv/rbenv.git ~/.rbenv
cd ~/.rbenv && src/configure && make -C src
echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
echo 'eval "$(rbenv init -)"' >> ~/.bashrc

# Restart shell
exec bash

# Check if rbenv is correctly installed
type rbenv

# Install ruby-build as rbenv plugin
git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build

Now that rbenv and ruby-build are installed, we will install the Ruby version which Soapbox Legacy uses. That version will also need to be enabled.

To enable Ruby, run:

rbenv install 2.6.1
rbenv global 2.6.1

This will take some time. Go stretch for a bit and drink some water while the commands run.

node.js And Ruby Dependencies

Now that Ruby is enabled, we will clone the Git Social git repository and install the Ruby and node.js dependancies.

Run the following to clone and install:

mkdir -p ~/projects
cd ~/projects

# Clone the Soapbox Legacy repository into ~/projects
git clone https://gitlab.com/soapbox-pub/soapbox-legacy

# Hop into the project directory (all are welcome!)
cd ~/projects/soapbox

# Install bundler
gem install bundler

# Use bundler to install the rest of the Ruby dependencies
bundle install

# Use yarn to install node.js dependencies
yarn install --pure-lockfile

# To setup the `soapbox_development` database, run:
bundle exec rails db:setup

# Use foreman to start things up
gem install foreman
foreman start

At this point, you should be able to open http://localhost:3000 in your browser and log in using the default credentials admin@localhost:3000 and password administrator.

Some additional useful commands:

# pre-compile the front-end assets and fun stuff
bin/rails assets:precompile

# manually start the webpack dev server
./bin/webpack-dev-server

# You can then run Soapbox Legacy with:
bundle exec rails server

Managing your development environment

It is assumed that development hosts are not publicly accessible. For best security, there should be no route from a public network to your Soapbox Legacy development workstation.

By default, your development environment will have an admin account created for you to use - the email address will be admin@YOURDOMAIN (e.g. admin@localhost:3000) and the password will be administrator.

Running tests

You can run tests with:

rspec

Managing localizations

There are two sets of locale files, one for the JavaScript front end and one for the Ruby back end. There are tools for automatically maintaining both sets of translations.

Ruby locale files

You can check localization status with:

i18n-tasks health

Or check it for a specific language with:

i18n-tasks health en

The problems on which this command might report include normalization (sorting and consistent YAML formatting), missing translations, unused translations and inconsistent interpolations.

What to do after adding or removing a message

The default locale file is config/locales/en.yml, so add any new messages to that file. Assuming that you place the new keys in alphabetical order with the others, no further action is immediately necessary, because languages without translations will fall back to using the messages from en.yml in production, and will show either the key or a missing translation message in the development environment.

If you want help alphabetizing, or to fix complaints from i18n-tasks health about normalization, use:

i18n-tasks normalize

If you removed a message from en.yml and the source code, and would like to remove all translations of it, use:

i18n-tasks remove-unused

Changing messages

If you change a message in such a way that the old translations are not going to work (an example would be removing a variable), the best strategy for getting rid of all the old invalid translations is to delete the key and message from en.yml and the source code, run i18n-tasks remove-unused, verify that it gets removed, and then add the key with its new value back into the source code. Some keys are excluded from remove-unused; see its configuration in config/i18-tasks.yml.

Dynamic keys

Rails i18n lets you dynamically create key values, such as:

I18n.t("category.\#{category.name}")

This feature is used in our codebase. Since i18n-tasks is a static code analyzer, it won't find dynamic keys, so if you add any new ones, protect them from being deleted by i18n-tasks remove-unused by adding them to the ignore_unused category in config/i18n-tasks.yml.

What is an "inconsistent interpolation"?

This means that the variable or variables used in a localized message do not match those used in the default locale, English. If a variable used in a message is not supplied by the source code, it may cause a 500 error in production. i18n-tasks health checks for all inconsistencies in interpolations. To check only for missing variables, and not unused ones, use:

i18n-tasks check-missing-interpolations

If a variable is supplied by the source code, not used in the default locale, but used in locales other than the default locale, this check will fail even though the code will work correctly. If this is the desired situation, or part of a work in progress as translations catch up to changes in the code, the check-missing-interpolations test may be overridden by adding the message key to the ignore_missing_interpolations section of config/i18n-tasks.yml.

Other things that i18n-tasks can do

The find command will print out the places a key is used in the source code.

With the appropriate API keys configured, the translate_missing command can use Google Translate or DeepL Pro to generate missing translations.

The mv and rm commands can be used to rename, merge and delete keys.

Javascript locale files

The files app/javascript/gabsocial/locales/en.json and app/javascript/gabsocial/locales/defaultMessages.json are automatically genenerated from the defaultMessage values in the source code, so do not edit them manually.

For more info see the i18n-tasks home page.

Start from a clean slate

The translation management tool uses the files created by webpack. These files can become out of date and lead to strings which have been removed from or edited in the source code being inserted into the locale files. To prevent this, clean up by removing the webpacker files and rerun it:

$ rm -r build public/packs tmp
$ mkdir build
$ yarn build:development

What to do after adding or removing a messsage

To recreate the default locale and the default messages file, after the above cleanup do:

$ yarn manage:translations en

Then check with git diff to make sure it did what you want.

If you have added or removed translation keys, and the above step worked out like you expected, the translation management tool can update all the translation files at once, if you leave off the locale argument. New keys will be added to the locale files with the defaultMessage value.

$ yarn manage:translations

This command also checks that the variable names used in the messages match in the default and the translations. If a translation uses a variable not in the default message, this command will report an error (and the behavior in production will be to show the default message, not the translated message). If a translation doesn't use a variable which is used in the default message, yarn manage:translations will print a warning.

Changing messages

If you change a message in such a way that the old translations are not going to work (an example would be removing a variable), the best strategy for changing it in all the locale files to the new default (English) version is to delete the key from the source code, run yarn build:development and yarn manage:translations, verify that it gets removed, and then add the key with its new value back into the source code and run the yarn commands again.

If you make an improvement to the English version of a message, but not a dramatic enough improvement to want to delete all of the existing translations, keep in mind that any locale file which didn't have a translation of that message will still have the old English version of the message, unless you fix it.

If you need to do something more complex than that, these are just JSON files, so it might be easiest to write a little program in Python or your scripting language of choice to do the job.

Linting

You can check code quality with:

$ rubocop
$ yarn test:lint

Federation development tips

Federation absolutely requires your Soapbox Legacy instance to have a domain name. If you want to operate a permanently-federated development server, set up a Soapbox Legacy instance with a domain, and update it against your development fork/branch while doing that development on your local workstation or as a team.

To test federation on a local developer workstation, localhost => world tunneling can be made possible yourself on a domain you manage or by using services like ngrok.

Ngrok and similar services give you a random domain on each start up and iteration of your development build. This is good enough to test how the code you're working on handles real-world situations. But, your instance domain name is unique every time you run it.

For managing a production server, a service like Ngrok is the definition of Doing It Wrong™.

Federation tips

Generally, federation is tricky to work on because it's hard to test. When you are testing with a disposable instance, you are polluting the database of the real server(s) you are testing against.

It is possible to use Ngrok for one session, record the exchanges from its web interface, and use that data to create fixtures and build test suites. From then on, the developer can continue working against the tests instead of live servers.

Study the code and RFCs before implementing federation features or changes.

Remote Development

If the development environment is running remotely, setting the REMOTE_DEV environment variable will instruct your instance to use "letter opener web"

Letter Opener launches a local browser. Letter Opener Web collects emails and displays them at /letter_opener.