Soapbox Legacy Developer Setup¶
Developers wanting to work on Soapbox Legacy source code and make changes to the system must configure a host for use with Soapbox Legacy's development environment.
The instructions in this file do not create a production-grade host that is secure and can scale. Instead, these instructions deliver a working environment tuned for making changes to Soapbox Legacy and for quickly iterating on those changes to get stuff done.
We are unlikely to support Windows as a host OS because no part of our software development infrastructure is based on Windows. We will, however, be happy to review and accept your pull requests adding Windows support for development and even production hosting if you think that's not too nutty.
Superuser Host Access Rights Required¶
This document describes commands intended to be run in a terminal. It also describes changes needed in some components' configuration files. Some of these actions must be performed with the user account you will use for day-to-day development. And, some of the commands need to be performed as the superuser (root) or a user with equivalent administrative privileges.
When superuser permissions are required,
Extend Ubuntu repositories when using Ubuntu 18.04.1LTS or later¶
Starting with Ubuntu 18.04.1 LTS, Canonical removed the multiverse and restricted repositories from the sources.list in
/etc/apt/. It is now necessary to add those repositories manually , otherwise the installation of the following dependencies will fail.
sudo add-apt-repository multiverse sudo add-apt-repository restricted sudo apt update
The following software components and libraries are required by Soapbox Legacy.
- ImageMagick - Soapbox Legacy uses imagemagick for image related operations
- FFMPEG - Soapbox Legacy uses ffmpeg for conversion of GIFs to MP4s
- libprotobuf-dev and protobuf-compiler - Soapbox Legacy uses these for language detection
- nginx - nginx is our frontend web server
- Redis - Soapbox Legacy uses redis for its in-memory data structure store
- postgresql - Soapbox Legacy uses PostgreSQL as its SQL database
- Node.js - Node is used for Soapbox Legacy's streaming API and other platform services
- Yarn - Yarn is a Node.js package manager
- gcc, g++, etc. - these are needed for the compilation of Ruby using ruby-build and to build Node.js extensions
All dependencies should be installed as the system superuser (root). Either use the
sudo command as required, or by first switching to the superuser using the following command:
If you become root, please be sure to switch back to your regular user account when instructed to do so later.
Install system components¶
apt-get install -y imagemagick ffmpeg libpq-dev libxml2-dev libxslt1-dev file git git-flow g++ libprotobuf-dev protobuf-compiler pkg-config gcc autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm5 libgdbm-dev nginx redis-server redis-tools postgresql postgresql-contrib certbot libidn11-dev libicu-dev
Install Node.js 10.15.3 LTS¶
Node.js is required for running the Soapbox Legacy Streaming API server and for other system management tasks.
# Install nvm to manage Node.js versions curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash # Install the Node.js runtime nvm install 10.15.3 # Install Yarn npm install -g yarn
Create User Account¶
Soapbox Legacy requires a standard non-root user account for day-to-day operations and work. This can be your own account or (if following this document for the first time) the
soapbox user is simple and can make following the rest of this guide very simple.
adduser --disabled-password --quiet soapbox
PostgreSQL Database Creation¶
Create a user for a PostgreSQL instance:
# Launch psql as the postgres user sudo -u postgres psql # In the following prompt CREATE USER soapbox CREATEDB; \q
Note that we do not set up a password of any kind, this is because we will be using ident authentication. This allows local users to access the database without a password.
Switch back to your account¶
If you became the root user to install system dependencies, please relinquish superuser privileges and return to your user account.
Configure your working environment¶
git clone https://github.com/rbenv/rbenv.git ~/.rbenv cd ~/.rbenv && src/configure && make -C src echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc echo 'eval "$(rbenv init -)"' >> ~/.bashrc # Restart shell exec bash # Check if rbenv is correctly installed type rbenv # Install ruby-build as rbenv plugin git clone https://github.com/rbenv/ruby-build.git ~/.rbenv/plugins/ruby-build
To enable Ruby, run:
rbenv install 2.6.1 rbenv global 2.6.1
This will take some time. Go stretch for a bit and drink some water while the commands run.
node.js And Ruby Dependencies¶
Run the following to clone and install:
mkdir -p ~/projects cd ~/projects # Clone the Soapbox Legacy repository into ~/projects git clone https://gitlab.com/soapbox-pub/soapbox-legacy # Hop into the project directory (all are welcome!) cd ~/projects/soapbox # Install bundler gem install bundler # Use bundler to install the rest of the Ruby dependencies bundle install # Use yarn to install node.js dependencies yarn install --pure-lockfile # To setup the `soapbox_development` database, run: bundle exec rails db:setup # Use foreman to start things up gem install foreman foreman start
At this point, you should be able to open
http://localhost:3000 in your browser and log in using the default credentials
admin@localhost:3000 and password
Some additional useful commands:
# pre-compile the front-end assets and fun stuff bin/rails assets:precompile # manually start the webpack dev server ./bin/webpack-dev-server # You can then run Soapbox Legacy with: bundle exec rails server
Managing your development environment¶
It is assumed that development hosts are not publicly accessible. For best security, there should be no route from a public network to your Soapbox Legacy development workstation.
By default, your development environment will have an admin account created for you to use - the email address will be
admin@YOURDOMAIN (e.g. admin@localhost:3000) and the password will be
You can run tests with:
Ruby locale files¶
You can check localization status with:
Or check it for a specific language with:
i18n-tasks health en
The problems on which this command might report include normalization (sorting and consistent YAML formatting), missing translations, unused translations and inconsistent interpolations.
What to do after adding or removing a message¶
The default locale file is
config/locales/en.yml, so add any new
messages to that file. Assuming that you place the new keys in
alphabetical order with the others, no further action is immediately
necessary, because languages without translations will fall back to
using the messages from
en.yml in production, and will show either
the key or a missing translation message in the development
If you want help alphabetizing, or to fix complaints from
health about normalization, use:
If you removed a message from
en.yml and the source code, and would
like to remove all translations of it, use:
If you change a message in such a way that the old translations are
not going to work (an example would be removing a variable), the best
strategy for getting rid of all the old invalid translations is to
delete the key and message from
en.yml and the source code, run
i18n-tasks remove-unused, verify that it gets removed, and then add
the key with its new value back into the source code. Some keys are
remove-unused; see its configuration in
Rails i18n lets you dynamically create key values, such as:
This feature is used in our codebase. Since
i18n-tasks is a static
code analyzer, it won't find dynamic keys, so if you add any new ones,
protect them from being deleted by
i18n-tasks remove-unused by
adding them to the
ignore_unused category in
What is an "inconsistent interpolation"?¶
This means that the variable or variables used in a localized message
do not match those used in the default locale, English. If a variable
used in a message is not supplied by the source code, it may cause a
500 error in production.
i18n-tasks health checks for all
inconsistencies in interpolations. To check only for missing
variables, and not unused ones, use:
If a variable is supplied by the source code, not used in the default
locale, but used in locales other than the default locale, this check
will fail even though the code will work correctly. If this is the
desired situation, or part of a work in progress as translations catch
up to changes in the code, the
check-missing-interpolations test may
be overridden by adding the message key to the
ignore_missing_interpolations section of
Other things that
i18n-tasks can do¶
find command will print out the places a key is used in the
With the appropriate API keys configured, the
command can use Google Translate or DeepL Pro to generate missing
rm commands can be used to rename, merge and delete keys.
automatically genenerated from the
defaultMessage values in the
source code, so do not edit them manually.
For more info see the i18n-tasks home page.
Start from a clean slate¶
The translation management tool uses the files created by webpack. These files can become out of date and lead to strings which have been removed from or edited in the source code being inserted into the locale files. To prevent this, clean up by removing the webpacker files and rerun it:
$ rm -r build public/packs tmp $ mkdir build $ yarn build:development
What to do after adding or removing a messsage¶
To recreate the default locale and the default messages file, after the above cleanup do:
$ yarn manage:translations en
Then check with
git diff to make sure it did what you want.
If you have added or removed translation keys, and the above step
worked out like you expected, the translation management tool can
update all the translation files at once, if you leave off the locale
argument. New keys will be added to the locale files with the
$ yarn manage:translations
This command also checks that the variable names used in the messages
match in the default and the translations. If a translation uses a
variable not in the default message, this command will report an error
(and the behavior in production will be to show the default message,
not the translated message). If a translation doesn't use a variable
which is used in the default message,
yarn manage:translations will
print a warning.
If you change a message in such a way that the old translations
are not going to work (an example would be removing a variable), the
best strategy for changing it in all the locale files to the new
default (English) version is to delete the key from the source code,
yarn build:development and
yarn manage:translations, verify
that it gets removed, and then add the key with its new value back
into the source code and run the
yarn commands again.
If you make an improvement to the English version of a message, but not a dramatic enough improvement to want to delete all of the existing translations, keep in mind that any locale file which didn't have a translation of that message will still have the old English version of the message, unless you fix it.
If you need to do something more complex than that, these are just JSON files, so it might be easiest to write a little program in Python or your scripting language of choice to do the job.
You can check code quality with:
$ rubocop $ yarn test:lint
Federation development tips¶
Federation absolutely requires your Soapbox Legacy instance to have a domain name. If you want to operate a permanently-federated development server, set up a Soapbox Legacy instance with a domain, and update it against your development fork/branch while doing that development on your local workstation or as a team.
To test federation on a local developer workstation, localhost => world tunneling can be made possible yourself on a domain you manage or by using services like ngrok.
Ngrok and similar services give you a random domain on each start up and iteration of your development build. This is good enough to test how the code you're working on handles real-world situations. But, your instance domain name is unique every time you run it.
For managing a production server, a service like Ngrok is the definition of Doing It Wrong™.
Generally, federation is tricky to work on because it's hard to test. When you are testing with a disposable instance, you are polluting the database of the real server(s) you are testing against.
It is possible to use Ngrok for one session, record the exchanges from its web interface, and use that data to create fixtures and build test suites. From then on, the developer can continue working against the tests instead of live servers.
Study the code and RFCs before implementing federation features or changes.
If the development environment is running remotely, setting the
REMOTE_DEV environment variable will instruct your instance to use "letter opener web"
Letter Opener launches a local browser. Letter Opener Web collects emails and displays them at /letter_opener.