pixelfed/docker/migration.md

Migrating to the new Pixelfed Docker setup

There is a lot of changes in how Pixelfed Docker/Docker Compose images work - it's a complete rewrite - with a bunch of breaking changes.

No more anonymous Docker Compose volumes

The old docker-compose.yml configuration file declared four anonymous volumes for storing Pixelfed related data within.

These are no longer used, instead favoring a Docker bind volume approach where content are stored directly on the server disk, outside of a Docker volume.

The consequence of this change is that all data stored in the - now unsupported - Docker volumes will no longer be accessible by Pixelfed.

• The db-data volume definitely contain important data - it's your database after all!
• The app-storage volume definitely contain important data - it's files uploaded to - or seen by - your server!
• The redis-data volume might contain important data (depending on your configuration)
• The app-bootstrap volume does not contain any important data - all of it will be generated automatically in the new setup on startup. We will not be migrating this!

Migrating off anonymous Docker Compose volumes

Caveats and warnings

Important

This is a best-effort guide to help migrate off the old system, the operation is potentially rather complicated (and risky), so please do be careful!

Caution

PLEASE MAKE SURE TO BACKUP YOUR SERVER AND DATA BEFORE ATTEMPTING A MIGRATION

YOUR INSTANCE WILL BE DOWN WHILE DOING THE MIGRATION, PLEASE PLAN ACCORDINGLY, DEPENDING ON DATA SIZE IT COULD TAKE ANYWHERE FROM 5 MINUTES TO 5 HOURS

Warning

It's important to note that this is a copy operation - so disk usage will (temporarily) double while you migrate We provide a "migration container" for your convenience that can access both the new and old volumes, allowing you to copy the data into the setup.

Step 0) Backup, rollout, and rollback plan

1. Make sure to backup your server (ideally after step 1 below has completed, but before is better than not at all!)
2. Capture the current Git version / Pixelfed release you are on (e.g. git --no-pager log -1 outputs the commit reference as the 2nd word in first line)
3. Backup your .env file (we will do this in step 3 as well)
4. Backup your docker-compose.yml file (cp docker-compose.yml docker-compose.yml.old)
5. Read through the entire document before starting

Step 1) Migrate your ".env" file

The new .env file for Docker is a bit different from the old one (many new settings!) so the easiest is to grab the new .env.docker file and modify it from scratch again.

$ cp .env .env.old
$ wget -O .env.new https://raw.githubusercontent.com/pixelfed/pixelfed/dev/.env.docker

Then open your old .env.old configuration file, and for each of the key/value pairs within it, find and update the key in the new .env.new configuration file.

Don't worry though, the file might look different (and significantly larger) but it behaves exactly the way the old file did, it just has way more options!

Tip

Don't worry if a key is missing in .env.new, you can add those key/value pairs back to the new file - ideally in the Other configuration section near the end of the file - but anywhere should be fine.

This is a great time to review your settings and familiarize you with all the new settings.

Note

In particular the following sections

• PHP configuration section (near the end of the file) where
  • The DOCKER_APP_PHP_VERSION settings controls your PHP version
  • The PHP_MEMORY_LIMIT settings controls your PHP memory limit
• Docker Specific configuration section (near the end of the file) where
  • The DOCKER_ALL_HOST_DATA_ROOT_PATH setting dictate where the new migrated data will live.
  • The DOCKER_APP_RUN_ONE_TIME_SETUP_TASKS controls if the One time setup tasks should run or not. We do not want this, since your Pixelfed instance already is set up!
• Frequently Asked Question / FAQ
  • How do I use my own Proxy server?
  • How do I use my own SSL certificate?

Step 2) Stop all running containers

Caution

This will take your Pixelfed instance offline

Stop all running containers (web, worker, redis, db)

$ docker compose down

Step 3) Pull down the new source code

Update your project to the latest release of Pixelfed by running

$ git pull origin $release

Note

The $release can be any valid git reference like dev, staging or a tagged release such as v0.12.0.

Step 4) Run the migration container

You can access the Docker container with both old and new volumes by running the following command

$ docker compose -f docker-compose.migrate.yml run migrate bash

This will put you in the /migrate directory within the container, containing 8 directories like shown here

|-- app-storage
|   |-- new
|   `-- old
|-- db-data
|   |-- new
|   `-- old
`-- redis-data
    |-- new
    `-- old

Step 5) Check the folders

Old folders

The following commands should all return SOME files and data - if they do not - then there might be an issue with the anonymous volume binding.

$ ls app-storage/old
$ ls db-data/old
$ ls redis-data/old

New folders

The following commands should all return NO files and data - if they contain data - you need to either delete it (backup first!) or skip that migration step.

If you haven't run docker compose up since you updated your project in step (2) - they should be empty and good to go.

$ ls app-storage/new
$ ls db-data/new
$ ls redis-data/new

Step 6) Copy the data

Warning

This is where we potentially will double your disk usage (temporarily)

Now we will copy the data from the old volumes, to the new ones.

The migration container has rsync installed - which is perfect for that kind of work!

NOTE It's important that the "source" (first path in the rsync command) has a trailing / - otherwise the directory layout will turn out wrong!

NOTE Depending on your server, these commands might take some time to finish, each command should provide a progress bar with rough time estimation.

NOTE rsync should preserve ownership, permissions, and symlinks correctly for you as well for all the files copied.

Lets copy the data by running the following commands:

$ rsync -avP app-storage/old/ app-storage/new
$ rsync -avP db-data/old/ db-data/new
$ rsync -avP redis-data/old/ redis-data/new

Step 7) Sanity checking

Lets make sure everything copied over successfully!

Each new directory should contain something like (but not always exactly) the following - NO directory should have a single folder called old, if they do, the rsync commands above didn't work correctly - and you need to move the content of the old folder into the "root" of the new folder like shown a bit in the following sections.

The redis-data/new directory might also contain a server.pid

$ ls redis-data/new
appendonlydir

The app-storage/new directory should look something like this

$ ls app-storage/new
app  debugbar  docker  framework  logs  oauth-private.key  oauth-public.key  purify

The db-data/new directory should look something like this. There might be a lot of files, or very few files, but there must be a mysql, performance_schema, and ${DB_DATABASE} (e.g. pixelfed_prod directory)

$ ls db-data/new
aria_log_control  ddl_recovery-backup.log  ib_buffer_pool  ib_logfile0  ibdata1  mariadb_upgrade_info  multi-master.info  mysql  performance_schema  pixelfed_prod  sys  undo001  undo002  undo003

If everything looks good, type exit to leave exit the migration container

Step 8) Starting up the your Pixelfed server again

With all an updated Pixelfed (step 2), updated .env file (step 3), migrated data (step 4, 5, 6 and 7) we're ready to start things back up again.

But before we start your Pixelfed server back up again, lets put the new .env file we made in step 1 in its right place.

$ cp .env.new .env

The Database

First thing we want to try is to start up the database by running the following command and checking the logs

$ docker compose up -d db
$ docker compose logs --tail 250 --follow db

if there are no errors and the server isn't crashing, great! If you have an easy way of connecting to the Database via a GUI or CLI client, do that as well and verify the database and tables are all there.

Redis

Next thing we want to try is to start up the Redis server by running the following command and checking the logs

$ docker compose up -d redis
$ docker compose logs --tail 250 --follow redis

if there are no errors and the server isn't crashing, great!

Worker

Next thing we want to try is to start up the Worker server by running the following command and checking the logs

$ docker compose up -d worker
$ docker compose logs --tail 250 --follow worker

The container should output a lot of logs from the docker-entrypoint system, but eventually you should see these messages

• Configuration complete; ready for start up
• Horizon started successfully.

If you see one or both of those messages, great, the worker seems to be running.

If the worker is crash looping, inspect the logs and try to resolve the issues.

You can consider the following additional steps:

• Enabling DOCKER_APP_ENTRYPOINT_DEBUG which will show even more log output to help understand whats going on
• Enabling DOCKER_APP_ENSURE_OWNERSHIP_PATHS against the path(s) that might have permission issues
• Fixing permission issues directly on the host since your data should all be in the ${DOCKER_ALL_HOST_DATA_ROOT_PATH} folder (./docker-compose-state/data by default)

Web

The final service, web, which will bring your site back online! What a journey it has been.

Lets get to it, run these commands to start the web service and inspect the logs.

$ docker compose up -d web
$ docker compose logs --tail 250 --follow web

The output should be pretty much identical to that of the worker, so please see that section for debugging tips if the container is crash looping.

If the web service came online without issues, start the rest of the (optional) services, such as the proxy, if enabled, by running

$ docker compose up -d
$ docker compose logs --tail 250 --follow

If you changed anything in the .env file while debugging, some containers might restart now, thats perfectly fine.

Step 9) Verify

With all services online, it's time to go to your browser and check everything is working

1. Upload and post a picture
2. Comment on a post
3. Like a post
4. Check Horizon (https://${APP_DOMAIN}/horizon) for any errors
5. Check the Docker compose logs via docker compose logs --follow

If everything looks fine, yay, you made it to the end! Lets do some cleanup

Step 10) Final steps + cleanup

With everything working, please take a new snapshot/backup of your server before we do any cleanup. A post-migration snapshot is incredibly useful, since it contains both the old and new configuration + data, making any recovery much easier in a rollback scenario later.

Now, with all the data in the new folders, you can delete the old Docker Container volumes (if you want, completely optional)

List all volumes, and give them a look:

$ docker volume ls

The volumes we want to delete ends with the volume name (db-data, app-storage, redis-data, and app-bootstrap.) but has some prefix in front of them.

Once you have found the volumes in in the list, delete each of them by running:

$ docker volume rm $volume_name_in_column_two_of_the_output

You can also delete the docker-compose.yml.old and .env.old file since they are no longer needed

$ rm docker-compose.yml.old
$ rm .env.old

Rollback

Oh no, something went wrong? No worries, we you got backups and a quick way back!

Move docker-compose.yml back

$ cp docker-compose.yml docker-compose.yml.new
$ cp docker-compose.yml.old docker-compose.yml

Move .env file back

$ cp env.old .env

Go back to old source code version

$ git checkout $commit_id_from_step_0

Start things back up

docker compose up -d

Verify it worked