Upgrade from Archivematica 1.11.x to 1.12.0¶

Clean up completed transfers watched directory¶

Upgrading from Archivematica 1.10.x or older to Archivematica 1.12.0 can result in a number of completed transfers appearing as failed in the Archivematica dashboard, as well as corresponding failure notification emails being sent. These are not actual failures, but are unintentional side effects of changes made in Archivematica 1.11 to the workflow and to how metadata files are stored and copied into the SIP.

To prevent these failures from occuring during an upgrade from Archivematica 1.10 or earlier:

1. Confirm that all transfers and ingests are complete.

   Check that there are no transfers or SIPs that are still being processed or awaiting decisions in the Transfer and Ingest tabs. If there are, finish processing the transfers/ingests before proceeding.

2. Delete all contents of the completedTransfers watched directory.

   sudo rm -rf /var/archivematica/sharedDirectory/watchedDirectories/SIPCreation/completedTransfers/*
   

3. Perform the upgrade as described below.

Create a backup¶

Before starting any upgrade procedure on a production system, we strongly recommend backing up your system. If you are using a virtual machine, take a snapshot of it before making any changes. Alternatively, back up the file systems being used by your system. Exact procedures for updating will depend on your local installation. At a minimum you should make backups of:

• The Storage Service SQLite (or MySQL) database
• The dashboard MySQL database

This is a simple example of backing up these two databases:

sudo cp /var/archivematica/storage-service/storage.db ~/storage_db_backup.db
mysqldump -u root -p MCP > ~/am_backup.sql

If you do not have a password set for the root user in MySQL, you can take out the ‘-p’ portion of that command. If there is a problem during the upgrade process, you can restore your MySQL database from this backup and try the upgrade again.

If you’re upgrading from Archivematica 1.8 or lower to the 1.9 version or higher, the Elasticsearch version support changed from 1.x to 6.x and it’s also recommended to create a backup of your Elasticsearch data, especially if you don’t have access to the AIP storage locations in the local filesystem.

You can follow these steps in order to create a backup of Elasticsearch:

# Remove and recreate the folder that stores the backup
sudo rm -rf /var/lib/elasticsearch/backup-repo/
sudo mkdir -p /var/lib/elasticsearch/backup-repo/
sudo chown elasticsearch:elasticsearch /var/lib/elasticsearch/backup-repo/
# Allow elasticsearch to write files to the backup
echo 'path.repo: ["/var/lib/elasticsearch/backup-repo"]' |sudo tee -a /etc/elasticsearch/elasticsearch.yml
# Restart ElasticSearch and wait for it to start
sudo service elasticsearch restart
sleep 60s
# Configure the ES backup
curl -XPUT "localhost:9200/_snapshot/backup-repo" -H 'Content-Type: application/json' -d \
'{
     "type": "fs",
     "settings": {
     "location": "./",
     "compress": true
     }
 }'
# Take the actual backup, and copy it to a safe place
curl -X PUT "localhost:9200/_snapshot/backup-repo/am_indexes_backup?wait_for_completion=true"
cp /var/lib/elasticsearch/backup-repo elasticsearch-backup -rf

For more info, refer to the ElasticSearch 6.8 docs.

Upgrade on Ubuntu packages¶

1. Update the operating system.

   sudo apt-get update && sudo apt-get upgrade
   

2. Update package sources.

   In Ubuntu 16.04:

   wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
   echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
   echo 'deb [arch=amd64] http://packages.archivematica.org/1.12.x/ubuntu xenial main' >> /etc/apt/sources.list
   echo 'deb [arch=amd64] http://packages.archivematica.org/1.12.x/ubuntu-externals xenial main' >> /etc/apt/sources.list
   

   Optionally you can remove the lines referencing packages.archivematica.org/1.11.x from /etc/apt/sources.list.

   In Ubuntu 18.04:

   echo 'deb [arch=amd64] http://packages.archivematica.org/1.12.x/ubuntu bionic main' >> /etc/apt/sources.list
   echo 'deb [arch=amd64] http://packages.archivematica.org/1.12.x/ubuntu-externals bionic main' >> /etc/apt/sources.list
   

   Optionally you can remove the lines referencing packages.archivematica.org/1.11.x from /etc/apt/sources.list.

3. Update the Storage Service.

   sudo apt-get update
   sudo apt-get install archivematica-storage-service
   

4. Update Archivematica. During the update process you may be asked about updating configuration files. Choose to accept the maintainers versions. You will also be asked about updating the database - say ‘ok’ to each of those steps. If you have set a password for the root MySQL database user, enter it when prompted.

   sudo apt-get install archivematica-common
   sudo apt-get install archivematica-dashboard
   sudo apt-get install archivematica-mcp-server
   sudo apt-get install archivematica-mcp-client
   

5. Restart services.

   sudo service archivematica-storage-service restart
   sudo service gearman-job-server restart
   sudo service archivematica-mcp-server restart
   sudo service archivematica-mcp-client restart
   sudo service archivematica-dashboard restart
   sudo service nginx restart
   

6. Depending on your browser settings, you may need to clear your browser cache to make the dashboard pages load properly. For example in Firefox or Chrome you should be able to clear the cache with control-shift-R or command-shift-F5.

Upgrade on CentOS/Red Hat packages¶

1. Upgrade the repositories for 1.12:

   sudo sed -i 's/1.11.x/1.12.x/g' /etc/yum.repos.d/archivematica*
   

2. Remove the current installed version of ghostscript:

   sudo rpm -e --nodeps ghostscript ghostscript-x11 \
                        ghostscript-core ghostscript-fonts
   

3. Upgrade Archivematica packages:

   sudo yum update
   

4. Once the new packages are installed, upgrade the databases for both Archivematica and the Storage Service. This can be done with:

   sudo -u archivematica bash -c " \
       set -a -e -x
       source /etc/default/archivematica-dashboard || \
           source /etc/sysconfig/archivematica-dashboard \
               || (echo 'Environment file not found'; exit 1)
       cd /usr/share/archivematica/dashboard
       /usr/share/archivematica/virtualenvs/archivematica-dashboard/bin/python manage.py migrate --noinput
   ";
   
   sudo -u archivematica bash -c " \
       set -a -e -x
       source /etc/default/archivematica-storage-service || \
           source /etc/sysconfig/archivematica-storage-service \
               || (echo 'Environment file not found'; exit 1)
       cd /usr/lib/archivematica/storage-service
       /usr/share/archivematica/virtualenvs/archivematica-storage-service/bin/python manage.py migrate
   ";
   

5. Restart the Archivematica related services, and continue using the system:

   sudo systemctl restart archivematica-storage-service
   sudo systemctl restart archivematica-dashboard
   sudo systemctl restart archivematica-mcp-client
   sudo systemctl restart archivematica-mcp-server
   

6. Depending on your browser settings, you may need to clear your browser cache to make the dashboard pages load properly. For example in Firefox or Chrome you should be able to clear the cache with control-shift-R or command-shift-F5.

Upgrade on Vagrant / Ansible¶

This upgrade method will work with Vagrant machines, but also with cloud based virtual machines, or physical servers.

1. Connect to your Vagrant machine or server

   vagrant ssh # Or ssh <your user>@<host>
   

2. Install Ansible

   sudo pip install ansible
   

3. Checkout the deployment repo:

   git clone https://github.com/artefactual/deploy-pub.git
   

4. Go into the appropiate playbook folder, and install the needed roles

   Ubuntu 16.04 (Xenial):

   cd deploy-pub/playbooks/archivematica-xenial
   ansible-galaxy install -f -p roles/ -r requirements.yml
   

   Ubuntu 18.04 (Bionic):

   cd deploy-pub/playbooks/archivematica-bionic
   ansible-galaxy install -f -p roles/ -r requirements.yml
   

   Centos 7:

   cd deploy-pub/playbooks/archivematica-centos7
   ansible-galaxy install -f -p roles/ -r requirements.yml
   

   All the following steps should be run from the respective playbook folder for your operating system.

5. Verify that the vars-singlenode.yml has the appropiate contents for Elasticsearch and Archivematica, or update it with your own

6. Create a hosts file.

   echo 'am-local   ansible_connection=local' > hosts
   

7. Upgrade Archivematica running

   ansible-playbook -i hosts singlenode.yml --tags=elasticsearch,archivematica-src
   

Upgrade in indexless mode¶

As of Archivematica 1.7, Archivematica can be run in indexless mode; that is, without Elasticsearch. Installing Archivematica without Elasticsearch, or with limited Elasticsearch functionality, means reduced consumption of compute resources and lower operational complexity. By setting the archivematica_src_search_enabled configuration attribute, administrators can define how many things Elasticsearch is indexing, if any. This can impact searching across several different dashboard pages.

1. Upgrade your existing Archivematica pipeline following the instructions above.

2. Modify the relevant systemd EnvironmentFile files by adding lines that set the relevant environment variables to false.

   If you are using Ubuntu, run the following commands.

   sudo sh -c 'echo "ARCHIVEMATICA_DASHBOARD_DASHBOARD_SEARCH_ENABLED=false" >> /etc/default/archivematica-dashboard'
   sudo sh -c 'echo "ARCHIVEMATICA_MCPSERVER_MCPSERVER_SEARCH_ENABLED=false" >> /etc/default/archivematica-mcp-server'
   sudo sh -c 'echo "ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_SEARCH_ENABLED=false" >> /etc/default/archivematica-mcp-client'
   

   If you are using CentOS, run the following commands.

   sudo sh -c 'echo "ARCHIVEMATICA_DASHBOARD_DASHBOARD_SEARCH_ENABLED=false" >> /etc/sysconfig/archivematica-dashboard'
   sudo sh -c 'echo "ARCHIVEMATICA_MCPSERVER_MCPSERVER_SEARCH_ENABLED=false" >> /etc/sysconfig/archivematica-mcp-server'
   sudo sh -c 'echo "ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_SEARCH_ENABLED=false" >> /etc/sysconfig/archivematica-mcp-client'
   

3. Restart services.

   If you are using Ubuntu, run the following commands.

   sudo service archivematica-dashboard restart
   sudo service archivematica-mcp-client restart
   sudo service archivematica-mcp-server restart
   

   If you are using CentOS, run the following commands.

   sudo -u root systemctl restart archivematica-dashboard
   sudo -u root systemctl restart archivematica-mcp-client
   sudo -u root systemctl restart archivematica-mcp-server
   

4. If you had previously installed and started the Elasticsearch service, you can turn it off now.

   sudo -u root systemctl stop elasticsearch
   sudo -u root systemctl disable elasticsearch
   

Upgrade with output capturing disabled¶

As of Archivematica 1.7.1, output capturing can be disabled at upgrade or at any other time. This means the stdout and stderr from preservation tasks are not captured, which can result in a performane improvement. See the Task output capturing configuration <task-output-capturing-admin> page for more details. In order to disable output capturing, set the ARCHIVEMATICA_MCPCLIENT_MCPCLIENT_CAPTURE_CLIENT_SCRIPT_OUTPUT environment variable to false and restart the MCP Client process(es). Consult the installation instructions for your deployment method for more details on how to set environment variables and restart Archivematica processes.

Back to the top

Update search indices¶

Archivematica 1.12.0 introduces new fields to the search indices and makes some changes to text field types in the Elasticsearch index mappings. To ensure that sorting works as expected on columns in the Backlog and Archival Storage tabs, it is necessary to update the search indices as the final step of the upgrade to Archivematica 1.12.0.

This can be accomplished one of two ways. Minimally, you can update the Elasticsearch mappings. This will ensure that sorting works correctly on existing data, but will not populate the new fields introduced in Archivematica 1.12.0 for transfers and AIPs created prior to the upgrade. To fully populate all of the fields available in Archivematica 1.12, you’ll need to recreate the indices.

sudo -u archivematica bash -c " \
    set -a -e -x
    source /etc/default/archivematica-dashboard || \
        source /etc/sysconfig/archivematica-dashboard \
            || (echo 'Environment file not found'; exit 1)
    cd /usr/share/archivematica/dashboard
    /usr/share/archivematica/virtualenvs/archivematica-dashboard/bin/python \
        manage.py update_elasticsearch_mappings
";

sudo -u archivematica bash -c " \
    set -a -e -x
    source /etc/default/archivematica-dashboard || \
        source /etc/sysconfig/archivematica-dashboard \
            || (echo 'Environment file not found'; exit 1)
    cd /usr/share/archivematica/dashboard
    /usr/share/archivematica/virtualenvs/archivematica-dashboard/bin/python \
        manage.py rebuild_transfer_backlog --from-storage-service --no-prompt
";

sudo -u archivematica bash -c " \
    set -a -e -x
    source /etc/default/archivematica-dashboard || \
        source /etc/sysconfig/archivematica-dashboard \
            || (echo 'Environment file not found'; exit 1)
    cd /usr/share/archivematica/dashboard
    /usr/share/archivematica/virtualenvs/archivematica-dashboard/bin/python \
        manage.py rebuild_aip_index_from_storage_service --delete-all
";