Transfer

In Archivematica, Transfer is the process of transforming any set of digital objects and/or directories into a SIP. Transformation may include appraisal, arrangement, description and identification of donor restricted, private or confidential contents. The Transfer tab prepares your content for preservation in Archivematica.

In the Transfer tab of the Dashboard, the user moves digital objects from source directories accessible via the Storage Service into Archivematica. See Administrator manual - Storage Service for instructions on how to set up shared transfer source directories. Once uploaded to the dashboard, transfers run through several micro-services: UUID assignment; checksum verification (if checksums are present); package extraction (i.e. unzipping of zipped or otherwise packaged files); virus checking; indexing; format identification and validation; and metadata extraction.

At the end of transfer, the user creates a SIP from one or more standard transfer(s). Once this is done, the SIP can be moved into ingest or sent to a backlog for processing later.

On this page:

See also

If you would like to import lower-level metadata with your transfer (i.e. metadata to be attached to subdirectories and files within a SIP), see Metadata import.

If your transfer is a DSpace export, please see DSpace export.

If your transfer is a bag or a zipped bag, please see Bags.

If your transfer includes access and service/mezzanine files, please see Access and service files.

If your transfer is composed of digital forensic disk images, please see Forensic image processing.

If you would like to skip some of the default choices for dashboard decision points or make preconfigured choices for your desired workflow, see Processing configuration.

Create a transfer

Open the web browser and the Archivematica dashboard and sign in with your username and password. Please note that if this is your first time logging in to a newly installed instance of Archivematica, you will see a log-in that allows you to register your repository and get updates to the format policy registry. For more information, see Format Policy Registry (FPR) and the Administrator manual - Format Policy Registry (FPR).

  1. In the transfer tab, select your transfer type in the dropdown menu. Types include Standard, Unzipped Bag, Zipped Bag, DSpace, maildir, and Disk Image.
  2. In transfer tab, name your transfer and select your source directory. Click Browse to select the directory containing your object(s) for upload. If you would like your transfer to be composed of multiple directories from different sources, repeat this step.
Select transfer(s) from source directory(ies)

Select transfer for transfers from source directories

As shown above, Archivematica will display the number of objects in each available transfer directory.

  1. If applicable, enter an accession number for the transfer.
  2. Once all of your digital object sources have been uploaded, hit the Start Transfer button for the transfer processing to begin.
Start transfer in dashboard

Start transfer

Create a transfer with submission documentation

The structured directory in Archivematica is the basic configuration of the transfer. If you just add a directory to the dashboard and start transfer processing, Archivematica will restructure it so it complies with this structure. Another option is to create a transfer in a structured directory prior to beginning processing in Archivematica.

To create a transfer that includes submission documentation, such as a donor agreement, you must create a structured directory prior to beginning processing in Archivematica.

  1. Your source directory can be called anything (transfer in this example). Place all of the digital objects that you are preserving in the transfer directory (you can add subdirectories if you wish, but they are not required).
  2. Create a subdirectory called metadata (no capitalization).
  3. Create a subdirectory within metadata called submissionDocumentation (in camel-case). Place any transfer forms, donation agreements or any other documents that relate to the acquisition of the records inside submissionDocumentation.

Important

Please do not include submission documentation that has non-standard characters in the filename, as submission documentation names are not sanitized. Any filenames other than plain ASCII names may cause errors in processing.

Create a transfer with existing checksums

If you have existing checksums that you would like to preserve along with your transfer, you can include them in your transfer directory. Archivematica will verify .md5, .sha1 and .sha256 checksums in the Verify checksums micro-service during the Transfer workflow in the dashboard.

  1. Your source directory can be called anything (transfer in this example). Place all of the digital objects that you are preserving in the transfer directory (you can add subdirectories if you wish, but they are not required).
  2. Create a subdirectory called metadata (no capitalization).
  3. Your checksums should be contained in text files named the following way: checksum.md5, checksum.sha1 or checksum.sha256. Place the checksum files in the metadata directory.
  4. Begin your standard transfer in the Archivematica dashboard interface.
  5. If your checksum check fails, this micro-service will show an error. The micro-service line will turn red. The log for this is sent to the /logs folder of the transfer. The file in the log folder should look like “checksum.md5 -Check-Tue Oct 14 13:56:51 PDT 2014”

Create a transfer with an existing structmap

It’s possible to apply an externally-created logical METS structmap to a transfer by including an XML file with the transfer. This can be used for internal purposes, like identifying the sections of a book, or to apply levels of description to be used when uploading the DIP to AtoM.

  1. Your source directory can be called anything (transfer in this example). Place all of the digital objects that you are preserving in the transfer directory (you can add subdirectories if you wish, but they are not required).
  2. Create a subdirectory called metadata (no capitalization).
  3. Your structmap must be contained in an XML file named mets_structmap.xml (no capitalization). the structmap must refer to the filenames of the objects in the transfer exactly. Place the structmap file in the metadata directory.
Structure of the logical mets_structmap.xml.

Logical mets_structmap.xml file showing the first few pages of a book, with AtoM levels of description applied (item, series, fonds).

  1. Begin your standard transfer in the Archivematica dashboard interface.
  2. In order to confirm that your structmap was correctly applied to the contents of the transfer, check the METS file contained within the AIP. The structmap is usually near the very end of the METS file. It should reflect the logical structmap that you uploaded with the transfer.
  3. If you are uploading a DIP to AtoM, your structmap should include the applicable Atom level of description as the mets:div TYPE (see image above). Once you have uploaded the DIP to AtoM, the objects should have the levels of description as defined in the structmap.

Important

In order to use the structmap to assign levels of description to the digital objects, you must use the levels of description defined in your AtoM instance without capital letters (for example, item instead of Item).

Note

It is not required to use AtoM levels of description for the mets:div TYPE. The value can be anything and does not need to adhere to any convention. For example, the sample provided in the image above could use the terms “book”, “chapter”, and “page” instead of “fonds”, “series”, and “item”.

Process the transfer

  1. In the dashboard transfer tab, the transfer will appear in the dashboard with a bell icon next to it. This means that it is awaiting a decision by the user.
  2. Click on the micro-service to display jobs that have completed, including the one requiring action.
  3. In the Actions drop-down menu, select “Approve transfer” to begin processing the transfer. You may also “Reject transfer” and quit processing.
In the Actions drop-down menu, select "Approve transfer"

In the Actions drop-down menu, select “Approve transfer”

  1. The transfer will now run through a series of micro-services. These include:
  • Verify transfer compliance: verifies that the transfer is properly structured - i.e. with the logs, metadata and objects folders.
  • Rename with transfer UUID: assigns a unique universal identifier for the transfer as a whole; directly associates the transfer with its metadata.
  • Assign file UUIDs and checksums to objects: assigns a unique universal identifier and sha-256 checksum to each file in the /objects directory.
  • Verify transfer checksums: verifies any checksums included with the transfer in its metadata directory.
  • Generate METS.xml document: creates a METS file capturing the original order of the transfer. This METS file is added to any SIPs generated from this transfer.
  • Quarantine: quarantines the transfer to a set duration based on preconfiguration settings in the Administration tab of the dashboard. This can be used to allow virus definitions to update before a virus scan.
  • Scan for viruses: scans for viruses and malware.
  • Generate transfer structure report: generates a directory tree of the original transfer and places as a text file in the AIP.
  • Clean up file and directory names: removes prohibited characters from folder and filenames, such as ampersands.
  • Identify file format: this is the identification that normalization will be based upon, the user can choose between FIDO and extension or skipping format identification at this stage. See Format identification below for more information.
  • Extract packages: extracts contents from zipped or otherwise packaged files. You can change your preconfigured workflow settings to allow for some choices about package extraction. See Extract Packages below for more information.
  • Characterize and extract metadata: identifies and validates file formats; extracts technical metadata embedded in the files. If you have preconfigured it to do so, Archivematica will stop during this micro-service and allow the user to choose a file identification command from a dropdown menu. To learn about preconfigured options, please see Administrator manual - Processing configuration. Archivematica’s file identification default is set to identification by file extension. You can also choose to skip identification and run it later, during Ingest, instead.
  • Complete transfer: Includes indexing the transfer.
  1. A transfer that is in the middle of processing will show which micro-services have been completed (green) and which are in progress (orange).
  2. When a micro-service fails or encounters an error, the micro-service background turns from green to pink and a “failed” icon appears next to the transfer or SIP name. See Error handling for more information about how to handle an error.
  3. Once the transfer micro-services are completed, a bell icon will appear next to the transfer. This means that the transfer is ready to be packaged into a SIP for ingest or sent to a backlog, indexed and stored to be retrieved for processing at a later date
A transfer that is ready to be packaged into a SIP or stored in backlog

A transfer that is ready to be packaged into a SIP or stored in backlog

  • Option 1: Select “Create single SIP and continue processing”
  • Option 2: Select “Send transfer to backlog”. In this case, your transfer will be stored in a backlog in the same location as your AIP store so that you can retrieve one or more transfers from the Ingest tab for processing at a later date. See Managing a backlog.
  • Option 3: Select “Reject the transfer”.
  1. See Ingest for next steps.

Cleaning up the transfer dashboard

The dashboard in the Transfer tab should be cleaned up from time to time. As the list of transfers grows, it takes Archivematica longer and longer to parse this information which can create browser timeout issues.

1. When you are ready to remove a transfer from the dashboard, click the remove icon found next to the add metadata icon, to the right of the transfer name.

  1. Click the Confirm button to remove the transfer from the dashboard.
A transfer that is ready to be removed from the dashboard

Note

This does not delete the transfer or related entities, including the source directory.

Format identification

Archivematica’s default is to allow the user to choose identification options to base normalization actions upon during transfer and then use those results to base normalization upon in ingest. However, you can set your preconfiguration options to allow for the opposite (skip at transfer and identify before normalization) or for both transfer and ingest to allow for user choice in the dashboard.

Artefactual included the ability to skip identification at transfer and/or to change identification tool before normalization mainly to allow for the possibility that content in the transfer backlog may contain formats for which there are not currently entries in the Format Policy Registry (FPR). While the transfers are in the backlog, you can add rules that allow for the format(s) not identified or identifiable at time of transfer to the FPR so that, when they are processed through ingest, all formats will be identified and normalization attempted based on those identifications.

There may be other use case scenarios in the future that this configuration flexibility facilitates. In general, we aim to include as much flexibility as possible when it comes to workflow choices so that the archivist is as central as possible to AIP and DIP processing rather than hardcoding and automating so much that the archivist is left less influence on ingest.

Format identification is logged as a PREMIS event in the METS.xml using the results of running whichever tool chosen during processing.

Extract packages

If you adjust your processing configuration settings, Archivematica will stop after format identification and allow you to extract any packages in your transfer. Additionally, you can decide whether you would like to keep the package with the extracted objects or not.

Back to the top