Manual normalization¶

Workflow 1: Manual normalization prior to transfer to Archivematica¶

This workflow assumes that users have created access and/or preservation copies of their files before the material is transferred into Archivematica.

For example, during a digitization project, high-resolution original (master) images as well as lower-resolution access copies may be created. It would be redundant of Archivematica to duplicate this normalization work. By adding a folder called manualNormalization to the transfer prior to processing, the Archivematica can recognize that normalization work has already been done so that Archivematica does not replicate the effort.

1. Create your files - the original (master) copies and the manually normalized derivatives. Note that the filenames must match in order for Archivematica to recognize the links between the original file and its derivatives.

   to-be-preserved/
   ├── hooded-pitcher-plant.gif
   ├── hooded-pitcher-plant.png
   ├── hooded-pitcher-plant.tif
   ├── ornamental-onion.gif
   ├── ornamental-onion.png
   ├── ornamental-onion.tif
   ├── pepperbox-poppy.gif
   ├── pepperbox-poppy.png
   └── pepperbox-poppy.tif
   

   In this example, there are three copies of each file - a GIF, a PNG, and a TIFF. The PNG files will be used as the original files, the TIFF files as preservation derivatives, and the GIF files as access derivatives.

2. Create a transfer directory and place the original files inside at the top level.

   my-manually-normalized-material/
   ├── hooded-pitcher-plant.png
   ├── ornamental-onion.png
   └── pepperbox-poppy.png
   

3. Create a subdirectory called manualNormalization (note the capital N). Inside of the manualNormalization directory, create directories called access and preservation (you do not need both - if you only have preservation copies, create only the preservation directory, and vice versa). Place your manually normalized files in the appropriate directory.

   my-manually-normalized-material/
   ├── manualNormalization
   │   └── access
   │       ├── hooded-pitcher-plant.gif
   │       ├── ornamental-onion.gif
   │       └── pepperbox-poppy.gif
   │   └── preservation
   │       ├── hooded-pitcher-plant.tif
   │       ├── ornamental-onion.tif
   │       └── pepperbox-poppy.tif
   ├── hooded-pitcher-plant.png
   ├── ornamental-onion.png
   └── pepperbox-poppy.png
   

   In the example above, you can see that the GIF access derivatives have been moved to the manualNormalization/access directory and the TIFF preservation derivatives have been moved to the manualNormalization/preservation directory.

   Note that the filenames must match in order for Archivematica to recognize the links between the original file and its derivatives.

4. Place the transfer in your transfer source location and start a Standard transfer as usual. For instructions, see the Transfer and Ingest documentation.

5. When you reach the normalization microservice on the Ingest tab, choose whether to normalize for access, preservation, or both.

   Note

   If you choose to review normalization results prior to accepting, you will find your manualNormalization folder and its contents in the objects directory of your transfer, with its access and/or preservation folders and their content intact.

6. Once the AIP is created, you will find that the originals and their preservation derivatives are now located in the data directory of the AIP as usual. Note that the manualNormalization directory and subfolders were deleted once normalization was complete. If you chose to normalize for access, a DIP will be created as well.

my-manually-normalized-material/
├── manualNormalization
│   └── access
│       └── pepperbox-poppy.gif
│   └── preservation
│       ├── hooded-pitcher-plant.tif
│       └── pepperbox-poppy.tif
├── hooded-pitcher-plant.png
├── ornamental-onion.png
└── pepperbox-poppy.png

Workflow 2: Manual normalization during Ingest¶

This workflow assumes that users have created access and/or preservation copies of their files while the material is being processed by Archivematica. For example, during processing the user might decide that the material needs to be normalized in a specialized way (for example, using proprietary tools).

This workflow requires that a user start a transfer in Archivematica, then use external normalization tools to create the required derivatives, manually place the derivatives within the currently processing directory on disk, and then return to Archivematica to proceed.

1. Create a transfer directory and place the original files inside at the top level.

   my-manually-normalized-material/
   ├── hooded-pitcher-plant.png
   ├── ornamental-onion.png
   └── pepperbox-poppy.png
   

2. Place the transfer in your transfer source location and start a Standard transfer as usual. For instructions, see the Transfer and Ingest documentation.

3. When you reach the normalization microservice on the Ingest tab, choose Normalize manually from the dropdown menu.

4. Manually normalize your files.

5. Using an SFTP file browser or SSH in your command line, go to your pipeline’s Processing Location and find the approveNormalization watched directory. The default location is /var/archivematica/sharedDirectory/watchedDirectories/approveNormalization/preservation/, which will contain your transfer directory. You may need to look at your Storage Service configuration to confirm the Processing Location defined for your pipeline.

6. Copy your manually normalized derivatives into your transfer directory. You will need to create a directory structure containing a manualNormalization directory. Inside of the manualNormalization directory, create directories called access and preservation (you do not need both - if you only have preservation copies, create only the preservation directory, and vice versa). Place your manually normalized files in the appropriate directory.

   my-manually-normalized-material/
   ├── manualNormalization
   │   └── access
   │       └── pepperbox-poppy.gif
   │   └── preservation
   │       ├── hooded-pitcher-plant.tif
   │       └── pepperbox-poppy.tif
   ├── hooded-pitcher-plant.png
   ├── ornamental-onion.png
   └── pepperbox-poppy.png
   

   In the above example, preservation and access derivatives were created for pepperbox-poppy.png, while only a preservation copy was created for hooded-pitcher-plant.png.

   Note that the filenames must match in order for Archivematica to recognize the links between the original file and its derivatives.

7. Return to the dashboard and approve normalization.

8. Once the AIP is created, you will find that the originals and their preservation derivatives are now located in the data directory of the AIP as usual. Note that the manualNormalization directory and subfolders were deleted once normalization was complete. If you had access copies, a DIP will be created as well.

Adding PREMIS eventDetail for manual normalization¶

When Archivematica carries out normalization, a PREMIS event is created and added to the METS that describes the normalization event. However, when you manually normalize material, Archivematica does not create a PREMIS event. This feature allows you to write a PREMIS event for manually normalized materials.

Regardless of whether you include manually normalized files in the initial transfer (Workflow 1) or during processing (Workflow 2), the procedure for adding PREMIS eventDetail is the same.

1. Follow the steps in Workflow 1 or 2, above.

2. After normalization is approved, Archivematica will stop and remind you to add metadata. Click on the metadata icon.

   

3. Select “Manual normalization event detail” under metadata.

   

4. Add event detail. This might be technical tool output, like: program=ImageMagick; version=6.6.4.0; command=%convertPath% %fileFullName% +compress %preservationFileDirectory%%fileTitle%.%preservationFormat% or something more narrative like in the example below.

   

5. Save the metadata and click on the ingest tab to continue processing.

6. Select Continue from the metadata reminder dropdown box to continue processing.

Normalizing files with the same name¶

The manual normalization workflow outlined above assumes that there are no conflicts between filenames. For example, if the original filenames are file1.doc and file2.xls, there is no conflict. But if you are manually normalizing both file1.doc and file1.xls, even if the extensions of the normalized files are different, you will need to provide a CSV file to document the relationships between the original and normalized files.

1. Create your directory structure.

   manually-normalized-duplicate-names/
   ├── manualNormalization
   │   └── access
   │       ├── file1.jpg
   │       └── file1.pdf
   │   └── preservation
   │       ├── file1.pdf
   │       └── file1.tif
   ├── file1.crw
   ├── file1.doc
   └── file1.tga
   

   In the example above, file1.tga has a JPG access derivative and a TIFF preservation derivative. file1.doc has PDF access and preservation derivatives. file1.crw does not have any derivatives.

2. Create a CSV file called normalization.csv. The CSV must contain three columns: the path of the original file, the path of the access derivative, and the path of the preservation derivative. All the files with duplicate names need to be listed, even if they do not have manually normalized derivatives.

   file1.tga	manualNormalization/access/file1.jpg	manualNormalization/preservation/file1.tif
   file1.doc	manualNormalization/access/file1.pdf	manualNormalization/preservation/sfile1.pdf
   file1.crw

   Important

   normalization.csv is case-sensitive. Ensure that file paths are exact. Do not include column headers.

3. Include the CSV file at the top level of your transfer.

   manually-normalized-duplicate-names/
   ├── manualNormalization
   │   └── access
   │       ├── file1.jpg
   │       └── file1.pdf
   │   └── preservation
   │       ├── file1.pdf
   │       └── file1.tif
   ├── file1.crw
   ├── file1.doc
   ├── file1.tga
   └── normalization.csv
   

4. Transfer as per Workflow 1 or 2, above.

In the example above, there are manually normalized files for both preservation and access, however it is possible to have manually normalized files for only access or preservation by leaving the appropriate column blank.

Other scenarios are also possible:

Original file not in a subdirectory, normalized copies are:

file1.tga,manualNormalization/access/subdir1/file1.jpg,manualNormalization/preservation/subdir1/file1.tif
file1.doc,manualNormalization/access/subdir1/file1.pdf,manualNormalization/preservation/subdir1/file1.pdf

Only some normalized copies in a subdirectory:

file1.tga,manualNormalization/access/subdir1/file1.jpg,manualNormalization/preservation/subdir1/file1.tif
file1.doc,manualNormalization/access/file1.pdf,manualNormalization/preservation/subdir1/file1.pdf

Subdirectories used only to distinguish file names:

file1.tga,manualNormalization/access/subdir1/file1.jpg,manualNormalization/preservation/subdir1/file1.tif
file1.ppm,manualNormalization/access/subdir2/file1.jpg,manualNormalization/preservation/subdir2/file1.tif

If the file names contain spaces or forbidden characters, Archivematica will sanitize the file names by replacing the spaces/characters with underscores. The normalization.csv file will still work by using either the original, unsanitized file names, or by pre-sanitizing the normalized filenames and the corresponding columns in normalization.csv.

Back to the top