Metadata That Survives: Describing Digitized Records So They Can Still Be Found

A digitization project produces two things: the images, and the explanation of the images. Projects almost always protect the first and neglect the second. Ten years on, the TIFFs are sitting safely on redundant storage, checksummed and backed up, and nobody can say which deed book IMG_04417.tif came from, or whether the spreadsheet that would have told them is the current one or the copy someone saved to their desktop before they retired.

This piece is about metadata that survives that decade: what to record, where to put it, how to name things, and how to stop the index from quietly drifting away from the files. I'll finish with a complete worked record so you can see all the parts in one place.

Three kinds of metadata

People use "metadata" to mean very different things. Splitting it into three types makes it easier to decide who records what, and when.

Technical metadata is the easiest to get right because machines produce it. Tools such as ExifTool and JHOVE will read it straight out of the files; there is no reason for a person to type pixel dimensions into a spreadsheet. For still images the reference data dictionary is NISO Z39.87, often called MIX, and for preservation events (format migrations, fixity checks, who did what) the PREMIS data dictionary is the usual model. You don't need to implement either in full to borrow their field lists.

Descriptive metadata is where projects stumble, so most of the rest of this piece is about it.

Dublin Core: fifteen fields that travel

Dublin Core is a set of fifteen general-purpose elements maintained by the Dublin Core Metadata Initiative (see dublincore.org). It is deliberately simple, which is exactly why it is useful: almost every repository platform, digital collections system and aggregator can take Dublin Core in and hand it back out. If your local fields map cleanly to it, your descriptions can move to the next system without a rescue project.

Two habits make Dublin Core work for records. First, decide locally what each element means for each record series and write it down; "creator" for a deed could mean the grantor, the scribe or the registry, and three volunteers will choose three answers unless told. Second, keep richer local fields in your own system and map them to Dublin Core on export, rather than squeezing everything into fifteen boxes.

Archivist’s kit

• Drives

  UGREEN NASync network storage

  A network drive for home or office that keeps a full set of masters on your own shelf, reachable from every computer.

  • Models from 2 to 8 drive bays
  • Takes third-party hard drives
  • Two-year maker warranty
  Open UGREEN

Picked from maker specifications and published reviews. We have not bench-tested these units.

More in the Archivist’s kit

Controlled vocabularies

Free text is where findability goes to die. One cataloger writes "Quit-claim deed," another "quitclaim," a third "QCD," and a search for any one of them misses two-thirds of the set. Controlled vocabularies fix this by giving each concept one preferred form.

Most small offices end up with a local list for the things that matter most to them, usually document types and place names, and a published vocabulary for everything else. That's fine. What matters is that the list exists, lives in one place, and gets used by every person entering data.

File names that still make sense

A file name is the one piece of metadata that can't be separated from the file. Make it carry an identifier, not a description.

Keep names lowercase, keep them reasonably short, and never reuse one. If a page is rescanned, give the new file a version suffix and record why in the administrative metadata.

Embedded or sidecar

Metadata can live inside the file (XMP, EXIF and IPTC blocks in a TIFF, JPEG or PDF) or beside it, in a sidecar file or a database. Each has a real weakness.

My recommendation for most projects: keep the authoritative record in one database or catalog, export sidecars (CSV, JSON or XML) alongside the files for every delivery, and embed a small, stable set inside each master: identifier, title, source, rights. Embed it at the production stage, then generate your checksums, and don't touch the embedded block afterward. Otherwise every metadata correction changes the master file and sets off a fixity alarm.

The index nobody updates

Almost every project I've been brought into late has the same artifact: a spreadsheet that was accurate on the day the scanning ended. Since then files have been renamed, a volume was rescanned, two boxes turned out to be misfiled, and corrections were made in the access system but not in the spreadsheet, or the other way round. Nobody is wrong, exactly. There are just three versions of the truth.

The fix is organizational more than technical:

• Name one authoritative source for descriptive metadata, and say so in writing.
• Make corrections there and only there; regenerate everything else from it.
• Give one person the job of owning it, with a backup.
• Run a reconciliation every quarter: list every file in storage, list every record in the index, and compare. Files with no record and records with no file both need an explanation.
• Record changes with a date and initials, even if that's just a change-log tab.

If you are running OCR or handwritten text recognition, treat the transcription as metadata with the same rules: tied to the identifier, versioned, and stored with the rest. Our piece on OCR and handwritten text recognition for archives covers how to judge and store that output.

Worked example: one deed book page

Here is a full record for a single page from a fictional town, Pell Harbor. The source is page 212 of Deed Book 14, imaged as part of a larger project to make a preservation copy of the land records.

Notice what the file name holds (only the identifier) and what it doesn't. Notice that grantor and grantee are recorded as contributors with roles in parentheses, which is a local decision written into the project's data dictionary. And notice that nothing here depends on a single spreadsheet cell staying in sync with a file name somewhere else. The master TIFF carries the identifier, title, source and rights embedded; the rest lives in the catalog and goes out as a sidecar with every export. For why the master is a TIFF in the first place, see our guide to file formats for long-term preservation.

If you only have time to do one thing on a project already underway, do the reconciliation. It will tell you how much of your explanation has already gone missing, while the people who remember are still around to put it back.