And you thought government records were boring . . . .

Government archives sometimes get a bad rap: a lot of people are under the impression that they're stuffed full of old policy documents and other legally necessary but deadly dull stuff, and even archivists who work in other settings sometimes think that government records are pretty dry.

I could go into my standard spiel about how government records document the rights of citizens to hold property, receive benefits that they have earned, and participate in civic life and how government archives promote government transparency and accountability. I could also go on about how government archives attract not only academic historians and genealogists but also biologists, engineers, historic preservationists, linguists, epidemiologists, attorneys, documentary filmmakers, and all sorts of other users. All of these things are true.

However, one of the things I most like about working in government archives is that even the most humdrum-seeming of records series can contain the unexpected. Seven or eight years ago, several colleagues and I were moving a very large and red rot-plagued series of 19th century financial records and discovered that the series included a little volume bearing a crudely inked title: "No-Good Lawyers." It was a listing of Victorian-era attorneys who had, in various ways, run afoul of the New York State Banking Department -- and a welcome little diversion from a laborious and dirty task.

Sometimes the unanticipated finds are amusing, and sometimes they're horrifying. When I was still in grad school, I was examining a series of photographs taken at a psychiatric facility and found that, in addition to images of female patients playing around with cosmetics and staff-patient softball games, it included a series of photographs documenting the administration of electroconvulsive therapy (ECT). It took me a little while before I figured out precisely what was going on in those photos, and when I finally did, I was completely unnerved. I hastily put the photos back in their box, told the reference archivist what I had found, and fled the research room. I haven't seen those photographs in over a decade, but the thought of them is still unsettling.

And sometimes, of course, the finds are hilarious. Every day, the Web site of the U.S. National Archives and Records Administration (NARA) highlights one of the records in NARA's holdings. Many of the featured records are historically significant; for example, yesterday's featured document consists of the U.S. Congress's official copy of the Twelfth Amendment, which it passed on 9 December 1803. However, today's document, which was issued on 10 December 1959, concerns a less weighty matter: the United States government's efforts to find the Yeti.

Clicking on the image below will bring up a much larger and more legible version. I'm particularly fond of regulation no. 2.

"Regulations Governing Mountain Climbing Expeditions in Nepal - Relating to Yeti"; UD-WW, 1454, , Box 252, Accession #64-9-0814, folder 5.1 Political Situation - General, File ended Dec 31, 1959; Records of the Agency for International Development; Record Group 286; National Archives. Image courtesy of the U.S. National Archives and Records Administration.

TSA's bad PDF redaction . . . and tips on redacting PDFs properly

The Transportation Security Administration (TSA) is the latest in a long line of Fortune 500 companies and federal government agencies to discover that information can all too easily be recovered from an improperly redacted PDF document. On Sunday, blogger The Wandering Aramean announced that the TSA had posted a copy of its Screening Management Standard Operating Procedure manual, which provides detailed information about how TSA personnel screen passengers and luggage, on a federal contract soliciation Web site.

Portions of the manual, which is identified as containing Sensitive Security Information, were redacted, but . . . whoever did the redactions simply used Adobe Acrobat or other PDF-compatible software to draw black boxes over the information that should have been redacted. As I've noted before, it doesn't take tons of computer know-how to recover the information hiding under those black boxes, and The Wandering Aramean and lots of other people were able to do so. The TSA has pulled the manual off the federal contract site, but you can find a complete and unredacted copy here and on lots of other sites.

The TSA has stated that the version of the manual it posted has been superseded repeatedly, that it was never actually used by TSA personnel, and that TSA security procedures have changed substantially since it was written. However, the damage has been done: the blogosphere and the news media are having a field day, and Congress is demanding an investigation. I know that beating up on the TSA is something of a sport (and, believe me, I have some issues with its 3-1-1 policy), but I really do feel for the folks at TSA HQ who have to clean up this mess.

Putting poorly redacted PDFs on the Web seems to be something of a fad these days -- Google did it a few weeks ago -- but I don't want to see archivists or records managers fall prey to the pitfalls that have ensnared so many others. If you're trying to figure out how to provide access to PDFs that contain information restricted by law or donor agreement, here are a few pointers:

• If you're working with a PDF file, never, ever use Adobe Acrobat's Draw or Annotate tools (or comparable tools in other programs) to place black, white, etc. boxes over the information you wish to redact. All a savvy user needs to do is to copy the PDF in its entirety and paste it into a word processing document. Moreover, someone with ready access to Adobe Acrobat or comparable software can skip the copying and pasting and simply open the PDF and remove the boxes that you drew. Don't think that locking your PDF will keep this from happening: shareware that promises to unlock PDFs is all over the Interwebs.

• If you're working with a word processing document that you plan to convert to PDF format, never, ever attempt to redact information by changing the font color to white or using a shading or highlighting feature to obscure the text and then converting the document to PDF format. The copy-and-paste technique outlined above will reveal the hidden text; users might have to play with the font colors a bit, but doing so won't take them more than a few seconds.

At present, there are several good tools for redacting PDF files, and you'll need to assess your current software setup, the amount of redaction work you'll have to do, and your budget in order to decide which one works best for you.

• If you've got Adobe Acrobat 8 or 9 Professional, take a look at its built-in redaction tool. Adobe's Adobe for Legal Professionals blog provides detailed advice on how to use it.

• If you've got an older version of Acrobat, two third-party plug-ins for Adobe Acrobat, Redax and Redact-It, are time-tested and have substantial followings in the legal community.
  

• If you are using an older version of Adobe Acrobat and can't or don't want to upgrade or purchase an add-on tool, the National Security Agency has produced a document that outlines a laborious but effective redaction procedure.

• If you've got an old version of Acrobat, no money for an upgrade or a plug-in, and only a handful of documents to redact, you might want to consider printing out the documents, whipping out a black magic marker, and redacting information the old-fashioned way. Photocopy the redacted printouts to reduce the chance that the text can be read through the marker, then scan the photocopies.
  

If you do commit to redacting documents electronically:

• Make sure you know how to use your chosen redaction tool. Most of them are pretty straightforward, but slip-ups are possible, and you don't want slip-ups circulating on the Web. All of the software tools listed above are well-documented, so take the time needed to review and digest said documentation.

• Prepare a test file and familiarize yourself with your chosen software tool before you start working with real live documents. If you can get a disinterested third party (preferably one with lots of IT or digital forensics experience) to review your test file and verify that the information you've redacted really is gone, by all means do so.

• This may seem a bit obvious, but someone once asked me, so I'm going to come right out and say it: don't redact your original e-documents. Chances are, your documents will one day be fully discloseable, so make electronic copies of them, redact the copies, and keep both the copies and the originals. Doing so increases your storage and preservation commitments, but there really aren't any good alternatives, particularly for records warranting permanent retention.
  

• Keep abreast of the relevant legal and digital forensics literature: people are trying to figure out how to "break" all of the tools listed above and recover information redacted with these tools. One of them may eventually succeed, at which point all bets are off.

Finally, a gentle disclaimer: the above information is . . . simply information, not legal, financial, medical, dental, or any other kind of advice. As is the case with everything on this blog, it's not necessarily reflective of the opinions and policies of my employer, either. It does reflect my own knowledge at the time of this writing, but, as is the case with all things electronic, electronic redaction technology and best practices change rapidly. It's really up to you to investigate the options for yourself and to make sure that the electronic information you redact really can't be recovered.

Happy redacting!

"So far, it's the best job in the country"

Last week, David Ferriero, the new Archivist of the United States, delivered his first State of the Archives address. I was particularly cheered by his continuing emphasis on the challenges posed by electronic records and electronic records management, which he likened to the problems faced by Robert Digges Wimberly Connor, the first Archivist of the United States, who fought valiantly to ensure that the nation's long-neglected records were properly housed:

. . . . It seems to me that we are at a similar crossroads in the history of the Archives in the challenges we face with the electronic records of the agencies we serve. Varieties of technology, platforms, software, practice, and lack of standards complicate the work of ingesting, preserving, and making available the records of the government. The work we have undertaken with Lockheed Martin is, of course, being watched closely by our funders, our stakeholders, and the rest of the archival community who is grappling with similar issues of born digital records. We have to get this right.

I also see the Electronic Records Archives initiative as a vehicle for reestablishing our oversight of the records management programs of each agency—working with agencies to establish protocols, practices, and annual audits.

I also like that Ferriero recognizes the larger archival community's interest in the Electronic Records Archives, and I hope that he continues predecessor Allen Weinstein's effort to bring the U.S. National Archives and Records Administration into closer alignment with archival professional organizations and other repositories throughout the nation.

If you want a sense of Ferriero's background and personality, check out the lengthy profile in today's Washington Post, which highlights his decades of work in libraries and includes video footage of him examining materials in the stacks of the Archives I facility in Washington, DC. The video's only 42 seconds long, but it reveals that the new Archivist has a puckish sense of humor:

An archivist responds to Jon Stewart

As you all know, a few weeks ago, Jon Stewart had a little fun at the archival profession's expense. Now, the Woody Guth3 (who may or may not be archivist/lyricist David Kay) explain -- for the benefit of Mr. Stewart and all the other uninformed souls out there -- what we do and why most of us have at least one graduate degree:

Something to ponder

The Post Carbon Institute is a think tank that seeks to supply "individuals, communities, businesses, and governments with the resources needed to understand and respond to the interrelated economic, energy, and environmental crises that define the 21st century."

A couple of months ago, the Institute published "Our Evanescent Culture and the Awesome Duty of Librarians," in which Richard Heinberg outlined the macro-level threats to the survival of digital information. Among them: failure to maintain reliable sources of power generation and delivery, nuclear war, and the systemic vulnerabilities associated with living in an increasingly interconnected world.

Sometimes, those of us charged with preserving digital information are so focused on the very real short-term threats such as file corruption, hardware failure, and software obsolescence that we sometimes forget that, as Heinberg asserts, "digitization represents a huge bet on society’s ability to keep the lights on forever."

Can we keep the lights on forever? Even if you think that the Post Carbon Institute's being overly alarmist about global warming and fossil fuel supplies, you have to admit that we're taking an awfully big gamble. A number of years ago, an historian of medicine told me that, statistically speaking, humanity is really overdue for a pandemic that combines the mortality rate of AIDS with the contagiousness of the common cold -- and for the social, political, and economic havoc that such pandemics wreak. We haven't experienced a "hot" global war for over sixty years, but in the larger scheme of things, sixty years is the mere blink of an eye.

And, of course, it's quite likely that one day our culture will be known chiefly through archaeological digs and a few surviving artworks and texts. Woe betide the 30th century archeologist who unearths a cache of data tapes!

Heinberg concludes that, given the very real risk that digital information will be lost, librarians (and, by extension, archivists) should be mindful of the importance of "conservation of essential cultural knowledge in non-digital form." Maybe he's right: perhaps we should devote a little effort to leading the public discussion about how our culture should be remembered and making sure that at least some information about our values and accomplishments is preserved in human-readable form.

Read the whole article. It's really good.

(Hat-tip: Alan's Notes on Digital Preservation.)

A new archival use for Twitter

I've been resisting Twitter mightily -- my tolerance for information clutter is pretty low -- but this splendid Twitter-driven initiative just might induce me to take the plunge . . . .

Archivists at Cambridge University's Scott Polar Research Institute are using Twitter to draw attention to the diary that Captain Robert Falcon Scott kept during his ill-fated 1910-1912 Antarctic expedition: every day, they tweet the first 140 characters of the diary entry that Scott penned exactly 99 years before.

Each tweet also includes a link to a blog, Scott's Last Expedition, that provides the full text of the diary entry and archival images. The blog itself is pretty neat, too: the only dates that appear on it are those of the diary entries, and as a result the blog looks as if it were written by Scott himself. The entries thus have an immediacy and intimacy that they would lack if accompanied by scholarly commentary.

The Scott Polar Research Institute plans to launch similar Web 2.0 initiatives focusing on other British polar explorers. More information about the Scott project and the Institute's future plans is available here.

Scholars and archivists have long known that archival records tell compelling stories, but most people tend to think of records as being of interest to history buffs, civil servants, and a few other quiet oddballs. The Scott Polar Research Institute's project promises to present Scott's diary in a way that will grab and hold the attention of 21st-century people, and other repositories certainly hold materials that could be brought to life in the same way. I really want to see lots and lots of archives follow in the Institute's footsteps. I promise I'll follow all your projects on Twitter.

Happy Thanksgiving

"World War II-Kids Reenacting Thanksgiving." Image courtesy of the New York State Archives. (New York (State). Dept. of Health. Bureau of Communications Production. Photography Unit. Photographic prints and negatives of department officials, facilities, and activities, ca. 1920-1983. Series 14655-88, Box 58.)

Wherever you are and whatever you're doing, I wish you a happy and safe Thanksgiving. Blogging is probably going to be light during the next few days, so here are a few holiday-related links to tide you over:

• The U.S. National Archives and Records Administration offers a handy explanation and digitized records that detail how Thanksgiving came to be a national holiday. The process wasn't as straightforward as you might think: for the first two thirds of the 19th century, the day of observance shifted around quite a bit, and in 1939 and 1940 some states celebrated it on the second to last Thursday of November while others waited until the last Thursday of the month. The current day of observance -- the fourth Thursday of November -- was enshrined in federal law in 1941. (Somewhere, someday, someone's going to win a trivia contest because s/he knows this.)
  

• The State Library of Massachusetts has posted on its blog a listing, with images, of some of its Thanksgiving-related holdings, which date from ca. 1650 to the present day. (If you're a Bay Stater, please let your legislators know that you're thankful for your State Library and that you want it to remain open! Contact info is here.)

• The Navy Department Library has digitized menus documenting Thanksgiving (and Christmas) meals served on several ships and at several bases from 1905-1950.

• The New York Public Library has digitized lots of great 19th and early 20th century Thanksgiving-related material: photos of parades and other observances, hotel and railroad menus, engravings and holiday postcards such as the Arts and Crafts beauty below.

"Thanksgiving Day Greetings." Image courtesy of the New York Public Library. Vintage Holiday Postcards Collection. Mid-Manhattan Library -- Picture Collection. Record ID 1065827. Digital ID 1588308.