BPE 2011: emerging trends

The 2011 Best Practices Exchange (BPE) proceeds apace, and today I’m going to focus upon yesterday’s plenary session, which featured Leslie Johnston, the Director of Repository Development at the Library of Congress (LC). Johnston devoted a lot of time to discussing ViewShare, LC’s new visualization and metadata augmentation tool, but I’ll discuss ViewShare in a forthcoming post about some of the new tools discussed at this year’s BPE. Right now, I want simply to furnish an overview of her exhilirating and somewhat unsettling assessment of the changing environment in which librarians and archivists work:

• Users do not use digital collections in the same way as they use paper collections, and we cannot guess how digital collections will be used. For example, LC assumed that researchers would want textual records, but a growing number of researchers want image files of textual records.

• Until recently, stewardship organizations have talked about collections, series, etc., but not data. Data is not just generated by satellites, experiments, or surveys; publications and archival records also contain data.

• We also need to start thinking in terms of “Big Data.” The definition of Big Data -- what can be easily manipulated with common tools and can be managed and stewarded by any one institutions -- is rather fluid, but we need to start thinking in these terms. We also need to be aware that Big Data may have commercial value, as evidenced by the increasing interest of firms such as Ancestry.com in the data found in our holdings.

• More and more, researchers want to use collections as a whole and to mine and organize the collections in novel ways. They use algorithms to do so and new tools that create visual images that transform data into knowledge. For example, the Digging into Data project examined ways in which many types of information, including images, film, sound, newspapers, maps, art, archaeology, architecture, and government records, could be made accessible to researchers. One researcher wanted to digitally mine information from millions of digitized newspaper pages and see whether doing so can enhance our understanding of the past. LC’s experience with archiving Web sites also underscores this point. LC initially assumed that researchers would browse through the archived sites. However, researchers want access to all of the archived site files and to use scripts to search for the information they want. They don’t want to read Web pages. Owing to the large size of our collections, the lack of good tools, and the permissions we secured when LC crawled some sites, this is a challenge.

• The sheer volume of the electronic data cultural stewardship organizations need to keep is a challenge. LC has acquired the Twitter archive, which currently consists of 37 billion individual tweets and will expand to approximately 50 billion tweets by year’s end. The archive grows by 6 million tweets an hour. LC is struggling to figure out how best to manage, preserve, and provide comprehensive access to this mass of data, which researchers have already used to study the geographic spread of the dissemination of news, the spread of epidemics, and the transmission of new uses of language.

• We have to switch to a self-serve model of reference services. Growing numbers of researchers do not want to come to us, ask questions of us, and then use our materials in our environment. They want to find the materials they need and then pull them out of our environment and into their own workspaces. We need to create systems and mechanisms that make it easy for them to do so. As a result, we need to figure out how to support real-time querying of billions of full-text items and the frequent downloading by researchers of collections that may be over 200 TB each. We also need to think about providing tools that support various forms of collection analysis (e.g., visualization).

• We can’t be afraid of cloud computing. Given the volumes of data coming our way and mounting researcher demands for access to vast quantities of data, the cloud is the only feasible mechanism for storing and providing access to the materials that will come our way. We need to focus on developing authentication, preservation, and other tools that enable us to keep records in the cloud.

There’s lots and lots of food for thought here -- including a few morsels that will doubtless induce indigestion in more than a few people -- and it’s just a taste of what’s coming our way. If we don’t come to terms with at least some of these changes, we as a profession will really suffer in the coming years. Let's hope that we have the will and the courage to do so.

A bottle of locally brewed Kentucky Bourbon Barrel Ale at Alfalfa Restaurant, Lexington, Kentucky, 20 October 2011. I highly recommend both the ale and the restaurant, but please note that Kentucky Bourbon Barrel Ale is approximately 8 percent alcohol. Just like the BPE, it's a little more intoxicating than one might expect.

Archives matter

Yesterday, the New York Times posted an article concerning a Dept. of Justice report documenting the work of the U.S. Department of Justice's Office of Special Investigations (OSI). OSI, which was recently folded into the department's new Human Rights and Special Prosecutions Section, was responsible for initiating denaturalization and deportation proceedings against American citizens found to have participated in the persecution of civilians in Nazi-occupied Europe, and ensuring that foreign nationals who took part in persecuting civilians are denied entry to the United States.

The Department of Justice has refused to release the report, which was written in 2006, in its entirety, but the Times somehow obtained an unredacted copy and has made it publicly accessible; it also created a supplement contrasting the redacted and unredacted versions. For reasons that are completely understandable, the Times article emphasizes the report documents instances in which the United States government gave "safe haven" to people who had been actively involved in wartime persecution or enslavement of civilians; the Central Intelligence Agency, in particular, comes off quite badly.

However, one of the most striking things about the report itself is the manner in which it highlights the centrality of archival records to the work of the OSI, which had a professional archivist on staff almost from the moment of its creation. Unlike other Department of Justice officies, OSI relies not upon interviews and surveillance conducted by law enforcement personnel but upon archival research conducted by academic historians. The report, which discusses not only the OSI's investigative techniques but also the state of and access challenges associated with archival documentation of Nazi atrocities, emphasizes: "Given the advanced age of survivors and questionable value of eyewitness testimony, a[n OSI] case is generally only as good as the archival evidence."

To date, OSI's work has resulted in the denaturalization of 83 people, the permanent departure from the United States of 62 people, and denial of entry to more than 170 people.

One could argue that, in the grand scheme of things, the denaturalization and deportation of these people doesn't mean much: nothing bring back the men, women, and children who perished at the hands of the Nazis or make whole those who survived, and even the youngest perpetrators likely aren't long for this world. However, in spite of the Cold War-era actions of the CIA and other U.S. government bodies, war criminals and human rights violators have no place within a society that values individual freedom and dignity, equality before the law, and democratic governance. We cannot undo old crimes, but we can bring them to light and ensure that those who perpetrated them -- whether in Central and Eastern Europe 60 years ago, in Guatemala 30 years ago, in the former Yugoslavia 15 years ago, or in any place at any time -- do not find sanctuary in this country.

Without archives, justice would be an even rarer thing.

"The war is still there"

We archivists are always telling funders and other stakeholders that archival records are of interest not only to genealogists and academics but also to a wide array of other people and that the information contained in some of our records may be life-saving.

Case in point: photographs and other records held by the Air Force Historical Research Agency, which is part of Maxwell-Gunter Air Force Base in Montgomery, Alabama. An article in yesterday's Wall Street Journal highlights how its holdings enable German construction forms to locate, remove, and safely detonate unexploded ordinance:

During World War II, American planes dropped 1.5 million tons of bombs on Europe. Perhaps nine out of 10 exploded on impact.

In a library in Alabama, Dietmar Staude hunts for the 10th.

Mr. Staude, a German consultant, has perfected the art of using World War II air-raid records and aerial photographs to find unexploded bombs. It's a lucrative business designed to help German construction companies comply with laws requiring them to clear live ordnance before breaking ground . . . .

Over the years, Mr. Staude's work has uncovered, among other explosives, 21 live bombs on the site of a planned industrial park, a 500-pound bomb just 50 yards from a new water park and another weapon that landed and then tunneled 45 feet without exploding . . . .

Several German states require builders to conduct unexploded-ordnance surveys. Failure to carefully do so can prove deadly. In 1993, a crew building a retaining wall in Berlin hit the nose fuse on a British bomb, killing three people.

Although Staude does a lot of research at the Air Force Historical Research Agency, his work has also taken him to many other repositories in the United States and Europe:

One of Mr. Staude's projects has been going on for 14 years. On Aug. 16, 1944, B-17 heavy bombers from the 8th Air Force raided an oil refinery at Rositz, in eastern Germany. Half a century later, the state government decided it wanted to turn the site into an industrial park for environmental-technology companies.

Searching through document collections in Germany, England, Alabama and the U.S. National Archives in College Park, Md., Mr. Staude pieced together a history of the raid.

He discovered photos, taken from a hot-air balloon in 1917 and printed on glass, that showed the refinery surrounded by farmland. He found an RAF reconnaissance photo taken in 1943, showing tar-storage tanks ill-concealed under camouflage netting. He found the briefing document used to guide the pilots to the most vital target buildings; on top it read, "Not to be taken into the air." He found photos showing 500-pound bombs tumbling through the air toward the complex that day in 1944, as well as pictures showing dark smoke, sprouting like a giant cauliflower from the burning factory . . . .

Five days after the raid, a Royal Air Force plane snapped clear pictures that showed the facility and nearby farms pocked by bomb craters. Most resembled moon craters, with a big hole in the middle surrounded by blast marks in the dirt.

One, in a field, showed a circular spray, but no large hole in the center. Mr. Staude concluded that the bomb had landed in soft dirt, splashing mud but failing to detonate.

Another showed a tiny mark alongside what appear to be bales of hay. He concluded that a bomb had punctured dry earth on that spot, but didn't explode. "This is a dud," [Mr. Staude] said, pointing to the spot on the photo.

Both bombs, he figured, were still buried on the site. Over the years, explosives experts have dug up 21 bombs, including one that penetrated about 20 feet into the soft ground, bounced off a subterranean gravel bed, traveled sideways 45 feet underground and finally came to rest six feet from the surface, pointed upward.

Staude cannot always travel to the Air Force Historical Research Agency to do research, so Archie DiFante, one of the archivists on staff, sends copies of photographs and maps to Germany upon request. Information supplied by DiFante helped Staude and his colleagues locate one of the two unexploded bombs at the Rositz site; it was buried adjacent to the site of a future water park, and the park's developers had already begun digging.

As DiFante, who has become a good friend of Staude's, points out: "We bombed the living daylights out of everything in Germany. The war is still there." However, thanks to archival records documenting the bombings and to the work of archivists such as DiFante, Mr. Staude and other researchers trying to track down unexploded bombs are helping to ensure that it doesn't claim any more lives.