Catching up

A few things you might have missed:

• Late last month, President Obama issued a memorandum directing each federal government agency to perform a comprehensive review of its records management program and then prepare a report for the Archivist of the United States and the Director of the Office of Management and Budget that outlines its plans to maintain and improve its program, "particularly with respect to managing electronic records, including email and social media, deploying cloud based services or storage solutions, and meeting other records challenges." These reports are due on 27 March 2012.

• Paper records created during an internal military investigation of a November 2005 massacre of civilians in the Iraqi city of Haditha were slated for destruction. However, the records, many of them marked as being secret, ended up in trailers purchased by a local businessman, who hauled the trailers to a Baghdad junkyard. Several weeks ago, a New York Times reporter covering the American withdrawal from Iraq inadvertently found them there. At present, it is unclear whether the military will open an investigation into the handling of these records.
  

• After a legal review, the Massachusetts State Archives has decided to open approximately 460 boxes of paper records of former Governor and current Presidential candidate Mitt Romney to researchers. Staff will review the files prior to disclosure and either remove or redact legally restricted information. The repository initially restricted access to the records as a result of a court ruling stating that gubernatorial records were exempt from the state's freedom of information law. As you'll recall, during the last days of the Romney administration, all of the files on its e-mail servers were deleted, several high-ranking officials were allowed to purchase the state-owned hard drives they used, and leased computer equipment was replaced.
  

• The administration of South Carolina Governor Nikki Haley routinely deletes internal e-mails. The administration claims that it does so in order to free up storage space on its server, but Erik Emerson, Director of the state's Department of Archives and History, asserts that it violates state records laws.
  

• OccupyArchive is George Mason University's Roy Rosenzweig Center for the History of New Media effort to capture digital items documenting Occupy Wall Street and other Occupy movements throughout the world. As Rosenzweig Center director Sharon Leon notes, they're "documenting a post-print movement" -- something that archivists must do if they want to ensure a complete and accurate documentary record.
  

• Finally, on a lighter note, here's why we need to caution teens about sexting: sooner or later, their sexts will be all over the Internet for everyone to read.
  

Barry Landau: motion to suppress

Yesterday, attorneys representing Barry Landau, the collector accused of stealing and selling records held by the Maryland Historical Society, the Historical Society of Pennsylvania, the Connecticut Historical Society, and the Franklin D. Roosevelt Presidential Library, filed a motion to suppress all evidence found during a search of Landau's Manhattan apartment. Judging from its content, it seems likely that Landau's attorneys plan to argue that Landau's companion, Jason James Savedoff, who pled guilty to the charges against him on 20 September 2011, committed all of the crimes of which Landau is accused.

Given that Saveoff physically possessed all the stolen materials recovered after he and Landau were arrested at the Maryland Historical Society in July 2011 and is cooperating with prosecutors, I'm not particularly surprised by this line of argument. Do I think it's a good one? Emphatically, no -- by all accounts, Landau is a seasoned researcher and Savedoff was new to the ways of archival research -- but I won't be sitting on the jury that determines Landau's guilt or innocence.

The copy of the motion that appears below was downloaded from Public Access to Electronic Court Records (PACER). I am posting it here so that interested archivists can read it. Downloaded PACER documents may be freely distributed, so please feel free to share it with others.

At present, Landau's trial is currently scheduled to begin on 13 February 2012 and is expected to last 5-6 days.
Barry H. Landau Motion to Suppress 2011-12-13

Thanks to "anonymous archivist" for drawing my attention to the filing of this motion.

Salvage and recovery of water-damaged solid-state electronic media

In the wake of tropical storms Irene and Lee, I've done some research into how to salvage and recover data housed on flood-damaged electronic media. There are some great, media-specific resources out there:

• Hard drives: Library of Congress; U.S. National Archives and Records Administration (NARA)
• Data tapes and analog audio and video tape: Association of Moving Image Archivists; Library of Congress; NARA; Specs Brothers
• CDs and DVDs: Library of Congress; NARA
• Floppy diskettes (yep, some people still use them): NARA

However, at present, information about how to salvage and recover data housed on solid-state media such as flash drives and digital camera and smartphone memory cards and solid-state devices such as portable music players (sometimes used to record audio), tablet devices, and computers with solid-state drives (e.g., MacBook Airs) isn't readily available. As a result, I contacted several vendors who specialize in recovering data from electronic media and devices damaged in floods, fires, and other disasters and asked for their advice. What follows is an initial summary of these conversations. I hope that it fills a gap in the existing professional literature -- and that no one who reads this blog ever has cause to make use of the following advice.

First, a few general guidelines:

• Restoring data from backups is always easier and cheaper than recovering data housed on damaged electronic media. Back up your data!
• A good disaster management plan will reduce the risk that your media will be damaged. For more information about developing such plans, consult the New York State Archives publication Preparing for the Worst: Managing Records Disasters.
  
• Floods and burst pipes aren't the only water-based disasters. First responders use water to fight fire and to keep down dust from collapsed structures. If your media is burned or crushed and wet, treat it as water-damaged.
  
• In some instances, you may have no choice but to try to recover data from damaged media. Backups may be incomplete or become corrupt, and sometimes records created immediately before disaster strikes (e.g., photographs documenting a crime scene) are so valuable that the time and expense associated with recovery is warranted.
• When disaster strikes, salvage damaged media and stabilize it long enough to determine whether your backups are complete and intact. If your backups are complete and readable or the records on the damaged media are less than essential, don't attempt to recover the data stored on the damaged media; however, as noted below, the cost of attempting to recover non-essential data from water-damaged flash drives and memory cards is so low that you might want to give recovery a shot. If the records are essential and backups don’t exist, are incomplete, or have been corrupted, attempt to recover the data housed on the damaged media.
• Actions suitable for water-damaged paper records may destroy electronic media. Although solid-state media should be air- or rice-dried (see below), some types of electronic media (e.g., hard drives) should be kept wet. Freeze- or vacuum-drying or using heat to speed air drying will likely destroy most forms of electronic media, and using heat to speed air-drying may also damage or destroy media.
• Protect yourself. Before you enter a flooded area, consult with emergency personnel and make sure that it's safe to enter. Contaminated water and live electricity -- keep in mind that uninterruptible power supplies attached to hardware may be live well after the power goes off -- pose serious safety risks, and noxious gases can build up, particularly in basements. Wear appropriate protective gear.
• Be prepared to document the disaster. If you need to file an insurance claim, your insurer will likely want photographs illustrating the extent of the damage. If the disaster is small (e.g., you drop a thumb drive housing important records into a cup of coffee), you may want pictures for your own records. If you're an archivist, records manager, or conservator, you may also want images to incorporate into presentations, publications, or other training materials. You may also need to take notes about the scope of the disaster and the location of hardware and media (first responders sometimes disconnect stuff and move it around).

Now, down to the nitty-gritty of salvaging and recovering water-damaged solid-state media and devices. If you're confronted with water-damaged solid-state media or devices, the following guidelines will maximize your chances of recovering your data.

Before you begin your initial salvage and stabilization effort, make sure you have the appropriate supplies on hand. For solid-state media and device(s), you'll need, at minimum, some clean, dry, lint-free cotton cloths (in a pinch, old bedsheets or garments will do) and some gallon- or quart-sized zippered plastic storage bags. Odd as it may seem, you may also want to have some uncooked white rice on hand.

Salvage and stabilization of flash drives and memory cards

• Remove memory cards from devices and disconnect drives from powered-down hardware.
• Wipe off any surface dirt and water with a clean, dry, lint-free cloth and then air-dry the media as soon as possible: place the media on a clean, dry, lint-free cotton cloth and prop it up in a way that speeds drainage.
• You may use fans and dehumidifiers to facilitate the drying process.

Salvage and stabilization of solid-state devices (e.g., cell phones, tablet devices, computers with solid-state drives)

• Unplug or remove the battery as soon as possible and gently shake the device to remove water lodged in ports and other openings.
• Wipe off surface dirt and water with a clean, dry, lint-free cotton cloth and then air-dry or "rice-dry" the device. To air dry the device, place it on a clean, dry, lint-free cotton cloth and prop it up in a way that facilitates drainage. You may use fans and dehumidifiers to speed the process. To rice-dry the device, place it in a zippered plastic storage bag and then fill the bag with uncooked white rice. If you must retain the device for more than 2-3 days, replace the rice to reduce the risk of mold growth. (FYI, this "rice-dry" technique may also bring water-damaged cell phones or digital cameras back to life . . . but I don't think I would trust such a device in a mission-critical situation.)

After you've salvaged and stabilized the media or device(s), assess whether recovery is warranted. Do you have complete, uncorrupted backups of the records stored on the media or device? If you do, restored the data from the backups and discard your damaged media. If you don't, how valuable are the records? Are they essential to your business operations or a court proceeding? Are they of immense historical (or, in the case of personal files, sentimental) value? How great is the cost of recovering the data? As noted below, the cost of attempting to recover non-essential data from a flash drive or memory card is quite low. The cost of having a vendor recover data from a solid-state device can be quite high. You have to determine whether the value of the records warrants the cost of recovering them.

If you determine that the data is essential and warrants the cost of recovery, you'll need to contract with a vendor that specializes in data recovery work. Many state archives maintain lists of such vendors, and a quick Web search will identify many others.

If the data is non-essential, discard the media or device appropriately; however, if the data is stored on a flash drive or memory card, you may want to try to recover it yourself. Damaged flash drives and memory cards that house legally restricted or sensitive data should be physically destroyed (by a recycling vendor or with a hammer or shredder), and damaged devices that house such data should be sent to a vendor that will destroy their drives and recycle their other components. Damaged media and devices that don't contain such data can probably be recycled by vendors who specialize in processing electronic waste.

Recovering data from flash drives and memory cards

• If the data is essential, send the drive or card to a qualified disaster recovery vendor.
• If the data is non-essential, attempt to read the files on the damaged device. If you are successful, copy the files onto new media and discard the damaged media. If you are not successful, admit defeat and discard the media or, if you are attempting to recover data from a memory card, decide whether the purchase of commercial recovery software (prices begin at around $30.00) is warranted.

Recovering data from solid-state devices

• Air- or rice-dry the device(s) and then send the device(s) to a qualified disaster recovery vendor. These devices are difficult to open and require special handling. Do not attempt to recover the data yourself.

Establish a relationship with your disaster recovery vendor as quickly as possible. Most vendors have 24/7 phone coverage, and they may be able to provide additional stabilization and recovery advice, offer pickup service (particularly in major metropolitan areas), and provide special handling or packing instructions. Moreover, the sooner recovery begins, the greater the chance it will be successful.

The U.S. National Archives and Records Administration has a great list of security and other considerations that should be discussed with prospective vendors and incorporated into service contracts. I have only one thing to add: be honest about the nature of your disaster. If your media or device came into contact with water that may have contained biological or chemical hazards, tell the vendor about it. Vendors have the protective gear and equipment needed to work with contaminated material and they deal with embarrassing situations (e.g., "I dropped my camera in the toilet!") all the time, but they need to know what's coming their way.

As far as sending the media or device(s) to the vendor is concerned, follow the instructions provided by the vendor. However, you will probably be asked to do the following:

• Place each piece of media and each device into a zippered plastic storage bag.
• Surround each bagged piece of media or device with bubble wrap.
• Pack the media or device(s) appropriately.
• If sending portable media to a vendor, you may be able to use a rigid shipping envelope. You can also use a box at least twice as large as the media.
• If sending device(s) to a vendor, use a box at least twice as large as the device
• If using a box, immobilize the media or device(s) with packing material (N.B.: some vendors will request that each piece of media and each device be placed in its own box)
  
• Ship to the vendor via overnight delivery service

Disclaimer: I am not liable for any losses or damages resulting from following any of the advice contained within this post.

Robert A. Caro in conversation with Harold Holzer, Albany, N.Y., 5 December 2011

Next Monday, the New York State Archives Partnership Trust will host an evening's conversation between Robert A. Caro and Lincoln scholar Harold Holzer. Caro is the biographer of President Lyndon Johnson (The Years of Lyndon Johnson: The Path to Power, Means of Ascent, and Master of the Senate) and of urban planner Robert Moses (The Power Broker: Robert Moses and the Fall of New York).

The Power Broker is widely regarded not only as a compelling (and not particularly flattering) biography of Moses but also a carefully researched history of modern New York City. Several scholars have concluded that Caro did not appreciate the extent of Moses' decline in power in the 1960s, and a number of twenty-first century New York politicians frustrated by legislative and procedural gridlock have come to see Moses's unparalleled ability to transform the built environment in a positive light, but no one contests Caro's assessment of Moses' impact:

It is impossible to say that New York would have been a better city if Robert Moses had never lived. It is possible to say only that it would have been a different city.

This event will take place on Monday, 5 December 2011, from 7:30-8:30pm at The Egg (Center for the Performing Arts) at the Empire State Plaza, in Albany, New York. Tickets, which are $10.00 apiece, may be purchased by calling The Egg's box office (518-473-1845) or visiting The Egg's Web site.

Image: Robert Moses speaks at the dedication of Bethpage State Park near Farmingdale, N.Y., 1933. From New York State Archives, New York (State). Conservation Dept. Photographic Prints and Negatives, [ca. 1904-1949], 14297-87_4232. Image courtesy of the New York State Archives. Click here for a magnifiable version of this image.

Freedom of information laws throughout the world

Things are going to be quiet around here this week: my modem abruptly ceased working on Sunday afternoon. Owing to the holiday I expect that the new one won't arrive until Friday at the earliest and that the coffeeshop in which I'm writing this post and all of my other usual wifi hotspots will be closed.

However, I wanted to draw your attention to a recent Associated Press article highlighting the results of its first-ever test of freedom of information laws throughout the world. At present, 105 countries have such laws, but the experience of the AP, which submitted requests for information to all of these nations and to the European Union, reveals that the extent to which these laws are observed varied widely.

• Only 14 countries supplied all the information requested within the time frame specified in their laws, and 38 more eventually complied. More than half ignored the AP's requests altogether.
• Newer democracies often complied more swiftly than mature democracies. Moreover, newer democracies' laws, which tend to reflect the existence of the Internet, are often better suited to today's world than the laws that mature democracies enacted decades ago, when the overwhelming majority of records were created on paper and the Xerox machine was the height of technological sophistication.
  
• Many countries adopted freedom of information laws as a condition of securing foreign assistance, and most of these countries ignore or seek to circumscribe these laws as quickly as possible.
• In some countries, citizens who file freedom of information requests may be targeted for retaliation. In India, where activists are using such requests to expose and combat entrenched governmental corruption, at least a dozen people who have filed freedom of information requests have been killed and dozens more have been violently attacked.
  

It's an interesting, thought-provoking piece, and it bears close reading.

State government electronic records in the news

Two stories relating to the management and continued accessibility of state government records popped up on my radar screen earlier today. Both of them warrant watching; it doesn't seem as if either situation will be resolved any time soon.

The first involves gubernatorial records, an ever-present matter of interest and concern. Earlier today, the Boston Globe reported that during the last days of Republican presidential candidate Mitt Romney's tenure as governor of Massachusetts, eleven of his high-ranking staffers used personal funds to purchase their state-supplied hard drives and laptops, staff replaced all of the other computers in the governor's office, and all Romney-era e-mail was deleted from the office's e-mail servers. When Deval Patrick, a Democrat, took office, he and his staffers found an electronic blank slate.

Romney's position is that staffers who purchased hardware did so openly and that he and his staffers complied with all records laws. It does seem that the Romney administration did transfer a substantial body of records to the Massachusetts Archives: according to the Globe, the the repository holds 700-800 boxes of paper records documenting the Romney administration. However, it's not clear whether these records include print copies of the e-mails. The Globe doesn't provide detailed information about them, and the Massachusetts Archives doesn't have an online catalog or detailed online finding aids.

Secretary of State Bill Galvin, who oversees the Massachusetts Archives, told the Globe that the hardware purchases strike him as odd and that the gubernatorial e-mail should have come to the archives: "Electronic records are held to the same standard as paper records. There’s no question. They’re not in some lesser standard."

Romney's campaign manager asserts that the Patrick administration is making a stink about the hardware purchases, computer replacement, and e-mail deletion because it is acting as "an opposition research arm of the Obama reelection campaign." After the Globe story appeared this morning, he filed a state Freedom of Information Act request seeking "all email correspondence, phone logs, and visitor logs" documenting contacts between Patrick administration staffers and prominent Obama political advisers David Plouffe, David Axelrod, and Jim Messina. Governor Patrick’s chief legal counsel has stated that staff will "be happy to fulfill" this request.

I'm not an expert on Massachusetts records laws, so I'm going to have wait for the experts to weigh in on whether the actions of Governor Romney and his staff were legal. Do I wish that the e-mail had been preserved? Of course I do. I'm an archivist, and my job is to preserve records of enduring value and to provide access to them. Gubernatorial correspondence and internal memoranda, regardless of format, do have enduring value. Do I think that Governor Romney should be pilloried for destroying the e-mail? If he violated the law, I hope he gets what's coming to him. If he didn't, I hope that Governor Patrick and other Massachusetts politicians focus on strengthening laws concerning the retention and disposition of gubernatorial records.

Do I think that Governor Patrick brought up these issues in an effort to give President Obama a boost? I don't know. Patrick and Obama are close allies, so it's possible. However, I'm also under the impression that Governor Patrick has his own reasons for disliking Governor Romney, and I'm open to the possibility that he and his staffers are discussing the matter because they keep getting freedom of information requests for Romney-era records. I must admit that I am curious as to how well the Patrick administration is managing its own records.

The second relates to an outrage. As anyone who's been paying even the slightest attention to the American news media knows, former Penn State assistant football coach Jerry Sandusky was recently arrested on charges that he sexually molested eight young boys. Two university administrators have been charged with perjury, and the university's president and football coach have lost their jobs.

Questions as to precisely what the president, the coach, and other university administrators knew about Sandusky and when they knew it are rampant. However, Pennsylvania's Right to Know Law, which was extensively revised in 2008, explicitly exempts most records created by Penn State, Lincoln University, the University of Pittsburgh, and Temple University. As a result, there is a distinct possibility that only those e-mails, phone records, and other Penn State records introduced in open court will be disclosed to the public -- unless, as the New York Times urged earlier today, the Pennsylvania legislature and governor move to lift this exemption.

Publicly funded universities in many other states -- New York included -- are subject to freedom of information laws. For what it's worth, I really don't see why Penn State, Pitt, Temple, and Lincoln should be granted such sweeping exemptions, and I hope that Pennsylvania's law changes. At the very least, I hope that Penn State's new administrators recognize that openness and honesty are essential to restoring the university's good name and start releasing records of their own accord as soon as prosecutors permit them to do so.

Yes, I know that Penn State is going to be hit with civil lawsuit after civil lawsuit and that its lawyers would probably jump for joy if a fire or flood destroyed a ton of university records. However, the lawsuits will come and the cost of settling them will be staggeringly high no matter what the university does.

Of course, Penn State is not the only entity that has relevant records: Sandusky met the boys he is accused of sexually assaulting through The Second Mile, a charitable organization that he founded. However, earlier today, the New York Times reported that investigators have yet to locate some important Second Mile records:

Officials at the Second Mile . . . reported that several years of the organization’s records were missing and had perhaps been stolen. The missing files, investigators worry, may limit their ability to determine if Sandusky used charity resources — expense accounts, travel, gifts — to recruit new victims, or even buy their silence . . . .

Much of the [charity's] older paperwork was stored at an off-site records facility. The travel and expense records, for instance, had been sent over several years earlier. But select members of the charity’s board of directors were alarmed to learn recently that when the records facility went to retrieve them, some of those records — from about 2000 to 2003 — were missing.

. . . . Subsequently, the [Second Mile] foundation located apparently misfiled records from one of the years, but the rest seem to have disappeared.

As awful as the Sandusky-Penn State situation currently appears, I can't help but think that we've seen only the tip of the iceberg. All the more reason to be as honest and as open as possible. The sooner the truth comes out, the sooner the victims can focus on rebuilding their lives and the sooner Penn State can focus on rebuilding itself.

BPE 2011: building digital repositories

Still playing catch-up re: the 2011 Best Practices Exchange (BPE). After the BPE ended, I spent a few days in Ohio with my parents, came back to Albany, prepped for and gave a presentation on salvaging and recovering data from electronic media, got sick, got well, got sick again, and got well again. Now I’m barreling through all kinds of personal and professional backlogs.

I took decent notes, but three weeks have elapsed. If you were there and your memory differs from mine, please let me know. I’ll update/correct as needed.

One of the most interesting BPE sessions I attended featured two speakers who focused on the creation of digital repositories. The first, Mitch Brodsky of the New York Philharmonic, discussed the creation and evolution of the Philharmonic’s repository. At present,staff are digitizing the organization’s international era (1943-1970) and will result in the digitization of 3,200 programs, 72 scrapbooks, 4,200 glass lantern slides (older but easy to do), 8,500 photographs, 8,000 business record folders. By the end of 2012, 1.3 million pages of paper records will be digitized and the repository will house 15 terabytes of data. Digitization of audiovisual materials will add another 2 TB of data to the system. However, the organization also plans to add materials created during the first 98 years of the Philharmonic’s existence (1842-1940) and to incorporate late 20th and 21st century electronic records into the repository.

The project’s larger goals are equally impressive:

• Accurate representation of originals. The Philharmonic’s archivists want the digital repository experience to match the research room experience as closely as possible. They don’t want to flatten curled records, disassemble bound volumes, or do anything else that would make the digital surrogates noticeably different from the originals. As a result, they’re using a digital camera (and the photographer who produced the digital surrogates of the Dead Sea Scrolls) to capture the originals, and many of the digital surrogates have a three-dimensional look. (Click here for an example.)
• Comprehensiveness. Staff are sensitive to privacy concerns, but want the digital repository to be as complete as possible.
• Easy and free accessibility. The Philharmonic expects that its digital repository Web site will be the public access mechanism for its archives.
• A new, sharable model for digitizing large collections.

As you might expect, the repository’s technical infrastructure is pretty sophisticated -- and entirely open source:

• ImageMagick is used to convert images delivered by the photographer into various formats and sizes.
• OpenMigrate is used to channel data into and out of Alfresco.
• Alfresco, the open source content management system, serves as the repository’s core. (At present, the New York Philharmonic may be the only institution using it to build a repository of archival materials, so this project really bears watching.)
• Alfresco is not yet developed enough to meet the Philharmonic’s data entry standards, and as a result it enters metadata into homegrown databases and then ingests the metadata into Alfresco.
• The repository’s search functionality is handled by Solr, Apache’s search server.
• The repository’s viewer component is a Javascript tool developed by the Internet Archive.
• A suggested materials component based upon end user suggestions ties together related materials of different types and other end user input will be added via phpList.
  
• Vanilla forums will promote end user discussions.

Brodsky also shared a number of lessons learned. As far as I’m concerned, anyone thinking of undertaking any sort of large systems development project should devote a substantial amount of thought to each of them:

• You don’t know what you don’t know. Brodsky never expected that he would learn PHP, become a bugtracker, or proof code. However, he’s an on-the-ground project manager, and the Philharmonic had problems with its vendor.
• Do it manually before you automate. The Philharmonic started out doing a lot of manual review and dragging and dropping. However, doing lots of hands-on work before setting up an automated system revealed where errors pop up and enabled Brodsky to figure out how to correct them. Deep and intricate understanding of every phase of your project is a must.
• Vendors need to earn it. Do not be laid back. The vendor is there to do right by you, and it’s their job to convince you that they can be trusted. (Hear, hear! Managing vendor relationships and retaining or taking control of projects on which vendors work was a recurring BPE 2011 theme).
• Archivists who develop systems are product developers. As Brodsky put it: “You are not the same sort of archivist you were before you went digital.” People are actively accessing your online resources from all over the world, and they expect that your system will be reliable.

John Sarnowski of the ResCarta Foundation then gave a demonstration of the ResCarta Toolkit, an open source, platform independent bundle of tools that enables institutions to create digital collections that range from the very small to the very large.

The toolkit contains a variety of useful, easy-to-use tools:

• Metadata creation: assigns institutional identifier, adds directory organization with aggregator/root identifiers, adds metadata to image files using forms, and writes Metadata Encoding and Transmission Standard (METS) XML files to root directory.
• Data conversion: converts JPEG, PDF, TIFF, or existing ResCarta data to TIFF with embedded metadata, writes a final object metadata.xml file with checksum. Archives and libraries have the option of using preconfigured METS XML (ResCarta metadata schemas are registered METS profiles) or apply a custom metadata template to all of the files in a given directory or tree.
• Textual metadata editor: enables viewing and editing of OCR metadata and addition of descriptive metadata.
• Collection manager: creates collections, manages digital objects, allows editing or augmenting object metadata, outputs METS collection level XML file, and can output Dublin Core or Open Archives Initiative_Dublin Core data from the collection-level metadata.
• Indexer: creates a Lucene index of collection contents, indexes the collection level metadata, indexes all textual metadata from each TIFF, rebuild and optimize options.
• Checksum verification: creates a checksum and verifies against the original checksum.

A separate ResCarta Web Application facilitates Web publishing of ResCarta digital collections. Simply download the application and drop your ResCarta data directory into the application.

Libraries and archives can also use ResCarta to create metadata before adding objects to CONTENTdm, and the ResCarta Foundation is thinking of creating a tool that will enable METS and Metadata Object Description Schema (MODS) metadata to be moved into CONTENTdm in a streamlined, easy fashion.

I haven’t yet had the opportunity to play around with ResCarta --I just bought a new computer, but haven’t had the chance to get the thing hooked up to the Internet or do miscellaneous software installs -- but I was pretty intrigued and impressed. I’ll report back after I get the chance to play around with it a little bit.

I would be remiss if I did not point out that ResCarta may not be an appropriate solution for everyone: at present, only images and textual files can be added to ResCarta repositories: the ResCarta Foundation is, understandably, waiting for the emergence of good, widely accepted metadata standards for audiovisual materials. However, if you want to build a simple digital repository to house digital images and textual records, by all means check ResCarta out.

Image: Mary Todd Lincoln Home, Lexington, Kentucky, 22 October 2011. William Palmateer built this two-story brick, late Georgian house, which originally served as an inn, in 1803-1806. It was soon purchased by Robert Smith Todd, one of Lexington's most affluent men, and became a home for the growing Todd family. Mary Todd Lincoln was born in 1818 and resided in this home, which is a stone's throw away from the hotel at which the BPE was held, until she married Abraham Lincoln in 1842.

A day in Oneonta, New York

Yesterday was Veteran's Day, and as a result all New York State offices were closed and I had the day off work. On a lark, I accompanied my friend Ron, who lives in Albany but teaches at the State University of New York at Oneonta, to Oneonta for the day.

Oneonta, which is about 80 miles to the southwest of Albany, is a community of approximately 14,000 nestled in the rolling hills of the Susquehanna River valley. It's a college town, and it has the lively, slightly off-kilter charm that one often finds in such communities.

The land now occupied by the City of Oneonta was originally settled by the Algonquin and the Iroquois. The first Europeans to move into the area were Dutch and Palatine German settlers who moved out of the Schoharie and Mohawk valleys shortly before the American Revolution. The city experienced a boom with the coming of the Delaware and Hudson railroad during the late 19th century and during the early 20th century was home to the largest railroad roundhouse in the world.

The roundhouse was demolished long ago, and the city's economy now centers around higher education (the State University of New York at Oneonta and Hartwick College sit on the hillsides that overlook downtown Oneonta), health care, and retail; Oneonta may have only 14,000 residents, but it's surrounded by numerous small towns and villages whose inhabitants come to Oneonta to shop, eat in restaurants, see movies, and attend concerts and other cultural events.

Oneonta's late 19th century boom is manifest in its architecture, much of which dates from that era. The Wilber Mansion at 11 Ford Street is an excellent example. It was built by George I. Wilber, who was the son of the founder of the nearby Wilber National Bank and who served as president of the bank from 1890-1923. The mansion's inner core was built in 1875, and the porches, turret, port-cochere, and high Victorian decorative elements were added during an 1890 renovation. Since 1999, the Wilber Mansion has served as the headquarters of the Upper Catskill Community Council of the Arts.

The Chapin Memorial Church at 12 Ford Street is directly across the street from the Wilber Mansion. The church, which was dedicated in 1894, is home to the Unitarian Universalist Society of Oneonta. It is the second house of worship that the congregation has built at this site. A 1941 lightning strike destroyed its steeple, which was never rebuilt, and, sadly, most of the congregation's historical records.

Main Street is, as its name implies, Oneonta's main thoroughfare. Two- and three-story commercial buildings dominate the streetscape, and the image above should give you a sense of what Oneonta's downtown looks like. My friend Ron and I had a leisurely brunch at the always awesome Autumn Cafe at 244 Main Street (look for the red awning). If you're ever in Oneonta, this is the place to eat.

After brunch, we headed to the State University of New York at Oneonta campus so that my friend could teach a trio of courses. While he was in class, I took in the exhibits at the Martin-Mullen Art Gallery in the Fine Arts Building (and highly recommend Recent Work: Faculty Art Exhibition, which will be open until 16 December) and spent a little time pondering Joseph Kurhajec's Twisting Force (2005), which occupies a prominent position in the courtyard of the Fine Arts Building.

I then headed over to the James M. Milne Library, where I spent a couple of hours working on an upcoming presentation on disaster recovery and electronic records (posts on this subject are forthcoming), then stopped by the adjacent Jazzman's Cafe for a cup of coffee.

It started snowing while I was at Milne Library and continued snowing as I made my way back to the Fine Arts Building to meet Ron. A few minutes after I took this picture, snow stopped falling in the courtyard and the sun started coming out. However, snow continued to fall on the western side of the Fine Arts Building for at least ten minutes afterward. The SUNY Oneonta campus is no stranger to this sort of highly localized precipitation.

After doing a little shopping, Ron and I headed back to the Autumn Cafe for dinner. We sat in one of my favorite spots, a very Maxfield Parrish-ish elevated alcove, to which has been added a tree full of crows and the Wicked Witch of the West.

On our way back to the car, we took a few minutes to contemplate the Municipal Building at 238-242 Main Street, next to the Autumn Cafe. This Beaux Arts structure was built in the early 20th century and was originally the Oneonta City Hall. It now serves as an Otsego County satellite office building.

I haven't been able to find much information about Oneonta's current City Hall, which sits at 252 Main Street, but I suspect that it was built in the 1930s and that it was a Works Progress Administration project.

The above images are only a taste of what this fun, funky little community has to offer. If you ever get the chance to spend a little time in Oneonta, by all means take the opportunity to do so.

Landau-Savedoff indictment and Savedoff plea

Sorry about the seemingly nonstop focus on the Landau-Savedoff case as of late -- this blog will return to its customary focus on electronic records very, very soon -- but I now have a Public Access to Electronic Court Records (PACER) account, which, for a nominal fee, allows me to access federal court documents, and have pulled up some of the records relating to the case.

It costs $.08 per page to view and download PACER records, but the documents themselves can be freely distributed. I've uploaded a copy of the July 2011 indictment against Barry H. Landau and Jason James Savedoff and a copy of Savedoff's plea agreement to Scribd so that anyone else who is interested in reading them may do so at no cost.
Savedoff Landau Indictment

These documents don't contain much information that hasn't been reported in the media, but there are a few small details that may be of interest. Most notably, Landau's plea agreement, which was signed on 20 September 2011 but not introduced in court until 27 October, includes a (very) partial list of the documents found in the Manhattan apartment that Landau and Savedoff shared.

Savedoff Plea

Savedoff will be sentenced on 10 February 2012. He faces a maximum of five years imprisonment for conspiring to steal documents and a maximum of ten years imprisonment for actually stealing them.

The case against Landau is still ongoing. According to a 4 November 2011 memorandum to Landau's counsel that is available in PACER, his trial is currently scheduled to begin on 13 February 2012 and is expected to last 5-6 days.

Jason Savedoff pleads guilty

Last Thursday, Jason James Savedoff, one of the two men caught attempting to steal documents from the Maryland Historical Society on 9 July 2011, pled guilty to charges of conspiring to steal materials from the Maryland Historical Society, the Connecticut Historical Society, the Historical Society of Pennsylvania, and the Franklin Delano Roosevelt Presidential Library. He will be sentenced on 10 February 2012. He faces a maximum of sentence of 15 years in prison and a maximum fine of $500,000.

Savedoff's plea agreement states that his criminal misdeeds were performed "solely at the direction of" Barry Landau, the prominent collector with whom he was apprehended. Landau has pled not guilty to all of the charges lodged against him, and his lawyer insists that Savedoff masterminded the theft of the mass of materials found in the Manhattan apartment the two men shared and pled guilty in an effort "to save his own hide." Landau's lawyer went on to assert that prosecutors had no evidence proving any "misappropriation of documents before Mr. Savedoff came into his life a year and a half ago."

Forgive my skepticism of this claim. First, investigators found approximately 10,000 documents in Landau's apartment. Busy as Landau and Savedoff seem to have been, it just doesn't seem likely that they amassed this volume of material in a mere eighteen months. Second, as evidenced by articles in the Washington Post and the Daily Beast, Landau has long had, to put it charitably, a most flexible relationship with the truth.

In the meantime, the Federal Bureau of Investigation press release summarizing Savedoff's plea agreement (the full text of which I'm having difficulty accessing via PACER) contains details about his and Landau's activities that should make archivists, manuscript curators, and other cultural heritage professionals sit up and pay attention:

• "Savedoff, under the direction of his co-conspirator, conducted research, including via the Internet, to identify collections containing valuable documents, which, when located, were targeted for theft." Making finding aids accessible via the Internet has many, many pluses, but those of us who hold materials that have market value should also be keenly aware that may also increase security risks.

• "Savedoff and his co-conspirator visited numerous museums posing as researchers; accessed collections of documents which they had determined to be of significant value; reviewed the documents from the collections; and used various techniques to steal them. These techniques included concealing documents inside sports coats and other outerwear which had been modified to contain hidden pockets, as well as distracting museum curators to disguise their actions." Some repositories simply bar researchers from wearing sport coats and like garments while in their research rooms, but many women's suit coats are designed to be worn without a blouse underneath. In other instances, research rooms are so cold that rules concerning sport or suit coats or even outerwear can't be reasonably enforced. The overwhelming majority of researchers who wear sport or suit coats or other garments with pockets are decent, honest people, but all of them should be monitored closely.

• "A checklist was prepared for each stolen document which identified the author and date of the document; the collection from which it was stolen; whether the museum card catalogue had been collected; whether there existed any microfilm or other 'finding aid' for the document at the museum; the nature of any markings on the document: and whether any museum markings had been removed from the document." Wow. I'm simultaneously impressed by the strength of the recordkeeping urge, praying that these checklists are now in the hands of prosecutors, and agog at the monumental hubris and stupidity that prompted the creation of these records.
  

• "In an effort to conceal the theft, Savedoff and his co-conspirator often took the card catalogue entries and other “finding aids,” making it difficult for the museum to discover that an item was missing." And here's the plus side of putting finding aids and other access tools online: it's a lot easier to swipe a paper finding aid or catalog card than to destroy every electronic copy of a descriptive resource. Those of us who still have lots of single-copy, paper-based finding aids need to think seriously about devoting some time to converting finding aids that make mention of valuable materials to electronic form -- even quick-and-dirty scans to PDF, TIFF, or JPEG format should be sufficient to document ownership of an item in the event that both the item and the finding aid disappear.
  

Finally, the U.S. National Archives and Records Administration's Office of the Inspector General, which is leading the effort to identify the approximately 10,000 documents found in Landau's apartment and return them to the repositories from which they were stolen. Tricia Bishop of the Baltimore Sun recently wrote a great article highlighting the work being done by Office of the Inspector General staff and the scant attention and resources that American law enforcement agencies typically give to crimes involving cultural heritage materials. It's fascinating and, all too often, frustrating reading. Let's hope that the amount of media attention focusing on what is, in all likelihood, the largest archival theft in United States history changes this state of affairs.

ARMA International and San Jose State records management Web events

If you're searching for low-cost professional development opportunities, trying to figure out how to manage and preserve social media content, or seeking to develop workable records management policies, check out these upcoming online events.

First, Dr. Patricia C. Franks, coordinator of the Master of Archives and Records Administration (MARA) degree program at the San Jose State University School of Library and Information Science (SLIS), will facilitate an ARMA International webinar, Records Management Policies in a Social Media World, later this month, that will focus on practical tools and strategies for managing social media records. Those who complete this webinar will be able to:

• Analyze the impact of various social media technologies on records management
• Apply current records retention schedules to records residing in social networking sites
• Identify and use existing tools to capture and manage social media records.
  

This webinar will be available online from 14-29 November 2011 and is free to both ARMA International members and non-members alike, but you must register no later than 25 November 2011 in order to access it.

I heard Dr. Franks speak about strategies for managing social media records at an ARMA Central New York meeting in October 2010, and I was really, really impressed. I refrained from blogging about it only because I was coming down with what turned out to be a really rotten cold; by the time I recovered, I wasn't 100 percent sure that my notes and memories were complete and accurate. I'm really looking forward to this webinar, and I hope that you check it out as well.

Second, SLIS itself is making an upcoming MARA guest lecture available live via the Web. On 14 November 2011 at 1:00 PM PST, Fred Diers, vice president and general manager of GRM's Solutions Group, will discuss "How to Create a Credible Retention and Information Governance Package." Mr. Diers will discuss:

• Proven steps to develop a retention schedule that is realistic and sustainable
  
• How to reduce the risk of litigation, government investigations, and audits

Instructions for viewing this lecture online via Elluminate (and for accessing recorded Webcasts of past SILS events) are available here.

Happy Halloween

New York State Governor Hugh Carey greeting a young guest at a Halloween party at the Executive Mansion, 30 October 1981. New York (State). Governor. Public information photographs, 1910-1992. Series 13703-83, Box 11, No. 020. Image courtesy of the New York State Archives.

Sorry for the light posting lately. After the 2011 Best Practices Exchange -- about which I'm not done posting -- ended, I headed to Ohio to spend a few days with my parents and took a bit of a break from the Internet. After I got back to Albany, I spent a few days digging out from under the mass of work that accumulated in my absence. Now that I've had a little time to recover, you'll see things perk up around here.

In the coming days, you'll see a couple of additional posts about the Best Practices Exchange, at least one post concerning the Barry Landau-Jason Savedoff case, and some other tidbits. However, today I want simply to wish you a happy Halloween and to share with you the above photograph, which was taken thirty years and one day ago and is found within my employer's holdings, and the recording below, which is of less certain provenance but was produced on 28 October 1940 by a San Antonio, Texas radio station. It brings together Orson Welles and H.G. Wells, who discuss Welles' infamous radio adaptation of Wells's War of the Worlds, which aired exactly seventy-three years and one day ago; you'll also hear a few words about Citizen Kane, which was released approximately six months after this conversation took place. Enjoy.

BPE 2011: tidbits and lessons learned

The 2011 Best Practices Exchange (BPE) wrapped up about an hour ago, and I am both invigorated and completely exhausted. I’ll be posting more about the BPE either over the next few days or late next week (I may be without Internet access for a few days) , but I wanted to take this opportunity to share a few insights and lessons learned.

In yesterday’s morning plenary session, Doug Robinson of the National Association of State Chief Information Officers provided an overview of the challenges that state CIOs are facing. Those of us who work in state government need to be aware of the issues that these important partners face:

• At present, 25 state and territorial CIOs are new appointments. Another 7 CIOs are serving in an acting capacity, and several new governors(Cuomo among them) have yet to appoint a CIO. About half of the CIOs report directly to the governor, but a growing number of states are moving the position of CIO out of the cabinet and placing it under the supervision of budget or procurement director. The CIOs of roughly half the states are currently situated within a budget or procurement office.

• State CIOs come in from and return to the private sector, serve at the pleasure of the governor, and are in office for an average of 21 months. In contrast, private sector CIOs are in office for more than four years.

• State CIOs are more interested in rationalizing and centralizing IT services than in introducing new technology. Their budgets are being cut, the budgets of the agencies that pay user fees for IT services are being cut, and they are very focused on saving money by consolidating and streamlining services. They’re also starting to explore sharing services across states. However, they’re pushing against agency resistance to consolidation, lack of a shared enterprise technology vision, the persistence of large legacy systems, outmoded and cumbersome IT procurement processes, and a host of other concerns. Many of them also have more accountability than authority.

• State CIOs have identified electronic records management/digital preservation, authentication of data, social media, and the presence of state data on mobile devices as pressing concerns, but they’re doing much about them -- in large part because they’re not sure what they should do. Archivists and records managers should seize the opportunity to help CIOs address these issues.

• Many state CIOs are also moving to private clouds, especially for e-mail, and this has records management implications: in many instances, individual users are getting very large inboxes, and many cloud contracts specify that service providers must destroy messages after a set period of time and certify that the destruction was carried out properly. It also has workforce implications: at present, states are accustomed to having one IT staffer administer roughly 20 servers. Google and other cloud service providers are accustomed to having one staffer oversee approximately 1,000 servers, and they’re always seeking greater efficiencies.

• Other possible points of archivist/records manager-CIO intersection include enterprise architecture and policy, IT consolidation, shared services, and demands for government openness and transparency.

Some other interesting tidbits and lessons learned include:

• In the past six years, state archives have accessioned over a million cubic feet of paper records. Agency consolidations, staff reductions, and budget cuts have propelled many records to push paper records out of storage space they pay for, and state archives are struggling to find space for masses of incoming records.

• The Uniform Electronic Legal Material Act a piece of model legislation governing how states authenticate, preserve, and provide access to electronic copies of the state constitution, session laws, codified laws or statutes, state agency rules with the effect of law, and, optionally, court rules and decisions, state administrative agency decisions, and other legal material, will soon be the subject of a lot of state-level deliberation. Archivists and records managers must be actively involved in these discussions.

• Redaction remains a bottleneck and a burden. We’ve made considerable strides in automating the processing of electronic records and making them accessible shortly after we accession them. Even though new search tools that facilitate identification and redaction of legally restricted or classified information are starting to appear, we’re still doing a lot of record-by-record review, particularly when less cut-and-dried forms of information (e.g., attorney work product) are present. In an era in which citizens and oversight bodies increasingly expect that records will made accessible quickly, this is a challenge.

• Adobe Acrobat X is currently creating regular PDFs that don’t fully conform to the published PDF technical documentation; at least one digital collection’s validation tool consistently rejects PDFs created with Acrobat X. PDF/A files created with Acrobat X do conform to the published specification.

BPE 2011: emerging trends

The 2011 Best Practices Exchange (BPE) proceeds apace, and today I’m going to focus upon yesterday’s plenary session, which featured Leslie Johnston, the Director of Repository Development at the Library of Congress (LC). Johnston devoted a lot of time to discussing ViewShare, LC’s new visualization and metadata augmentation tool, but I’ll discuss ViewShare in a forthcoming post about some of the new tools discussed at this year’s BPE. Right now, I want simply to furnish an overview of her exhilirating and somewhat unsettling assessment of the changing environment in which librarians and archivists work:

• Users do not use digital collections in the same way as they use paper collections, and we cannot guess how digital collections will be used. For example, LC assumed that researchers would want textual records, but a growing number of researchers want image files of textual records.

• Until recently, stewardship organizations have talked about collections, series, etc., but not data. Data is not just generated by satellites, experiments, or surveys; publications and archival records also contain data.

• We also need to start thinking in terms of “Big Data.” The definition of Big Data -- what can be easily manipulated with common tools and can be managed and stewarded by any one institutions -- is rather fluid, but we need to start thinking in these terms. We also need to be aware that Big Data may have commercial value, as evidenced by the increasing interest of firms such as Ancestry.com in the data found in our holdings.

• More and more, researchers want to use collections as a whole and to mine and organize the collections in novel ways. They use algorithms to do so and new tools that create visual images that transform data into knowledge. For example, the Digging into Data project examined ways in which many types of information, including images, film, sound, newspapers, maps, art, archaeology, architecture, and government records, could be made accessible to researchers. One researcher wanted to digitally mine information from millions of digitized newspaper pages and see whether doing so can enhance our understanding of the past. LC’s experience with archiving Web sites also underscores this point. LC initially assumed that researchers would browse through the archived sites. However, researchers want access to all of the archived site files and to use scripts to search for the information they want. They don’t want to read Web pages. Owing to the large size of our collections, the lack of good tools, and the permissions we secured when LC crawled some sites, this is a challenge.

• The sheer volume of the electronic data cultural stewardship organizations need to keep is a challenge. LC has acquired the Twitter archive, which currently consists of 37 billion individual tweets and will expand to approximately 50 billion tweets by year’s end. The archive grows by 6 million tweets an hour. LC is struggling to figure out how best to manage, preserve, and provide comprehensive access to this mass of data, which researchers have already used to study the geographic spread of the dissemination of news, the spread of epidemics, and the transmission of new uses of language.

• We have to switch to a self-serve model of reference services. Growing numbers of researchers do not want to come to us, ask questions of us, and then use our materials in our environment. They want to find the materials they need and then pull them out of our environment and into their own workspaces. We need to create systems and mechanisms that make it easy for them to do so. As a result, we need to figure out how to support real-time querying of billions of full-text items and the frequent downloading by researchers of collections that may be over 200 TB each. We also need to think about providing tools that support various forms of collection analysis (e.g., visualization).

• We can’t be afraid of cloud computing. Given the volumes of data coming our way and mounting researcher demands for access to vast quantities of data, the cloud is the only feasible mechanism for storing and providing access to the materials that will come our way. We need to focus on developing authentication, preservation, and other tools that enable us to keep records in the cloud.

There’s lots and lots of food for thought here -- including a few morsels that will doubtless induce indigestion in more than a few people -- and it’s just a taste of what’s coming our way. If we don’t come to terms with at least some of these changes, we as a profession will really suffer in the coming years. Let's hope that we have the will and the courage to do so.

A bottle of locally brewed Kentucky Bourbon Barrel Ale at Alfalfa Restaurant, Lexington, Kentucky, 20 October 2011. I highly recommend both the ale and the restaurant, but please note that Kentucky Bourbon Barrel Ale is approximately 8 percent alcohol. Just like the BPE, it's a little more intoxicating than one might expect.

BPE 2011: ERA and the move to the cloud

This week, I’m spending a little time with my parents in Ohio and at the 2011 Best Practices Exchange (BPE) in Lexington, Kentucky. The BPE, which brings together state government, academic, and other archivists and librarians and other people seeking to preserve state government enduring information of enduring value, is my favorite archival conference. The Society of American Archivists annual meeting is always first-rate, but it’s gotten a little overwhelming, and I love the Mid-Atlantic Regional Archives Conference (MARAC), but nothing else has the small size, tight focus on state government records, informality, and openness that characterize the BPE.

Before I start detailing today’s highlights, I should say a few things about the content of these posts. For the past few years, those of us who have attended the BPE have tried to adhere to the principle that “what happens at BPE, stays at BPE.” This doesn’t mean that we don’t share what we’ve learned at the BPE (hey, I’m blogging about it!), but it does mean that we’re sensitive to the fact that candor is both essential and risky. The BPE encourages people to speak honestly about how and why projects or programs went wrong and what they learned from the experience. Openness of this sort is encouraging; all too often, we think that we’re alone in making mistakes. It's also helpful: pointing out hidden shallows and lurking icebergs helps other people avoid them. However, sometimes lack of senior manager commitment, conflicts with IT personnel, and other internal problems contribute to failure, and colleagues and supervisors occasionally regard discussion of internal problems as a betrayal. As a result, BPE attendees should exercise some discretion, and those of us who blog about the BPE should be particularly careful; our posts are a single Web search away. As a result, in a few instances I may write about the insights and observations that attendees have shared but obscure identifying details.

Moving on to this year's BPE itself, I'm going to devote the rest of this post to the insights and predictions offered up by U.S. National Archives and Records Administration (NARA) Chief Information Officer Mike Wash, who spoke this morning about the Electronic Records Archives (ERA), NARA’s complex, ambitious, and at times troubled electronic records system, and some changes that are on the horizon.

At present, ERA sort of works: staff use it to take in, process and store electronic records. At present, ERA holds approximately 130 TB of data. The Office of Management and Budget wants NARA to take in 10 TB of data per quarter, and NARA is working with agencies to meet this benchmark. However, ERA lacks an integrated access mechanism, and it contains multiple modules. The Base module handles executive agency data, the EOP module handles presidential records (and includes some internal access mechanisms), the Classified module holds classified records, and several other modules were built to deal with specific problems.

Building ERA taught NARA several lessons:

• Solution architecture is critical. ERA’s multiple modules are a sign of a failed system architecture. Anyone building such a system must carefully consider the business and technical architecture carefully during the planning stage and must manage the architecture carefully over time.
• The governance process must be clear and should start with business stakeholders. What do they really need the system to do, and how do you ensure that everyone stays on the same page throughout the process? Information technology invariably challenges control and authority, but if you set up your governance process properly, you should be able to retain control over system development.
• Over communicate. Funders and other powerful groups need frequent updates; failure to keep feeding information to them can be profoundly damaging.
• You must manage the project. The federal government tends to hire contractors to develop IT systems, and contractor relationships tend to deteriorate about six months after the contract is awarded. Most federal agencies cede authority to contractors because they are loath to be seen as responsible in the event that a project fails, but staying in control of the project increases your chances that you'll get the system you want.
• Watch costs closely. Cost-escalating provisions have a way of sneaking into contracts.
• Be mindful of intellectual property issues. The federal government typically reserves the right to all intellectual property created as a result of contracts, but this doesn’t always happen, and the vendor that built the first iteration of ERA has asserted that it controls some of the technology that now makes the system work; NARA will be much more assertive in working with future ERA vendors.

Wash also made some intriguing observations about some of the challenges that NARA and other archives are confronting:

• At present, our ability to acquire data is limited by bandwidth limitations. It takes more than three days to convey 20 TB of data over a 1 gbps data line and at least a month to convey it via the Internet. NARA recently took custody of 330 TB of 2010 Census data, and it did so by accepting a truckload of hardware; at present, there are no alternatives to this approach.
• The rate of data creation continues to accelerate. The administration of George W. Bush created 80 TB of records over the course of 8 years, but the Obama administration likely created more than 80 TB of data during its first year.

Wash indicated that NARA is starting to think that federal records should be created and maintained in a cloud computing environment and that transfer of custody from the creating agency to NARA should be effected by changing some of the metadata associated with the records being transferred.

Wash noted that the move to cloud computing will bring to the fore new preservation and authentication concerns. It also struck me that the transition that Wash envisions assumes the existence of a single federal government cloud that has adequate storage, security, and access controls and that, at least at this time, many states aren’t yet thinking of constructing such environments. Individual state agencies may be thinking of moving to the cloud, but most states don't seem to be preparing to move to a single, statewide cloud environment. Moreover, owing to its sheer size, the federal government is better able to negotiate favorable contract terms than state or local governments; the terms of service agreements that the feds hammered out with various social media providers are an excellent example. I have the uneasy feeling that some governments will accept, out of lack of knowledge, desperate financial straits, or inability to negotiate optimal terms, public cloud service contracts that prove problematic or outright disastrous.

Its nonetheless apparent that government computing will move into the cloud, that this transition offers both new challenges and new opportunities for managing and preserving records, and that archivists and records managers are going to have come to grips with these changes. The next decade promises to be most interesting.

The Lexington Laundry Company building on West Main Street, Lexington, Kentucky, 20 October 2011. This little gem was built ca. 1929, is an outstanding example of Art Deco architecture in the city, and is part of Lexington's protected Downtown Commercial District. It now houses an art gallery.

Drew University employee sentenced

Last week, a federal judge sentenced 20 year-old William Scott, the former Drew University student employee who admitted to stealing 31 historical documents from the university's United Methodist Archives Center, to three years of probation and three hundred hours of community service.

When I first learned of Scott's sentence, I was a little steamed. In my opinion, just about anyone convicted of stealing cultural heritage materials deserve to spend at least a little time in a correctional facility; I might be willing to make exceptions for people who steal to feed their families or to pay for lifesaving medial treatment for a loved one, but that's about it.

I nonetheless recognize that imprisonment is expensive and that incarcerating a young, non-violent offender who does not have a prior criminal record might not be the best use of our limited resources. Moreover, the sentencing judge and prosecuting U.S. attorney clearly wanted to make sure that Mr. Scott's will have ample cause and opportunity to reflect upon his misdeeds. While on probation, Mr. Scott:

• Must adhere to a 9:00PM curfew. (Most people would find such a curfew restrictive, but such restrictions are particularly painful for younger adults such as Mr. Scott, who once described himself as a night person who enjoys partying.)
• Is barred from working any job that would give him access to cultural heritage materials.
• Must write a monthly letter to the court describing the progress of his life.
• Must write to each of the 72 people who submitted character letters to the court on his behalf and explain what his experience of theft, prosecution, and conviction has taught him.
• May list his 300 hours of community service on his resume only if he specifies that said community service was court-ordered.
  

If I were 20 years old and forced to adhere to these conditions for 3 years, I suspect that the thought of spending 6 or 12 months in a minimum-security facility might seem like a reasonable alternative . . . .

Of the 31 documents that Mr. Scott stole, 30 have been recovered. The missing item is the two-sided second page of a letter that Charles Wesley wrote in 1755. United Methodist Archives Center staff scanned the Wesley letters in its holdings some time before Scott arrived on campus. If you ever come across an incomplete, double-sided document bearing Charles Wesley's autograph (stranger things have happened), you can compare it to the digital images of Wesley family letters that the repository contributed to the American Theological Library Association's Cooperative Digital Resources Initiative.

Former NARA employee pleads guilty to theft

On Tuesday, Leslie Charles Waffen, a career U.S. National Archives and Records Administration (NARA) archivist who ultimately became head of its Motion Picture, Sounds and Video Recording Branch, pleaded guilty to stealing at least 955 NARA-held sound recordings worth approximately $30,000. Dozens of boxes of recordings were found were found when officials raided his home, but he sold others on eBay using the account name "hi-fi_gal."

The Baltimore Sun reports that the investigation into Waffen's criminal activities cost approximately $48,000. The sale that led investigators to swoop in -- a 1937 audio recording of baseball legend Babe Ruth -- netted him $34.74.

As an archivist who has lived through a major internal theft, I have immense sympathy for all of the NARA employees whose lives have been turned upside down as a result of Waffen's illicit activities, which came to light last October and will continue to affect NARA's operations for years to come. Internal theft leaves in its wake powerful feelings of outrage, betrayal, and humiliation, and it takes a long time for those emotions to become manageable. Some of my colleagues have said that it took about a year after my former co-worker's theft came to light for them to come to grips with our experience, and some of us (myself included) needed even more time. All of us will carry the experience with us throughout the remainder of our lives and our careers; if you look at the membership roster of the Society of American Archivists' Security Roundtable, you'll note the presence of a healthy contingent of New York State Archives employees.

Moreover, internal theft always prompts -- as it should -- changes in security procedures and protocols. It's not unusual to understand intellectually the need for these changes while at the same time resenting the ways in which they make it harder to do one's job. I'm a big proponent of improving security in archival repositories -- as evidenced by numerous past posts on this blog -- but every now and then I can't help but blame my thieving former co-worker for some minor security-related inconvenience.

I realize that the above statements may seem a bit gloomy, but I do want to say to any current or former NARA employee who reads this blog that things will get better. You and your employer will both come to terms with this experience, and you will eventually adjust to the "new normal," whatever it may be. It won't happen quickly or easily, but it will happen.

Leslie Charles Waffen will be sentenced on 5 March 2012. As noted in his plea agreement, he faces a maximum sentence of ten years in prison and a maximum fine of $250,000.

A few small changes

I've never been particularly attached to the visual appearance of this blog. When I started it, I simply chose the least obnoxious Blogger template available and made some modest alterations. However, l'Archivista the blog is now a little more than four years old -- which means l'Archivista the blogger has maintained it for approximately one-tenth of her life -- and its baby clothes don't fit so well anymore.

I'll continue playing around with the design of this blog during the next few days, so you might see a few tweaks, quirks, and works in progress. Apologies in advance for any confusion or disruptions resulting from my experiments.

If you're new to this blog (hello!) and would like to see what it looked like before today's design changes took effect, the Internet Archive captured it in April of this year.

Interesting thing

Every archivist has at least one interesting-thing-I-found-in-a-box story, but an unnamed archivist working for the Central Arkansas Library System has what may well be the best interesting-thing-I-found-in-a-box story ever.

Government IT investments

A couple of days ago, the New York Times published an article by David Halbfinger that highlights major problems with a City of New York information technology project. Shortly after Mayor Michael Bloomberg took office, the city opted to allocate $66 million to modernizing the system that manages information about the municipal workforce. At the time of this writing, the city has spent $363 million for a system that does far less than initially planned. What went wrong? According to Halbfinger, who relied heavily upon records requested in accordance with the New York State Freedom of Information Law, several things:

• The Bloomberg administration was so intent on proving that outside consultants could streamline government that it failed to heed early signs that things were going wrong
• The individuals given responsibility for administering the project did not have the power to make pivotal decisions and overlooked numerous cost-saving opportunities
• The administration reacted to an early security failure by giving Accenture, the consulting firm responsible for developing the software, responsibility for defining precisely what the new system should do -- something that private corporations are loath to do for cost reasons
• City agencies kept fighting over the system's features and functionality -- the city was slow to give a single person responsibility for setting policy -- and Accenture repeatedly made and undid changes as a result
• Managers realized late in the game that they never wrote user instructions or provided training for staff responsible for using the new system and scrambled to cobble together documentation in the days prior to its rollout

At present, the new system is functional -- sort of. Although most City of New York employees can now access their personnel information online, retired employees and tens of thousands of Dept. of Education paraprofessonals and support staffers are currently not represented in the system. Moreover, at some point the city decided not to integrate management of its civil service system into its new personnel system. As a result, the civil service system that was developed in the 1980s is going to remain operational for the foreseeable future.

This is not the only government IT investment gone horribly wrong. The Bloomberg administration planned to spend $70 million developing CityTime, a modern municipal employee payroll system. To date, the city has spent $740 million, has a system that still isn't fully functional, and has suffered the humiliation of having the feds indict a bunch of CityTime contractors and consultants on charges of offering and accepting kickbacks. Several years ago, the Federal Bureau of Investigation junked its Virtual Case File system, which cost over $100 million, after it determined that the system was already outdated and simply couldn't replace its paper-based records management system. Similar horror stories abound at the federal, state, and local government level. Billions of taxpayer dollars that could have been put to better use have been squandered, and I don't even want to think about how difficult it's going to be to pull archival data out of some of these hideously expensive, deeply flawed systems.

Given the frequency with which government IT investments go bad, it's not surprising that some people -- including several of the readers who commented on Halbfinger's article -- conclude that governments are inherently bad at making IT purchases and that we simply shouldn't expect otherwise.

I don't think that this is the case, and I started writing this post with the intent of articulating a few strategies for avoiding the sorts of problems that the City of New York, the Federal Bureau of Investigation, and countless other government entities have encountered. However, upon second thought, I think I'm going to save these ideas for another post, simply because the problems that Halbfinger's article highlights are so fundamental that mere "strategies" can't solve them.

Personnel system project leaders and the Bloomberg administration itself knew that major problems existed and did nothing about them. The rollout of the first component of the personnel management system, which was completed in 2002, resulted in a massive security breach. Staff responsible for monitoring the system's development asserted in 2003 that "no sense of economy, efficiency or value is evident in any area of the project," but no one in a position of power paid any attention to their findings. Moreover, no government official has been fired or demoted as a result of the problems associated with this project.

In essence, what we have here is a woeful failure of leadership. No one in a position of power kept the scope and mission of the project from shifting and expanding, prevented city agencies from issuing multiple, competing change requests, or intervened as the contractor's bills skyrocketed. The City of New York did hire a private-sector project manager who was nominally responsible for keeping the project on track, but he quit after less than a year and has become an outspoken critic of the city's handling of the project. It also hired a human relations expert who was supposed to establish one city-wide personnel policy -- more than eight years after the personnel system project got underway. He returned to the private sector after less than six months in the city's employ.

At best, the Bloomberg administration failed to appreciate the differences between a multi-agency, public-sector operating environment and a single-entity, private-sector operating environment. At worst, it simply assumed that the public sector was inherently incapable of operating effectively and that the consultant would simply do an end run around the dysfunction -- real and perceived -- of the public sector. Instead of tackling the dysfunction head-on and ensuring that the personnel system project remained on track and under control, it sought to take the easy way out and gave its contractor free rein. City of New York taxpayers are now paying the price: the unanticipated $300 million that this project consumed would pay the salaries and benefits of more than a few teachers, nurses, firefighters, police officers, and sanitation workers.