SAA day two: electronic records

Comb jellyfish at the Georgia Aquarium, Atlanta, Georgia, 2 August 2016.

Even though I always make it a point -- at least when I'm paying my own way -- to attend a few Society of American Archivists conference sessions that have nothing to do with my current job responsibilities, I also seek out electronic records sessions that intrigue me or push me a little past my comfort zone. I attended two such sessions this morning: session 309, "DWG, RVT, BIM: A New Kind of Alphabet Soup, with a Lot More Heartburn," and session 409, "Working Together to Manage Digital Records: A Congressional Archives Perspective."

"I lost archival perspective and made wrong choices"

Earlier today, Leslie Waffen, the former head of the Sound and Video Branch of the U.S. National Archives and Records Administration (NARA), was sentenced to eighteen months in prison and two years of post-prison probation and ordered to pay a $10,000 fine.  His crime?  Stealing the materials for which he was supposed to care and selling them on eBay.  In an odd twist of fate, the chain of events that culminated in his arrest and prosecution began when an item he put up for sale -- a 1937 recording of Babe Ruth -- was spotted by J. David Goldin, who had given the recording to NARA.

At his sentencing, Waffen tearfully apologized for his misdeeds, noting that he had "lost archival perspective and made wrong choices" and that he had "violated, personally, the archivist's code of ethics." His statement isn't accessible via Public Access to Court Electronic Records (PACER), but judging from various media reports, Waffen's criminal career began when he began taking recordings that were being considered for inclusion in NARA's holdings to his house so that he could work at home.  At some point, his passion for collecting old sound recordings overrode his professional and moral judgment.  Eventually, he apparently decided that selling the stolen recordings on eBay was a good idea.

Every archival security expert I've ever met has emphasized that collectors of and dealers in historical materials deserve special scrutiny. Waffen, who most assuredly got off easy, exemplifies why we need to be mindful of the fact that expertise and passion -- which are, by and large, wonderful things -- can lead some people horribly astray.

 I didn't have a PACER account at the time Waffen entered his guilty plea, but I've got one now.  For your reference, here's a copy of Waffen's October 2011 guilty plea, which states that he could have been imprisoned for ten years and fined up to $250,000 but broadly hints that he would likely receive an eighteen-month sentence, and a copy of the supplementary statement of facts that accompanied the plea. 
Waffen Plea 2011-10 Waffen Supplement 2012-10

NARA releases 2011 records management assessment

Since 2009, the U.S. National Archives and Records Administration (NARA) has conducted annual surveys of federal government agencies' records management practices.  All of these surveys have revealed that electronic records management is a particular challenge for the federal government, and the 2011 assessment, the results of which NARA released earlier this week, is no exception.  Although NARA identified some modest successes, most notably increased transfers of archival electronic records, it's plain that management of electronic records remains an area of particular concern.  NARA found that:

< snip >

• Many respondents do not know or understand key terms and concepts pertaining to electronic records;
• Many respondents consider various aspects of electronic records management to be the purview of information technology staff;
• A significant number of agencies do not have migration procedures in place to ensure that electronic records are retrievable and usable to conduct agency business;
• Many respondents believe that media neutral records schedules eliminate the need for records management policies and procedures specific to electronic records;
• A significant number of agencies use backup tapes, which NARA does not consider a recordkeeping system, to preserve electronic documents and e-mail records;
• A third of agencies are using an ERMS [Electronic Records Management System] or RMA [Records Management Application] to manage their electronic records;
• Over 40 percent of agencies use e-mail archiving applications to manage e-mail messages . . . .

< /snip >

These findings are depressing but not particularly surprising.  Electronic records management remains a real challenge for many public- and private-sector organizations.  I would be willing to bet that the feds are actually ahead of most (but by no means all) state and local governments, and I suspect that many corporations -- even those whose stock in trade is digital information -- are similarly challenged. Earlier this week, I blogged about the near-disaster that Pixar (which should be applauded for its candor) experienced, and Twentieth-Century Fox and Paramount have discarded or lost digital files that have monetary and artistic value.  A host of other corporations are probably hoping that their records and information management nightmares remain out of the public eye.

What does NARA propose to do about the sorry state of federal records management?  Appendix I of the recently released report offers a detailed plan of action, and I encourage you to read it -- and the rest of the report -- in its entirety.  However, I will say that I'm particularly pleased that NARA wants agencies to incorporate records management plans -- with benchmarks and resource allocations -- into their annual budget submissions to the Office of Management and Budget (OMB).  I'm also glad that NARA to work with OMB to ensure that records management and archival functions are incorporated into new electronic recordkeeping systems and into the federal "IT governance process."  When a fiscal control entity demands something, government agencies tend to listen.

NARA seeks your input

Last November, President Obama issued a memorandum that marked the start of a sweeping effort to reform federal records management practices. Among other things, the memorandum directs the U.S. National Archives and Records Adminstration (NARA) and the Office of Management and Budget (OMB) to consult with individuals and organizations -- government and non-government alike -- who have an interest in improving the management of federal government records and increasing government openness.

NARA, which particularly interested in ideas that will promote government openness, use records and information to enhance federal agency performance, and reduce unnecessary costs and burdens, is fulfilling this requirement in a variety of ways. It's hosting public meetings and has created a Managing Government Records IdeaScale community that enables archivists, records managers, vendors, and anyone else who cares to do so to offer their suggestions, read suggestions advanced by others, and comment and vote up or down on each suggestion. You'll need to create a free IdeaScale account in order to do so, but signing up is quick and painless.

The IdeaScale interface is pretty intuitive, but if you're new to IdeaScale (which is one of my favorite Web 2.0 tools), you may want to consult NARA's IdeaScale guide.

The Managing Government Records IdeaScale community will be open for comments until Friday, 6 April.

Catching up

A few things you might have missed:

• Late last month, President Obama issued a memorandum directing each federal government agency to perform a comprehensive review of its records management program and then prepare a report for the Archivist of the United States and the Director of the Office of Management and Budget that outlines its plans to maintain and improve its program, "particularly with respect to managing electronic records, including email and social media, deploying cloud based services or storage solutions, and meeting other records challenges." These reports are due on 27 March 2012.

• Paper records created during an internal military investigation of a November 2005 massacre of civilians in the Iraqi city of Haditha were slated for destruction. However, the records, many of them marked as being secret, ended up in trailers purchased by a local businessman, who hauled the trailers to a Baghdad junkyard. Several weeks ago, a New York Times reporter covering the American withdrawal from Iraq inadvertently found them there. At present, it is unclear whether the military will open an investigation into the handling of these records.
  

• After a legal review, the Massachusetts State Archives has decided to open approximately 460 boxes of paper records of former Governor and current Presidential candidate Mitt Romney to researchers. Staff will review the files prior to disclosure and either remove or redact legally restricted information. The repository initially restricted access to the records as a result of a court ruling stating that gubernatorial records were exempt from the state's freedom of information law. As you'll recall, during the last days of the Romney administration, all of the files on its e-mail servers were deleted, several high-ranking officials were allowed to purchase the state-owned hard drives they used, and leased computer equipment was replaced.
  

• The administration of South Carolina Governor Nikki Haley routinely deletes internal e-mails. The administration claims that it does so in order to free up storage space on its server, but Erik Emerson, Director of the state's Department of Archives and History, asserts that it violates state records laws.
  

• OccupyArchive is George Mason University's Roy Rosenzweig Center for the History of New Media effort to capture digital items documenting Occupy Wall Street and other Occupy movements throughout the world. As Rosenzweig Center director Sharon Leon notes, they're "documenting a post-print movement" -- something that archivists must do if they want to ensure a complete and accurate documentary record.
  

• Finally, on a lighter note, here's why we need to caution teens about sexting: sooner or later, their sexts will be all over the Internet for everyone to read.
  

BPE 2011: ERA and the move to the cloud

This week, I’m spending a little time with my parents in Ohio and at the 2011 Best Practices Exchange (BPE) in Lexington, Kentucky. The BPE, which brings together state government, academic, and other archivists and librarians and other people seeking to preserve state government enduring information of enduring value, is my favorite archival conference. The Society of American Archivists annual meeting is always first-rate, but it’s gotten a little overwhelming, and I love the Mid-Atlantic Regional Archives Conference (MARAC), but nothing else has the small size, tight focus on state government records, informality, and openness that characterize the BPE.

Before I start detailing today’s highlights, I should say a few things about the content of these posts. For the past few years, those of us who have attended the BPE have tried to adhere to the principle that “what happens at BPE, stays at BPE.” This doesn’t mean that we don’t share what we’ve learned at the BPE (hey, I’m blogging about it!), but it does mean that we’re sensitive to the fact that candor is both essential and risky. The BPE encourages people to speak honestly about how and why projects or programs went wrong and what they learned from the experience. Openness of this sort is encouraging; all too often, we think that we’re alone in making mistakes. It's also helpful: pointing out hidden shallows and lurking icebergs helps other people avoid them. However, sometimes lack of senior manager commitment, conflicts with IT personnel, and other internal problems contribute to failure, and colleagues and supervisors occasionally regard discussion of internal problems as a betrayal. As a result, BPE attendees should exercise some discretion, and those of us who blog about the BPE should be particularly careful; our posts are a single Web search away. As a result, in a few instances I may write about the insights and observations that attendees have shared but obscure identifying details.

Moving on to this year's BPE itself, I'm going to devote the rest of this post to the insights and predictions offered up by U.S. National Archives and Records Administration (NARA) Chief Information Officer Mike Wash, who spoke this morning about the Electronic Records Archives (ERA), NARA’s complex, ambitious, and at times troubled electronic records system, and some changes that are on the horizon.

At present, ERA sort of works: staff use it to take in, process and store electronic records. At present, ERA holds approximately 130 TB of data. The Office of Management and Budget wants NARA to take in 10 TB of data per quarter, and NARA is working with agencies to meet this benchmark. However, ERA lacks an integrated access mechanism, and it contains multiple modules. The Base module handles executive agency data, the EOP module handles presidential records (and includes some internal access mechanisms), the Classified module holds classified records, and several other modules were built to deal with specific problems.

Building ERA taught NARA several lessons:

• Solution architecture is critical. ERA’s multiple modules are a sign of a failed system architecture. Anyone building such a system must carefully consider the business and technical architecture carefully during the planning stage and must manage the architecture carefully over time.
• The governance process must be clear and should start with business stakeholders. What do they really need the system to do, and how do you ensure that everyone stays on the same page throughout the process? Information technology invariably challenges control and authority, but if you set up your governance process properly, you should be able to retain control over system development.
• Over communicate. Funders and other powerful groups need frequent updates; failure to keep feeding information to them can be profoundly damaging.
• You must manage the project. The federal government tends to hire contractors to develop IT systems, and contractor relationships tend to deteriorate about six months after the contract is awarded. Most federal agencies cede authority to contractors because they are loath to be seen as responsible in the event that a project fails, but staying in control of the project increases your chances that you'll get the system you want.
• Watch costs closely. Cost-escalating provisions have a way of sneaking into contracts.
• Be mindful of intellectual property issues. The federal government typically reserves the right to all intellectual property created as a result of contracts, but this doesn’t always happen, and the vendor that built the first iteration of ERA has asserted that it controls some of the technology that now makes the system work; NARA will be much more assertive in working with future ERA vendors.

Wash also made some intriguing observations about some of the challenges that NARA and other archives are confronting:

• At present, our ability to acquire data is limited by bandwidth limitations. It takes more than three days to convey 20 TB of data over a 1 gbps data line and at least a month to convey it via the Internet. NARA recently took custody of 330 TB of 2010 Census data, and it did so by accepting a truckload of hardware; at present, there are no alternatives to this approach.
• The rate of data creation continues to accelerate. The administration of George W. Bush created 80 TB of records over the course of 8 years, but the Obama administration likely created more than 80 TB of data during its first year.

Wash indicated that NARA is starting to think that federal records should be created and maintained in a cloud computing environment and that transfer of custody from the creating agency to NARA should be effected by changing some of the metadata associated with the records being transferred.

Wash noted that the move to cloud computing will bring to the fore new preservation and authentication concerns. It also struck me that the transition that Wash envisions assumes the existence of a single federal government cloud that has adequate storage, security, and access controls and that, at least at this time, many states aren’t yet thinking of constructing such environments. Individual state agencies may be thinking of moving to the cloud, but most states don't seem to be preparing to move to a single, statewide cloud environment. Moreover, owing to its sheer size, the federal government is better able to negotiate favorable contract terms than state or local governments; the terms of service agreements that the feds hammered out with various social media providers are an excellent example. I have the uneasy feeling that some governments will accept, out of lack of knowledge, desperate financial straits, or inability to negotiate optimal terms, public cloud service contracts that prove problematic or outright disastrous.

Its nonetheless apparent that government computing will move into the cloud, that this transition offers both new challenges and new opportunities for managing and preserving records, and that archivists and records managers are going to have come to grips with these changes. The next decade promises to be most interesting.

The Lexington Laundry Company building on West Main Street, Lexington, Kentucky, 20 October 2011. This little gem was built ca. 1929, is an outstanding example of Art Deco architecture in the city, and is part of Lexington's protected Downtown Commercial District. It now houses an art gallery.

New NARA trustworthy digital repositories guidance

I've made it back to Albany and am wading through piles of mail and other stuff that either accumulated in my absence or simply wasn't dealt with before I left town. Included in that pile are a few nuggets of information that I wanted to pass on to you, and over the next few days, I'm going to do just that.

The U.S. National Archives and Records Administration has just released Establishing Trustworthy Digital Repositories: A Discussion Guide Based on the ISO Open Archival Information System (OAIS) Standard Reference Model. It contains a brief overview of the six core business processes -- ingest, archival storage, data management, administration, preservation planning, and access -- outlined in the OAIS Reference Model and a series of nicely thought-out questions that enable agencies to assess how well their current high-level policies, practices, and procedures support each of the six processes and to identify needed improvements.

As might be expected, this publication is directed at federal agency CIO's, program managers, and records officers. However, anyone charged with developing and implementing policies, procedures, and processes that support the long-term preservation of college/university, local or state government, or corporate digital materials should examine it closely. It's only twelve pages long and refreshingly free of jargon, which means that non-archivists and non-records managers may actually read it, and almost all of the questions it asks are broadly applicable. Strongly recommended.

U.S. Federal Cloud Computing Strategy released

Last week, the Chief Information Officer of the United States released the Federal Cloud Computing Strategy, which outlines how the federal government anticipates saving money, increasing the efficiency of its IT operations, and delivering better service to the public via cloud computing.

This strategy, which will help federal agencies migrate at least some of their IT infrastructure to commercial or government cloud environments, is intended to:

• Articulate the benefits, considerations, and trade-offs of cloud computing
• Provide a decision framework and case examples to support agencies in migrating towards cloud computing
• Highlight cloud computing implementation resources
• Identify Federal Government activities and roles and responsibilities for catalyzing cloud adoption (p. 2)
  

I haven't had the chance to give this document a close reading and likely won't have the chance to do so for a couple of weeks, but I have skimmed it and was pleased to note the following:

Storing information in the cloud will require a technical mechanism to achieve compliance with records management laws, policies and regulations promulgated by both the National Archives and Records Administration (NARA) and the General Services Administration (GSA). The cloud solution has to support relevant record safeguards and retrieval functions, even in the context of a provider termination (p. 14) [emphasis added]

As a state government employee, I'm also intrigued by this statement:

Federal Government contracts will also provide riders for state and local governments. These riders will allow all of these governments to realize the same procurement advantages of the Federal Government. Increasing membership in cloud services will further drive innovation and cost efficiency by increasing market size and creating larger efficiencies-of-scale (p. 29) [emphasis added].

And this one:

To effectively manage these governance issues in the long-term, the Federal Government needs to lay a stable governance foundation that will outlast single individuals or administrations. To the best extent possible, individuals or committees should have explicitly defined roles, non-overlapping responsibilities, and a clear decision-making hierarchy. These steps will empower the government for action, minimize unnecessary bureaucracy, and ensure accountability for results.

The following bodies will therefore have these roles and responsibilities:

• National Institute of Standards and Technology (NIST) will lead and collaborate with Federal, State, and local government agency CIOs, private sector experts, and international bodies to identify and prioritize cloud computing standards and guidance . . . . (p. 31) [emphasis added]

I'm looking forward to seeing how all of this plays out.

Lincoln document at NARA altered by researcher

U.S. National Archives and Records Administration, "National Archives Discovers Date Change on Lincoln Record," 25 January 2011.

Archivists devote a lot of attention to ensuring that researchers don't take records out of our facilities, but we don't always think about what they might be bringing in to our repositories. However, as recent events in the United States and the United Kingdom reveal, theft merely one of a host of malicious threats to the documentary record: fraudulent alteration and augmentation can be every bit as damaging.

Earlier today, the U.S National Archives and Records Administration (NARA) revealed that a researcher clandestinely changed the date on a 14 April 1864 pardon issued by President Abraham Lincoln by changing the "4" in "1864" to a "5" -- and thus giving the misleading impression that signing the pardon was one of Lincoln's last official acts. (Lincoln was shot at approximately 10:13 PM on 14 April 1865 and died nine hours later.)

Researcher Thomas Lowry snuck a pen into NARA's research room and the altered the date [images here] in an effort to bolster his scholarly reputation, and it worked: his "discovery" of the pardon garnered a substantial amount of media attention, resulted in the exhibition of the pardon in the rotunda of NARA's Archives I facility in Washington, DC, and helped to propel sales of his book.

Kudos to Trevor Plante, the eagle-eyed NARA archivist who suspected that something was amiss, contacted NARA's Office of the Inspector General, and did the historical research that confirmed his suspicions, and to NARA's Office of the Inspector General, which investigated the matter and secured Lowry's confession. Lowry's misdeed came to light well after the statute of limitations on prosecution expired, but he has been banned from NARA. Moreover, his scholarly reputation -- which is obviously of great importance to him -- is ruined.

NARA is not the only institution that has had to deal with a researcher seeking to alter the documentary record: in 2008, the National Archives of the United Kingdom discovered that researcher Martin Allen had clandestinely inserted 29 forged documents into 12 separate files documenting the activities of British secret agents during the Second World War. Allen, who used a laser printer to produce the forged records, then cited them in his books, which claimed, among other things, that the Duke of Windsor was a traitor, that Winston Churchill had ordered British agents to murder Nazi leader Heinrich Himmler, and that the British had entered into clandestine peace negotiations with the Nazis. Much to the dismay of many historians and archivists, the Crown Prosecution Service ultimately determined that prosecuting Allen, whose health was fragile, would not serve the national interest.

Let's start taking a closer look at the writing implements and research materials that our users bring to our research rooms -- doing so protects our researchers, protects us, and, most importantly, protects our records.

(A big tip o' the hat goes to Richard Pearce-Moses for alerting me to today's NARA story.)

The infopocalypse is upon us

Last week, the Boston Phoenix published an article by Chris Faraone highlighting how local, state, and federal governments are struggling to manage ever-increasing amounts of digital data. Provocatively titled "Infopocalypse: The Cost of Too Much Data," Faraone notes that:

The United States Census Bureau alone maintains about 2560 terabytes of information -- more data than is contained in all the academic libraries in America, and the equivalent of about 50 million four-drawer filing cabinets of documents.

Other federal agencies have similarly mind-boggling quantities of data, and state and local governments are also amassing vast stores of digital information.

Not surprisingly, "public data remains, by and large, a disorganized mess." Governments don't know precisely what they have or how to make best use of it, and old, paper-centered ways of responding to freedom of information requests and performing other essential functions persist.

Why does this situation exist? In my humble opinion, Faraone has nailed the root causes:

There is too much data. Digital storage is not a natural resource. The amount of information that government agencies may be required to keep — from tweets and e-mails to tax histories — is growing faster than the capacity for storage.

There's not enough manpower to manage all this data. The Obama administration hopes to make more information freely available online. But in the meantime, the old method of requesting data from the government -- filing a FOIA request -- is bogged down due to an insufficient workforce and long request backlogs.

Private companies are storing public data. This trend in outsourcing, largely the result of too much data and too little manpower, is a potential threat to both access and security, as resources that belong to the people are entrusted to outside vendors, raising new privacy concerns.

What to do about this situation? As Faraone notes, the data center consolidation strategy being pushed by Vivek Kundra, the Chief Information Officer of the United States, may help, but it's only a start. Faraone also suggests -- correctly -- that hiring additional staff who can process freedom of information requests and making readily available online data that doesn't contain legally restricted or, in the federal environment, classified information would also improve things a bit.

However, none of these things will solve the problem, which, as Sunlight Foundation policy director John Wunderlich pointed out to Faraone, is in many ways akin to that posed by the explosive growth of paper government records during the first two-thirds of the 20th century:

"Back then [government agencies] didn't know what to throw out, what to standardize, or how to organize. The challenges we face in data are in similar scope -- that's why it's so important that these issues are addressed head-on before it's too late."

Surprisingly, Faraone makes no mention of the U.S. National Archives and Records Administration (NARA), which works with agencies to figure out how to standardize and organize their records and how and when to dispose of records that have reached the end of their useful life, or of the role that agency records managers have -- or, as is all too often the case, should have -- in ensuring that all agency records are properly managed. Hiring some records management personnel -- at NARA, the 50 state archives, larger local governments, and larger government agencies -- would no doubt help to reduce agency storage pressures.

However, the more I work with electronic records, the less convinced I am that simply hiring a few more records managers will make everything better. We forget sometimes that formalized records management theory and practice are not mere outgrowths of common sense. They were practical responses to the challenges posed by the deluge of paper records created by ever-larger and ever more complex organizations. The infopocalypse that we face is in some respects quite similar to that which confronted our mid-20th century predecessors, but it is also, at least in some respects, unique. Addressing the challenges associated with our infopocalypse successfully will likely mean a shift in thinking no less monumental than that which propelled the rise of records management as a discipline and More Product, Less Process archival processing.

What will this shift in thinking look like? I don't know. I anticipate I that we're going to focus less on one-on-one guidance and more on standards development and automation of tasks now performed by humans. I also expect that our definitions of "record" and "records series" will be altered significantly and I suspect that, at some point in the future, be discarded altogether.

Yeah, I'm scared, too. However, our mid-20th century predecessors were as shaken by the changes in their record-keeping environment as we are by the changes in ours. They chose to meet those challenges head-on, and, after a lot of hard work and mistakes along the way, eventually developed workable solutions to complex problems. If we have any interest in surviving -- which may well mean evolving from "archivists" and "records managers" into "digital preservationists" or "data curators" or somesuch -- we'll take our lumps and do the same.

80 TB of presidential records

Here's a not-surprising-but-nonetheless-mind-boggling snippet of news: the U.S. National Archives and Records Administration (NARA) reports that it has received approximately 80 terabytes of electronic records generated by President George W. Bush and his staff. As if this weren't astounding enough, consider these other facts:

• The transfer of the Bush records marks the first time that NARA has received more electronic than paper records from a presidential administration.
• If printed out, the Bush administration's e-mail -- which makes up a fraction of the electronic records transferred to NARA -- would comprise approximately 600 million sheets of paper. NARA estimates that all of the other presidential libraries that it administers hold a combined total of roughly 550 to 580 million sheets of paper records.

Is your head spinning yet? If not, just start contemplating the volume of electronic records that NARA will likely receive from the Obama administration.

The Bush administration records, which have been ingested into NARA's Electronic Records Archives (ERA) system, won't be fully processed for quite some time, but archivists are busy preparing for an anticipated deluge of researcher requests in January 2014, when some of the records will become accessible under the provisions of the federal Freedom of Information Act.

By the way, NARA and the federal Office of Management and Budget are halting further development of the ERA system earlier than planned. NARA will focus instead on getting federal agencies to use ERA and may "recompete" the operation and maintenance components of its contract with Lockheed Martin, which built the system. Interesting.

FBI systems development problems -- and how to keep big IT projects on track

Prompted by the release of a scathing April 2005 U.S. House of Representatives Committee on Appropriations report, in June 2005 U.S. News and World Report devoted a lot of attention to information technology and systems design problems within the Federal Bureau of Investigation (FBI), which was then working with Science Applications International Corporation (SAIC) to develop a system that would enable FBI agents throughout the world to create and exchange case file data and to search a wide array of government databases.

The magazine's interview with the FBI's CIO and an accompanying article make for interesting reading: general-interest publications typically devote scant attention to recordkeeping issues. Moreover, the interview and article reveal that the FBI was beset by a host of problems, among them: it had difficulty maintaining control over the development process and kept amending its list of system requirements, technological change rendered SAIC's products obsolete even before they could be put into production, cost overruns were mammoth, and no one wanted to tell FBI Director Robert Mueller that the project was in deep trouble.

Shortly after the U.S. News and World Report articles appeared, the FBI scrapped the project, in which it had invested $124 million, and terminated its contract with SAIC (which has an interesting track record with big federal contracts). The agency ultimately started developing a new case management system, this time with Lockheed Martin. Unfortunately, earlier this month, the U.S. Department of Justice's Office of the Inspector General issued a report questioning whether the new case file management system, named Sentinel, will be finished on time and at cost and whether it will truly meet the FBI's needs.

Given that Lockheed is building the Electronic Records Archives system for the U.S. National Archives and Records Administration (NARA), this report is particularly worrisome. However, NARA identified its system requirements at the outset and hasn't substantially modified them and seems to have retained control over the development process, which suggests that NARA is going to have a better experience with Lockheed.

In the meantime, how can government agencies ensure that their systems development projects don't end up like the FBI's Virtual Case File and Sentinel initiatives? On Monday, the TechAmerica Foundation’s Commission on Government Technology Opportunity in the 21st Century, which consists of 31 federal agency and information technology industry representatives, issued a 33-step action plan for federal agencies and federal contractors.

The commission's core recommendations for agencies, which are detailed in full in its report, are:

• Develop professional, in-house program/project management capability
• Embrace iterative and incremental approaches to systems development
• Subject all major information technology acquisitions to independent, third-party risk review
• Improve communication and engagement with both the contractor developing the system and the internal staff who will become its end users

The report goes on to outline the benefits associated with each recommendation, the obstacles that inhibit their implementation, performance measures that assess the impact of implementing these recommendations, and a suggested timeline for federal implementation of these recommendations.

The commission's report was written with federal agencies in mind, but its plain-spoken assessments and recommendations and the vast experience of its authors make it required reading for any state or large local government contracting out the development of an information system. Moreover, any government archivist or records manager seeking to understand the potential pitfalls associated with large-scale systems development projects should study this report and its appendix, which summarizes the findings of past reports concerning federal government systems procurements.

New NARA guidance re: Web 2.0 and social media records

Late last week, the U.S. National Archives and Records Administration released a long-anticipated bulletins concerning management of records created as a result of federal agency usage of Web 2.0 and social media tools to conduct government business. It details how to determine whether social media and Web 2.0 content meets the Federal Records Act definition of a government record, highlight the records management challenges particular to Web 2.0 and social media records, and outline how agencies can address some of these challenges. It also stress that agencies are responsible for managing records that are housed by third-party service providers.

If you're interested in managing or preserving government social media and Web 2.0 content, be sure to check out this bulletin. I'll be giving several of my colleagues a heads-up about it.

Documenting Leadership: keynote and first session

Today was the first day of Documenting Leadership: A Symposium on Public Executive Records in the 21st Century. About 100 journalists, current and former public officials, policy researchers, archivists, and records managers were present, and more will be coming tomorrow.

I’ve had a very long day -- it began at 8:00 AM and ended at 8:30 PM, and the festivities begin again tomorrow at 8:00 AM -- so this post is going to focus on today’s keynote address and the first panel discussion. I’ll be playing catch-up tomorrow and over the weekend.

Richard Norton Smith

Richard Norton Smith, who has written biographies of several presidents and New York governors Theodore Roosevelt, Thomas Dewey and who is currently working on a biography of Nelson Rockefeller, got the day off to a great start: Noting that 16 U.S. Presidents have had gubernatorial experience, he asserted that the fact that 4 of these presidents came from New York really shouldn’t be surprising: New York’s governors have reformed the civil service, altered industrial and labor practices, spearheaded mammoth public work projects, oversaw the development of an effective public health system, championed civil rights, and fought water and air pollution. For decades, New York functioned as the laboratory for federal reform.

He then discussed how his research has led him to value both archival records, which shed unique light on the personal qualities of executive leaders and the nuances of their thoughts and deeds, and oral histories, which may be flawed but help to fill in the gaps in the documentary record. I was particularly struck by his assertions concerning executive leaders’ attitudes toward their records: in his experience, presidents think they did a pretty good job, and they don’t lie awake trying to figure out how to cover up their mistakes or misdeeds. However, they do lie awake thinking of how to prove the Washington Post and the New York Times wrong. Noting that history is always more generous than headline writers, he expressed the hope that New York’s governors would eventually recognize that history is not a continuation of the New York Post.

Robert Ward and Robert Sink

Some of the themes that Smith touched upon returned during the first panel discussion of the day, “Public Policy and the Public Interest.” Robert Ward (Rockefeller Institute of Government) highlighted several factors that may help to account for the divergence between federal and New York State laws governing executive records: declining gubernatorial and legislative interest in tackling the large-scale issues that commanded the attention of the state’s governors for much of the 20th century, the “broad degrading” of political discourse, the State Legislature’s efforts to limit executive power, and growing disbelief in the value of public institutions. Moderator Rex Smith (Albany Times-Union) noted that the relatively new but widely held belief that government is inherently ineffective has also contributed to the situation.

Rex Smith and Ned Regan

Former State Comptroller Ned Regan (Baruch College, CUNY), who transferred his official records to the State Archives when he left office and periodically consults them, asserted that New York’s political circumstances may also be to blame. New York is a “strong governor” state, and its current “three men in a room” approach to politics -- only the governor, the Senate majority leader, and the Assembly speaker wield any real power -- emerged because it’s the only way that the legislative branch can stand up to the executive. However, this system breeds legislative dysfunction, and the fate of executive records is one of many important matters that legislators have not addressed. Robert Ward, who asserted that the “three men in a room” phenomenon is rooted in legislators’ self interest and their constituents’ failure to hold them to account, agreed that legislative dysfunction helps to account for the state’s failure to modernize laws relating to executive records.

Gerald Benjamin (SUNY New Paltz), who will participate in one of tomorrow’s panels, offered an interesting comment from the floor: unlike Smith, whose work centers upon people who have been out office for some time, he’s found that executives dealing with the day-to-day grind of politics often have a short-term interest in tampering with the historical record. Moreover, New York is facing an even more unsettling problem: loss of institutional capacity to deal with records. Responding in part to this comment, Ned Regan offered an incisive observation: “systems are more important than competence.” Indeed: while at the reception, a colleague and I concluded it’s better for an organization to have a long-lived, workaday records management program than to have a records management dynamo whose influence fades after his or her departure.

In retrospect, I’m kind of surprised that none of the panelists devoted much attention to Watergate. I’ve always suspected that one of the reasons that New York lacks a strong executive records law is that, to date, it hasn’t had a governor who has abused the powers of the office as egregiously as Richard Nixon abused those of the presidency. However, all of the other factors identified by the panelists -- the narrowed focus of gubernatorial and legislative effort, the strident tone of current political discourse, cynicism about government, and legislative dysfunction -- are certainly in play.

Robert Sink (formerly of the New York Public Library) highlighted how the problems identified by the other panelists fed a scandal centering upon municipal executive records. New York City had a weak public records law and a weak archival program and Mayor Rudolph Giuliani, who was intent upon reorganizing city government and had a “defiant attitude” toward the concept of freedom of information, slashed funding for the Department of Records and Information Services (DORIS), appointed a commissioner who had no archival or records management experience, and sought to rob it of its independence.

As Mayor Giuliani was preparing to leave office, he asserted that DORIS, which lost half of its staff as a result of budget cuts that he himself had made, couldn’t properly care for his records. He wanted to transfer them to a non-profit organization that bore his name, allow the organization to limit access to certain materials, and engage an archival consulting firm to process them; it soon came to light that he had offered his records to several non-governmental repositories, all of which turned him down.

These actions prompted a public outcry, and the records were returned to DORIS after processing. However, the city has not substantially increased support for DORIS, and its public records law, while strengthened, is still relatively weak. Moreover, even though a reputable archival consulting firm processed the records, the possibility remains that the records were “cleansed” while they were outside public control. If New York City had a strong public records law and a strong government archival program that was insulated from political pressure, the integrity of the records wouldn’t be subject to question.

A lively discussion of the merits of preserving public records in public archives ensued. Bob Sink noted that if a gubernatorial administration works with state archives staff to manage the transfer of records, the records’ chain of custody will be firmly established. Moreover, New York’s state archivist is not a political appointee, and the State Archives itself is not directly controlled by the governor. Archivists in the audience pointed out that state archives are repositories of other government records and knowledge of the workings of state government and that, unlike some private repositories that mandate that the archivists who process a politician’s records share that politician’s political views, they do not factor political beliefs into their hiring decisions.

All in all, a great start. More tomorrow . . . .

Reminder: Executive Records Symposium, Albany, New York, May 20-21

The New York State Capitol, as seen from the Empire State Plaza, 24 November 2008.

If you have a personal or professional interest in ensuring that the official records of presidents, governors, attorneys general, mayors, and other elected executives are preserved and made accessible, you might want to come to Albany, New York on 20-21 May -- and make arrangements to do so by 17 May.

The records of elected executives document important policy and resource allocation decisions. However, in New York and many other states, the records of elected executives are not always transferred to the state archives or to other repositories. The gaps in New York State laws concerning the disposition of gubernatorial records have gotten some media attention lately, but other states, local governments, and the federal government face similar problems. Moreover, many executive records are now created or maintained in electronic systems; as a result, the days of acquiring executive records via dumpster diving or negotiation with an executive's heirs are pretty much over.

On 20-21 May 2010, the New York State Archives Partnership Trust and the Albany Law School’s Government Law Center will join forces to highlight the need for effective executive recordkeeping at all. Documenting Leadership: A Symposium on Public Executive Records in the 21st Century will explore the importance of preserving the records generated by governors and other elected public executives, including presidents, attorneys general, and mayors.

Former U.S. Attorney General and Governor of Pennsylvania Richard Thornburgh will deliver the symposium's keynote address, "The Legacy of an Executive: A Governor’s Perspective," and nationally recognized Presidential historian Richard Norton Smith will deliver an address entitled "Telling the Executive Story: The Thrill of the Chase."

In addition, attorneys, journalists, and past and current government officials will take part in panel discussions focusing on:

• Public Policy and the Public Interest
• Transparency, Executive Records, and the Media
• Executive Records: Access and Disclosure
• Access in the Digital Age
• Executive Records as Legacy

This event, which will be held at Albany Law School, is free and open to the public and will include a May 20th reception at the Governor's Mansion. However, attendees must register by May 17. Click here for online registration, detailed location information, and the full program.

Executive records symposium, Albany, New York, 20-21 May 2010

Do you have a personal or professional interest in ensuring that the official records of presidents, governors, attorneys general, mayors, and other elected executives receive the proper care? If so, you might want to come to Albany, New York on 20-21 May.

The records of elected executives document important policy and resource allocation decisions, and in many instances are essential to the study of state history and state government. However, in New York and many other states, the records of elected executives are not always transferred to the state archives or to other repositories. Moreover, most of these records are now created or maintained in electronic systems and are thus at risk of being rendered inaccessible by changes in technology or lost as a result of human error, decay of storage media, or equipment failure.

On 20-21 May 2010, the New York State Archives Partnership Trust and the Albany Law School’s Government Law Center will join forces to highlight the need for effective executive recordkeeping at all. Documenting Leadership: A Symposium on Public Executive Records in the 21^st Century will explore the importance of the records generated by governors and other elected public executives, including presidents, attorneys general, and mayors.

Richard Thornburgh, the former Governor of Pennsylvania (1979-1987) and U.S. Attorney General (1988-1991), and Richard Norton Smith, the nationally recognized Presidential historian, will deliver keynote addresses.

In addition, panelists drawn from the ranks of government, the news media, historians, public policy researchers, and the legal community will discuss the issues associated with managing, preserving, and accessing records of elected public officials who have executive responsibilities:

• The importance of executive-level records for the administration of government and implementation of public policy, and, at an administration’s end, for the historical record
  
• The special challenges of protecting sensitive information while assuring government transparency and accountability
  
• Best practices and model programs in other states and at the federal level
• Elements of model legislation

And to top it all off, on the evening of 20 May a reception for speakers and attendees will be held at the New York State Executive Mansion (reservation absolutely, positively required!)

For more details about Documenting Leadership symposium, which will be held in the Dean Alexander Moot Court Room of Albany Law School's 1928 Building, check out the online program. Please note that although this event is free and open to the public, advance registration for both the symposium and the reception is required.

"Removing the Shroud of Secrecy": records management and open government

Last Tuesday (23 March), the U.S. Senate Subcommittee on Homeland Security and Government Affairs, Subcommittee on Federal Financial Management, Government Information, Federal Services, and International Security held a hearing on "Removing the Shroud of Secrecy: Making Government More Transparent and Accountable." Members of the subcommittee, which is chaired by Senator Thomas R. Carper (D-DE) heard testimony from, among others, Archivist of the United States David Ferriero and Federal Chief Information Officer Vivek Kundra.

In his written testimony and in the Webcast of the hearing, Ferriero stressed the centrality of records management to good government: "the government cannot be accountable if it does not preserve -- and cannot find -- its records." He went on to assert that heads of federal agencies and other senior agency personnel "need to understand that the records and information they and their organizations are creating are national assets that must be effectively managed and secured so that the public can be assured of the authenticity of the record [emphasis added]."

Well said. It's all too easy to forget that government records and information are, fundamentally, public property and that they warrant the same careful stewardship as other public assets, and we archivists and records managers need to keep reminding others of this essential fact.

Open Government in the Digital Age Summit: videos, slideshow, and other resources

On 19 March, the New York State Chief Information Officer/Office for Technology (CIO/OFT)and the New York State Archives jointly hosted the Open Government in the Digital Age Summit in Albany. Speakers included Archivist of the United States Ferriero, U.S. Deputy Chief Technology Officer for Open Government Beth Noveck, and e-Republic Inc. Chief Content Officer Paul Taylor.

CIO/OFT's Web site now features video recordings of all Summit speeches and panel discussions, Paul Taylor's PowerPoint presentation, and the final Summit agenda. It has also prepared a news release that captures many of the day's highlights and posted photos of the Summit on its Facebook page.

Over 150 people attended the Summit, which brought together information technology professionals, archivists and records managers, public policy experts, journalists, and others interested in the relationship between information technology, recordkeeping, and government accountability and transparency. If you're interested in these issues, be sure to check out the resources above.

Open Government in the Digital Age Summit: David Ferriero

Approximately 150 people -- IT professionals, archivists and records managers, public policy experts, and journalists -- attended the Open Government in the Digital Age Summit jointly sponsored by the New York State Office of the Chief Information Officer/Office for Technology and the New York State Archives.

In lieu of a lengthy recap, I’m going to put up two or three shorter posts -- one centering on the keynote address delivered by David Ferriero, the 10th Archivist of the United States, and at least one other post highlighting the key threads of the day’s discussion. I have several reasons for doing so:

• I expect that all you archivists out there are at least a little curious about the new Archivist.
  
• Certain points and themes kept coming to the fore throughout the day, and it makes more sense to take a little time and tease them out than to do a session-by-session summary.
• I’m on vacation. I have some pretty intense sightseeing plans for the next few days -- which means that I need to break things up a bit and that posts about the Summit are likely going to alternate with posts about my travels. (Where am I? Come back tomorrow, and you’ll find out.)

First, however, a few words about “open government.” As a number of speakers and panelists pointed out, “openness” is commonly seen as being synonymous with transparency of operations and decision-making processes, accountability to citizens, and promotion of citizen participation in the creation of policy and development of services. Moreover, as several Summit participants noted as the day went on, open government requires an ongoing commitment to transparency and accountability -- and to ensuring that essential government information remains uncorrupted and accessible over the long term.

David Ferriero really helped to bring into focus the relationship between records and openness by stressing that if open government is your goal, you should focus on records management. In many respects, his point is obvious, but it’s all too often overlooked: how can you promise transparency and accountability if you can’t find the records that provide insight into government operations or policy development or can’t guarantee their completeness, accuracy, or integrity?

He then focused on the management of electronic records and the immense amount of work awaiting the federal government: a recent U.S. National Archives and Records Administration (NARA) survey of federal agencies revealed that agency records management and information technology staff, who really need to work together in order to manage electronic records appropriately, rarely do so. Moreover, almost every agency surveyed had records subject to a moderate or high risk of loss.

(Sadly, this situation isn’t unique to the federal government: the records management-IT disconnect is a big problem in New York State government and, judging from what I’ve heard from colleagues elsewhere, most other state and local governments are in the same boat. We’re all falling far short of the mark.)

Ferriero then discussed NARA’s own efforts to become more open and citizen-centered. Its Open Government Working Group, which is responsible for enhancing NARA’s ability to interact and collaborate with agency personnel and the general public, will post a formal plan for doing so on NARA’s Web site on 7 April. However, Ferriero gave us a preview of some of the recommendations that may appear in it:

• Establish an ongoing group charged with figuring out new ways of doing business and increasing openness
• Retool NARA’s strategic plan to include open government
• Create staff-only Web 2.0 tools that will enable NARA personnel to share ideas and collaborate more effectively
• Seek to engage the public via Facebook and other popular social media sites -- in essence, go where users are instead of waiting for users to come to NARA’s site
• Redesign NARA’s Web site with end users in mind, update the records management section, incorporate an “Ask an Archivist” interactive feature, and set up a wiki so that researchers can share the results of their research
• Sponsor an Apps for Archives competition (akin to the Apps for Army and other federal competitions) competition for development of applications that will improve access to NARA’s holdings
• Realign digitization priorities and give users ability to see what’s in the digitization queue
• Continue publishing high-value datasets in open formats on www.data.gov -- in essence, move from providing services to providing a platform for others to develop services
• Investigate the possibility of developing a Freedom of Information Act (FOIA) Web dashboard on FOIA services and responsiveness
• Actively declassify records and streamline declassification protocols
• Create an open government Web portal

Ferriero also took questions from the audience and offered up a number of fascinating tidbits:

• NARA is working with the secretaries of the Senate and the House of Representatives to ensure that draft versions of bills are captured and preserved appropriately
• NARA holds about 1 million e-mails from the Reagan administration, 250 million e-mails from the administration of George H.W. Bush, and anticipates getting approximately 1 billion e-mails from the Obama administration. Several of the people sitting around me gasped audibly.
• In an effort to breach the records management-IT divide, he’s convening the first-ever joint meeting of the federal CIO Council and the federal Records Management ??
• Like everyone else, NARA is still trying to figure out how to preserve Web sites, which change constantly; however, it’s plain that sporadic crawls such as those performed by the Internet Archive aren’t sufficient. (I agree; however, it’s just about the only practical approach available to most archivists at this time.)
• When developing new online tools and services, you really need to focus on users’ circumstances and preferences, not your own internal uses of technology; if you don't, you're not really committed to openness. For example, only fifty percent of New York City residents have home Internet connections, but most of them have cell phones -- and want mobile services and applications.
  

The last couple of questions centered on e-mail, BlackBerries, and cell phones, all of which pose particular challenges for archivists and records managers.

• Even the most conscientious employees occasionally use them to send and receive personal messages. Ferriero noted that, in the absence of automated tools that can pick out personal messages, it’s probably easier to keep everything than to conduct a manual review of messages. As an archivist who works with a small but growing volume of e-mail “archives,” I agree wholeheartedly: weeding these archives would be a time-consuming, soul-sucking task, and the resulting reduction in storage costs simply wouldn’t justify the investment of staff time.
• Determining which messages to keep is a particular challenge. I disagree with the approach that Ferriero advocated -- keeping everything -- but I understand why someone charged with upholding the Presidential Records Act and acutely cognizant of the research potential of routine correspondence would advance this argument. There has to be a way to preserve an adequate historical record -- sampling (by individual or by agency), targeting the accounts of key personnel -- without committing to saving every message that passes through an agency’s e-mail servers.

Ferriero’s speech was really well received, and I’m really looking forward to seeing how NARA evolves under his direction. The next few years ought to be really interesting.

White House IT + Web 2.0 = records management issues

Earlier today, Slate's Farhad Manjoo examined why the Obama administration hasn't been able to live up to its campaign promises to use social media and other Web tools to promote government accountability and responsiveness. His discussion of one of the problems facing the administration -- antiquated information technology -- sent all of my archivist/records manager concerns into overdrive.

According to Manjoo, the Obama campaign had access to state-of-the-art information technology and hired incredibly talented social media personnel (a Facebook co-creator among them) and found that entering the White House was like stepping back in time:

"The computers were so old they couldn't actually run social-media Web sites" like Facebook, says Alan Rosenblatt, the associate director of online advocacy for the Center for American Progress Action Fund. Even when the staff got better tech, federal computing policies restricted access to many sites; Rosenblatt says that some staffers had to bring in their own laptops with wireless modems in order to get on the Web.

Some of these difficulties have since been ironed out, and the White House managed to work out special terms of service with some social-networking sites in order to post content online. But Rosenblatt says that there are still staffers who need to bring in their own machines to get anything done [emphasis added].

President Obama has been in office for more than a year, and EOP staff are still using their own laptops do official government work? I'm a civil servant, and I fully recognize that laws and policies must be observed. I also realize that in some instances, people use their own computers to get work done at home or on the road. I've done it myself on occasion, and I'm generally fine with the practice provided that a) employer policy doesn't prohibit it; b) legally restricted or classified information isn't involved; and c) record copies of all work product make their way into the employer's servers, content management system, or other storage resources as appropriate.

Nonetheless, I really would like to know why moving the White House's information technology infrastructure into the 21st century is taking such a long time. I suspect that legitimate security concerns are slowing things down, but allowing government information to reside on computers that may be used to check personal e-mail, watch videos on YouTube, access Facebook, or play World of Warcraft really isn't an appropriate stopgap measure.

Moreover, how is EOP ensuring that the government information on these computers is being managed properly and that the Presidential Records Act isn't being violated? Is use of personal computing equipment limited to creation of social media content that will likely be preserved via other means, or is it common practice throughout EOP? Is there some sort of policy governing use of personal computing equipment to create official records, and if so, is it backed up with training and credible enforcement efforts? If EOP isn't addressing these issues, it might have a big problem on its hands one day . . . .