Adopting DocTracker in Editorial Practice

§Introduction

This essay was initially formulated as a review of the DocTracker (DT), based on the experiences of two documentary editors in adopting the tool in their respective editorial practices. The resulting review, beyond critiquing the merits of DT, served as a reflection on how technology informs and impacts the practice of documentary editors and the operations of a project. At the suggestion of co-editors-in-chief Noelle Baker and Kathryn Tomasek, and former reviews editor Christopher Minty, we are presenting this piece as two editorial reflections.

§The Papers of Dolley Madison and DocTracker

Holly C. Shulman

§Introduction and Context

In this reflection and as editor of the Dolley Madison Digital Edition (DMDE), I discuss my history and experience with this innovative and invaluable software. Before considering DT and the DMDE, however, I would like to acknowledge the role played by David Chesnutt, an editor of the Papers of Henry Laurens. In the 1980s, before the introduction of the web browser, Chesnutt and Charles Cullen, an editor with the Papers of Thomas Jefferson, realized that the embryotic digital world would change the work environment of the documentary editor. To address the rapidly evolving nature of this work, they helped develop Cindex, an indexing program designed for the complex indexes of scholarly editions. Chesnutt, one of the founders of the Association for Documentary Editing, also created the “Model Editions Partnership,” or MEP, in 1994–1995. Working with the Text Encoding Initiative (TEI) team, he adapted technical standards and practices for digital scholarly editing that fit the needs and characteristics of historical documents. It is worth pointing out the prescient and forward-thinking nature of Chesnutt’s work in the history of digital publication. It was not until 1993 that Mark Andreessen established Mosaic (which became Netscape in 1994), the system that then lost the first web-browser war to Internet Explorer. It was the web browser that made digital publications such as the DMDE possible.

§Early Experiences with Digital Editions

Less than a decade after the establishment of the MEP, I began working on the DMDE, having gained some experience in web-page building in the 1990s at the University of Maryland.¹ By 2003 I had edited The Selected Letters of Dolley Payne Madison with co-editor David Mattern as a letterpress edition, and had secured a publisher for my projected expanded born-digital edition: the still-nascent Rotunda (the electronic imprint of the University of Virginia Press). My initial staff consisted of two graduate students funded by a grant from the Division of Preservation and Access at the National Endowment for the Humanities. We had, however, no born-digital model to follow, beyond what MEP had created.² It was a given that a born-digital edition should follow the editorial practices long established in the field, and that the transcription of documents would remain at the heart of the project. We understood that we would publish documents that had been accessioned, transcribed, proofread, tandem read, and finally annotated, but the technical workflows of how to accomplish that were undetermined. As our work progressed, the DMDE received funding from the scholarly editions divisions of the National Endowment for the Humanities (NEH) and the National Historical Publications and Records Commission (NHPRC).

At first we hand-coded everything, both documents and annotations, using a rudimentary system for assigning unique identifiers to both annotations and documents. We needed a way to keep track of all the documents digitally, accounting for what had been accessioned, who had committed specific tasks on each document, and when that individual work had been completed. We required electronic versions of the traditional tools we adopted in creating our paper editions. Such letterpress tools included, for example: 3 × 5 cards recording correspondents, date, and archive; binders of letters with corrections and yellow sticky notes; and the cover sheets detailing workflow. Before we sent our digital manuscript to the press, all of our XML files had to be verified. Each text was run through Oxygen, a visual XML editor. It was tedious, boring, and yet essential. Producing each document, note, and essay was laborious.

As digital editors we were moving away from the more cumbersome physical management on paper and via file folder of the the records of a complex edition of hundreds of documents that passed through several stages of editorial work. Until the development—in our case, of DocTracker—of a digital equivalent of those file cabinets, index card drawers, and other physical tools for project management, the DMDE and the early experiments of the MEP operated under a kind of hybrid system. Some of the editorial work was made easier by the digital tools of word processing, but other parts of it required a continued use of the physical tools, or cobbling together other digital tools such as spreadsheets and DBase 2, which were not integrated with each other. I have spelled out my argument for how to rethink digital editions in an essay entitled “Television Is Not Radio with Pictures.”³

§Early Experiences with DocTracker

Mary MacNeil joined the DMDE around 2010. Mary was then on the technical staff of the University of Virginia Press. Rotunda had just published the first volume of our project (1788 to 9 June 1836), and Mary realized that we could transform our process if we worked inside a relational database. Along with Robert Oeste, programmer and analyst at Johns Hopkins University Press, she created DT. It profoundly altered our work process.

DT allows us to record and preserve all the relevant metadata related to each document, as well as to manage the workflow of the editorial process. We record the type of document (correspondence, land records, legal documents, literary records), enter the sender and receiver, assign the document to a volume, and link keywords. We enter a partial transcription and a summary that appears alongside the published document. We record the specific date or a date range, as necessary. Later, we recognized the need for descriptive editorial notes to accompany each document, both for substance, such as noting why we assigned a date to an undated document, and for recording physical details of the document, such as an excised corner or a hole that had been nibbled by mice. DT also allowed us to link related documents. Mary made this all happen.

§Functionality of DocTracker

DocTracker works by creating separate sections for managing and integrating the intellectual and clerical work of preparing the text for publication, and the research work of creating the annotations. DT includes a field for the body of the text, with additional fields for dateline, salutation, closing, signature, inside address, endorsement, postscript, docket, envelope, and verso. We annotate by linking a unique annotation for a person, place, term, or title to the reference in the text; DT provides the means to make that link. Further, it offers a space to insert images of the original document; this not only simplifies the process of copy editing, tandem reading, and checking, but enables us to do so remotely. Likewise, it facilitates versioning. Critical to the whole process is a workflow chart that traces the life of a text from entry into database, to transcribing and checking, to marking and tagging, to editing. In other words, the editor can view the status of any document during its lifetime, from inclusion in the project through publication.

Because of our digitally driven system of annotation, we do that scholarly work in a separate part of the database. Each annotation is written within the “ID” section of the database as a term, place, or name. This section of the database has its own workflow chart. If we need to revise an annotation for a subsequent installment of the edition, we can document that change. There is a field for source citations and a place for notes in which the researcher may record the research they undertook in creating the annotation. This system keeps track of the work and also assists in fact-checking the annotation. DT even allows the user to paste images of newspaper articles and paragraphs from scholarly works into the notes field. The annotation function allows the editor to see where a person, place, term, or other category has been referenced in the edition previously. DT provides the same process for the writing of editorial notes. Finally, DT makes it possible for the technology editor to check the edition through synchronizers, and for every team member to check constantly updated work reports.

I have found DocTracker invaluable. I am not sure how I would have finished the DMDE had it not been for the brilliance of Mary MacNeil and Bob Oeste. Connie Schulz will continue our reflection in her discussion of the revised system that Mary MacNeil developed for adoption by the Pinckney Papers Project.

§The Pinckney Papers Projects and DocTracker

Constance Schulz

§Introduction and Context

I cannot pretend to offer an independent or disinterested evaluation of DocTracker (DT), the database program that has been the single most valuable tool my staff and I have used in imagining, creating, and publishing two born-digital scholarly editions over the past fifteen years. As Holly Shulman proposes in the first part of this reflection essay, I am approaching my discussion through the history of our projects and our use of DT.

When Holly Shulman contacted me in the summer of 2006, asking me to help expand the number of editions of the papers of women from the American founding era, I never imagined that the partnership she proposed would last for fifteen years. I had just finished the research for a chapter on Eliza Lucas Pinckney and her daughter, Harriott Pinckney Horry, for a volume of biographies of South Carolina women, and I knew that the documentation of both their lives was much more extensive than the well-known edition of a letterbook by Eliza Pinckney’s descendant.⁴ With the offer from the DMDE of the use of DocTracker, a successful NEH grant application to fund a comprehensive edition of all known documents by or to the two South Carolina eighteenth-century women, and a publication commitment from the University of Virginia Press Rotunda digital imprint, my staff and I began work in the summer of 2008. We purchased inexpensive individual licenses of FileMaker for installation on our computers, the commercial database of which DT is a customized version, structured to meet the database linkage needs and TEI XML tagging of a scholarly historical documentary edition. We negotiated use of their more expensive server license and server space with the university’s Honors College. Use of the server license gave us the right to use several of their individual device access “desks” that enabled several editors to access the server simultaneously to work with the accession records data, texts, and editorial annotation we entered into DT. Mary MacNeil came to our campus as a consultant shortly after we began, and she held a two-day workshop to teach our small staff of editors and student assistants how to enter and then manipulate our data in DT.

§Adopting DocTracker: Early Experiences

In creating the digital edition of the Papers of Eliza Lucas Pinckney and Harriott Pinckney Horry (which we refer to as ELP),⁵ we were copycats of the DMDE. Essentially, ELP adopted the first version of DT as MacNeil and Oeste had designed it for the papers of Dolley Madison, with only a few minor adjustments to elements such as the “keyword” subject list and additional types of documents that reflected the papers of women living in different circumstances. Four years later, the result was a well-received digital edition of approximately 800 documents—including correspondence, household inventories, wills and land records, travel journals, and “receipt” or recipe books. The original DT system managed all this editorial labor seamlessly.

Our work with the papers of the Pinckney women introduced us to the much larger body of family records that documented the revolutionary and early national-era careers of Eliza Pinckney’s two sons and nephew, and by 2013 we had successfully applied to the NEH for funding. We naively envisioned this new edition as a six-year project to publish a selective edition of 3,000 of the 6,000 records we estimated were extant to document the military, political, and diplomatic writings of these Revolutionary-era Pinckney “statesmen” and their immediate families—and their management of their extensive plantations and enslaved labor force. Fortunately for us, Holly Shulman and the University of Virginia had received an NHPRC grant to enhance and revise DocTracker to meet the varied needs of more complicated projects and documents. The result, DT2, turned out to be ideal for the “Pinckney statesmen” editorial project.

§Adopting DocTracker: Refinements and Innovations

Several key elements had been added to the DT2 database by 2013. First, a more complex “table” for document sources enabled us to enter once, with a unique linkable identifier, not only the correct name or title of a repository or other source but also subentries with accurate titles of all individual collections that we had consulted within that source. This innovation greatly simplified the creation of accurate citations for all the published documents. The “Source” table also provided a template for preserving contact information for all source repositories linked with a record of our correspondence, enabling us to efficiently seek publication permissions and accurately provide acknowledgment credit. It also afforded a means of keeping track internally of “other copies” of complicated documents and allowed us to provide users of the final edition a detailed source list of all known versions of a document. As of this writing, we have accessioned materials from 150 repositories and 15 private collections, for which the “Source” table is an invaluable management tool.

A second enhancement in DT2 was the separation of the database sections for recording our accession records, and the templates into which we promoted for publication the transcribed texts and individual document annotations. DT2 provided two separate components called the “DocLog” and “Edition” tables. Our edition is highly selective. We accessioned nearly 13,000 individual unique manuscripts (many of which have more than one copy or version), of which we are publishing only 4,500. In addition to the “scratch transcription” field, the “DocLog” now offers other useful tools for managing the selection process: a “dating note” field to explain date discrepancies; a set of publication status “radio buttons” (“yes,” “no,” “maybe,” and “calendar”) that allows us to keep track of the selection decisions already made—and to determine which documents still need to be evaluated; a “sort” tool that permits searching in all fields of each section of the database in any hierarchical order of fields; and a “note” field in which staff can enter brief summaries, keywords, and other useful information about a document to allow for more precise search and retrieval. This enhanced search capability of DT2 enables us to manipulate our accession record of documents by date, sender and/or receiver, topic, publication status—or any combination of such elements—and to create and save reports. Such reports include, for instance, a spreadsheet of all documents to or from an individual, sorted by date with their current publication status.

The “Edition” table of DT2 now has templates and features that automate several of the TEI tagging operations necessary for the accurate rendering of transcriptions. Basic information from the “DocLog” accession record (to, from, date, source, type of document) transfers automatically into the “Control” template of the edition table when a document is promoted. The “Text” component of the edition table provides one-click tagging elements for all formatting, and some DT2 users transcribe documents directly into the text field. DT2 also allows us to transcribe documents and correct transcriptions in a word-processing program and upload the files into the text field of DT2, which then translates the formatting “tagging” in the word-processed file into TEI-compatible XML tagging, eliminating the need for even DT2’s streamlined hand-tagging. Other features of the edition table permit us to link transcriptions in another language to their associated translations and to toggle between a “clear text” view of a document or one showing all its markup—all the while creating and maintaining a workflow record of who worked on the document and when.

In spite of the seeming complexity of DT2, the system’s design is intuitive. In the past decade our project has trained two new editors, a half-dozen graduate assistants, and a dozen undergraduates, as well as four visiting scholars wishing to begin editions based on our methods and our use of DT. All of them quickly became proficient with the basic operations of data entry into the various tables, templates, and fields; they also learned how to search and to conduct the simplest manipulations of the data with relatively little assistance. Those who require more complicated functions have been able to investigate them and—with a little practice—to use them.

For the Pinckney Papers projects, DT in both its iterations has been an invaluable digital tool for both the project and content management aspects of our edition. Problems remain, however. First, the TEI guidelines on tagging for geographic entities embedded in the DT program for identification of places is not as self-evident or agile as writing and tagging other IDs. Second, as currently formulated, DT2 does not provide an immediate means for publishing the documents it has aided editors to prepare for publication. We were fortunate that Rotunda already had in place a system for converting the Oxygen-vetted XML files we submitted to them into a robust XTML publication accessible to browsers worldwide. Mary MacNeil and Bob Oeste were working on a simple publication process for DT, but they did not have the time or funding to complete work on developing the sophisticated linking capabilities that Rotunda has created using XML exported from DT’s database. While both Holly and I would, without hesitation, encourage new projects to adopt DT, one caution is the lack of a robust network of projects to support new users as problems arise, as the programmers have retired.

§Conclusions

DocTracker was an important innovator, taking the concepts developed by the Model Editions Partnership and creating a robust digital system that provides the tools an editorial project needs to manage its day-to-day work and integrate all aspects of its documents’ metadata and intellectual content. More recently, process- and content-management digital edition systems such as Drupal and Omeka have adopted many of DT’s assumptions and basic structures, replicating the capability we had developed in DT to provide linkages between different types of editorial data (transcriptions, editorial and bibliographic notes, source citations, correspondence with source repositories, workflow tracking of multiple documents and multiple staff, and research notes), all of which are crucial for internal management of a project as well as for successful digital publication.

DT is well worth saving for future editors and their editions, but it requires a mechanism for long-term support. As two editors who have relied upon DT, we believe the effort would be well worth the cost and labor. We are editors, not programmers. A system such as DT—which reduces the burden of electronic work and allows the editor to concentrate on the essential tasks of imagining, funding, creating, and publishing a digital scholarly edition—is worth keeping.

§Footnotes

1. At the University of Maryland, I had created a class in building websites, and with one of my students created the “The Dolley Madison Project,” http://www2.vcdh.virginia.edu/madison/. ↩

2. We did have Mary-Jo Kline and Susan Holbrook Perdue’s A Guide to Documentary Editing (Charlottesville: University of Virginia Press, 2008). ↩

3. Holly Cowan Shulman, “Television Is Not Radio with Pictures,” Historically Speaking 14, no. 5 (November 2013): 16–18. ↩

4. Constance B. Schulz, “Eliza Lucas Pinckney and Harriott Pinckney Horry: A South Carolina Revolutionary-Era Mother and Daughter” in South Carolina Women, ed. Marjorie Spruill, Valinda Littlefield, and Joan Johnson, vol. 1: Their Lives and Times (Athens: University of Georgia Press, 2009); Elise Pinckney, ed., The Letterbook of Eliza Lucas Pinckney, 1739–1762 (Chapel Hill: University of North Carolina Press, 1972). ↩

5. Constance B. Schulz, ed., The Papers of Eliza Lucas Pinckney and Harriott Pinckney Horry (Charlottesville: University of Virginia Press, 2^nd edition, 1998). ↩