One Culture. Computationally Intensive Research in the Humanities and Social Sciences

pub151. One Culture. Computationally Intensive Research in the Humanities and Social Sciences

Williford, Christa; Henry, Charles (Jun 2012, 44 pp.) ISBN: 978-1-932326-40-6.

DOI

This report culminates two years of work by CLIR staff involving extensive interviews and site visits with scholars engaged in international research collaborations involving computational analysis of large data corpora. These scholars were the first recipients of grants through the Digging into Data program, led by the NEH, who partnered with JISC in the UK, SSHRC in Canada, and the NSF to fund the first eight initiatives. The report introduces the eight projects and discusses the importance of these cases as models for the future of research in the academy. Additional information about the projects is provided in the individual case studies below (this additional material is not included in the print or PDF versions of the published report).

Citation

View

Files

Introduction

This study afforded opportunities to spend extensive time talking to the eight international teams of researchers supported by the first Challenge. CLIR conducted interviews with multiple participants in each project, and site visits with one of the institutional partners from each of the eight groups. Although the number of projects was small, the range of disciplines they represent are diverse, including classical and medieval literatures, British and American histories, the history of European cartography, nineteenth and twentieth century American quilt-making, twentieth century popular music, the study of British and American speech, the application of advanced computer algorithms for pattern recognition in digital images, and more. Each project described below serves as a touchstone for considering the changing nature of research in its particular domain(s), and what directions it these fields might take in the future.

Collaborations were of varying sizes, and the extent to which partners’ work was interdependent also varied. The common interests among the partners also ranged broadly; whereas for some there was a high degree of shared expertise in a particular subject, discipline, or theoretical approach, for others the basis for collaboration was the establishment of a shared, more discipline-agnostic, methodology. In some collaborations certain individuals contributed expertise in both a subject domain and the development of tools for data analysis, but in other cases these kinds of expertise were contributed by separate individuals representing quite different backgrounds and perspectives. In every case, however, principal investigators of the funded projects were highly experienced, well known scholars with a history of successful collaborations involving advanced technology, in many cases with the same partners with whom they worked on their Challenge projects. The vast majority of principal investigators held secure, senior-level, permanent positions, although the contributions of junior scholars, graduate students, and even undergraduates were without exception vital to project success.1 Notably, all the Digging Into Data projects were extensions of prior work requiring major investments of time and money in the preparation of data and analytical tools.

This relative maturity, at least when compared to many other e-research initiatives, is hardly surprising. Because the focus of the Digging Into Data program was on questions and methodologies rather than on the creation and maintenance of data corpora, there was an underlying assumption that respondents to the Challenge would build upon their own or others’ work in amassing data that would be both significant enough and reliable enough to be meaningfully queried. At the same time, in practice the availability and reliability of the data upon which the work depended varied greatly: these differences necessitated differences in approach for the researchers and to a large extent determined what kinds of research outcomes were possible during the brief grant period. While some work relied upon highly structured data, other efforts were built upon unstructured or raw digital files of multiple types, or harvested or produced specifically for the purposes of the project. We present our case studies along a rough continuum,2 from work focused upon the most uniform and organized kinds of data to those built upon less structured, more abstract, and heterogeneous forms. This ordering is necessarily subjective and is by no means an indicator of the relative value, impact, or success of these efforts.

The Eight Case Studies


1 All principal investigators were men; however women researchers did play key roles in several of the projects. At the conclusion of the Challenge, investigators expressed concern at the gender imbalance at the level of project leadership. This topic merits deeper exploration. For the second round of the Digging into Data Challenge funded in December 2011, nine of the fourteen funded projects have a woman as a principal investigator.

2 Some of the factors we considered in determining this order: data consistency, reliability, and completeness; the existence and reliability of metadata; the homogeneity of data types incorporated into the project; the proportion of text vs. non-text data; uniform vs. multi-layered analyses of data.

Using Zotero and TAPOR on the Old Bailey Proceedings: Data Mining with Criminal Intent (DMCI)


[Note: Additional details about this project are contained in the main body of this report (PDF).]

Using Zotero and TAPoR on the Old Bailey Proceedings: Data Mining with Criminal Intent serves as a classic example of expanding the impact of prior investment in constructing a reliable, highly structured digital repository. The Old Bailey Online was first launched in 2003 and has been in development for over a decade. It now includes complete trial records, including transcripts, of 197,000 trials held at the Central Criminal Court in London between 1674 and 1913. In size and scope, the Old Bailey corpus is broadly useful for answering a wide range of questions relating to social and cultural history; worldwide, Old Bailey Online users number in the hundreds of thousands.3

The scholars involved in Data Mining with Criminal Intent (DMCI) represent seven different institutions. Their joint goal was to expand the utility of the Old Bailey corpus by integrating it with online research tools for collecting and exploring textual data. These tools were Zotero, a personal research environment that helps users harvest and manage texts, and Voyeur Tools (now called Voyant Tools), a suite of visualization applications for text analysis, part of TAPoR. Zotero is based at the Roy Rosenzweig Center for History and New Media, George Mason University, and Voyant Tools have been developed by Canadian scholars Geoffrey Rockwell (University of Alberta) and Stéfan Sinclair (McGill University). Like the Old Bailey Online, both Zotero and Voyant Tools were already relatively mature initiatives at the start of the joint effort funded through Digging Into Data.

Before DMCI, scholars could search the Proceedings in Old Bailey Online by keyword or by predetermined categories such as ‘offence type,’ ‘verdict,’ or ‘punishment.” By bringing groups of trial texts into Zotero and/or Voyant Tools, scholars could be free from the “restricted research pathways” built into the Old Bailey Online architecture, opening up possibilities of exploring and analyzing the texts in alternate ways. In order to achieve the integration of the three projects, however, each of the teams had to make significant additions or changes to the software upon which their individual projects depended. They (1) developed a new application programming interface (API) for Old Bailey Online (OBAPI) so that other applications could call up, aggregate, and (if desired) export trial data, (2) created a “translator” for Zotero so that individual trial records could be described and managed along with other texts, and (3) simplified and extended the functionality of the user interface for Voyant Tools to make the exploration, visualization, and analysis of those texts more intuitive and powerful for scholars. Once the newly developed pieces of each project were in place, the team demonstrated the value of the integration using example searches.

The DMCI project cohered around a cluster of historical questions about crime in London society, as well as a broader scope of methodological questions about the application of data mining technologies to textual corpora. The UK, US, and Canadian teams each also had their own goals for strengthening and expanding the impact of their individual initiatives. This posed some challenges, since each of the three teams had to continue to serve the needs of their existing user bases while collaborating on making their work interoperable. Despite these challenges, team members observed that the collaboration was successful in both anticipated and unanticipated ways. It increased the range of expertise to which the participants had regular access, strengthened trust among the partners, and kept individual morale high in the process of meeting project goals. Teams in each country, and individuals within each of these teams, each followed independent work plans. Since the goals of the project required adherence to a common timeline, however, they held joint weekly conference calls during which all three groups shared progress reports. Participants also met in person on several occasions throughout the project, scheduling meetings around conferences and workshops of mutual interest.

One of these workshops involved exploring the application of machine learning techniques used to calculate degrees of similarity between records or texts-familiar to most users of search engines and e-commerce websites as links labeled “more like this.” The teams immediately saw the potential of implementing machine learning on the Old Bailey site: as a researcher explores the Old Bailey trial texts, selecting relevant records, machine learning technologies would result in increasingly accurate predictors of materials of interest to that researcher, all along suggesting trials that he or she might not otherwise have found if searching by keyword, category, or date. Excited by the potential of this kind of machine learning to facilitate serendipitous discovery (characteristic of so much scholarly inquiry), the Canadian team is now preparing to implement these technologies within Voyant Tools.


We realized that to pursue intellectual agendas such as the differing crime patterns of women and men we needed multiple points of entry rather than a single massive visualization. We thus followed several tracks at the same time, including data warehousing, mathematical models, and small-to-large visualizations.–DMCI team (white paper)


Throughout their work together and in their joint white paper, the DMCI group stresses the importance of communicating the value and utility of the research methods they facilitate and employ to “ordinary working historians,” as well as the importance of optimizing the usability of research tools to broaden their impact. To this end, they recruited an advisory team of a dozen professional historians with expertise in eighteenth and nineteenth century London and the history of crime to test the use of Old Bailey texts with Zotero and Voyant Tools. While the collection and management of trial records within Zotero came naturally to the “ordinary” user group, textual analysis and visualization was not so intuitive. Reactions from their focus group prompted the creation of a new, simplified user interface for Voyant Tools and documentation and tutorials about the project specifically tailored to the needs of historians.


What has become self-evident to me, is that “big data,” and even “pretty big data” inevitably creates a different and generically distinct form of historical analysis, and fundamentally changes the character of the historical agenda that is otherwise in place.–Tim Hitchcock


At the conclusion of the grant term, participants began pursuing publication opportunities within their own individual areas of historical expertise that demonstrate how “textual analysis of the infinite archive” can complement and transform existing practices and suggest new interpretations. In their white paper, the group explains how extracting records containing the term “poison” from the Old Bailey Archive with Zotero and importing these records into Voyant Tools helped project participant and developer Fred Gibbs identify frequent co-occurrences of “poison” with “drank” and “coffee,” while verifying that co-occurrences with “ate” or “eaten” are relatively rare, suggesting to him that coffee was the poisoner’s medium of choice in eighteenth and nineteenth century London. Cohen used Criminal Intent methods to identify and extract trial transcripts with references to bigamy and demonstrated a significant decline in the severity of punishments for female bigamists in the latter part of the nineteenth century. Tim Hitchcock and Bill Turkel took the extracted Old Bailey corpus into the sophisticated computation and visualization tool Mathematica. After numerous iterations of visualization and interpretation, they arrived at the graph below, which plots trial transcript length across the two hundred years of legal history represented in the corpus, color-coding the transcripts by offence. The results show a significant  increase in short trial reports for both serious offences such as forms of “killing”, and for all trials.  Through this process, Turkel and Hitchcock were able to identify an important phenomenon in the history of the criminal trial and to evidence the rise of ‘plea bargaining’ as a common characteristic of the British criminal justice system from the second quarter of the nineteenth century. In the process they were able bring the results of datamining to bear on a current historical debate on the timing and character of the  “modern ‘adversarial trial.'”

Data Mining With Criminal Intent is an ambitious initiative that required high investments of time and talent during a relatively short period. Participants also reported higher than anticipated expenses that had to be absorbed by the institutional budgets of the main partners, and that the computational demands of the project stretched the limits of the resources investigators had at hand: this was a pattern that we saw in every Digging Into Data project. Even with the advantage of a reliable, highly structured, and relatively uniform data set, the integration of textual analysis and visualization methodologies into historians’ search and discovery experience has not been a straightforward process, nor is it complete. Software development is ongoing and will continue to be informed by active historical research; by their own testimony, participants’ scholarship has also been richly informed by the development process. The pattern of collaboration between partner developers and scholars, including many partners who play both roles, is one that re-appears consistently in the other Digging into Data projects.


3 Statistics kept for The Old Bailey Online suggest 23 million visits to the site since 2003.


Project Participants


Other contributors and stakeholders


Project Outcomes


Other writings and media

Presentations, lecture notes and slides

Tools and documentation

Digging into the Enlightenment: Mapping the Republic of Letters

Grappling with very different kinds of visualization and analysis of pre-existing structured data were the teams engaged in Digging into the Enlightenment: Mapping the Republic of Letters. Like DMCI, Digging Into the Enlightenment is built upon significant prior work done by collaborating partners. The Electronic Enlightenment, based at the Bodleian Libraries, University of Oxford, continues to collect and make available to subscribers digitized scholarly editions of historical letters dating from the seventeenth through the nineteenth centuries.  In addition to the source texts, the resource provides chronological, biographical, and geographical information for more than 50,000 letters. In aggregate, these letters provide a remarkable portrait of the correspondence networks of Europe and North America throughout the Enlightenment period. Stanford University’s Mapping the Republic of Letters is an ongoing effort to visualize certain of these correspondence networks as a way of exploring a bundle of historical questions about the geographic range, diversity, and interactions among intellectuals during seventeenth, eighteenth, and nineteenth centuries. The Electronic Enlightenment team has collaborated with the Stanford Humanities Center on a series of related initiatives since 2008. In their Digging into Data Challenge project, the two teams recruited new partners Chris Weaver and colleagues at the University of Oklahoma, experts in the emergent field of visual analytics, who have been developing the Improvise tool to allow researchers to work interactively with complex data sets in an intuitive fashion.

Each of the three teams has followed an independent work plan designed to help them meet the goals of their individual projects while sharing data and communication tools throughout their work. During the course of the project (some of which at the time of this writing is still ongoing) all partners agreed that the collaboration had enriched their understanding of their own prior work and had been a major influence on their future plans. Notably, students at both the graduate and undergraduate level have made significant contributions to the Mapping effort. The partners are making active use of their work on the project for teaching as well as research.

Unlike the DMCI project, the Enlightenment team’s mapping methodology highlighted one of the most common challenges faced by humanists and other scholars who rely on historical evidence: despite long and painstaking work in collecting and describing the letters in the Electronic Enlightenment collection, inevitably much of the contextual information, including dates and locations, has not survived with the original documents. While these difficulties soon made the collaborators realize that their original ambition to create a complete and definitive visualization of the Republic of Letters would need to be postponed beyond the life of the grant, in this challenge they saw another opportunity, and one that would be a chance to make broader impacts well beyond the discipline of history. Strategies for representing absent or uncertain data have not yet been deeply explored in visual analytics. The collaborators describe the desired functionality as “ampliation,” which they define as “interpretation‐driven extension of data through visual interaction.” Developing and implementing an “ampliable” web-based environment would be revolutionary for humanists, whose task is most often to fill in the gaps in knowledge through interpretation. As an added feature for researchers using collections such as the Electronic Enlightenment, a data-driven, interactive map would expose the rich collections to researchers at the outset of their investigations and allow them to explore the collection through space and time, rather than through words. While the University of Oklahoma team continues to develop Improvise to strengthen its capacity to facilitate user ampliation, the Oxford team plans to seek funding to support a new interactive data mapping tool for its website.

At time of writing, the Stanford team had completed several working prototype visualizations as “case studies” in connection with Mapping the Republic of Letters. An early visualization of the Electronic Enlightenment texts helped them to develop functional requirements for a recent, more flexible interactive tool, which allows a user to explore the correspondence of Athanasius Kircher. This new tool was developed along with a new collaborative partner, DensityDesign Research Lab of Milan. While still in an experimental stage, their visualizations have already led the historians on the project to a new understanding of the Republic of Letters, questioning the characterization of the exchange of ideas during the Enlightenment as a “global” phenonmenon. The visualizations have also taught researchers a number of important lessons that are extensible to other disciplines. Exploring historical data geographically makes noticeable “outlying” evidence that might not otherwise be apparent to the researcher. Since humanistic inquiry is as much about identifying what is unique or exceptional about a subject as it is about observing trends and generalities, exposing these aspects of large data corpora can prove highly useful. Alternating from a geographic to an “interpretive” data histogram, as the new Athanasius project allows, permits that researcher “to explore many more dimensions of a correspondence collection, while also addressing, in part the problem of incomplete data.”

Finally, in their words, the Mapping project “has changed the way that all parties involved do research.” This was the result of extending their own collaborative network to incorporate experts in visual analytics. They explain:

For one thing, formulating humanities research questions in terms of machine‐readable queries required us to think systematically, across our areas of expertise and our individual research interests, and come up with broad, generalizable concepts. Humanistic inquiry-by contrast- is freeform, fluid, and exploratory; not easily translatable into a computationally reproducible set of actions. Much of our work throughout this project has been bridging this cultural divide: learning more about each other’s underlying assumptions in order to communicate the needs of humanistic research within the constraints of visual analytics.

This observation about the need to “bridge” a gap between automated computational analysis and interpretive reasoning that must make allowances for doubt, uncertainty, and/or multiple possibilities, is characteristic of the Challenge projects. But rather than seeing this mismatch as a barrier, for most of the researchers in this study this seems to have been a highly productive tension that prompted rethinking the potential of computer science to address the more amorphous kinds of questions that preoccupy scholars of human culture and behavior as well as redrawing the conceptual boundaries of those kinds of scholarship.

Project Participants

Other contributors and stakeholders

Density Design Research Lab (Polytechnical Institute, Milan)
Keith Baker (Stanford University, US)
John Bender (Stanford University, US)
Giovanna Ceserani (Stanford University, US)
Jon Christensen (Stanford University, US)
Dario Generali (National Publication of the Works of Antonio Vallisneri, Italy)
Anthony Grafton (Princeton University, US)
Carl-Olof Jacobson (Uppsala University, Sweden)
Wijnand W. Mijnhardt (Utrecht University, Netherlands)
Peter M. Miller (Bard Graduate Center, US)
Guliano Pancaldi (International Center for the History of Universities and Science, Italy)
Mark Peterson (University of California at Berkeley, US)
Jessica Riskin (Stanford University, US)
Jacob Soll (Rutgers University, US)
Francoise Wacquet (French National Center for Scientific Research, France)
Caroline Winterer (Stanford University, US)

See also:

Project website

Towards Dynamic Variorum Editions (DVE)

The final Digging into Data initiative that focuses primarily on textual data is Towards Dynamic Variorum Editions (DVE), a collaboration of scholars from Tufts University, Mount Allison University, and Imperial College, London. This project also represents a continuation of decades of prior work that includes digitizing and organizing primary and secondary resources pertinent to Classical Studies and making them available to researchers. The thirteen participants named in the project include scholars, computer scientists, scholar technologists, and one specialist librarian. Since all participants had a long history of working together on numerous grant projects, collaboration at an international level was a well-established practice for this group.

Much of the inspiration for the DVE project comes from partners’ experiences building the Perseus Digital Library, which is based at Tufts and is now one of the oldest and most widely used online research environments. The creation of Perseus was motivated by very basic, longstanding concerns for a variety of disciplines concerned with the history and culture of the Classical world, including documenting changes in language use and meaning throughout human history. Until recently, for most scholars answering such questions required lifetimes of effort working with centuries of editions, lexica, and commentaries, and the product of these scholars’ work has been new works built upon these same publication models.

In the past few decades, optical character recognition, text mining, and machine learning technologies have matured to a point at which researchers might discover word usages that would be impossible to find by other means, holding the potential to revolutionize our understanding of classical languages and, in turn, the many other languages they have influenced over time. The aspiration of the scholars working together on the DVE project is audacious: to create an online “edition of editions” that would make freely available and in one environment every edition and scholarly work ever published pertaining to Greek, Latin, or Arabic classical texts.

The deliberate choice to call the project Towards Dynamic Variorum Editions is telling. As in the examples of the Criminal Intent and Mapping the Republic of Letters projects, much work remains to be done in adapting already available technologies so that they are suitable to the purposes of classicists. Among the project’s technical concerns is the customization of current optical character recognition algorithms for classical languages, in particular for Greek. Secondly, the DVE partners have developed an infrastructure capable of handling the volume of text data now available that fits into their scope: rather than describing this corpus as “a million books,” they have chosen to reframe their corpus as “a billion words,” and are currently explicating the scale and implications of this corpus for classical scholarship in various publications. Finally, many hundreds of works, many of which are inaccessible to all but the most determined researchers, still remain to be digitized. These include the vast majority of works that survive in Greek and Latin, since the challenging nature of scholarship in an “analog” world has precluded most scholars from venturing outside the established canon. The task the partners have set themselves is to create an environment in which the rich array of translations and editions of canonical sources can be queried and intellectually accessed, while continuing to build upon this canon with less well studied works made accessible at the lexical level through machine-assisted techniques. Fulfilling both goals is critical to the team, since texts in classical languages span many more centuries than have been sufficiently explored. Texts that have drifted into obscurity often hold critical clues to shifts in meaning and use over time. Without these clues, scholars are liable to misinterpret what they read.

Unlike the Criminal Intent and Mapping the Republic of Letters projects, the DVE project depends upon data that is largely unstructured. These include page images of 28,000 works identified as being primarily in Latin selected from a corpus of 1.2 million books freely available through the Internet Archive (collected by the DVE team’s collaborators at the University of Massachusetts Amherst), 1000 scanned books from Google’s collection to which optical character recognition has been applied, and 8 million words of Greek and Latin that has been carefully marked up in XML for the Perseus Digital Library. The heterogeneity of the content available presents its own challenges, not the least of which is the flawed quality of much of the metadata. Using language detection software, the team found 4000 works in the Internet Archive collection which had been incorrectly identified as Latin, and a similar number of Latin works within the corpus of 1.2 million that had been misidentified as being in another language. The date information provided for most of the Internet Archive works was also unhelpful to the team, since the date of publication for the scanned volumes was most often drastically different from the original dates of the works contained within the editions. To correct these errors, the partners employed a team of students to hand correct metadata for 9000 of the Latin volumes in their corpus. The resulting 385 million word corpus of dated Latin was by far the largest amount of text the collaborators had worked with to date. Rather than the first 500 years of Classical Latin represented in the contents of the Perseus project, their new corpus contained the writings of two millennia. To conduct semantic analysis of the Latin corpus, the team used a morphological analyzer for Latin and a 2.9 million word collection of Latin words and their English translations. To these tools, Bamman, Babeau, and Crane (2010) have added a methodology for aligning multiple editions of the same text so that the scholarly markup of one edition may be mapped onto another.

Building and aligning a similar corpus of Greek texts is more difficult, because of the limitations of current optical character recognition technology for the Greek language. While current systems were reasonable for small quantities Greek-only texts, the team was interested in performing OCR on large-scale corpora. Furthermore, a large number of works of interest to Greek scholars include quotations in Greek embedded in works in other languages, so a methodology had to be developed to recognize Greek in these contexts as well. To this end, building upon work Federico Boschetti had done while a resident fellow at Perseus, Mount Allison University professor Bruce Robertson led a team of students in customizing open source OCR engines for Greek, introducing facilities to detect page layouts and to identify and classify Greek characters in multiple font families. Their customizations have improved the accuracy of OCR for pre-20th century Greek texts significantly. They have also begun work on a new graphical user interface (GUI) to enable easy correction of OCR.

In a similar way to the other text-based projects, working with data at a large scale has required access to infrastructure much larger and more powerful than the average desktop computer provides. The task of the Imperial College London team was to create and implement such an infrastructure, one that would not only process the mass quantities of text data and page images available today, but also work with the even larger quantities of texts relevant to Classical studies that have not yet been scanned, as well as those not yet written. This infrastructure had to privilege speed of character analysis and retrieval of comprehensive results across a large corpus over the precision and accuracy only possible using human-mediated OCR. The DVE project supported the system’s design, which is now undergoing testing. Preliminary results show a processing speed for Greek of approximately two minutes per page. DVE scholars believe that the distributed system, which takes advantage of parallel processing and cloud computing, can serve as a model for managing intensive processing of other types of research data in the future.

While it represents just one step in a much larger effort, the Dynamic Variorum Editions project has much wider implications for the future of humanities research and education which are laid out very clearly in the project white paper. By breaking away from the boundaries of the traditional classical canon, its scholars have begun to expose the deeper ties between Classical Studies, itself an interdisciplinary field, and the humanities writ large, encompassing all studies of human history and culture since the Classical period. At the same time, they have exposed the limitations of the current scholarly labor force to cope with the scale of interrelated, interlinked knowledge now at our disposal online. The deep relationships between Greco-Roman and the early Islamic world, for example, have gone largely unexplored due to the restrictions on physical and intellectual access. As technologies loosen these restrictions by making more accurate text mining, semantic analysis, and machine translation possible, scholars, students, and ordinary citizens can begin to make meaningful contributions to our knowledge and understanding of the past.

Project Participants


Project Outcomes

Project website

Peer-reviewed publications

Bamman, David and David Smith. 2012. “Extracting Two Thousand Years of Latin from a Million Book Library.” In Journal of Computing and Cultural Heritage, 5 (1), 2012. http://doi.acm.org/10.1145/2160165.2160167. Open access preprint available at: http://www.perseus.tufts.edu/publications/01-jocch-bamman.pdf

Bamman, David and Gregory Crane. 2011. “Measuring Historical Word Sense Variation.” In Proceedings of the 11th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL 2011), pp 1-10. http://dx.doi.org/10.1145/1998076.1998078. Open access preprint available at: http://hdl.handle.net/10427/75561

Bamman, David, Alison Babeu, and Gregory Crane. “Transferring Structural Markup Across Translations Using Multilingual Alignment and Projection.” In Proceedings of the Tenth ACM/IEEE-CS Joint Conference on Digital Libraries, Gold Coast, Australia, June 21-25, pp. 11-20. http://dx.doi.org/10.1145/1816123.1816126. Open access preprint available at: http://hdl.handle.net/10427/70398

Crane, Gregory, Bridget Almas, Alison Babeu, Lisa Cerrato, Matthew Harrington, David Bamman and Harry Diakoff. 2012 (To appear). “Student Researchers, Citizen Scholars and the Trillion Word Library.” Paper accepted to JCDL 2012. Available from Tufts Digital Library, Digital Collections and Archives, Medford, MA. Open access preprint available at: http://hdl.handle.net/10427/75559

Other writings and media

Robertson, Bruce. “Optical Character Recognition of 19th Century Polytonic Greek Texts: Results of A Preliminary Survey.” 19 January 2012.

Lectures, talks, workshops

Immergo: a workshop for dynamic variorum editions (4-7 August 2010)

Mining a Year of Speech

Two of the inaugural Digging into Data projects were based in the field of computational linguistics, a discipline with well established methodologies honed over decades by theorists and practitioners from multiple academic and corporate research environments. However, gathering the audio data to support research in this domain has, until recently, been performed more often in laboratory settings than in real life. Audio recordings collected in the laboratory suffer from obvious limitations: the artificiality of the laboratory setting, the framing of data collection around the specific interests of the researcher, and the “observer effect” being just a few. Mining a Year of Speech is a project that seeks to move beyond the laboratory to investigate very large-scale corpora of English recorded “in the wild.” These corpora include the recently digitized British National Corpus (BNC) from the British Library Sound Archive, and diverse recorded speech collections at the Linguistic Data Consortium (LDC) at the University of Pennsylvania. For the purposes of this project, the team dealt only with recorded language collections for which there were pre-existing transcriptions and for which there was a reasonable amount of descriptive information and would present relatively few legal difficulties for publication.

The 10 million-word spoken part of the BNC includes 4.2 million words BNC of everyday speech recorded by volunteers throughout the United Kingdom during the 1990s. The LDC’s collections include political speeches, news broadcasts, Supreme Court oral arguments, anonymous telephone and face-to-face conversations, audio book recordings, and interviews. Together the corpora analyzed for this project total over 5000 hours of speech, far more than would be possible for any one researcher to listen to in a lifetime. In data terms, the aggregated corpora for this project compare with some of the largest “Big Science” research data sets in use today.

Relative size of “Big Science” and “Big Humanities” corpora, by John Coleman.4

Rather than framing their project around a single question, the eight researchers working on Mining a Year of Speech focused their attention on establishing a methodology for making their corpora searchable by the scholarly community and wider public, in other words, a creating a “speech search engine.” When they reach this goal, their corpora will be able to support investigations of a wide variety of subjects across many disciplines. Naturally, the potential of using such a resource to test theories of word use, pitch, stress, and dialectical differences would be vast, and the team provides several interesting examples of these in their white paper, but a “speech search engine” could have impacts on other disciplines that would be equally wide-ranging. Students of English as a second language could retrieve multiple use cases and pronunciations for the vocabulary they are studying; scholars in media studies could identify and examine changes in the way certain words and phrases are stressed across multiple news outlets over time; given sufficient data about the gender, geographic origin, and socioeconomic backgrounds of speakers featured in large-scale audio corpora, psychologists or anthropologists could collect and study examples of recorded speech to explore theories of human behavior.

Graphical user interface for aligning transcripts to audio files used for the Mining a Year of Speech project.

The team’s central challenge is this: while speech transcriptions can already be easily searched, searchers cannot retrieve the snippets of audio data that match these transcriptions until those transcriptions are precisely time-coded to the appropriate places in the recordings. Manual time-coding is a time-consuming endeavor. For corpora at the scale of the Mining project, such a task would be impossible. Fortunately, recent developments in speech technology offer a solution: forced alignment. Roughly, forced alignment requires the automatic conversion of textual transcriptions into their phonetic equivalents, then using a phonetic dictionary to match the phonetic transcriptions to the recorded speech.

Performing forced alignment of mass quantities of transcribed speech has required the Mining team to deal with a number of complicating factors, including incomplete transcriptions, the existence of transcriptions for which the corresponding audio has been lost, varied dialects, varied qualities of audio data, the presence of background noise, moments when multiple speakers speak simultaneously, the use of “non-standard” words that cannot be found in a phonetic dictionary, and the presence of “disfluencies” such as “um” or “uh.” Each of these factors affects the accuracy of forced alignment. As they describe in their white paper for their ongoing project, the Mining team have already managed to address disfluencies and dialect differences sufficiently to allow reasonably accurate alignments. Producing phonetic transcriptions of non-standard words for the diverse corpora included in the project was also a greater challenge than anticipated, requiring the generation of multiple word and phoneme pairings and training the alignment software to select the most likely one. Once they addressed these issues, it was possible to align transcripts and audio for hundreds of hours of news broadcasts, two-thirds of the U.S. Supreme Court arguments and the BNC, two hundred fifty hours of two-party telephone conversations, and more.

Other issues have proved more obstreperous. For example, adapting the alignment software to cope with missing portions of audio was more important for this project than researchers initially anticipated, so the team has had to divert some attention to finding ways to identify portions of transcripts that have no corresponding audio. Another challenging problem arises from the need to preserve the anonymity of the people whose speech is included in the BNC: many of the instances in which speakers disclose personally identifiable information are marked in the transcripts, but it will be necessary for those portions of the audio files to be cut or altered (“bleeped out”) before the corpus is made available in searchable form. Still another challenge that continues to occupy the researchers is how to devise a way to measure the accuracy of alignment across large corpora, so that future users of the audio search engine can tailor their search methods to retrieve the audio they need as precisely and comprehensively as possible.

Work on Mining a Year of Speech continues,5 and it is clear that a refinement to the forced alignment algorithms used by this team will be required before the thornier issues affecting its accuracy are resolved. Some of the error issues will never be resolved; instead, as the team argues compellingly in their white paper, scholars will need to devise statistically valid ways to account for it as they use large-scale data resources for their work. Nevertheless, the team has already demonstrated that automatic alignment of large-scale transcribed audio is possible, and that soon web-based searching of automatically aligned corpora will be a reality. With more and more digital audio and video becoming freely available over the web, not to mention the vast quantities of analog data held in libraries throughout the world awaiting digitization, this project will make it possible for future scholars to use this information in ways we can only begin to imagine.


4 John Coleman comments, “I was astonished that the quantities of data involved in audio collections worldwide are on the same order of magnitude as ‘big science’ projects from the Human Genome project through the Hubble Space Telescope data right up to the scale of data collected over several years by the Large Hadron Collider…this insight will be useful for libraries, archives and humanities scholars in making the case for financial support for research needs that are in some respects comparable to ‘big science’ projects.”

5 There was a nine-month delay between the times the participating partners in this project received their funding from their respective agencies. This meant that the research timeline extended beyond other projects funded through the Digging into Data program, and beyond the time allowed for composition of this report.

Project Participants

Project Outcomes

Project website

Media

John Coleman interview on BBC World Service Digital Planet.

Lectures and talks

“Mining a Year of Speech,” at “New Tools and Methods for Very-Large-Scale Phonetics Research“, at the University of Pennsylvania, a workshop organized by our collaborators Jiahong Yuan and Mark Liberman. http://www.phon.ox.ac.uk/jcoleman/MiningVLSP.pdf

Baghai-Ravary, Ladan, Sergio Grau and Greg Kochanski. “Detecting gross alignment errors in the Spoken British National Corpus” (poster). 10/19/2010.

Coleman, John. “Large-scale computational research in Arts and Humanities, using mostly unwritten (audio/visual) media” (invited lecture), at the Universities UK-sponsored “Future of Research” conference. See slides.

The British National Corpus Spoken Audio Sampler continues to be actively developed and extended; in 2012 the team will be releasing a further substantial portion of the Spoken BNC. Even more extensive sections of the American English corpora used in Mining a Year of Speech have been published by the Linguistic Data Consortium.

Tools and documentation

A key tool vital to all the work of this project is the Penn Phonetics Lab Forced Aligner. This was developed prior to this project.

Harvesting Speech Datasets for Linguistic Research on the Web

The second audio-centric project among the Digging into Data pioneer initiatives is another methodologically focused linguistics project, Harvesting Speech Datasets for Linguistic Research on the Web, led by researchers at Cornell and McGill universities. Rather than devising a method for conducting analysis of specific transcribed speech corpora, however, the Harvesting partners focused on demonstrating a way to explore linguistic theories at “web scale,” effectively opening up all web-based audio corpora for scholarly investigation. To handle what is for all practical purposes a limitless (and ever-changing) resource, the Harvesting team uses the power of computing to collect and organize data for testing targeted, specific theories.

The Harvesting team’s expertise is primarily in the area of prosody, or the combination of rhythm, stress, and intonation speakers use when speaking the same word sequences in different contexts. To use an example from their project proposal, a theory of prosody called “contrastive prominence” would explain the difference between the different stress patterns for the phrase “than I did” in the following two contexts:

a.     She did more than I did.
b.    I wish I had done more than I did.

To study these kinds of linguistic theories, researchers need many different examples (or “tokens”) of the same short word sequence. Traditionally methods involve producing these tokens in a laboratory, but audio data available on the web offers researchers an alternative to retrieve them from “the wild.”

Just as with the team from Mining a Year of Speech, the partners on the Harvesting project have necessarily chosen to work with audio recordings for which there are pre-existing transcriptions. However, rather than relying on human-made transcriptions, Harvesting researchers relied on transcriptions produced through automatic speech recognition (ASR). Since they are reliant on still-maturing technologies, automatically generated transcriptions contain frequent errors, but the accuracy of ASR transcriptions “is often better than 50% at the level of short, common word sequences.” For this reason, automatic transcriptions are suitable for the kinds of research that most interest the Harvesting partners, when the goal is not to obtain an exhaustive listing of every possible use of a given phrase but simply to assemble enough tokens with which to test key theories about prosody.

The partners’ first goal was to develop and demonstrate a reliable technique for harvesting and analyzing audio snippets from the web. As the team explains in detail in the white paper, their harvesting technique is a multi-step process that is a combination of computationally intensive and manual work. First, the researchers use command-line programs first to create a list of potential “hits” for a particular target word or phrase in a web-based audio repository, including portions of the transcript that both precede and follow those hits. Using a second script, they retrieve the relevant audio files. The two scripts that identify and retrieve audio data must be tailored to work with each online resource from which data is harvested.  The researchers use a third script to cut segments from the harvested audio that surround the target word or phrase. Using an alignment method that is essentially an earlier version of the forced aligner employed in Mining a Year of Speech, the team then matches the retrieved transcriptions to their corresponding sounds in the audio snippets. Finally, they use two different sets of machine learning techniques to analyze and classify each snippet into two pre-defined categories according to its specific prosodic features.6

The partners divided the responsibility for each step in the process according to their own areas of expertise: Mats Rooth at Cornell handled the programming required for the initial harvest of target files; Michael Wagner at McGill handled the alignment of the transcription and audio data; and Jonathan Howell, a recent Cornell graduate who worked under Rooth and who now holds a postdoctoral fellowship at McGill, handled the machine classification of the retrieved, aligned audio files. This division of labor allowed each researcher to work somewhat independently, although they communicated frequently throughout the project. In a group interview with CLIR representatives, the team observed that having one member with experience working at both partner institutions was a key advantage.

The group’s second goal was to compare the implications of web-harvested data for specific linguistic theories to the implications derived from data produced in a traditional manner, in a laboratory. The McGill-based partners assumed responsibility for producing lab data that corresponded to the harvested data sets. The team was pleased to find that the lab data supported most of the same conclusions as the web harvested data. The team took this as an affirmation of the validity of traditional data collection methods. Said Michael Wagner, “This is a little bit surprising, and a nice result because the main acoustic cues are working both in the lab and in nature. Our lab data are almost as good [as data produced ‘in the wild.’]”

To meet the final goal for their project, the teams will harvest datasets from multiple audio repositories corresponding to a variety different theories of prosody, such as the aforementioned theory of contrastive prominence (“than I did” vs. “than I did”). At the time of writing, work on producing these datasets is still underway. It deserves mention that these research outcomes will be equally (if not more) important for facilitating future research as the team’s publications and conference presentations will be. Also among their most significant contributions are the series of computer programs they have created, which they have made freely available to other researchers on their personal and institutional websites.

One major challenge to their research, completely outside the researchers’ control, deserves mention: the web-based audio repositories with which the team has chosen to work were not always reliably accessible, with at least one key repository vanishing from the web without warning during the course of their work. For this reason, researchers working with “web scale” data must be prepared to adapt to the ever-changing nature of that data. Nevertheless, as ASR technologies continue to improve, including automatic “word spotting” techniques that can perform searches of auto data without recourse to transcriptions, it is easy to imagine the method demonstrated in this project being employed at greater and greater scales. Both separately and together, the harvesting as well as the machine analysis techniques now under development will have immediate applications in related academic disciplines such as speech recognition and artificial intelligence, as well as more long-term value in the humanities and social sciences broadly construed. In time, these methods will make the ever-expanding masses of audio and audiovisual data accessible to the general public in ways that were previously impossible.


6 These techniques are much more thoroughly explained in the project white paper, to be released after the conclusion of this project’s term in July 2012.

Project Participants

Project Outcomes

Gorman, Kyle, Jonathan Howell and Michael Wagner (forthcoming). “Prosodylab-Aligner: A tool for forced alignment of laboratory speech.” Proceedings of Acoustics Week, the annual conference of the Canadian Acoustical Association  (draft here).

Howell, Jonathan and Mats Rooth. 2009. “Web Harvest of Minimal Intonational Pairs.” Open access pre-print: http://ecommons.library.cornell.edu/handle/1813/13079.

Howell, Jonathan, Mats Rooth, and Michael Wagner. 2011. Acoustic classification of focus in a web corpus of comparatives. Presented at New Tools and Methods for Very-Large-Scale Phonetics Research, University of Pennsylvania, January 28-31. Open access pre-print available on Howell’s website: [Penn_presentation.pdf

Howell, Jonathan and Mats Rooth. “A corpus search methodology for focus realization” (poster). 157th Meeting of the Acoustical Society of America. Abstract appears in J. Acoust. Soc. Am. Volume 125, Issue 4, pp. 2573-2573. http://hdl.handle.net/1813/13093.

Prosody Datasets

Structural Analysis of Large Amounts of Music Information (SALAMI)

The final audio-oriented project focuses on recorded music. The stated goal of Structural Analysis of Large Amounts of Musical Information (SALAMI) is “to develop a state-of-the-art infrastructure for conducting research in music structural analysis.” In its emphasis on tool development and research methodology, it shares a great deal with the other Digging Into Data initiatives such as Data Mining with Criminal Intent and Mining a Year of Speech. Also like the Mining project, SALAMI is solidly based in an interdisciplinary research tradition: music information retrieval (MIR). Although the history of MIR is not so long as computational linguistics, its leading researchers have been developing tools and conventions for working with digital music data over the past decade and are poised to “scale up” to work with “large amounts of music information.”

While from a strictly computational point of view digitized speech recordings may share a lot with music recordings, the kinds of questions one might ask about music data are naturally very different from the kinds of questions one might ask about recorded speech. Musicologists are interested in audio signals in all their complexity, so MIR specialists interested in developing computer tools must go beyond the simple alignment of notation with a recording to identifying the formal and structural characteristics of pieces, collections, and genres of music.

Structural analysis, through musical annotation, is a skill widely taught to students learning music theory. The sequencing of elements such as verse, chorus, and bridge can form recognizable patterns characteristic of different musical genres, and the variations in these patterns are part of what makes an individual piece unique. By comparing annotations of different compositions it is possible to explore theories about how different composers stretch the limits of the genres in which they work, or how one composer’s work may influence another. It would be impossible, however, to perform structural analysis of very large music corpora manually: for one, it would be too time-consuming and expensive, and for another, since structural analysis is an interpretive exercise rather than an empirically exact science, it is impossible to guarantee that one person’s analysis of a piece of music would be the same as another’s.

The SALAMI team sought to test the accuracy of a range of computer algorithms designed to detect musical structures, measure the performance of these algorithms against human annotators, make adjustments to the algorithms as necessary, and, finally, produce a large web-accessible corpus of analyses of several hundred thousand recordings. With this data, musicologists could explore structural similarities among diverse pieces as well as examine in greater detail the empirically measurable characteristics of music that relate to how we understand a piece’s individual parts. The team was divided into three groups, each of which worked independently on a set of clearly defined tasks. In this way, the distributed team worked together in a way similar to the DMCI partners.

Partners at McGill University worked with a team of students to produce a “ground truth” set of musical annotations against which to measure the importance of the computer algorithms. In order to ensure that the “ground truth” set was representative of the range of genres included in the entire corpus assembled for the project, researchers focused on using recordings for which they had reasonably detailed and reliable metadata. Partners at Oxford and the University of Southampton devised a theoretical model that provides a way to express the relationship between machine analysis of recordings and the musicological concepts that govern how scholars understand their structures. This includes a method for representing musical annotations graphically into sections and hierarchies and a way of describing these annotations using Linked Open Data. The team at the University of Illinois built the computer infrastructure for collecting and analyzing hundreds of thousands of musical works, including popular and classical music, jazz, folk music, world music, and a variety of live recordings.

When the results of five different structural analysis algorithms were compared to the human annotations of the “ground truth” recordings, the segmentations identified by some algorithms aligned more closely to the human annotations than others did. The highest level of similarity between analyses, however, was achieved when more than one person analyzed a single piece. In the judgment of the SALAMI team, the results stressed the need for methodologies that incorporated comparisons of multiple analyses of recordings rather than seeking to perfect one definitive algorithm. To this end, they have designed an interactive interface within which scholars can examine the results of multiple structural analysis algorithms at the same time [See below]. Within this interface, a user may examine and play back individual segments of a piece and decide for him or herself how accurate or inaccurate each method of analysis is for that piece. The different algorithms as well as human-generated annotations are aligned across a timeline and color-coded for easy comparison. By facilitating interactions with complex data sets through visual means, the SALAMI visualizer provides for musicologists what Voyant Tools offers to historians working with the Old Bailey database using the DMCI methodology. The multi-layered and color-coded display achieves something like what the collaborators on Digging Into the Enlightenment hope ultimately to achieve: a way of indicating a range of possible options, or a level of uncertainty about the information depicted in a visualization.

Comparison of the segmentation analysis of one musical piece performed by various algorithms with ground-truth data produced for the SALAMI project

The SALAMI team’s final goal is to produce multi-layered analyses of approximately 200,000 pieces of music and make them accessible to users through their interactive visualizer. This “scaling up” of the project posed a couple of problems for the team. Performing analyses with the algorithms tested for this project consumed roughly five to six minutes of compute time per piece. To generate analyses for all of the pieces in the corpus, a typical computer would take five to six years, a highly impractical prospect. For this reason, completing the project requires the team to prepare to run calculations on a supercomputer. The University of Illinois partners have reconfigured each of the algorithms they have tested so that they can work on a standard supercomputer, and they have compressed and migrated the corpus to a mass storage device that can be searched, retrieved from, and decompressed by the supercomputer as needed during calculations. Once they have identified and scheduled time on a supercomputer cluster, they plan to publish data for the entire corpus. While the team has already presented at numerous conferences about their work, arguably the most important outcome of the project will be its published data.

Another aspect of this project that it is important to emphasize is the labor necessary to prepare the “ground truth” set, and how funding this labor constituted a significant amount of the project’s resources. As in the case of copious tests of the forced alignment algorithms required for the Mining project and for the tedious hand-correction of metadata necessary for the Dynamic Variorum Editions initiative, computationally-intensive research, especially if it employs new methodologies, often requires the validation of results through other means. While perhaps disappointing to the creators of the structural analysis algorithms used for SALAMI, the finding that these algorithms did not perform as well as human annotators on the test corpus was highly significant, raising new questions about how the human brain processes and understands music. This will give MIR specialists much food for thought in the coming years.

Project Participants

Advisors, data contributors, and other contributors

Music annotators

Christa Emerson, David Adamcyk, Elizabeth Llewellyn, Meghan Goodchild, Michel Vallières, Mikaela Miller, Parker Bert, Rona Nadler, and Rémy Bélanger de Beauport

Project Outcomes

Project Blog

Peer-reviewed publications

De Roure, D., K. R. Page, B. Fields, T. Crawford, J. S. Downie, and I. Fujinaga. 2011. An e-Research approach to Web-scale music analysis. Philosophical Transactions of Royal Society A. 369: 3300–17.

Smith, J. B. L., J. A. Burgoyne, I. Fujinaga, D. De Roure, and J. S. Downie. 2011. Design and creation of a large-scale database of structural annotations. Proceedings of the International Society for Music Information Retrieval Conference. Miami, FL. 555–60.

Ehmann, A., M. Bay, J. S. Downie, I. Fujinaga, and D. De Roure. 2011. Exploiting music structures for digital libraries. Proceedings of the Joint Conference on Digital Libraries. Ottawa, ON. 479–80.

Other writings and media

De Roure, D., J. S. Downie, and I. Fujinaga. 2010. SALAMI: Structural analysis of large amounts of music information. Proceedings of the UK e-Science All Hands Meeting 2010, Cardiff, Wales.

Lectures and talks

Osaka Symposium on Digital Humanities. Osaka, Japan. 2011. J. S. Downie, D. De Roure and I. Fujinaga. Large-scale music audio analyses using high performance computing technologies: Creating new tools, posing new questions.

Joint Conference on Digital Libraries, Ottawa, ON. 14 June 2011. I. Fujinaga. The structural analysis of large amounts of music (SALAMI) project.

Tools and documentation

http://salami.music.mcgill.ca/index.php/2010/12/updated-annotators-guide/

Digging into Image Data to Answer Authorship Related Questions (DiD-ARQ)

Digging into Image Data to Answer Authorship Related Questions (DiD-ARQ) is founded on arguably the most audacious “What if…?” proposition of all of the inaugural Digging into Data initiatives: in brief, what can we learn about authorship if we apply the same bundle of advanced image analysis algorithms across diverse, otherwise unrelated, digitized collections of anonymous and corporately created works of art? Formally characterizing the construct of an “authorship related question” is a task comparable to the challenge of digging into the data. Identifying the most salient questions required careful thinking on the part of humanities experts about what constituted meaningful authorship related questions in their domain, without overlooking new, compelling directions.

The corpora selected for the project included high resolution page images of fifteenth-century illustrated manuscripts, a select group of seventeenth, eighteenth, and early nineteenth century maps of North America, and more than 50,000 low-resolution digital photographs of nineteenth and twentieth century American quilts; taken together, these archives hold rich potential for exploring the histories of literature, art, culture, technology, science, and ecology as well as for advancing the science of adaptive image analysis.

DiD-ARQ’s three collaborating teams comprised one of the largest groups of individuals involved in any of the Digging into Data projects and necessarily incorporated experts involved in the creation of the three corpora, the diverse scholarly domains concerned with studying those archives, as well as scientists adept at varied kinds of image analysis. The three teams, based at Michigan State University (funded by NEH), University of Illinois (NSF), and the University of Sheffield (JISC), as well as the diverse experts represented in each of those teams, brought a different set of questions to the project; these questions were just as varied as the manuscripts, quilts, and maps with which they are working.

Our notions of what constitutes a viable research question have changed over time, and the humanities, social sciences, and sciences naturally have divergent understandings of the formality, originality, specificity, and breadth of questions appropriate to projects commonly undertaken in those traditions, whether they be conference presentations, single-authored journal articles, monographs, fieldwork, or lab-based research. “Mash-up” projects that bring together experts trained within very different traditions face special challenges when it comes to framing their collaborations. At the level of the overall collaboration, the motivating question frames the project along methodological, rather than disciplinary lines, e.g. “How accurate and scalable are adaptive image analysis methodologies when they are applied to diverse collections of image data?” At the domain level, the questions are equally broad but focus on bounding more specific topics within the subject areas represented by the three collections. The three teams diverged at this level to focus on (1) the characteristics of different scribal and artistic hands represented in ten fifteenth-century illustrated manuscripts, (2) the variety and accuracy of 17th-19th century French and English cartographic depictions of the Great Lakes, and (3) the relative popularity or uniqueness of specific color and pattern choices for nineteenth and twentieth century American quilts.

It is at a third, experimental level that they saw the strongest benefits of the collaboration, in terms of methods, results and new working hypotheses about authorship. Here the teams were able to select from a common set of image analysis tools to help them frame more specific questions relevant to each topic area [See figure below].  For example, shape analysis algorithms developed by the team enable comparisons of illuminated manuscripts as well as geographical shapes across maps.

“Illustration of the need to share software that can perform model-based segmentation of images. The same segmentation algorithm is applied to finding armor in Froissart’s manuscripts and lakes in historical maps.” (Simeone et al.)

The Sheffield team experimented with shape to identify a “digital fingerprint,” first, for the unique scribes’ hands known to have contributed to the copying of the manuscripts. While at the project’s conclusion the team was still awaiting the results of their calculations with reference to the characteristic features of the scribes’ hands found in the manuscripts, workshops held in the last quarter of 2011 enabled them to pinpoint key formal criteria which, it is anticipated, will help the team move towards establishing “a progressively more objective digital fingerprint for some of our hitherto rather shadowy medieval copyists.” This work is ongoing and has most recently been refined and enriched by the practical insights of palaeographers and a professional calligrapher (Colin Dunn, who also photographed the digital corpus).

The humanists working in cartography and climatological history at the University of Illinois used a segmentation process and algorithm for estimating scale to extract and compare depictions of the Lakes across forty different maps. To date, results clearly show that some of the lakes were more consistently mapped than others during this period, leading credence to the theory that extreme weather in some locations may have prevented cartographers from obtaining accurate measurements. Some of the analysis suggests “continuities between and within the [British and French] national traditions,” but the team considers these notions to be tentative pending further research. Despite the need for further investigation, from a methodological perspective the DiD-ARQ cartographers consider the project to be a major step forward in historical climatology.


“Climatologists know that historically water levels of the Great Lakes have varied significantly, but they have not yet included differing levels in climate models because data before the late 19th century is almost nonexistent. Because the segmentation algorithm may offer a way to estimate water levels and persistent ice cover…, we can begin to analyze maps prior to 1800 in order to provide usable data for historical climate models and future projections. Without Digging into Data funding for the segmentation algorithm and the quantitative analysis of data from forty maps of the Great Lakes between 1650 and 1800, we would not have been able to imagine the questions we’re asking, let alone provide useful data. Our preliminary reports have been well-received by climatologists, historians, historians of science, and literary scholars, so we believe we have begun to open a new avenue of productive and timely research.” –Robert Markley


The quilt historians working on DiD-ARQ used color space conversion to track the popularity of a particular shade of blue fabric across their quilt corpus, as well as a newly developed pattern recognition algorithm specifically tailored to measure the degree of symmetry in a design, or, in quilt parlance, to identify “crazy” quilts – an asymmetrical and highly embellished quiltmaking style that flourished in the United States from the 1880s until the 1920s. The algorithm was designed to assess the degree of regularity or divergence between compared regions of the quilt and distinguish highly irregular “crazy” quilts from their more regular counterparts. Intriguing “false positives” identified by the “craziness” algorithm categorized significant numbers of quilts as being visually related to crazy quilts.  These results suggest new directions for textile researchers as well as for digital humanities work. For scholars, there may be antecedents to the development of the crazy quilt tradition that have not yet been identified by researchers, as well as influences of the “crazy quilt” movement on subsequent visual arts that valued disorder and juxtaposition over symmetry and harmony. Further analysis of the large set of results is needed as well as new tools to facilitate interdisciplinary collaboration and iteration on results of computational analysis of data sets, particularly in a manner that can incorporate the participation of citizen scholars. These new directions and working hypotheses emphasize the complex relationship of authorship to so many dimensions of quilting: history, design, technology, etc. The relationships require explication and definition, to help experts understand what kinds of investigations are best pursued by authorship studies. These results suggest that authorship has the potential to be a canonical window through which humanists pursue a variety of questions. Obviously the potential rewards and risks need to be better understood before such epistemic status can be conveyed on authorship related questions and the investigations they might inform.


“While there have been numerous insights into the cultural history…of our individual datasets, the most significant findings of DiD-ARQ for the general audience is the applicability of computer algorithms for humanities analysis–no matter the image dataset. We have clearly illustrated that computer-assisted analysis can reveal new areas of exploration for humanists and that we can provide clarity regarding the underlying rationale of why [images] are interpreted in particular ways. For the general public, this means that we can more easily illustrate the value of humanities to discussions of things like how space, place, or race are depicted and remembered.” –Jennifer Guiliano


The size, scope, and complexity of this project’s network of nested research questions, not to mention the iterative way that these questions have necessarily developed and changed over time, have prompted the participants to adopt a very formal approach to their collaboration. They drafted a memorandum of understanding at the outset that outlined the responsibilities of each of the three teams as well as established practices for communication, for data, hardware, and software sharing, and guidelines for citation practice and credit sharing in project publications. Michael Simeone, Jennifer Guiliano, Rob Kooper, and Peter Bajcsy report some of the details of the project’s organization in a May 2011 issue of First Monday, including the reasons that supported their decisions to approach the project as they did. Weekly recorded web conferences, an email discussion list, and a secure, shared storage server were critical to daily project operations. While free open source, commercial, and institutional applications of these kinds are widely available and familiar to many researchers, the choices of what tools to adopt for a collaboration, the authors argue, are critical determinants of a project’s ultimate success, and deciding upon the appropriate level of openness, authentication, and functionality of tools to be used by all team members can be a complicated negotiation. The establishment of and adherence to rules for project documentation and communication helped the large numbers of scholars and students involved in the project engage with one another’s work, kept morale high, and bonds of trust among the collaborators strong throughout the ongoing project. Researchers indicated that they believed their experience on the DiD-ARQ project would be an important influence on the ways they planned and organized their future research.

Project Participants

Core Participants involved in all project elements

Contributing additional expertise in computer science

Wayne Dyksen (Michigan State University, US)
Alhad Gokhale (Independent Researcher)
Zach Pepin (Michigan State University, US)
William Punch (Michigan State University, US)
Tenzing Shaw (University of Illinois Urbana Champaign, US)

Contributing additional expertise in quilt making and quilt history

Beth Donaldson (Michigan State University Museum, US)
Amy Milne (Alliance for American Quilts, US)
Marsha MacDowell (Michigan State University and MSU Museum, US)
Amanda Silkarskie Michigan State University, US)
Mary Worrall (Michigan State University Museum and Quilt Index Project, US)

Other consulting quilt experts

Karen Alexander, Barbara Brackman, Janneken Smucker, Merikay Waldvogel, Jan Wass and members of the American Quilt Study Group email discussion list.

Heather Tennyson (University of Illinois Urbana Champaign, US)
Colin Dunn (Scriptura Limited, University of Oxford, UK)
Godfried Croenen (University of Liverpool, UK)
Caroline Prud’homme (University of Toronto, Canada)
Victoria Turner (University of Warwick, UK)
Anne D. Hedeman (University of Illinois Urbana Champaign, US)
Natalie Hanson (University of Illinois Urbana Champaign, US)

Contributing expertise in historical cartography and environmental literatures

Robert Markley (University of Illinois Urbana Champaign, US)

Project Outcomes

Project website

Peer-reviewed publications

Ainsworth, Peter and Meredith, Michael. “Breaching the Strongroom: a Pervasive Informatics Approach to Working with Medieval Manuscripts,” Proceedings of the KMIS 2011 International Conference on Knowledge Management and Information Sharing, Joachim Felipe and Kecheng Liu, eds. 2011: Setúbal, Portugal. pp. 264-71. ISBN 978-989-8425-81-2.

Ainsworth, Peter. “Digital Attraction: from the real to the virtual in manuscript studies,” Forum : University of Edinburgh Postgraduate Journal of Culture & The Arts, issue on Authenticity (May 2011), 14 p. http://www.forumjournal.org/site/issue/12/peter-ainsworth

Simeone, Michael, Jennifer Guiliano, Rob Kooper, and Peter Bajcsy. “Digging into data using new collaborative infrastructures supporting humanities-based computer science research.” First Monday 16.5 (2 May 2011).

Presentations and posters

Ainsworth, Peter, Presentation of the DID and Online Froissart projects, seminar on “Temporality and Value at the Intersection of the Arts and Humanities,” University of Southampton, UK, 12 April 2012.

Bajcsy, Peter. Presentation at Wolfram Technology conference in IL; October 13, 2010, http://www.wolfram.com/events/techconf2010/speakers.html.

—. Presentations at Imaging at Illinois workshop in IL, October 14-15, 2010, http://www.imaging.beckman.illinois.edu/imaging2010/.

—. Presentation at the Gordon Challenge in Data-Intensive Discovery conference in CA, October 26-29, 2010, http://www.sdsc.edu/gordongrandchallenge/.

—. Presentation at the Supercomputing Conference 2010, NSF funded panel on Grand Challenges in Humanities, Arts and Social Sciences, New Orleans, Louisiana , November 14-16, 2010; http://sc10.supercomputing.org/schedule/event_detail.php?evid=stpan108.

—, Rob Kooper, Luigi Marini, Tenzing Shaw, Jennifer Guiliano, Anne D. Hedeman, Robert Markley, Michael Simeone, Natalie Hanson, “Supporting Scientific Discoveries to Answer Art Authorship Related Questions Across Diverse Disciplines and Geographically Distributed Resources,” Microsoft Research eScience Workshop, October 11–13 in Berkeley, CA,  http://research.microsoft.com/en-us/events/escience2010/default.aspx (accepted as poster August 2010)

— and Maryam Moslemi, “Discovering Salient Characteristics of Authors of Art Works,” IS&T/SPIE Electronic Imaging, 17 – 21 January 2010, San Jose Convention Center, Section – Computer Vision and Image Analysis of Art, Paper 7531-10  presented on January 18th at 1:20pm.

Gokhale, Alhad and Peter Bajcsy, “Automated classification of quilt photographs into crazy and non-crazy,” IS&T/SPIE Electronic Imaging 2011, January 23-27; (Poster presentation).

MacDowell, Marsha. “The Quilt Index: Digging Into and Broadening Content, Current Challenges and Future Opportunities.” Closing Keynote Address, American Quilt Study Group Annual Seminar, October 2010.

Meredith, Michael and Peter Ainsworth, “Answering Medieval Authorship Questions using e-Science”, UK All Hands eScience meeting, 13 Sep 2010 – 16 Sep 2010, Cardfiff Wales.

—, “Digging into Image Data to Answer Authorship-Related Questions”, UK All Hands eScience meeting, 13 Sep 2010 – 16 Sep 2010, Cardiff, Wales.

Rehberger, Dean. “What to do with a Million Images: Rhetoric, Composition and High Performance Computing.” Conference on College Composition and Communication. Atlanta, Georgia, April 6-9, 2011. (Invited Featured Speaker)

—.”What to do with a Million Images: Rhetoric, Composition and High Performance Computing.” 2011 CCCC Virtual Conference. April 27, 2011.

—.”Digging into Data.” Computers and Writing 2011. University of Michigan. Ann Arbor, MI, May 19-22, 2011.

—.”Corporate Authorship and the Classification of Quilts.” Imaging without Boundaries: Exploring the Science, Technology, and Applications of Imaging and Visualization, Beckman Center, University of Illinois, Champaign IL, October 14-15, 2010.

Richardson, Justine. “The Quilt Index International and Digging into Data: Two Material Culture Digital Repository Initiatives Advancing Global Knowledge Production in the Humanities – There’s a Quilt for That.” December 1, 2011, at HASTAC: Humanities, Arts, Science and Technology Advanced Collaboratory, Ann Arbor, Michigan.

—. “Visually Digging into Museum Data.” Poster presentation on Digging into Image Data research project interdisciplinary collaboration with computer scientists and humanities scholars. April 8, 2011, Museums and the Web 2011, Philadelphia, PA.

—. “Supporting Scientific Discoveries to Answer Art Authorship Related Questions Across Diverse Disciplines and Geographically Distributed Resources”: Paper Presentation. Digital Humanities 2011, June 19-22, 2011, Stanford University, Palo Alto, California. Co-authored with: Jennifer Guiliano of the University of Maryland; Peter Bajcsy, Rob Kooper, Luigi Marini, Tenzing Shaw, Anne D. Hedeman, Robert Markley, Michael Simeone, Natalie Hanson, and Simon Appleford of the University of Illinois; Peter Ainsworth and Michael Meredith, University of Sheffield (UK); Dean Rehberger, Justine  Richardson, Matthew Geimer, and Steve M. Cohen, Michigan State University.

Shaw, Tenzing and Peter Bajcsy, “Automation of Digital Historical Map Analyses,” IS&T/SPIE Electronic Imaging 2011, January 23-27; (accepted as an oral presentation).

Shaw, Tenzing Michael Simeone, Robert Markley, and Peter Bajcsy, “Quantifying Historical Geographic Knowledge From Digital Maps,” Microsoft Research eScience Workshop, October 11–13 in Berkeley, CA, (accepted as oral presentation August 2010).

Shaw, Tenzing, Natalie Hansen, Anne D. Hedeman, and Peter Bajcsy, “Quantifying Differences between Medieval Artistic Hands Using Statistical Analyses in Multiple Color Spaces,” UK All Hands eScience meeting, 13 Sep 2010 – 16 Sep 2010, Cardiff, Wales, http://www.allhands.org.uk/events/all-hands-meeting-2010

Worrall, Mary. “The Quilt Index: On-Line Tool for Education and Research.” Roundtable, American Quilt Study Group Annual Seminar, October 2010.

Sample groups of images and metadata for each content area were uploaded to Illinois’ Medici collaboration platform for internal research access by DiD-ARQ researchers. Full datasets are accessible by each data set as follows:

Manuscripts:
The current collection of early 15th-century manuscripts comprises in excess of 6,100 images mainly at 500 DPI, hosted on a federated Storage Resource Broker (SRB) facility between UoS and UIUC using a web-front end collaboratively developed between the two sites (see http://cbers.shef.ac.uk). The images can also be retrieved from the SRB system via an API which provides direct access to the image dataset within a programming environment. See also the Online Froissart project: http://www.hrionline.ac.uk/onlinefroissart/.

Quilts:
The Quilt Index comprises more than 60,000 images of quilts (ranging from 150 to 600 dpi) with associated metadata regarding technical textile features as well as provenance and historical background documentation. All images are available online at the Quilt Index (http://www.quiltindex.org), hosted in the open source KORA repository at Michigan State University. The KORA repository has a web management interface, an API, as well as OAI-PMH capacity.

Maps:
For DiD-ARQ research, maps of the Great Lakes dating between 1747 and 1797 were selected and digitized from the Map Library at the University of Illinois Urbana-Champaign. High resolution TIFF files produced for the project were indexed and contributed to the library. Researchers may contact the library for further access or research needs. http://www.library.illinois.edu/

Tools and documentation

Tools and documentation for the algorithms created and tested through this project will be available on each participating institution’s project site. The direct code repository URL is: http://did.ncsa.illinois.edu/svn/did/trunk/.

Railroads and the Making of Modern America

[Note: Additional details about this project are contained in the main body of this report (PDF).]

The final of the eight inaugural Challenge initiatives is exceptional in that it is not only multi-disciplinary in terms of its applications but also in terms of data types integrated into a single project. Rather than being centered on one, or a small handful, of discrete data corpora, or on one particular type of research methodology, Railroads and the Making of Modern America is focused squarely on its subject-railroads-but also informed by a distinct theoretical framework for historiography. This framework, not unlike the position taken by the DMCI scholars, incorporates historical evidence into an integrated knowledge system that affords an examination of that evidence both from the broad, macroscopic and the minute, microscopic levels, and at every level in between. If realized, such an integrated system would hold the potential to make “invisible history” visible; in other words, help show correlations between continental trends and local acts that might not otherwise be noticeable in an archive, on a map, or in a narrative alone. In a way, the ambition for comprehensive coverage of the topic of American railroads is comparable to the ambition of the Dynamic Variorum Editions project to build an infrastructure that could encompass all primary and secondary materials related to Classical Studies, except in this case the goal is to create an infrastructure for using “big data” to do “big history” within a “continental-scale” geographic landscape, revealing relationships across both time and space. The Railroads project shares with Digging Into the Enlightenment the aim of constructing interactive data visualizations from evidence that is often incomplete.

The “big data” the two partners in the Railroads project seek to harness to the North American map include scanned books, serials, city directories, census documents, maps and geographic data, and, ultimately, all forms of railroad-related archives and ephemera, such as timetables, payroll documents, annual reports, etc. Naturally, not all information of these types is presently available in digital form. Libraries, archives, and online publishers are nevertheless making enormous strides in making these kinds of materials available. For this reason, efforts like the Railroads project, that seek to bring them together in new ways, are essential for informing libraries’ ongoing work. In the project white paper, the Railroads team cites numerous barriers to access posed by current digitization practice; these include limited access to many online sources, the difficulty of digitally assembling complete runs of rare serials from multiple repositories, to the neglect of fold-out maps in mass digitization projects such as Google Books. Despite huge investments and increasing availability, access to digitized newspaper collections is still particularly fraught for researchers who wish to repurpose that content in computationally intensive projects, since they are “designed with the needs of individual researchers in mind, rather than the requirements of automated and exploratory data analysis tools.” Licensing agreements or rights restrictions imposed by the suppliers of digitized content can often be too limiting to permit this kind of work.

Added to these barriers on access for the Railroads team are the thorny issues of data quality and consistency. To start, the data types considered for this project are inherently diverse, inconsistent, and variable as any advanced archival research practitioner may find in a career, but since digitization procedures refashion them as images which are then indexed through optical character recognition (OCR) in order to make them machine-readable numerous errors and omissions arise that hinder results. When research depends upon identifying and linking together all mentions of particular names, places, dates, and other numbers across varied data types, OCR frequently falls short. While automated methods of indexing can do a great deal, the limitations of the effectiveness of these technologies are such that a certain amount of hand-correction of these errors is unavoidable in order to insure precise and accurate visual analysis. Gaining access to their data was “simply the first stage” for the Railroads collaborators. Just as in the case of errors arising in the OCR of Greek text for the DVE team or the incomplete transcripts or recordings of the British National Corpus for Mining a Year of Speech, problems with data required large investments in hand cleaning, standardization, and coding. As in the other projects for which this was an issue, student labor made these tasks possible.

In order to help themselves manage the risks posed by barriers to access and variable data quality, the project partners have sub-divided their project into five work packages they call “Apps,” each of which explores a discrete railroad-related topic from the standpoint of a particular set of evidence. Built on a common platform called the Aurora Engine, each App contains an interactive visualization, raw data files and descriptive metadata for those files, a summary of the significance of the visualization to the study of the subject domain, documentation for those who wish to repurpose the visualization or data in other work, and (for project participants) a curatorial interface for editing content. While the work to complete the Apps had not yet concluded by the end of the project’s term, the white paper describes their envisioned functionality. For example, their “Network Connectivity” App draws upon historical and archival evidence to display the geographic expansion of the American railroad on an interactive map that allows visitors to explore at five-year intervals the growth and, in some cases, loss of railroad coverage across the United States throughout the latter half of the nineteenth century. “The Civil War and Mobility” App marks references to geographic locations that appear near any given keyword used in editions of the Richmond Daily Dispatch from 1860-1865.

Because of the aforementioned difficulties attendant to integrating heterogeneous data, the project serves an excellent test case for confronting challenges to be faced by large numbers of humanists as they enter the realm of computationally intensive research. As they continue their work assembling, correcting, and integrating their data into their five Apps, the team continues to think critically about that data, its limitations, and the manners in which it is contingent upon both its original cultural context and our own interpretations and/or mis-interpretations of its content. The value of deep engagement with evidence is familiar to researchers across the disciplines, but in a product-oriented research culture this is less frequently captured and touted as a beneficial outcome of the research process. Unless one recognizes the full range of outcomes of large-scale collaborations that incorporate data integration, including perhaps the most important of those outcomes-standardized, annotated data made openly accessible to others-it is impossible to understand the full import of this kind of long-term research initiative. But it is projects such as Railroads and the Making of Modern America that, by requiring such attention to detail, create opportunities for reflections on historical practice, the nature of evidence, and the limitations of computer technology for revealing new truths. Advocating a “critical middle ground” that accepts that reality is too complex to “be compressed into a very small number of categories,” yet not so “infinitely complex that any attempt to standardize data or search for regularities is fruitless”:

[T]he growing volume of digital source materials should not result in the increasing suspension of our critical faculties when terabytes of data wash over us. Instead, more data requires a much greater engagement of these critical faculties, since there is more scope for detailed investigation of within and between group variability, more opportunity for comparison and integration of different types of research resources, and more need for both simple and complex analyses, data mining and data visualization. [Healey and Thomas white paper, 42-43.]

Project Participants

Other participants and advisors

Project Outcomes

Main Project Website: follow links to “Aurora Project Apps”

Peer-reviewed publications

Thomas, William G. The Iron Way: Railroads, the Civil War, and the Making of Modern America (Yale University Press, 2011).

Healey, Richard G. (2012)   “Railroads and Immigration in the Northeast United States 1850-1900” Geography Compass, 6(8), 455-476.

Video

Thomas, William G. “The Civil War, Railroads, and the Making of Modern America.” Miami University Hamilton, Hamilton, Ohio, October 2011. [Introduction | Part One | Part Two | Part Three | Q&A Session]

Papers and Presentations

Thomas, William G and  R. G. Healey (2010) “Railroad Workers and Worker Mobility in the Great Plains” (Paper given at the Western History Association Conference, Lake Tahoe, October 2010)

Thomas, William G. and Doug Downey, (2010) “Digging into Railroads,” (paper given at Chicago Colloquium in Digital Humanities, Northwestern University, November 2010)

Healey, Richard G., W. G. Thomas, M. Johns,  I. Cottingham, E. White and L. Working (2010) “Digging into Railroad Data : a GIS and Visualisation-Based Approach” (Paper given at the SSHA Annual Conference, Chicago, November 2010).

Healey, Richard G.(2010) “Space: the Final Railroad Frontier?”  Commentator. Panel on Richard White’s “Constructing Railroad Space.” SSHA Annual Conference, Chicago, November 2010.

Thomas, William G. “Digital Analysis of Texts: The Mobility of African Americans After Emancipation,” Organization of American Historians, Houston, March 2011.

Healey, Richard G., M. Johns, M. Schaefer,  W. G. Thomas, I. Cottingham and L. Working (2011) “Railroads, Visualization and the Web: A Progress Report on the ‘Digging into Data Challenge’ Project”   (Paper given at the GISRUK Annual Conference, Portsmouth, UK, April 2011)

Thomas, William G., R. G. Healey  I. Cottingham, M. Johns, L. Working and M. Schaefer  (2011) “Railroads and the Making of Modern America: Tools for Spatio-Temporal Analysis and Visualization.” (Paper given at the JISC/NEH Digging into Data Challenge Conference, Washington D.C. June 2011)

Healey, Richard G.,  and M. Johns (2011) “Development of an Historical GIS of Railroads in the North-East USA 1826-1900. Phase II”  (Paper given at the Workshop on Railroads in Historical Context: Construction, Costs and Consequences  Foz Tua, Alto Douro, Portugal, October 2011 – now published in conference proceedings).

Thomas, William G. (2012) “Digital History: The State of the Field.” Chair, American Historical Association, Chicago, January 2012.

Thomas, William G. and Leslie Working (2012) “Railroads and the Making of Modern America: Tools for Spatio-Temporal Visualization” American Historical Association Conference, January 2012.

Thomas, William G. and Leslie Working (2012) “African American Mobility After Emancipation, 1865-1867,” (paper given at the Society of Civil War Historians, Lexington, June 2012)

Data related to the larger Railroads and the Making of Modern America initiative is available on the Project Website.

Tools

Exploratory tools related to this project are available through The Aurora Project website. These include :

Previous Report: "Rome Wasn’t Digitized in a Day": Building a Cyberinfrastructure for Digital Classicists Next Report: Core Infrastructure Considerations for Large Digital Libraries