Chapter 12 Reproducibility and Contestation in Humanities Digital Infrastructure

Contesting Reproducibility

A reproducibility crisis (also known as a replication crisis) has been ongoing for over a decade in the sciences, in which large-scale experimental studies across several scientific communities were reexamined in the context of the failure to replicate (Pashler and Wagenmakers; Baker; Hicks). Reproducibility here is broadly considered a means of obtaining the same results from the conduct of an independent study whose procedures are as closely matched to the original experiment as possible (Goodman, Fanelli, and Ioannidis).

The source of this crisis was located particularly in the fields of social psychology and biomedical research. For example, the Reproducibility Project (Open Science Collaboration) could confirm only thirty-nine of one hundred published social psychology studies were reproducible, and similarly preclinical cancer studies were identified as irreplicable (Begley and Ellis). A poll undertaken by the journal Nature reported that 52 percent of the scientists surveyed were convinced that science was facing a replication crisis (Baker).

Several types of poor practice (some of which continue) variously identified in the scientific literature suggest that a crisis is at hand—namely, the failure to reproduce the results of published studies in large-scale systematic replication projects (e.g., Open Science Collaboration); evidence of publication bias (Fanelli, “Negative Results Are Disappearing from Most Disciplines and Countries”); a high prevalence of questionable research practices (Fraser et al.); and a general lack of transparency in the reporting of methods, data, and analysis (Nuijten et al.).

Researchers in other fields have referred to the reproducibility crisis in science and a resulting lack of confidence (Pashler and Wagenmakers). Such conditions have led to a growing desire to accommodate support for reproducibility in research infrastructure (including scholarly publishing), as well as to consider the role of reproducibility in other scientific and scholarly fields, such as in the humanities (Peels and Bouter; Peels; Sikk), archaeology (Marwick), and linguistics (Berez-Kroeker et al.). An international cohort of linguists put forward a position statement on reproducibility in 2018 and expressed an overall need to establish protocols for verification and accountability, such as through data citation and attribution (Berez-Kroeker et al., 12). For other academics, however, this crisis has simply been mislabeled (Fanelli, “Opinion: Is Science Really Facing a Reproducibility Crisis, and Do We Need It To?,” 2630; Lash, Collin, and Van Dyke). They point to the fact that criticism and comments about reproducibility (real and perceived) largely focus on statistical and methodological approaches (Larregue).

The replication “crisis” has been instrumental in the move toward the adoption of open science, in which open data and open-source software and hardware are critical to enabling reproducible results by making the original data analysis and related processes more transparent (Terras; Munafò et al.; Schofield, Whitelaw, and Kirk). The open science movement is also underpinned by well-developed standards for open data, such as the FAIR (findable, accessible, interoperable, and reusable) principles,3 which have become increasingly adopted across scientific disciplines (Wilkinson et al.) and are now commonly invoked in library and archive settings (Koster and Woutersen-Windhouwer) and increasingly in the humanities. The European-funded All European Academies (ALLEA)4 initiative (Harrower et al.) is a recent sectoral example of key recommendations toward aligning digital humanities (DH) with the FAIR principles. Although data-sharing approaches have been ubiquitous in humanities computing (Terras; Sikk), there also have been criticisms of FAIR’s applicability in the humanities (Verhoeven, “Scholarship in a Clopen World”).

There is a smaller debate emerging over the definition, process, and value of reproducibility itself. The notion of reproducibility has been largely based on empirical, quantitative sciences where data are reified and viewed as objective. However, the terms reproducibility and replicability are not interchangeable in this sense; actually, they have variable meanings and requirements across different science domains (Plesser). Sabina Leonelli focuses on replicability and reproducibility as different processes that exhibit different characteristics. More broadly, the climate crisis and the complex disciplinary intersections in climatic data collection and reporting flag the limitations of a singular approach to reproducibility and replication (Bush et al.).

In the Reproducibility and Replicability report undertaken by the National Academies of Sciences, Engineering, and Medicine (National Academies), there is a concerted move to clarify the definitional boundaries of the two terms. Reproducibility involves the original data and code, and replicability involves new data collection to test for consistency with previous results of a similar study (46). Both processes differ in the type of results that may be expected. Of particular relevance to this chapter is that the contributors to this report consider reproducibility as being synonymous with “computational reproducibility” (obtaining consistent results using the same input data, computational workflow, methods, documentation and code, and conditions of analysis). This definition aligns with the acknowledgment that computational methods are becoming integral to research investigations across digitally transforming and “big data” disciplines, such as DH (Eijnatten, Pieters, and Verheul; Kitchin; Schofield, Whitelaw, and Kirk; McGillivray et al.). Although some scholars argue that computer replicability is possible and necessary in the humanities (Eijnatten et al.; Peels and Bouter; Peels; Sikk), the rationale supporting the practice remains open to challenge. The question around the value of reproducibility in the humanities is further exacerbated by evidence of unreliable, even unusable research in the sciences arising from the use of machine learning and artificial intelligence (AI). This has given rise to a full-circle debate around a renewed crisis in reproducibility (Ball), not least because many variants of AI remain “black boxes”— a system theory debated since the 1960s (Bunge, 346).

Scholars further point out, for example, that reproducibility can be a limited (and often forced) epistemic criterion for research quality and validity, articulating a specific technology of accountability and diverting humanities researchers from assuming the agency to explore innovative approaches or to add intrinsic forms of value to their data (O’Sullivan; Penders, de Rijcke, and Holbrook, “Rinse and Repeat: Understanding the Value of Replication Across Different Ways of Knowing”; Sui and Kedron). Furthermore, humanities researchers typically work within highly contextual and situated findings (Suchman; Cecire; Posner). Bart Penders, Sarah de Rijcke, and J. Britt Holbrook (“Rinse and Repeat”; “Science’s Moral Economy of Repair: Replication and the Circulation of Reference”) suggest that the wholesale adoption of epistemic standards from the sciences disregards humanities scholars as legitimate knowing subjects or subjects of agency in their own field, thus incurring a form of epistemic injustice (Suchman; Fricker).

In the digital information systems that matter to many humanities scholars, the collections of data describing the work of galleries, libraries, archives, and museums, an emphasis on positivist principles is especially pernicious. Typically, GLAM institutions organize their collections into structured hierarchies of knowledge that serve to reassert the patriarchal and colonial logics that underpinned their formation (Burrows, Jones, and Verhoeven). Seemingly in contradiction to this is the way that museums and libraries are positioning themselves as increasingly “participatory” knowledge organizations (Shilton and Srinivasan; Simon), engaging in high-profile advocacy addressing alternative truths, racism, homelessness, and migration, among other issues (Borda and Bowen; Lynch; Message; Sieg). In the next sections, we explore how contestability and contestation, as social and epistemological architectures, have been incorporated into alternative designs for digital infrastructure.

Contestability and Complexity in Galleries, Libraries, Archives, and Museums

For GLAM institutions, and museums in particular, the concept of “participation” is now so familiar as to be considered a defining characteristic of modern institutions. The Extraordinary General Assembly of the International Council of Museums in Prague on August 24, 2022, approved a new definition for museums that included the following: “Open to the public, accessible and inclusive, museums foster diversity and sustainability. They operate and communicate ethically, professionally and with the participation of communities” (ICOM).

Digital infrastructure is seen as an important space for inviting such participation, for example through crowdsourcing initiatives. When putting collections online, institutions can provide tools that allow users to tag, annotate, transcribe, describe, or otherwise contribute. As Trevor Owens writes: “[W]e might actually think about crowdsourcing as one of the most precious experiences we can offer our users. Instead of simply giving them the ability to browse or poke around in digital collections, we can invite them to participate. We are in a position to let the users of these collections leave a mark on the collections. Instead of browsing through a collection they literally become authors of our historical record” (128).

Such language emphasizes the agency of contributors. But as a portmanteau of crowd and outsourcing, crowdsourcing retains much of its original meaning: “The act of taking work once performed within an organization and outsourcing it to the general public through an open call for participants” (Ridge, 1). While institutions might benefit from external expertise, there is often little recognition that there are perspectives outside the institution that are not found within, let alone a move toward spaces outside the institution where people can challenge or contest the authority of that institution. Instead, contributing to “our historical record” becomes a precious gift from those select institutions that “let the users . . . leave a mark.”

What is more, these marks and the infrastructures used to capture them sometimes have little visible longevity. In 2014, Shelley Bernstein wrote about the Brooklyn Museum’s online collection, which at that time included opportunities for users to tag, comment, and “favorite” items, and contribute to the augmentation of collection records. She cites a simple example where a user’s comment allowed them to correct the orientation of an image displayed in collections online, providing a link to their comment in the footnotes (Bernstein, 18). Eight years later, the link returns a 404 error, and no comments are visible on this or any other collection record. Although tagging options remain, it appears that other user contributions have either been assimilated into the institutional record or rendered invisible (assuming that they have not been discarded altogether).

For many First Nations communities, control over records about artefacts and archives is yet another example of the ways in which large heritage institutions deny them agency, knowledge, and voice (Sentance). Therefore, the development of digital infrastructures that support more complex, polyvocal readings of Indigenous artifacts and archival material mostly have been developed separate from GLAM collection management systems. One of the most prominent examples of alternative infrastructures for representing collections is the digital access platform Mukurtu.

Mukurtu started as a grassroots collaboration between the Warumungu Aboriginal community in Australia’s Northern Territory and cultural anthropologist, ethnographer, and “accidental archivist” Kimberly Christen (Christen, “Opening Archives: Respectful Repatriation,” 185). The primary impetus behind the development of the tool was an identified need to manage access at a granular, contextual level based on “dynamic social and cultural systems, relationships, and cultural protocols” (Christen, Merrill, and Wynne). For example, communities might manage access based on gender, familial relationships, or social status. The name Mukurtu means “dilly bag” in the Warumungu language: “The dilly bags held sacred materials and elders kept and protected them as part of their obligations to care for their communities, relatives, places and ancestors, but the elders could not be ‘stingy’ and had to open them up when younger generations asked respectfully [. . .] like the dilly bag—Mukurtu—the digital archive we built should be ‘a safe keeping place’” (Christen, Merrill, and Wynne). The first version of the platform was deployed for the Warumungu community in 2007, followed by beta versions with Plateau tribes in the United States. This was followed in 2010 by the launch of Mukurtu CMS, an open-source platform “aiming to empower communities to manage, share, narrate, and exchange their digital heritage in culturally relevant and ethically-minded ways” (Mukurtu CMS).

In addition to access controls, Mukurtu includes options for capturing multiple perspectives. According to Christen (“Does Information Really Want to Be Free?,” 2887), it allows users “to infuse their voices, their cultural concerns, and their notions of sociality and historicity into the system.” Unlike typical crowdsourcing initiatives, the platform is designed to prioritize the unique knowledge of First Nations peoples regarding their own culture and collections. Where institutional collection records already exist, this is achieved by appending a parallel record that can include alternative narratives and classification structures, as well as additions or corrections to the existing record without altering or expunging those records. As Christen (“Opening Archives: Respectful Repatriation,” 201) writes: “We all understood the multiple benefits to tribes, scholars, and public users of the site, of seeing that history is indeed made, unmade, and negotiated over time; whereas ‘records’ often seem official and irrefutable, they are malleable and susceptible to change.”

Plateau Peoples’ Web Portal5 is an example of a collaboratively curated archive built on the Mukurtu CMS. It includes records created by the community and augmented records for digitized artifacts found in GLAM institutions such as the National Museum of the American Indian (NMAI). Whereas the catalogue record for a woman’s basket hat at NMAI includes little more than an image of the item and a set of fielded metadata (Spino), the Mukurtu record includes some of the NMAI metadata, as well as a separate Confederated Tribes of Warm Springs record that includes an extended cultural narrative section with a video and transcript where three Warm Springs women discuss the basket hat and its use as part of traditional food gathering practices (Plateau Peoples’ Web Portal; Spino).

Mukurtu takes important steps beyond annotation, tagging, and comments, providing opportunities for communities to capture and preserve their significant knowledge as part of a well-structured digital archive. However, some of the conceptual limitations of GLAM description remain evident. Visitors to the Plateau Peoples’ Web Portal looking for items like the woman’s basket hat are directed to the museum record first, and then they must select the community record to move beyond the institutional perspective. This gives a sense that the community record is appended to the museum record; community knowledge here is additional rather than foundational. Although users can compare the two records and see what (if anything) is different, this only allows indirect contestation. And there is no clear link to the source NMAI record, or to other relevant items or records in the Portal or elsewhere.

Perhaps most significantly, community records in Mukurtu remain aligned with the object-based segmentation of the world imposed by existing collections documentation, rather than providing space for alternative, more relational information structures and ways of knowing. As Michael Christie argues: “There is much stripping and splicing to be done to fit Aboriginal representations of ecological knowledge into an official archive” (62). Discrete object-based description does little to capture the complexities of stories, histories, and knowledge that weave together artifacts, places, people and other beings, events, and diverse temporalities. Writing about Indigenous thinking, Tyson Yunkaporta from the Apalech Clan says: “How might we identify and utilize the various sets of Indigenous Knowledge scattered throughout this kaleidoscope of identities? Not by simplistic categorization, that’s for sure. Through the lens of simplicity, historical contexts of interrelatedness and upheaval are sidelined, and the authenticity of Indigenous Knowledge and identity is determined by an illusion of parochial isolation, another fragment of primitive exotica to examine, tag and display” (13). More expansive, interconnected, relational approaches to collection documentation are required (Jones, 118–37), not just as a means for annotating and contesting the content of existing GLAM records at the level of the field and record, but as a first step toward opening up GLAM data, metadata structures, and infrastructures to new modes of thinking and ways of working.

Still, Mukurtu takes valuable steps in this direction. There is no requirement for consensus, nor is it considered necessary to try and resolve differences in perspective or disagreements about interpretation. Multiple voices, knowledge domains, and historicities sit alongside each other, sometimes with little duplication (let alone replication). The complexity of the layers and relationships grows with each new addition. Access control is a key part of the system, placing limits on who can contribute; but though access controls embody notions of collective rather than individual ownership, Mukurtu and the communities who use it show no desire to try and develop or replicate a singular collective voice.

Contestation and Collaboration: Humanities Network Infrastructure Project

One example of research infrastructure designed for the relational trail-making approach to humanities knowledge realization is the Humanities Networked Infrastructure (HuNI; pronounced “honey”). HuNI is a knowledge graph with eighteen million nodes, drawn from more than thirty data sources that reflect the perspectives of various disciplines across the humanities and creative arts (Verhoeven and Burrows). The data comes from libraries and museums, including Museums Victoria, the Australian National Maritime Museum, the Australian Film Institute (AFI) research collection, the library of the Australian Institute for Aboriginal and Torres Strait Islander Studies (AIATSIS), and the National Library of Australia (specifically the Trove digitized newspaper collections), as well as from research groups focused on subjects like literature (AustLit), performance (AusStage), art and design (Design and Art Australia Online, or DAAO), and endangered languages (PARADISEC). While the primary focus has been on Australian culture, a growing body of Canadian data has recently been added and annotated.

HuNI was developed by a consortium of humanities scholars in part to redress the way that collecting institutions minimized the possibilities for contestation through infrastructures with highly restricted access and an authoritative approach to information organization. It provides an environment for data collection that enables researchers to creatively reconfigure data relationships to refute hierarchies and enable alternative interpretations. In their original institutional environment, the data that are aggregated in HuNI have been heavily curated and intended to convey authority through their closely defined scope, through their controlled vocabularies, ontologies, hierarchical relationships and concepts, and even through their very data models. They present a view of the world that is intended to impose order and structure and that reflects the expertise of a particular academic or curatorial group. There is no room for the messiness and contestation that are integral to the humanities and creative arts. These databases are designed to exist as separate universes that take little, if any, notice of external sources—even similarly authoritative ones such as those belonging to other GLAM institutions.

Bringing the vocabularies and structures of these various data sources together could be an exercise in normalization, reconciliation, and conformity to a single, authoritative structure, as Europeana aims to do. HuNI, however, takes a fundamentally different approach, partly through its method of ingesting and minimally modeling the data and partly through what it enables its users to do with (and to) the data. The incoming data is mapped to a modest model, consisting of only six basic types of entity: Person, Place, Concept, Event, Organization, and Work. For example, while there are nearly eighteen million entities in HuNI, these may include multiple versions of the same person since no reconciliation or merging of entities from different sources has been imposed.

The platform’s design reflects the belief that there is value for researchers in knowing how often a record appears in different collections and how the differences and similarities between these records can yield valuable insights. Relationships between entities have not been imported from the source datasets, with one exception: the 43,000 persons, events, and organizations originating from DAAO have generic “associated with” and “participated in” relationships. As a result, HuNI is like a “connect-the-dots” puzzle, where HuNI provides the dots (in the form of records) and leaves users to draw their own connecting lines (and conclusions) by creating a picture of their own devising rather than following a predetermined pattern.

The key feature of HuNI is that users (essentially anyone with a social media or Australian Access Federation account; see AAF, “AAF 10”) can create their own links between entities. To create links, they can reuse existing expressions of a relationship (since HuNI offers these as prompts) or they can invent their own. Since multiple relationships can be made between the same two entities, a HuNI user can add a different relationship that contests an existing one—or simply expresses a different perspective or interpretation.

HuNI users can also group entities into public or private collections based on whatever categorizing or organizing principle they choose and share the collections publicly. There are almost 200 public collections in HuNI, ranging from scholarly subjects (“Literary Works About Bushfires”) to topics associated with popular culture (“Women Being Haunted by Things” or “Dead Guys Named David”). Users can also add their own entity records through comma-separated value (CSV) uploads. Recent uploads have included more than 1,100 entities from spreadsheets constructed by Canadian students in DH, women’s and gender studies, and library and information science. Through working with HuNI, these students learn how data and power are mutually implicated, especially when data is integrated, exchanged, and interoperated.

HuNI’s multiplicity of relationships and categorizations does create complexity and messiness. The graphlike visualizations of relationships between entities can quickly blow up into immensely complex networks with many interconnections. But this reflects the complexity and multiplicity of meanings inherent in the humanities and the social worlds that are their focus. At the same time, by noting the source of each entity, every relationship link, and every collection, HuNI ensures that users are accountable and responsible for their own interpretations and contributions. It is not a reproducible experiment in the scientific sense, but its infrastructure is designed to allow users to re-traverse the associations made by other users and replicate their connections, while at the same time contesting and reinterpreting them.

Over to You

Insistence on the value of replication and reproducibility centers on the success of elevating the purity of process over the vagaries of people. The idea that data and methods exist outside the self is anathema to humanities methods. In the humanities, social and ethical questions are typically paramount. Who has the authority to create public datasets, to control vocabularies and ontologies, and to define which data relationships are valid? What and, most important, who have been eliminated in the singular, universal logics of digital information infrastructure? A humanities-centered view of data acknowledges that information is woven in a relationship to the curator, creator, and keeper. Data sovereignty initiatives suggest that we should also think of information as “belonging to” its subjects. Mukurtu and HuNI are digital information platforms that recognize and attempt to redress the solipsistic curation of knowledge and understanding embedded in collections built on supposedly neutral “authoritative” vocabularies, ontologies, and relationships. By creating spaces in which existing meanings can be challenged, and new or alternative meanings and truths constructed, they offer a powerful opportunity to explore new processes of validation in the humanities and sciences alike by embracing complexity, multiplicity, and contestation.

Finally, we would be remiss if we did not acknowledge that these proposed approaches to contestation are themselves necessarily incomplete and partial in the sense of always being positioned, of being or having a-side. Contestation is critically a form of collaborative and social knowledge-making, of elaborating and adding nuance to existing knowledge with humility, a way of appreciating the manifold ways in which acts of revision, resistance, and resilience can be creative rather than simply reproductive.

Notes

1. 1. https://huni.net.au/#/search.

2. 2. https://mukurtu.org.

3. 3. https://www.go-fair.org/fair-principles/.

4. 4. https://allea.org.

5. 5. https://plateauportal.libraries.wsu.edu.

Bibliography