Notes
Chapter 19 Subjective Functions
How Should Humanistic Research Be Quantified?
Kyle Booten
Objective Functions
Humanists, like all scholars, are the objects of machine reading. Google Scholar keeps track of the simplest bibliometric data: how many times a paper or book has been cited. A researcher’s profile on the platform will list both “h-index” and “i10-index”1—two metrics that attempt to quantify something like “sustained excellence”—while plotting as a bar chart one’s citations per year. Bibliometric quantification advances a coherent but one-dimensional (1D) understanding of scholarly activity as a game in which the goal is to accrue the most citations. While humanists tend to play this game less aggressively than scientists, who have been known to form “citation farms” to gratuitously cite each other’s work (Van Noorden and Chawla), this does not mean that they are excluded from it. Since Google Scholar and related clearinghouses are important places for academics (including humanists) to see and be seen, making citation-based metrics so highly visible exerts a social pressure, just as the quantified “likes” and “friends” of other social media platforms inherently nudge their users to look for more such entities.
It is not hard to imagine the ways that these metrics may yield subtly perverse incentives. Another field-shifting paper, you wager, will likely bump up your h-index (the “largest number h such that h publications have at least h citations”2); hence you politely decline to contribute to a dear mentor’s Festschrift. Realizing that, alas, your niche intellectual obsession is too niche, you bend—almost to the point of breaking—your latest manuscript to touch on a more popular topic. The rise of “altmetrics,” which capture references to academic work on social media platforms (Piwowar), may encourage forays into public scholarship—or spending too much time on Twitter/X cultivating a sufficiently broad and chummy network.
The term objective functions is used in applied mathematics, and more recently machine learning, to describe an outcome to be optimized. This optimization is either a matter of making the output of the function smaller (e.g., minimizing the number of miles that the Amazon delivery truck must drive) or larger (e.g., maximizing the linguistic diversity of the chatbot’s utterances [Li et al.]). Academic metrics such as citation count and h-index are also “objective functions,” in that they may become targets of behavior, and in the additional sense that they presume to measure an objective, almost Newtonian quality: scholarly impact. This concept has roots in twentieth-century military logistics, emerging to help planners focus on defining goals while paying less attention to the typical way in which things were done (Cottle): the “specific operations whose relations to the accomplishment of basic ends could only be evaluated subjectively” (Wood and Dantzig, 196). But doesn’t being a humanist have something to do with attending carefully not just to the ends but also to the means, styles, and gestures of thought? Oblique allusion, obfuscatory digression, conceptual ellipsis, bizarre puns—the humanist reserves the right to all these and more, h-index be damned.
The question of how humanistic research is quantified is important even for those scholars who pay no heed to their own or their colleagues’ personal metrics. For many of us, it would now be difficult to imagine completing any sort of serious research (or even cobbling together a syllabus) without the aid of Google Scholar. And since this platform uses citation-based metrics to rank search results, the 1D quantification of scholarship becomes a core feature of the infrastructure of humanistic thought itself. When we cite an article (or neglect to), we guide the algorithms that will help resurface some texts while leaving others obscured, in part determining what we will read and not read, think and not think.
As a way of thinking beyond the conceptual blandness of standard bibliometrics (as well as most “alt” ones), I offer in this discussion four sketches of ways that humanistic research might be quantified—ways that are more in keeping with the complexity and diversity of humanistic habits and dispositions. These subjective functions differ from most bibliometric objective functions in two ways: (1) they aspire to subjectivity rather than objectivity, encoding obviously opinionated, controversial, nonuniversal notions of what virtues scholars should manifest; and (2) they attend not to impact but to the gesture, not to the ends but to the means.3 While my sketches of these subjective functions are merely speculative, I endeavor to place them on firm technical foundations, although I acknowledge that others would no doubt have different ideas about how best to engineer them.
Subjective Functions
Matchmaker Score
Take a paper with a title such as “The Languorous Abstraction of James Schuyler and Fairfield Porter.” Whatever it ends up arguing, this paper’s most basic rhetorical effect is to yoke together two figures by implying that it indeed makes sense to analyze them together. And not just sense but interest. Some conjunctions are more unexpected than others. A Google Scholar search of the combined terms “Frantz Fanon” and “Edward Said” returns around 19,600 results; “Frantz Fanon” and “Julia Kristeva” only around 3,800; “Frantz Fanon” and “Lev Vygotsky,” fewer than 1,000. An intuition: the more infrequently two authors or texts are discussed in combination, the more potential for surprise—one facet of interestingness—it will have.
How might one calculate a Matchmaker Score that describes how unexpected are the combinations of authors or texts discussed by an academic publication? I have already suggested the basis for one metric: the number of preexisting publications that contain a particular name and another name, relative to other names. Examining actual citations, rather than mere in-text references, would be more reliable. Based on the total occurrences of item A, the total occurrences of item B, and the number of co-occurrences of items A and B, pointwise mutual information (PMI) describes the strength of the association between items A and B. Calculating PMI scores has been used to surface “serendipitous” combinations within datasets—that is, those with lower PMI values (Jenders et al.).
However, one might wish to exclude those papers that are substantially about author/text A but only tangentially refer to author/text B, or even those that discuss both but do not truly “put them into conversation.” Computationally identifying this sort of discursive entanglement is a harder problem. One approach would be to make sure that a publication contains both names a certain number of times within a span of a certain number of characters. Thus, a paper’s Matchmaker Score could be operationalized as the minimum PMI between any two authors or texts within a publication (minimum PMI here meaning something like “least typically associated”), ignoring those pairs that do not seem to be truly discussed in tandem.
Persistence Score
Is it not better to love deeply, or to worry deeply, than to flit from fixation to fixation? Although the rhythms of scholarly publication seem to be ever-hastening, the academic career still allows one to return to a problem again and again over years or even decades, perhaps never quite satisfied with any solution. To measure this sort of relentless attention to a text or question, we might define a Persistence Score. A naive approach to defining such a metric is straightforward enough: a scholar who discusses a certain idea or text—for instance, Anne Bradstreet’s poem “Contemplations”—in three papers would earn a higher Persistence Score than a peer who discusses this poem in only one paper. Yet there are important decisions to be made: What if scholar A writes about Bradstreet’s poem in five papers over seven years, and scholar B in three papers over fifteen years? Which is the more persistent? The calculation of a Persistence Score could require separate coefficients for the number of publications referencing and the number of days between the first and last references.
Simply counting repeating references would provide no way of distinguishing between the true target of this proposed metric—the scholar who really does return to Bradstreet’s poem in different seasons of life, struggling to see it afresh—from the scholar who keeps handy a clever aperçu, repeating it more or less unchanged whenever it seems fitting. Telling apart these two scholars would require something more complicated: plagiarism detection. For an algorithm to detect plagiarism, it must deal with the fact that plagiarists often make superficial changes (e.g., substituting a word for its synonym) to hide the offending act from a mere string search. The scholar who reheats their own reading of Bradstreet’s poem may indeed introduce subtle changes. A typical approach within the field of automatic plagiarism detection is to estimate the semantic distance between two chunks of text using techniques that do not depend on the chunks sharing any of the same words; these include latent semantic indexing (or latent semantic analysis) and word embeddings (Foltýnek, Meuschke, and Gipp, 19–20). These same techniques could be used not just to exclude subsequent readings of Bradstreet’s poem that are too similar to one of the author’s previous readings but also to reward those recent readings that are strikingly different from those earlier ones. That would be the sign of an author who is stretching to make sense of the familiar text from an unfamiliar vantage.
Contrariness Score
A Contrariness Score would help identify those scholars who routinely disagree with opinions that their peers laud, or at least take for granted. An essential prerequisite of such a metric is the ability to distinguish between positive (“As Malabou lucidly argues . . .”) versus negative (“What Malabou’s account misunderstands . . .”) versus neutral (“For a different opinion, see Malabou (1994) . . .”). Fortunately, researchers in natural language processing (NLP) have made significant progress in the task of sentiment analysis for citations (Yousif, 2019), and large language models (LLMs) could also be well suited to this task.4 It might make sense for the Contrariness Score to consider not just a citation’s emotional polarity and intensity but also its duration—a barbed quip encased in parentheses versus an extended critique. Intensity of sentiment about cited text and percentage of publication devoted to cited text could each have its own coefficient. A paper could be given a Contrarianism Point for railing on a text that most other scholars tend to discuss in a positive or neutral manner. A scholar’s Contrariness Score might be the total number of points accrued across all their publications.
Conceptual Simplicity Score
A common (and indeed at this point quite exhausted) stereotype of certain veins of academic writing is that it is made nearly indecipherable by the thick impasto of jargon, with terms such as the abject and le Nom-du-Père daubed on every sentence, never to be explained or returned to. Humanists might want to highlight those works that do the opposite. By opposite here, I do not mean a piece of writing that is totally free of all terms of art, but rather one that takes a single such term, or perhaps two or three, and returns to the term or terms again and again so that, one hopes, they become that much clearer. The first step toward calculating such a Conceptual Simplicity Score would be to surface those words from a publication that could fairly be deemed jargon. One might begin by looking for those that are contextually marked as academic argot:
. . . what Lacan refers to as “le Nom-du-Père” . . .
. . . reworking Kristeva’s notion of “the abject” . . .
Or one might also attempt a more statistical approach, considering as jargon those words (or brief sequences of words) that (1) appear very infrequently in a general corpus of English and (2) appear preponderantly in one academic subfield relative to academic writing in general. To achieve a high Conceptual Simplicity Score, a paper would need to use one or a few pieces of jargon multiple times throughout a text while also avoiding or minimizing the use of any other jargon. A paper’s score could be increased should it include a linguistic marker that suggests a striving for clarity:
. . . Lacan’s concept of “le Nom-du-Père,” by which he means . . .
From Sketches to Infrastructure?
Just as citation counts determine what articles surface most readily on Google Scholar, subjective functions could be integrated into infrastructures of thought that allow more sophisticated and humane ways of navigating the vast archive of scholarship. For instance:
- Confused by a piece of jargon, a student or newcomer to a subfield could search for that esoteric term and filter results based on Conceptual Simplicity Score, assuming that a high-scoring paper could serve as a not-too-steep on-ramp.
- Constructing an introductory syllabus on a given topic, one might search for relevant papers that possess both very low and very high Contrariness Scores.
- A senior scholar might prioritize new publications with high Matchmaker Scores, thinking that these are the most likely to cast an unfamiliar light on a familiar field.
But subjective functions, if embedded into scholarly infrastructure, would also help scholars to better understand their own styles of thinking and to better represent these styles to their peers (as well as to administrators and other evaluators). Subjective functions might be translated into badgelike visualizations that make different scholarly dispositions easily apperceptible. What follows are a few sketches of such visualizations.
Aposematic Porcupine
Figures 19.1 and 19.2 present a visualization of a scholar’s Contrarianism Points. Each quill represents one such point, with the length of each representing the intensity of the critique.5 A widely praised text with which the scholar has frequently taken issue can be assigned the same color—or value, in the case of grayscale.6 Compared to Figure 19.1, Figure 19.2 indicates fewer Contrarianism Points, each earned by attacking a different target. The shorter quills suggest that these critiques are less intense.
Figure 19.1. An “aposematic porcupine” illustrating contrarianism.
Figure 19.2. Another porcupine, this one less aggressive.
Time Locket
Figures 19.3 and 19.4 offer a simple way of depicting the Persistence Score (or, at least, some version of it). Time is represented as a circle and publications as points on that circle. A shape is created by connecting adjacent points with chords; this shape gives a rough but immediate sense of how much of the scholar’s career has been spent turning and returning to the text in question. In this case, the area of the shape within Figure 19.3 is larger than the one within Figure 19.4, indicating greater persistence.7
Figure 19.3. A “time locket” visualization of an author’s references to a text over time. The author’s most recent reference has been selected and the title of the text revealed.
Figure Description
A circle with six dots placed around the circumference at irregular intervals. Each dot represents a reference to a particular text described above the circle (Anne Bradstreet’s “The Author to Her Book”). These dots represent years of publication along a circular timeline, starting at the year 2008. The first and last dots are connected by lines to the center of the circle; otherwise, lines have been drawn between adjacent dots. Together these lines form a polygon shape that has been colored blue, taking up most of the area of the circle. The last dot is marked “2021,” and this dot is highlighted and labeled with the name of the publication that refers to Bradstreet’s poem, called “Bradstreet’s ‘Rambling Brats’: Prosody as Parental Discipline.”
Figure 19.4. Another locket.
Figure Description
A circle with three dots placed around the circumference at irregular intervals. Each dot represents a reference to a particular text described above the circle (Langston Hughes’ “Blues in Stereo”). These dots represent years of publication along a circular timeline, starting at the year 2008. The dots are fairly close together, and the last one is labeled “2015.” Lines connect the three dots to form a red triangle, taking up less than a third of the circle’s total area.
Gradient Goblet
Figures 19.5 and 19.6 illustrate the Matchmaker Score. Each of a scholar’s top ten papers in terms of this score is depicted as a horizontal line between two black squares; higher scores are reflected by longer lines. These lines are placed in descending order over a field with a horizontal gradient so that color emphasizes the distance or nearness between two points. Two very distant points will cross from bright red to bright green (or, in grayscale representations, from dark to light gray). Figure 19.6, a slender vase (cast mostly in tepid ochre or, sans color, middling pewter), would represent the publications of a scholar who is not much of a matchmaker.
Figure 19.5. A “Gradient Goblet” for Matchmaker Scores. The third-highest score has been highlighted and its authors listed, flanking the goblet.
Figure Description
A wide symmetrical goblet shape. Along each side of the goblet are 10 small squares, each representing an author. For instance, a square on the right is labeled “Anne Bradstreet” and a square on the left is labeled “Lev Vygotsky.” The pairs of squares (one on the left, one on the right) are different distances apart (representing their Matchmaker Scores) and in descending order in terms of distance.That the goblet is wide—i.e., that the distance between the squares on the left and the squares on the right, until the last two pairs (the “bottom” of the goblet), is large—indicates that the Matchmaker Scores between the pairs of authors is high. This is visually indicated by a color gradient that goes from right to left; the left edge and right edge of the goblet, especially at the top (where the distances are greatest) have very different hues, while toward the center of the goblet the colors are more murky (a blend between the two poles of the gradient).
Figure 19.6. A slenderer goblet.
Figure Description
A wide symmetrical goblet shape. Along each side of the goblet are 10 small squares, each representing an author. The pairs of squares (one on the left, one on the right) are different distances apart (representing their Matchmaker Scores) and in descending order in terms of distance. That the goblet is generally narrow indicates that the Matchmaker Scores between the pairs of authors is generally low. This is visually indicated by a color gradient that goes from right to left; the left edge and right edge; the distinct hues of this gradient can only be seen at the very top of the goblet (where the distances are the greatest), but most of the goblet is a murky hue from between the two poles of the gradient.
Integrated into a Google Scholar–like platform, these sorts of miniature visualizations could be useful for comparing scholars within a field and noticing the specific charisms of those who may not rank as highly according to traditional metrics (Figure 19.7).
Figure 19.7. Comparing scholars beyond typical metrics. In the two rightmost columns, each scholar has selected one or two optional visualizations to represent themselves. (Image inspired by the way scholars are compared on Research.com. The two scholars here are fabricated, and their images are generated by artificial intelligence [AI].)
Figure Description
A dashboard featuring thumbnail photos of two scholars and organized in rows and columns. Each row is for a different scholar. The columns from left to right hold a scholar’s citation rank, photo, name with institutional affiliation, number of citations, H-Index score, and badges as described above. The fictional scholar in the top row (“Keely LaPrince,” Northwestern University, Citation Rank of 12, 4,002 total citations, and an H-Index of 20) has only one badge, the “locket” from Figure 19.3, indicating sustained attention to an author. The second hypothetical scholar (“Paulo V. Stefano,” University of Coimbra, Citation Rank of 30, 523 total citations, and an H-Index of 6) has two badges, one in each of the rightmost columns. The first is the “goblet” from Figure 19.5, indicating high Matchmaker Scores. The second is the “porcupine” from Figure 19.1, indicating high contrarianism.
Metrics very easily become goals, and so even subjective functions would lead to the pursuit of “subjective objectives.” Unlike h-index and other impact-based metrics, subjective functions could also support multiple, conflicting behaviors: a self-styled contrarian might become even more contrary, while another person might experience their high Contrariness Score as a discomfiting glance into the mirror. Like any metric, subjective functions might still feel alienating and stifling—especially if used clumsily by institutions to evaluate faculty or to allocate resources (see Strathern).8 Likewise, it would be important to attend to the aspects of humanistic thought that metrics—any metrics—would be powerless to capture.9 Still, just as machine reading can reveal surprising facts about a corpus of novels without exhausting their meaning, personalized subjective metrics could serve as starting points for understanding one’s own oeuvre. More strategically, subjective functions could come in handy should a researcher, a department, a field, or even a university need to push back against even more ill-fitting metrics.
We Need It All
To compute these or other subjective functions, one would want to have access to nothing less than the full texts of as many terabytes of scholarship as possible, yet efforts to rethink humanistic metrics suffer from “a lack of machine-readable data with permissive licensing” (Konkiel, 5). In theory, a researcher with a sufficiently capacious hard drive (and a sufficiently relaxed approach to questions of copyright) could begin by downloading a cache of papers and books from Library Genesis (LibGen) or another of the “shadow libraries” discussed by Martin Paul Eve in chapter 6 of this book. But running calculations locally is one thing, building infrastructure quite another. Humanists would need to make these metrics part of a public system of information retrieval, an alternative Google Scholar where one could search and be searched, see and be seen, according to a toggleable menu of diverse metrics.
The subjective functions sketched in this chapter represent only four points in a large and underexplored design space. None of the four attends to questions of scholarly ethics, such as “citational justice” (Kwon). None directly attempts to measure anything like erudition. Scholars should be able to design their own functions to represent their own values, and others should be able to discuss and critique these functions. Imagining, arguing about, and building this extensible algorithmic infrastructure-for-thought should be a core metaproject of the humanities. The alternative is to remain “objectified.”
Notes
1. As a tooltip on any scholar’s Google Scholar profile explains, this is “the number of publications with at least 10 citations.”
2. This is another definition provided via tooltip on a scholar’s profile; see Hirsch for the original definition of and rationale for this metric.
3. I am participating here in conversations about how metrics can be reimagined to better fit humanistic research. For instance, the HuMetricsHSS initiative has begun to imagine metrics that reflect the “shared values” of the humanities (Konkiel; see also Long). My goal is to consider how metrics could reflect values that may be largely unshared, if not totally idiosyncratic.
4. Zhang et al. found that LLMs (with some exceptions) performed competently on a variety of sentiment analysis tasks, though not always as well as smaller models trained specifically for these tasks.
5. This design is inspired by the Altmetric Attention Score “donut,” which represents an individual published work’s references on social media, the news, and elsewhere (Altmetric).
6. All badges were originally designed with color in mind. Where possible, these images have been adapted into grayscale for print.
7. In Figure 19.3, the first and last points representing references connect not to each other but to a central point; otherwise (if the first and last points were connected), the resulting shape would cover time prior to the first reference.
8. A funding body might bully a department or researcher into striving for a lower (or higher) Contrariness Score, for instance, regardless of whether this goal is at all meaningful from an emic/insider’s perspective. On the other hand, subjective functions would be easier to hack than traditional metrics; a disingenuous scholar could larder their books with half-hearted ripostes.
9. As Eileen A. Joy (26) asked provocatively in a manifesto against metrics, “How do you measure the time spent not doing anything at all in order to open a space for thinking differently?”
Bibliography
- Altmetric. “The Donut and Altmetric Attention Score.” Accessed September 20, 2023. https://www.altmetric.com/about-us/our-data/donut-and-altmetric-attention-score/.
- Cottle, Richard W. “Objective Function.” In International Encyclopedia of the Social Sciences, 2nd Edition, vol. 16, edited by William A. Darity Jr. (2008): 5–6.
- Foltýnek, Tomáš, Norman Meuschke, and Bela Gipp. “Academic Plagiarism Detection: A Systematic Literature Review.” ACM Computing Surveys, 52, no. 6 (2019): 1–42.
- “Google Scholar Metrics.” Google Scholar. Accessed August 20, 2022, https://scholar.google.com/intl/en/scholar/metrics.html#metrics.
- Hirsch, Jorge E. “An Index to Quantify an Individual’s Scientific Research Output.” Proceedings of the National Academy of Sciences 102, no. 46 (2005): 16569–72.
- Jenders, Maximilian, T. Lindhauer, Gjergji Kasneci, et al. “A Serendipity Model for News Recommendation.” In Joint German/Austrian Conference on Artificial Intelligence (Künstliche Intelligenz), 111–23. Springer, 2015.
- Joy, Eileen A. “‘An Instrument for Adoration’: A Mini-Manifesto Against Metrics for the Humanities (to Be Elaborated Upon at a Further Date).” In Humane Metrics/Metrics Noir, edited by meson press, 26–33. Post Office Press, Rope Press, and meson press.
- Konkiel, Stacy. “Approaches to Creating ‘Humane’ Research Evaluation Metrics for the Humanities.” Insights 31 (2018).
- Kwon, Diana. “The Rise of Citational Justice: How Scholars Are Making References Fairer.” Nature. March 22, 2022. https://www.nature.com/articles/d41586-022-00793-1.
- Li, Jiwei, Michel Galley, Chris Brockett, et al. “A Diversity-Promoting Objective Function for Neural Conversation Models.” In Proceedings of North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 110–19. ACL, 2016.
- Long, Christopher P. “Toxicity, Metrics, and Academic Life.” In Humane Metrics/Metrics Noir, edited by meson press, 14–25. Post Office Press, Rope Press, and meson press.
- Piwowar, Heather. “Introduction: Altmetrics: What, Why and Where?” Bulletin of the American Society for Information Science and Technology 39, no. 4 (2013): 8–9.
- Strathern, Marilyn. “‘Improving Ratings’; Audit in the British University System.” European Review, 5, no. 3 (1997), 305–21.
- Van Noorden, Richard, and Dalmeet Singh Chawla. “Hundreds of Extreme Self-Citing Scientists Revealed in New Database.” Nature. August 19, 2019. https://www.nature.com/articles/d41586-019-02479-7.
- Wood, Marshall K., and George B. Dantzig. “Programming of Interdependent Activities: I: General Discussion.” Econometrica, Journal of the Econometric Society (1949): 193–99.
- Yousif, Abdallah, Zhendong Niu, John K. Tarus, et al. “A Survey on Sentiment Analysis of Scientific Citations.” Artificial Intelligence Review, 52, no. 3 (2019): 1805–38.
- Zhang, Wenxuan, Yue Deng, Bing Liu, et al. “Sentiment Analysis in the Era of Language Models: A Reality Check.” May 24, 2023. https://arxiv.org/abs/2305.15005v1 (preprint).