Open Peer Review: New Rule

New rule! From this moment forward, in anything claiming to be a “discussion” of open peer review, no one is allowed to refer to the Nature experiment as evidence that open review can’t work, at least not unless you simultaneously demonstrate (a) that you’re aware of at least one experiment in which it worked quite well (hey, wait; the results were even reproducible!) and (b) that you’ve read at least one text that asks a question or two about the Nature experiment’s presuppositions, and thus its scientific merit. We can call this the Fitzpatrick variant of Godwin’s Law; once Nature gets trotted out, it’s evident that you’re not interested in a real discussion.

That is all.

Today’s (Apparently) the Day

According to Amazon, at least: today is the day that Planned Obsolescence has been released!

The link above is to the paperback version; here’s a link to the Kindle edition. And it’s the existence of the Kindle edition that makes the whole “release day” thing so amusing (to me, at least).

I got my first copy of the paperback about four weeks ago now. Granted, I got it the very day it arrived at the press, and it does take a couple of weeks for Amazon to get its first shipment, to get that shipment into its system, and to get ready to start shipping them out. So while I kinda expected them to update the book’s page from listing a November 1 release date to reflect its in-stock-ness before now, I wasn’t terribly surprised that it didn’t happen.

But… the Kindle edition. Has also been there, more or less ready to go. Folks even pre-ordered it! And apparently are receiving it today. Virtual copies of my book are zipping out across the WhisperNet, arriving like presents in people’s digital libraries.

Which is completely awesome, of course, but it does make me wonder: November 1 was not intended by NYU Press to be an official laydown date; there was no publisher-enforced embargo on sales, or reviews, or anything else before that. And given that one of the virtues of the Kindle is that you can have that text right now, why hold it back? Why turn what was meant to be an estimate by the press into an official release date?

It makes today pretty nifty for me personally, but I’m wondering whether it makes any kind of sense otherwise.

Moves and Updates

The news is starting to make its way out there: I’m thrilled to announce that I’ll be joining the Modern Language Association this July as the Director of Scholarly Communication. In this role, I’ll be leading a new office that will expand upon the existing book publications program, exploring new modes of publishing and exchange in order to support the changing needs of MLA members in the twenty-first century.

Pomona College, where I’ve been a member of the faculty for the last thirteen years, has generously granted me a leave of absence in order to explore this opportunity. My feeling is that it’s an extraordinary opportunity not just for me but for the profession at large — the MLA, which has been a leader in addressing the changes facing the academy today, will be thinking head-on about the future of scholarly publishing, supporting member initiatives and experiments and helping to shape the vibrant and creative ways that scholars will be communicating in the coming years.

There’s lots to be done before I get started — not least, moving to New York early this summer — but I’ll look forward to exploring the possibilities for this new office and its programs as we go forward.

The Never-Appeared

I’m thinking that I’m going to start a new publishing project around here, based around a cluster of essays that I’ve written for various collections that have never actually gotten published — because the editor lost interest in the project, or because the publisher dropped the book, or because of some situation that was never really explained. I’ve got at least three of these essays, things that I really rather like but that don’t fall into the main pathway of my current work, and so things for which I’ve never been quite motivated enough to seek a new home.

Those never-appeared essays sort of rankle; they’re imperfect and sometimes out of date, but they should at least see the light of day. Perhaps when I’ve cleared the last few pending writing projects off my plate, I’ll go ahead and publish them.

On Open Access Publishing

[The following article was originally published by the Society for Critical Exchange in January 2010; alas, that version has been overrun with spam comments, making further discussion of or linking to it unlikely. I’m thus republishing it here, in the interest of having a copy that’s viable into the future.]

Raising the idea of “open access publishing” among contemporary scholars produces an immediate and sometimes surprising set of responses, ranging from enthusiasm to anger to befuddlement. The open access movement has a wide range of proponents and an often entrenched opposition, and the depth of feeling on both sides often leaves those scholars in between scratching their heads, wondering exactly what the deal is.

A huge part of the confusion arises from the proliferation of misinformation and mythology around the notion of open access; opponents of open access alternately argue that making all scholarship available for free will destroy the economic model of the publishing industry, making it impossible for anything to get published, and that doing so will simultaneously undermine peer review, turning all scholarship into vanity publishing, allowing just anything to get published. Neither of these things is true; open access publishing does not necessarily mean making everything available free of cost, nor does it necessarily imply the absence of peer review processes. It doesn’t mean that scholars lose control of the copyright of their publications (from a certain perspective, we’ve long since given that away, but that’s a matter for another article), and it doesn’t mean that plagiarism will become more prevalent.

What does it mean, though? Why have a number of colleges and universities, including institutions as varied as MIT, the University of Kansas, Trinity University, and Oberlin College recently passed resolutions mandating the open access availability of the work of their faculty members? Why have similar initiatives failed at other institutions? And what’s actually at stake in such decisions?

Far more in depth histories and analyses of the open access movement are available — including John Willinsky’s The Access Principle and Gary Hall’s Digitize this Book!, among others — but in what follows I hope to present one reading of the issues at play in the debates around open access, and my own argument for the reasons that scholars and publishers alike should support and participate in open access publishing.[1]

The open access movement in contemporary scholarship began in large part with the sciences, as a response to the predatory practices of certain commercial journal publishers. By the early 1990s, a small number of large commercial publishers had acquired most of the top journals in many fields and had begun developing a range of profit-oriented pricing structures, including bundling together large groups of journals to which libraries are required to subscribe in order to gain access to the key journals that they actually want. Because of these practices, many less-affluent institutions in the U.S. — much less those institutions in developing nations — have become unable to afford to provide access to the most important research being done in what have come to be known as the STEM fields (Science, Technology, Engineering, and Mathematics). And, of course, scholars without official ties to a subscribing institution, including independent researchers and un- and under-employed faculty members, are often unable to access that scholarship as well.[2]

Open access publishing thus has its origins both in an economic imperative, to ensure that our institutions aren’t bankrupted by the commercial interests in the scholarly communication chain, and in an ethical imperative, to ensure that less affluent institutions and individual scholars without institutional support are able to gain access to current research. That ethical concern is heightened by the prevalence of public funds used in the development of this research, including funds provided by federal granting agencies. Grantors such as the National Institutes of Health have begun requiring scholars to publish or deposit their work in open-access venues as a condition of funding. Beyond such requirements supporting the public’s right of access to research for which it has paid, however, proponents of open access publishing also call upon scholars to consider the funding and support provided to research by their own institutions, which are then charged exorbitant subscription rates to buy back the products of the research that they have supported.

These concerns have been somewhat slower to develop in humanities-based fields than they have in the sciences, primarily because the monopolistic practices of STEM journal publishers haven’t affected humanities and social science journals to quite the same degree.


There are signs that we need to be paying attention, however; when Wiley recently acquired the rights to publish the American Anthropological Association’s journals (the association’s prior contract with the University of California Press having expired), the publisher proceeded to double the subscription fee for a number of the major journals[3] — an indication that commercial publishers do see the potential for profit in “softer” fields.

Beyond the economics of the matter, however, scholars in the humanities should of course be held to the same ethical obligations as those in the sciences; though the products of our research may not always appear to be as crucial to the health and well-being of diverse populations, our work nonetheless has potentially profound implications for popular discussions about the politics of cultural representations, about the meaning of human interactions, and so forth.

We in the humanities often resist opening our work to the broader public, fearing the consequences of such openness — and not without reason. The public at times fails to understand our work, and, because the content of the work seems as though it ought to be comprehensible (you’re just writing about books, or movies, or art, after all!), isn’t inclined to wrestle with the difficulties that our work presents; their dismissive responses[4] give us the clear sense that the public doesn’t take our work as seriously as, say, papers in high-energy physics, which few lay readers would assume their ability to comprehend without some background or training. As a result of these doubled misunderstandings, we close our work off from the public, arguing that we’re only writing for a small group of specialists anyhow. In that case, why would open access matter?

The problem, of course, is that the more we close our work away from the public, and the more we refuse to engage in dialogue with them, the more we undermine that public’s willingness to fund our research and our institutions. Closing our work away from the public, and keeping our scholarly conversations private, might protect us from public criticism, but it can’t protect us from public apathy, a condition that is, in the current economy, far more dangerous. This is not to say that such openness doesn’t bear risks, particularly for scholars working in controversial areas of research, but it is to say that only through open dialogue across the walls of the ivory tower will we have any chance of convincing the broader public, including our governmental funding bodies, of the importance of our work.

All of that having been said, it’s evident that the economics of humanities-based publishing is quite different from that in STEM fields, and many of the misapprehensions preventing the broad acceptance of open-access publishing derive from that difference.

To back up a bit: there are two primary avenues through which open-access publishing is being developed. First, what has been called the “green” road to open access, self-archiving in institutional and disciplinary repositories. Under this model of open-access publishing, which is what is covered by most of the institutional mandates referred to above, scholars agree to deposit copies of their published articles in online archives associated either with their institution’s library or with their field.

Such archives are surrounded by clouds of misinformation, however; some argue, for instance, that self-archiving mandates will prevent scholars from publishing in top-tier journals (thus endangering tenure and promotion bids). In fact, most journals permit some measure of self-archiving, whether of post-prints (the manuscript of an article as edited for print) or of pre-prints (the manuscript of an article as submitted for print).[5] And it’s arguable that scholars have the responsibility to demand that those publishers and journals that don’t, as yet, permit self-archiving change their policies.

But there are other anxieties surrounding self-archiving that demand address as well. Some scholars are concerned that material that hasn’t been subjected to peer review can be deposited in such archives, thus undermining the quality control of institutional repositories generally; others are concerned that making published material available through self-archiving will have the effect of undermining already declining journal subscription figures; still others are concerned that archives needlessly complicate citational practices, by providing multiple avenues of access to published work. None of these concerns are borne out by the facts, however. Material in institutional repositories can and should be labeled as “pre-print” or “post-print,” thus giving a clear indication of its status with respect to peer review and, where possible, directing the interested reader to the final, published version of the text, with its appropriate citation. Such links to journals, and the discoverability of material published in them via institutional repositories, may in fact help promote purchases of articles, issues, or subscriptions from publishers once desirable content has been found. And projects such as the Open Archives Initiative’s OAI-PMH protocol for harvesting the metadata provided by institutional and disciplinary repositories is increasingly making such archives interoperable, and their contents more easily discoverable.

And let this be said clearly: increasing the discoverability of one’s work on the web, making it available to a broader readership, is a Good Thing, not just for the individual scholar but for the field in which she works. The more that well-researched, thoughtful scholarship on contemporary cultural issues is available to, for instance, journalists covering those issues for popular venues, the richer the discourse in those publications will become — increasing, not incidentally, the visibility of institutions of higher education, and their importance to the culture at large.

Beyond self-archiving, however, lies what has been referred to as the “gold” road to open access: journals that are published online in a freely-accessible form. Such journals are surrounded by similar forms of misinformation — most notably, that they aren’t peer reviewed, and because they are made available for free, they must therefore be intellectually valueless, both of which assumptions are patently untrue — but the greatest concern that they raise for scholars and publishers is their economic model. After all, publishing still costs money, and if the journal’s subscribers aren’t financing it, who is?

Part of the reason for such concern has been the visibility of the Public Library of Science project; this non-profit open access publishing project launched its first journal, PLoS Biology, in October 2003, followed by several more such journals, all of which employ rigorous peer review and have developed high rankings in terms of selectivity and scholarly impact. However, the funding model for these journals, as for many other open-access journals in STEM fields, is author-pays, which is to say that authors are charged at times hefty page fees in order to publish their articles. PLoS Biology, in fact, charges $2900 in page fees to an author whose work is selected for inclusion in the journal.

Such a model works in the sciences, in large part because page fees have long been a part of the culture; scientists have for quite some time written publication costs into their funding proposals, and funders have agreed that the cost of publication should be funded as part of the cost of doing research. Transplanting such a model to the humanities will simply never work, as the vast majority of research in these fields is either self-funded or funded, directly or indirectly, by the scholar’s home institution; moreover, grants coming from agencies supporting humanities research are generally so small that there’s no room available for publishing costs.

This is only a problem, however, if “author-pays” is the only viable business model for open access journals — and it’s simply not. Many journals in the humanities have published in a free and open fashion since the early days of the web; the electronic book review, for instance, was founded in 1994, and has been in continuous, open publication since. Kairos, likewise, has been in open, online publication since 1996. And Open Humanities Press publishes a range of open-access, peer-reviewed journals online.[6] Journals such as these generally operate on very limited budgets, cobbling together a range of kinds of support, including grants from funding bodies and staff/in-kind support from the journal’s host institution. But much of the support that such journals rely upon is volunteer labor — unpaid editors and reviewers, volunteer designers and coders, and so forth.

This situation isn’t all that different from more traditional, publisher-based models of journal production; whether the end result is distributed by commercial or university presses, the support that those entities provide to a journal’s editors is generally slim at best. Economist Theodore C. Bergstrom argued this point in his 2001 paper, “Free Labor for Costly Journals?,” advocating that scholars refuse to publish in overpriced commercial journals. I, however, want to espouse a more radical position, and argue for what strikes me as the most important reason for scholars to espouse open-access publishing: reclaiming the value of our labor for the profession itself. I want to suggest, as I conclude this essay, that it isn’t just ethically incumbent on us as scholars to publish in open-access venues, but in fact to create more open-access publications, and more systems for their support. These systems might include new public or foundation-based granting agency programs specifically designed to support open-access publications. They might include more consortial agreements among universities to create and support open-access publications. And they might include the development of new tools to assist in the labor that goes into journal production, such as the Public Knowledge Project’s open-source project, Open Journal Systems, which helps to create a workflow that reduces a journal editor’s reliance on technical personnel and expensive web production.[7]

But the key point is that we need to take back our publications from the market-based economy, and to reorient scholarly communication within the gift economy that best enables our work to thrive. We are, after all, already doing the labor for free — the labor of research, the labor of writing, the labor of editing — as a means of contributing to the advancement of the collective knowledge in our fields. We should value our labor sufficiently to ensure that we, our institutions, our colleagues, and our students, have full and perpetual access to the results of our work — and promoting the development of open-access publishing venues, and contributing all of our work to them, are the best ways to meet that ethical imperative toward the widest possible distribution of the knowledge that we produce.

[1] I focus in what follows on journal and journal-like publishing, largely because of its role in the origins of the open access movement, but discussion of and projects supporting open-access book publishing are on the rise; see, for instance, the work of the National Academies Press, Rice University Press, and Open Humanities Press, and projects such as the Open Monograph Press, among many others.

[2] Contrary to some assumptions, the interlibrary loan (ILL) service of a smaller institutional or public library is not an adequate substitute for such direct access; though faculty at larger institutions rarely see the cost directly, ILL is not a free service, and many institutions are required to pass those charges on to the user.

[3] See Kelty, Christopher M. et al. “Anthropology of/in Circulation: The Future of Open Access and Scholarly Societies.” Cultural Anthropology 23.3 (2008): 559-588.

[4] For evidence of such dismissive responses, one might see the annual stories about those wacky papers being presented at the MLA.

[5] The SHERPA/RoMEO project maintains an extensive list of the copyright and self-archiving policies of publishers.

[6] An extensive list of open access journals may be found at the Directory of Open Access Journals project.

[7] A list of journals using Open Journal Systems is available on the PKP website.

On the Scholarly Press, the Manual of Style, and Intellectual Property

Stuart Shieber posted an interesting and troubling analysis a few days ago of the recommendations of the Chicago Manual of Style with respect to open access publishing. The upshot of these recommendations appears to be “fight it,” or at least “limit the threat it poses to publishers’ ownership of the materials of scholarship.” As Shieber points out, there’s no small irony in the fact that

the book is owned by a university (The University of Chicago, as stated in three copyright notices on each page) filled with faculty and students whose interests are not best served by this kind of short-term profit-maximizing attitude.

And yet, there’s the problem: while The University of Chicago claims ownership of the Chicago Manual of Style, that ownership comes through the intermediary of the University of Chicago Press. And the press, like nearly all US-based university presses — which is to say that I’m not particularly picking on Chicago here; this could have happened at any such university press that happened to be the publisher of such an influential style guide — isn’t part of the university, except in a most nominal sense. The existence of the press is meant to confer a kind of prestige on the university, but, as I discuss in chapter 5 of Planned Obsolescence, the trend over the course of the second half of the twentieth century and the first decade of the twenty-first has been to so severely attenuate the relationship between the press and the institution that, for all intents and purposes, most presses are now independently operating non-profit corporations that sometimes happen to reside on university campuses.

Over the course of those decades, most university presses successfully fought off the stigma of being seen as “vanity” publishing operations by professionalizing — turning away from local authors in favor of a field-based publishing model, seeking the “best” work being produced nationally and internationally. The result was increased prestige, and increased income from sales — but that last has proven to be a double-edged sword. Because university presses are no longer seen by their institutions as serving in-house needs, and because they now appear able to generate income from the broader academy, most such presses have had the financial support provided to them by their universities slashed, making them increasingly dependent upon commercial income and decreasingly a part of the broader university culture.

The result, as we see in the open access section of the Chicago Manual of Style, is, perhaps of necessity, a wholly commercial understanding of their function, their products, and their ownership thereof. The press’s survival might seem to depend upon it. And because of that understanding, Chicago, which one might in previous editions have understood to be addressing both authors and publishers, has here clearly announced its partisanship: it is a volume intended to serve publishers, and not the authors those publishers publish, or the universities those authors populate.

And that’s fine. I’m not here calling for a boycott of Chicago style by open access publications — if anything, Chicago could stand to learn from their example — but I do want us to look carefully at the financial implications for all of our universities of the style guide’s having staked out such a position with respect to scholarly publishing and intellectual property. And I want us to recognize that there is another way.

It’s not an easy option, to be sure: it’ll be resisted by everyone involved, from established presses to university administrations to scholars themselves. And there are lots of complexities that I haven’t fully worked out here, of course. But there are a few very basic, if massive, changes that can help get us out of this mess:

  1. Every institution that requires its faculty to publish needs to develop a scholarly publishing service. It might not necessarily be an entirely in-house, single-institution operation — it might be productive to think about consortial publishing arrangements paralleling our current library consortia — but every institution must have a publishing system of some sort.
  2. Those publishing systems must focus on publishing the work of the faculty at that institution, re-creating the connection between the publisher and the institution that has been allowed to deteriorate over the last several decades.
  3. All of the work published through these services must be released in open access form, saving our libraries from the slow death by budgetary strangulation they’ve been suffering, and making the work available to all students, scholars, and interested members of the public. The point of all this publishing, after all, is making the scholarship public; the more freely it can circulate, the better.
  4. Because of that open access imperative, the university publisher — once again genuinely a university publisher — must be fully supported by its institution. If it can find ways to recuperate some costs, perhaps through the sale of print-on-demand editions of work or through other specialized services, so be it, but the university cannot abdicate its responsibility with respect to supporting scholarly publishing, any more than it can expect the library to become self-supporting.

The first question, of course, is about the press once again coming to be seen as a vanity outfit: I can hear the cries of what about peer review? The bottom line here, of course, is that peer review is already the responsibility of scholars, though it’s currently facilitated by publishers; under a model such as this one, scholars will only be required to acknowledge and take charge of that responsibility. University publishers should of course continue to facilitate peer review, but will likely be best served by doing so openly, curating the kinds of crowd-sourced conversation that can genuinely help an author improve a text and that can give us a more detailed sense of the impact an author’s work is having on the field.

There are many, many other questions to be asked about a system like this one, some of which I take up in Planned Obsolescence, but the key point here is clear: as long as university publishing is beholden to the bottom line, it cannot serve the needs of the university community. Only in radically changing the relationship between the publisher and the institution can we set aside the misguided questions about ownership that Chicago has gotten caught up in, and instead genuinely meet the ethical imperatives of open access to knowledge that the university ought to serve.

[Update, 6.34am: edited to fix link problem.]
[Update, 5 Jan 2011, 7.04am: reverted to older version to fix WP iPhone app format hosing, and re-corrected spelling of Stuart Shieber’s name.]

Relaunching The Anxiety of Obsolescence

Back in 2006, a few months before the release of my first book, The Anxiety of Obsolescence: The American Novel in the Age of Television, I launched a small WordPress-driven site to promote it. The site contained the full introduction and first chapter of the text, plus the introductory bits of the subsequent chapters, as well as gathering blurbs and reviews and so forth.

In the four-plus years since then, that site came to seem dated: the text column was awfully narrow, the template a little constrained, and the content… well, it was border-pushing enough for its time to have as much content posted as I did, but there are better ways of going about it now, and I’d been itching to get the full text into circulation.

I had a brief chat earlier this year with the folks at Vanderbilt University Press, my publisher, asking whether they’d approve of a site redesign that would include posting the full text in CommentPress. Happily, they agreed with my reasoning: four-plus years in, sales have slowed considerably, and it’s unlikely that having the full text online will keep someone from buying a copy of the book. If anything, coming across it through Google might persuade a reader to buy the text. So sure, go for it, was their basic response.

So this morning, I’m happy to announce the relaunch of The Anxiety of Obsolescence for your full-text reading and discussion pleasure.

Enormous thanks are due to my friends at Vanderbilt University Press for allowing me to republish the text in this format. I hope that you’ll support them by purchasing a copy of the book, or by asking your library to purchase a copy. In fact, VUP is generously offering a 20% discount off your entire order if you enter the code ANXIETY when you check out.

The Future of the University Press

My friends at MPublishing have released a new issue of the Journal of Electronic Publishing, guest edited by the director of the University of Michigan Press, Phil Pochoda, and including extremely insightful essays from a number of key thinkers in contemporary scholarly publishing. Jen Howard reports on the issue for the Chronicle, thinking through the issue’s collective implications.

I’m very pleased to see that so many of the predictions and recommendations contained in the JEP issue align with my own, from the final chapter of Planned Obsolescence. It’s my expectation that there will be fewer of what we currently think of as “university presses” in the coming years, as the failure of the press-as-revenue-center model spreads, but that there will be a great increase in university publishing services, as more and more institutions realize that if they are going to require their faculty to publish, they’ve got to take responsibility for providing the means of publication. These publishing services are, true to that label, likely to focus more on a broad range of services and less on producing physical objects for sale. And they’re going to have to be supported, at the faculty level, by real innovation in thinking about how published work is evaluated, and at the administrative level, by an actual functioning budget rather than an expectation of cost recovery.

All of the essays in the JEP issue are worth attending to; I hope that faculty and administrators will do so, and will press these conversations forward.

Peer-to-Peer Review and Its Aporias

Over the course of last week, a huge number of friends and colleagues of mine posted links and notes on Twitter and around the blogosphere about Mike O’Malley’s post on The Aporetic about crowdsourcing peer review.

It probably goes without saying that I’m in great sympathy with the post overall. I’ve invested a lot of time over the last couple of years in testing open peer review, including the experiment that we conducted at MediaCommons on behalf of Shakespeare Quarterly, which has been written about extensively in both the Chronicle of Higher Education and the New York Times. And of course there was my prior experiment with the open review of my own book manuscript, whose first chapter focuses in great detail on this new model of peer review, and which has been available online for just over a year now.

It’s gratifying to see other scholars getting interested in these wacky ideas about reinventing scholarly publishing that I’ve been pushing for over the last several years. In particular, the entry of scholars who are relatively new to the digital into these discussions confirms my sense that we’re at a tipping point of sorts, in which these new modes, while still experimental, are beginning to produce enough curiosity in mainstream academic circles that they’re no longer automatically dismissed out of hand.

All that said, I do feel the need to introduce a few words of caution into these discussions, because the business of open peer review isn’t quite as straightforward as simply throwing open the gates and letting Google do its thing. O’Malley argues that Google “is in effect a gigantic peer review. Google responds to your query by analyzing how many other people — your ‘peers’ — found the same page useful.” And this is so, up to a point; Google’s Page Rank system does use inbound links to help determine the relevance of particular pages with respect to your search terms. But what relationship Page Rank bears to the category of folks you might consider your “peers” — however democratically you construct that term — needs really careful consideration. On the one hand, Google’s algorithm remains a black box to most of us; we simply don’t know enough about how its machine intelligence self-adjusts to take it on faith as a reliable measure of scholarly relevance. And on the other, the human element of Page Rank — the employment of Search Quality Raters who evaluate the relevance of search results, and whose evaluations then affect the algorithm itself — and the fact that this human element has been kept so quiet, indicates that we haven’t yet turned the entire business of search on the web over to machine intelligence, that we’re still relying on the kinds of semi-secret human ratings that peer review currently employs. [1]

To put it plainly: I am absolutely committed to breaking scholarly publishing of its dependence on gatekeeping and transforming it into a Shirkyesque publish-then-filter model. No question. But our filters can only ever be as good as our algorithms, and it’s clear that we just don’t know enough about Google’s algorithms. O’Malley acknowledges that, but I’m not sure he goes quite far enough there; the point of opening up peer review is precisely to remove it from the black box, to foreground the review process as a discussion amongst peers rather than an act of abstracted anonymous judgment.

That’s problem number 1. The second problem is that peer review as we currently practice it isn’t simply a mechanism for bringing relevant, useful work into circulation; it’s also the hook upon which all of our employment practices hang, as we in the US academy have utterly conflated peer review and credentialing. As a result, we have a tremendous amount of work to do if we’re going to open peer review up to crowd-sourcing and/or make it an even partially computational process: we must simultaneously develop credible ways of determining the results of that review and, even more importantly, ways of analyzing and communicating those results to other faculty, to administrators, and to promotion and tenure committees, such that they will understand how these new processes construct authority online. It’s clear that the open peer review processes that I’ve been working with provide far more information than does the simple binary of traditional peer review’s up-or-down vote, but how to communicate that information in a way that conventional scholars can hear and make use of is no small matter.

And the third issue, one that often goes unremarked in the excitement of imagining these new digital processes, is labor. Most journal editors will acknowledge that the hardest part of their job is reviewer-wrangling; however large their list of potential peer reviewers may be, a tiny fraction of that list does an overwhelming percentage of that work. Crowdsourcing peer review presents the potential for redistributing that labor more evenly, but it’s only potential, unless we commit ourselves to real participation in the work that open peer review will require. It’s one thing, after all, for me to throw my book manuscript open for review — a process in which I received nearly 300 comments from 44 unique commenters — but what happens when everyone with such a manuscript uses a similar system? How much time and energy are we willing to expend on reviewing, and how will we ensure that this work doesn’t end up being just as unevenly distributed as is the labor in our current systems of review?

This difficulty is highlighted by the fact that many of the folks who have written excitedly about the post on The Aporetic are mostly people who know me, who know my work, and yet who were not commenters on my manuscript. Not that they needed to be, but had they engaged with the manuscript they might have noted the similarities, and drawn relevant comparisons in their comments on this later blog post. This is the kind of collaborative connection-drawing that will need to live at the forefront of any genuinely peer-to-peer review system, not simply so that the reviews can serve as a form of recommendations engine, but in order that scholars who are working on similar terrain can find their ways to one another’s work, creating more fruitful networks for collaboration.

There are several other real questions that need to be raised about how the peer-to-peer review system that I hope to continue building will operate. For instance, how do we interpret silence in such an open process? In traditional, closed review, the only form of silence is a reviewer who fails to respond; once a reviewer takes on the work of review, she generally comments on a text in its entirety. In open review, however, and especially one structured in a form like CommentPress, which allows for very fine-grained discussion of a text section by section and paragraph by paragraph, how can one distinguish between the silence produced by the absence of problems in a particular section of a text, the silence that indicates problems so fundamental that no one wants to point them out in public, and the silence that results from the text simply having gone overlooked?

And that latter raises the further question of how we can keep such a peer-to-peer review system from replicating the old boys’ club of publishing systems of yore. However much I want to tear it down, the currently existing system of double-blind peer review was in no small part responsible for the ability of women and people of color to enter scholarly conversations in full; forcing a focus on the ideas rather than on who their author was or knew had, at that time, a profoundly inclusive result.

That blind review is now at best a fiction is apparent; that it has produced numerous flaws and corruptions is evident. It’s also clear from my work that I am no apologist for our current peer review systems.

But nonetheless: I’d hate to find us in a situation in which a community of the like-minded — the cool kids, the in-crowd, the old boys — inadvertently excludes from its consideration those who don’t fall within their sphere of reference. If, as I noted above, our computational filters can only ever be as good as our algorithms, the same is doubly so in a human filtering system: peer-to-peer review can only be as open, or as open-minded, as those who participate in it, those whose opinions will determine the reputations of the texts on which they comment and the authors to whom they link.

[1] Most of this information came to me through a conversation with Julie Meloni, who also pointed out that for a glimpse of what a purely machine-intelligence driven search engine might produce, we can look at the metadata train wreck of Google Books. For whatever reason, Google has refused to allow the metadata associated with this project to be expert-reviewed, a situation that becomes all the more puzzling when you take the Search Quality Raters into account.