Notes from this morning’s first session follow. Any misrepresentations herein are solely the fault of the note taker.
Dan Greenstein, Vice Provost, University of California
“New Directions, Different Possibilities”
I’m in Oakland for the day today, at a thoroughly exciting meeting: “New Structures, New Texts: A Summit on the Library and the Press as Partners in the Enterprise of Scholarly Publishing.” I’ll hope to post my notes either during the day today or in the coming days, as I process what’s said.
Major editorial project due June 1. Approximately 50% of work on project remains ahead of me. I hate deadlines.
About six months ago, I published a lengthy post, both on Planned Obsolescence and on if:book, about the future of peer review in electronic scholarly publishing. At least some portion of that post was occasioned by Nature’s experiment with an open peer-review system. That experiment was closed earlier this month, and the editors have now analyzed the data resulting from it, and have declared the experiment to have failed, and have announced that “for now at least, we will not implement open peer review.”
The statistics that they cite are indeed indicative of some serious issues in the open system they implemented: only 5% of authors who submitted work during the trial agreed to have their papers opened to public comment; only 54% of those papers (or a 38 of a total of 71) received substantive comments. But I have to wonder whether the experiment wasn’t rigged from the beginning, destined for a predictable failure because of the trial’s constraints.
First, no real impetus was created for authors to open their papers to public review; as I noted back in June, the open portion of the peer review process was wholly optional, and had no bearing whatsoever on the editors’ decision to publish any given paper. And second, no incentive was created for commenters to participate in the process; why go to all the effort of reading and commenting on a paper if your comments serve no identifiable purpose?
What I want to ask at this point is what MediaCommons can learn from the ostensible failures of Nature’s experiment. How can we develop a successful open peer-review process with adequate author and reviewer buy-in?
Right before I left for Paris and Vienna, I did an email interview with a writer from Media Life magazine who was working on an article about The Anxiety of Obsolescence
Given how little of my rambling made it into the article, though, I thought I’d post the entirety of the interview, for my own future reference, if nothing else.
And it’s really good advice, too: how to write an academic book that folks might actually want to read.
Though she doesn’t quite say so, many of the points she makes—particularly that, with respect to outside sources, “[e]ither they prove you right or you prove them wrong or they don’t get to be in your book,” and, with respect to audience, that you shouldn’t “write for the four people who know more about your subject than you do. They aren’t going to buy your book. Write for the five thousand people who are smart and curious but don’t know as much about your subject as they should”—are precisely the differences between a dissertation and a book.
The first trick, I think, in revising your dissertation into a book is to take all of that stuff—the outside sources who are there only to prove that you’ve consulted them; the references designed to demonstrate to the four people who are reading your dissertation that you really do know your field—and stick it in the footnotes. It’s not part of the flow of your argument, so into the backmatter it goes.
The second trick, however, is to eliminate the half of your footnotes that really aren’t that important, after all.
Are there other tricks and tips you’d care to share?
I’m still running pretty much a day behind—meant to post this yesterday, but never got to it. In any event, and in a hurry:
The Chronicle reported yesterday that the ACLS had released a report, “Our Cultural Commonwealth,” examining the state of “cyberinfrastructure” in the humanities and social sciences, arguing—unsurprisingly, perhaps—that these “softer” areas of the academy have a long way to go in order to catch up with the levels of development and support available to the hard sciences. Among their recommendations is one near and dear to my heart: “Encourage digital scholarship.”
Today is also the final day in the summer institute on Cyberinfrastructure for Humanities, Arts and Social Sciences at UC San Diego, sponsored by (among other organizations), HASTAC.
I’m very much hoping to hear what comes out of that institute, and looking forward to seeing how the ACLS’s report is received…
Hey, this is cool: the Amazon page for The Anxiety of Obsolescence
(cross-posted from if:book)
Over the last several months, as I’ve met with the folks from if:book and with the quite impressive group of academics we pulled together to discuss the possibility of starting an all-electronic scholarly press, I’ve spent an awful lot of time thinking and talking about peer review—how it currently functions, why we need it, and how it might be improved. Peer review is extremely important—I want to acknowledge that right up front—but it threatens to become the axle around which all conversations about the future of publishing get wrapped, like Isadora Duncan’s scarf, strangling any possible innovations in scholarly communication before they can get launched. In order to move forward with any kind of innovative publishing process, we must solve the peer review problem, but in order to do so, we first have to separate the structure of peer review from the purposes it serves—and we need to be a bit brutally honest with ourselves about those purposes, distinguishing between those purposes we’d ideally like peer review to serve and those functions it actually winds up fulfilling.
The issue of peer review has of course been brought back to the front of my consciousness by the experiment with open peer review currently being undertaken by the journal Nature, as well as by the debate about the future of peer review that the journal is currently hosting (both introduced last week on if:book). The experiment is fairly simple: the editors of Nature have created an online open review system that will run parallel to its traditional anonymous review process.
From 5 June 2006, authors may opt to have their submitted manuscripts posted publicly for comment.
Any scientist may then post comments, provided they identify themselves. Once the usual confidential peer review process is complete, the public ‘open peer review’ process will be closed. Editors will then read all comments on the manuscript and invite authors to respond. At the end of the process, as part of the trial, editors will assess the value of the public comments.
As several entries in the web debate that is running alongside this trial make clear, though, this is not exactly a groundbreaking model; the editors of several other scientific journals that already use open review systems to varying extents have posted brief comments about their processes. Electronic Transactions in Artificial Intelligence, for instance, has a two-stage process, a three-month open review stage, followed by a speedy up-or-down refereeing stage (with some time for revisions, if desired, inbetween). This process, the editors acknowledge, has produced some complications in the notion of “publication,” as the texts in the open review stage are already freely available online; in some sense, the journal itself has become a vehicle for re-publishing selected articles.
Peer review is, by this model, designed to serve two different purposes—first, fostering discussion and feedback amongst scholars, with the aim of strengthening the work that they produce; second, filtering that work for quality, such that only the best is selected for final “publication.” ETAI’s dual-stage process makes this bifurcation in the purpose of peer review clear, and manages to serve both functions well. Moreover, by foregrounding the open stage of peer review—by considering an article “published” during the three months of its open review, but then only “refereed” once anonymous scientists have held their up-or-down vote, a vote that comes only after the article has been read, discussed, and revised—this kind of process seems to return the center of gravity in peer review to communication amongst peers.
I wonder, then, about the relatively conservative move that Nature has made with its open peer review trial. First, the journal is at great pains to reassure authors and readers that traditional, anonymous peer review will still take place alongside open discussion. Beyond this, however, there seems to be a relative lack of communication between those two forms of review: open review will take place at the same time as anonymous review, rather than as a preliminary phase, preventing authors from putting the public comments they receive to use in revision; and while the editors will “read” all such public comments, it appears that only the anonymous reviews will be considered in determining whether any given article is published. Is this caution about open review an attempt to avoid throwing out the baby of quality control with the bathwater of anonymity? In fact, the editors of Atmospheric Chemistry and Physics present evidence (based on their two-stage review process) that open review significantly increases the quality of articles a journal publishes:
Our statistics confirm that collaborative peer review facilitates and enhances quality assurance. The journal has a relatively low overall rejection rate of less than 20%, but only three years after its launch the ISI journal impact factor ranked Atmospheric Chemistry and Physics twelfth out of 169 journals in ‘Meteorology and Atmospheric Sciences’ and ‘Environmental Sciences’.
These numbers support the idea that public peer review and interactive discussion deter authors from submitting low-quality manuscripts, and thus relieve editors and reviewers from spending too much time on deficient submissions.
By keeping anonymous review and open review separate, without allowing the open any precedence, Nature is allowing itself to avoid asking any risky questions about the purposes of its process, and is perhaps inadvertently maintaining the focus on peer review’s gatekeeping function. The result of such a focus is that scholars are less able to learn from the review process, less able to put comments on their work to use, and less able to respond to those comments in kind.
If anonymous, closed peer review processes aren’t facilitating scholarly discourse, what purposes do they serve? Gatekeeping, as I’ve suggested, is a primary one; as almost all of the folks I’ve talked with this spring have insisted, peer review is necessary to ensuring that the work published by scholarly outlets is of sufficiently high quality, and anonymity is necessary in order to allow reviewers the freedom to say that an article should not be published. In fact, this question of anonymity is quite fraught for most of the academics with whom I’ve spoken; they have repeatedly responded with various degrees of alarm to suggestions that their review comments might in fact be more productive delivered publicly, as part of an ongoing conversation with the author, rather than as a backchannel, one-way communication mediated by an editor. Such a position may be justifiable if, again, the primary purpose of peer review is quality control, and if the process is reliably scrupulous. However, as other discussants in the Nature web debate point out, blind peer review is not a perfect process, subject as it is to all kinds of failures and abuses, ranging from flawed articles that nonetheless make it through the system to ideas that are appropriated by unethical reviewers, with all manner of cronyism and professional jealousy inbetween.
So, again, if closed peer review processes aren’t serving scholars in their need for feedback and discussion, and if they can’t be wholly relied upon for their quality-control functions, what’s left? I’d argue that the primary purpose that anonymous peer review actually serves today, at least in the humanities (and that qualifier, and everything that follows from it, opens a whole other can of worms that needs further discussion—what are the different needs with respect to peer review in the different disciplines?), is that of institutional warranting, of conveying to college and university administrations that the work their employees are doing is appropriate and well-thought-of in its field, and thus that these employees are deserving of ongoing appointments, tenure, promotions, raises, and whathaveyou.
Are these the functions that we really want peer review to serve? Vast amounts of scholars’ time is poured into the peer review process each year; wouldn’t it be better to put that time into open discussions that not only improve the individual texts under review but are also, potentially, productive of new work? Isn’t it possible that scholars would all be better served by separating the question of credentialing from the publishing process, by allowing everything through the gate, by designing a post-publication peer review process that focuses on how a scholarly text should be received rather than whether it should be out there in the first place? Would the various credentialing bodies that currently rely on peer review’s gatekeeping function be satisfied if we were to say to them, “no, anonymous reviewers did not determine whether my article was worthy of publication, but if you look at the comments that my article has received, you can see that ten of the top experts in my field had really positive, constructive things to say about it”?
Nature‘s experiment is an honorable one, and a step in the right direction. It is, however, a conservative step, one that foregrounds the institutional purposes of peer review rather than the ways that such review might be made to better serve the scholarly community. We’ve been working this spring on what we imagine to be a more progressive possibility, the scholarly press reimagined not as a disseminator of discrete electronic texts, but instead as a network that brings scholars together, allowing them to publish everything from blogs to books in formats that allow for productive connections, discussions, and discoveries. I’ll be writing more about this network soon; in the meantime, however, if we really want to energize scholarly discourse through this new mode of networked publishing, we’re going to have to design, from the ground up, a productive new peer review process, one that makes more fruitful interaction among authors and readers a primary goal.
They arrived yesterday, proving that this project, begun in 1996, is finally, one-hundred percent, completely and totally done.
Celebration commenced roughly immediately. Recovering continues today.
(And at the risk of over-advertising, remember that you can get your very own copy here.)
