Peer-to-Peer Review and Its Aporias

Over the course of last week, a huge number of friends and colleagues of mine posted links and notes on Twitter and around the blogosphere about Mike O’Malley’s post on The Aporetic about crowdsourcing peer review.

It probably goes without saying that I’m in great sympathy with the post overall. I’ve invested a lot of time over the last couple of years in testing open peer review, including the experiment that we conducted at MediaCommons on behalf of Shakespeare Quarterly, which has been written about extensively in both the Chronicle of Higher Education and the New York Times. And of course there was my prior experiment with the open review of my own book manuscript, whose first chapter focuses in great detail on this new model of peer review, and which has been available online for just over a year now.

It’s gratifying to see other scholars getting interested in these wacky ideas about reinventing scholarly publishing that I’ve been pushing for over the last several years. In particular, the entry of scholars who are relatively new to the digital into these discussions confirms my sense that we’re at a tipping point of sorts, in which these new modes, while still experimental, are beginning to produce enough curiosity in mainstream academic circles that they’re no longer automatically dismissed out of hand.

All that said, I do feel the need to introduce a few words of caution into these discussions, because the business of open peer review isn’t quite as straightforward as simply throwing open the gates and letting Google do its thing. O’Malley argues that Google “is in effect a gigantic peer review. Google responds to your query by analyzing how many other people — your ‘peers’ — found the same page useful.” And this is so, up to a point; Google’s Page Rank system does use inbound links to help determine the relevance of particular pages with respect to your search terms. But what relationship Page Rank bears to the category of folks you might consider your “peers” — however democratically you construct that term — needs really careful consideration. On the one hand, Google’s algorithm remains a black box to most of us; we simply don’t know enough about how its machine intelligence self-adjusts to take it on faith as a reliable measure of scholarly relevance. And on the other, the human element of Page Rank — the employment of Search Quality Raters who evaluate the relevance of search results, and whose evaluations then affect the algorithm itself — and the fact that this human element has been kept so quiet, indicates that we haven’t yet turned the entire business of search on the web over to machine intelligence, that we’re still relying on the kinds of semi-secret human ratings that peer review currently employs. [1]

To put it plainly: I am absolutely committed to breaking scholarly publishing of its dependence on gatekeeping and transforming it into a Shirkyesque publish-then-filter model. No question. But our filters can only ever be as good as our algorithms, and it’s clear that we just don’t know enough about Google’s algorithms. O’Malley acknowledges that, but I’m not sure he goes quite far enough there; the point of opening up peer review is precisely to remove it from the black box, to foreground the review process as a discussion amongst peers rather than an act of abstracted anonymous judgment.

That’s problem number 1. The second problem is that peer review as we currently practice it isn’t simply a mechanism for bringing relevant, useful work into circulation; it’s also the hook upon which all of our employment practices hang, as we in the US academy have utterly conflated peer review and credentialing. As a result, we have a tremendous amount of work to do if we’re going to open peer review up to crowd-sourcing and/or make it an even partially computational process: we must simultaneously develop credible ways of determining the results of that review and, even more importantly, ways of analyzing and communicating those results to other faculty, to administrators, and to promotion and tenure committees, such that they will understand how these new processes construct authority online. It’s clear that the open peer review processes that I’ve been working with provide far more information than does the simple binary of traditional peer review’s up-or-down vote, but how to communicate that information in a way that conventional scholars can hear and make use of is no small matter.

And the third issue, one that often goes unremarked in the excitement of imagining these new digital processes, is labor. Most journal editors will acknowledge that the hardest part of their job is reviewer-wrangling; however large their list of potential peer reviewers may be, a tiny fraction of that list does an overwhelming percentage of that work. Crowdsourcing peer review presents the potential for redistributing that labor more evenly, but it’s only potential, unless we commit ourselves to real participation in the work that open peer review will require. It’s one thing, after all, for me to throw my book manuscript open for review — a process in which I received nearly 300 comments from 44 unique commenters — but what happens when everyone with such a manuscript uses a similar system? How much time and energy are we willing to expend on reviewing, and how will we ensure that this work doesn’t end up being just as unevenly distributed as is the labor in our current systems of review?

This difficulty is highlighted by the fact that many of the folks who have written excitedly about the post on The Aporetic are mostly people who know me, who know my work, and yet who were not commenters on my manuscript. Not that they needed to be, but had they engaged with the manuscript they might have noted the similarities, and drawn relevant comparisons in their comments on this later blog post. This is the kind of collaborative connection-drawing that will need to live at the forefront of any genuinely peer-to-peer review system, not simply so that the reviews can serve as a form of recommendations engine, but in order that scholars who are working on similar terrain can find their ways to one another’s work, creating more fruitful networks for collaboration.

There are several other real questions that need to be raised about how the peer-to-peer review system that I hope to continue building will operate. For instance, how do we interpret silence in such an open process? In traditional, closed review, the only form of silence is a reviewer who fails to respond; once a reviewer takes on the work of review, she generally comments on a text in its entirety. In open review, however, and especially one structured in a form like CommentPress, which allows for very fine-grained discussion of a text section by section and paragraph by paragraph, how can one distinguish between the silence produced by the absence of problems in a particular section of a text, the silence that indicates problems so fundamental that no one wants to point them out in public, and the silence that results from the text simply having gone overlooked?

And that latter raises the further question of how we can keep such a peer-to-peer review system from replicating the old boys’ club of publishing systems of yore. However much I want to tear it down, the currently existing system of double-blind peer review was in no small part responsible for the ability of women and people of color to enter scholarly conversations in full; forcing a focus on the ideas rather than on who their author was or knew had, at that time, a profoundly inclusive result.

That blind review is now at best a fiction is apparent; that it has produced numerous flaws and corruptions is evident. It’s also clear from my work that I am no apologist for our current peer review systems.

But nonetheless: I’d hate to find us in a situation in which a community of the like-minded — the cool kids, the in-crowd, the old boys — inadvertently excludes from its consideration those who don’t fall within their sphere of reference. If, as I noted above, our computational filters can only ever be as good as our algorithms, the same is doubly so in a human filtering system: peer-to-peer review can only be as open, or as open-minded, as those who participate in it, those whose opinions will determine the reputations of the texts on which they comment and the authors to whom they link.

[1] Most of this information came to me through a conversation with Julie Meloni, who also pointed out that for a glimpse of what a purely machine-intelligence driven search engine might produce, we can look at the metadata train wreck of Google Books. For whatever reason, Google has refused to allow the metadata associated with this project to be expert-reviewed, a situation that becomes all the more puzzling when you take the Search Quality Raters into account.

Past and Future

For the next few days, my “Five Years Ago” block at right will be filled with post-Katrina posts. After all these years with the blog, it still feels very odd to have such a record of past trauma, the detail of what was going through my head in those days when I desperately needed someone around me to understand how bad things were in New Orleans.

The power of this kind of record is part of what makes me agree with Paul Carr’s assessment: what I’ve gained in immediacy and community via Twitter, I’ve lost in preservation, longevity, even permanence.

It’s this kind of thing that has me torn between the precious little time I have for this kind of writing and the desire to keep those thoughts somewhere I might get back at them five years from now.

The Late Age of Print, Audio Edition

From Ted Striphas comes news of an exciting project: the crowd-sourced production of a text-to-speech audiobook version of his fantastic book, The Late Age of Print. Ted has opened a wiki for the project, through which interested volunteers can help him clean up the text for audio conversion. Instructions and details are available on the wiki.

This is an exciting project, not least for its attempt to manage the labor involved in creating a public resource that will be given away freely. I hope you’ll get involved.

The Future of Publishing?

A promo video produced by DK Books for a Penguin sales conference has gone something like viral in the last two days, getting a lot of attention in my circles. In case you haven’t seen it:

I saw this video when DK first posted it, and have been thinking about it since then, mostly because I’ve been trying to figure out what makes me crazy about it.

When I watched it again yesterday it started to hit me: couched in the “hey, maybe social media isn’t going to kill reading after all; hey, maybe we really do need to start thinking about a new business model” stuff is an essentially conservative message about the ongoing primacy of the book. That only by reading everything exactly backward can you turn “books are dying” into “books are not dying!” That either the kids today only care about pop media crap or they care about reading, with no possibility that they can care about both, or that what appears to be pop media crap might in fact be important.

I dunno. It’s clever. It’s very nicely designed. And I’m happy it’s undoing some of the “kids today” rhetoric. But I’m not sold on the message overall. I genuinely believe that publishing has a future, but my feeling is that the future is going to look more like this video than like the book as we have known it. And no amount of running the tape backward will change that.

My Secret Life

Oh, hi! I’m sure it appears that I’ve forgotten about this blog thing. Really, it’s less that I’ve forgotten than that my attention has gotten fragmented in a million different directions, both work-wise and internet-communication-wise. Much of the stuff that I would have blogged back in the day is getting super-condensed and landing on Twitter, and some of the link-sharing stuff that I might have done here has for the last little while wound up being shared via Google Reader. So this space has mostly served for longer thoughts, things I want to puzzle through, and there’s been precious little time for that of late.

And, of course, the introduction of Google Buzz further complicates my communication network, as Buzz allows not just the bite-size, Twitter-like thoughts to be shared, and not just links-and-comments, but also slightly longer, more discussion-oriented thoughts. There’s been a lot of criticism of the rollout of Buzz — criticism that’s so widespread I’m not going to bother tracking down links; you can, um, Google it — and much of it well-deserved, but I’ll admit that I’m enjoying it, generally speaking. It brings together several modes of communication that I already use in a way that’s useful to me, and it allows me to keep up with what a set of people I follow are reading or thinking about. And it does that without necessarily hopelessly blurring the lines among social spheres, which means — at least for right now — it’s got many of the best aspects of Facebook, but with much less noise.

Necessarily, though, is a big caveat: one of the primary criticisms of the rollout of Buzz was the way it auto-added many of your email contacts to your followers, creating the potential for all kinds of havoc, as things people expected to be private were suddenly defaulting to a more public setting than they’d intended, and as groups that people meant to kept separate were suddenly mingling. The whole thing would have gone much better if the whole follower/following thing had been made opt-in at the outset.

That said, though, it’s possible to create pretty fine-grained control of who sees what in Buzz, if you’re vigilant about your contact groups and the ways you share stuff. So taking my own advice, I spent much of this morning wading through my contacts — synchronizing my Address Book with Google Contacts, creating better groups, assigning each and every contact to the right group, and so forth. I feel so organized right now it hurts.

But here’s the reason for this post: when I got to my own Google Contact entry, I spotted an email address that I wasn’t expecting to see: basically, firstname-dot-lastname at gmail. And I was a bit surprised — I didn’t know I had that email address. In fact, I so didn’t know I had that email address that I’d actually tried a couple of times in the last couple of years to obtain that email address, only to be told that it was taken.

And it was. Apparently, by me.

When I spotted that address in my contact information, I assumed it was a mistake, but decided to check it out. I logged out of my usual gmail account, and attempted a password reset on the other one. The system asked me for my father’s middle name — and trust me, this is a real determiner; there’s no way that someone else out there with my name has a father with the same middle name. And then it took me to the password reset screen.

The account was not only actually mine, but it was apparently my first gmail account; I sent myself an invite from that account to create the one I now use. All of the sent mail in that account was from me, mostly sending out gmail invitations, and all of it was over five years old.

The inbox, however, had a backlog of 525 messages from the last five years, most of them from the last two years. Of course a bunch of it was spam that hadn’t gotten properly filtered. But a lot of it was sent by actual people who typed in an actual email address. And going through these messages was absolutely dizzying. It felt as though some little part of my internet persona had broken off from me and been living a life — or, more accurately, lives — of its own.

I apparently attended an entrepreneurial workshop at Stanford in 2009, where I used this address on several sign-in sheets, and got several followup messages. I also attended a spiritual/motivational retreat in Salt Lake City a while back, and have been deluged with mail from the organization that ran it, as well as the other folks who were at that retreat. My father (not mine; my breakaway persona’s) has invited me to keep in touch with him on LinkedIn. I went to law school and shared class notes with several of my fellow students. I am much loved by a woman who shares my last name, and emails me every single day to tell me so. And then there’s the woman who was emailing me about the arrangements for the rehearsal dinner for her wedding; she’s probably pretty pissed at me right now.

I’ve had internet doppelgangers before, folks who mistakenly think my email address is theirs, and who use it to sign up for some of the most annoying things. This felt different, however, as I’d completely forgotten the account existed, and let it go on to live several different lives without me.

The question now is whether to start using that account. It’ll take some time, I imagine, to persuade my law school colleagues, my friends from the retreat, and everyone else that I’m not who they think I am, but the address is a good one, and it would be worth doing a bit of rehab on that persona, I think.

Gee, Time Warner, Thanks for Asking

I’ve just gotten the following email message from my friends at Time Warner Cable:

We’ve got a hard choice…

Roll Over or Get Tough?

No one likes paying more. You don’t. We don’t.

Yet, every time our contracts with TV program providers come up for renewal, that’s what we face.

Price increases. Big ones. Up to 300% more.

Sometimes we can avoid passing them on to you. Sometimes we can’t. Sometimes, a network will threaten to take your shows away if we don’t roll over.

Whenever that’s happened in the past, we’d make the best deal we could and hope that would be the end of it. But it never was. So no more.

The networks shouldn’t be in the driver’s seat on what you watch and how much you pay. You’re our customers, so help us decide what to do.

Let us know if you want us to Roll Over or Get Tough.

We’re just one company, but there are millions of you.

Together, we just might be able to make a difference in what America pays for its favorite entertainment.

So, in effect, you’re using the power of crowdsourcing to find out whether your customers would prefer to pay the same amount for less entertainment, or to pay more for the same amount. By intimating that we all need to form a united front to stick it to the network Man.

One might also consider that the same united front could conceivably used to tell one’s cable provider where they can stick not only their bogus referendum but also their ridiculously overpriced service.

We may be your customers, but if you think you’re in the driver’s seat on what we watch and how much we pay, you might consider that piece of mail I got last week telling me that Fios is coming to my neighborhood.

Is all I’m saying.

The Rise of the Landscape Web

I’ve noticed over the last couple of months that several of my favorite websites were becoming, well, wide. It’s become increasingly common, in fact, for me to find myself scrolling sideways as well as up-and-down when out there browsing, and frankly, it was getting to be a bit annoying.

But with my entry (yes, at last!) into the ranks of those who are getting to play with the Google Wave preview, it hit me: the fundamental orientation of the web is changing. And Wave may well cement that change.

Here’s the thing. Early web pages were composed vertically, in portrait layout, partially because of the limitations of screen width and partially because of the rear-view mirrorism that caused us to think about these new digital forms as “pages.” That concept has proven surprisingly sticky: web “pages” scroll vertically to this day, and very few sites have played with the horizontal axis.

Enter Google Wave, however (and possibly, as its necessary precursor, Google Chrome, though being a Mac user I can’t really speak to that at all).


Its three-column orientation demands horizontality — if the columns are too narrow, you lose a lot of the toolbar options, and everything just feels out of proportion.

So this makes me wonder, if Wave gets the kind of buy-in that the hype suggests, whether we’re seeing the fundamental orientation of the web switching from portrait to landscape — not that we won’t still be scrolling vertically rather than horizontally, but that the basic screen unit will be wider than it is tall.

This has deep implications for contemporary web design, I think, and not least for me; the other Planned Obsolescence works quite well in a wide window: you can stretch the main text and comments columns to be as wide as you would like. But it doesn’t work well here at all, as I’ve been using a fixed-width theme, and that ugly gray background block at right just gets bigger and bigger.

I’ll be curious to see whether this shift becomes — no pun intended — broader. Is the basic assumption of web layout becoming landscape? How do we organize a wider window?

The Waiting…

You know what they say about it.

Google Wave went into a wider preview release yesterday, as the first of what was supposed to be 100,000 invites were sent out to folks who’d agreed to beta-test for the developers. And everyone, myself included, has been waiting with bated breath.

Well, not exactly bated breath. More like breath that’s being used to whine and wheedle, hoping for an invite to come sooner.

The developers, however, are (a) in Australia, and so not operating on my time zone!, and (b) apparently pushing out these invites by hand. One. At. A. Time.

So I have no idea whether I made the 100,000 cut or not. I do know, however, that a friend who did passed an invite my way last night, and that invite hasn’t yet been processed, so I’m still locked out.

It’s not like I don’t have other things I need to do (like respond to some of the awesome comments that have been left on [the other] Planned Obsolescence), but I’m still checking my Gmail account every three seconds just in case.