Comments on: Getting Serious About the Online Part of Research Online

By: Wendell Dryden

Wendell Dryden — Tue, 31 Mar 2009 19:52:54 +0000

This sounds very interesting and exciting.

Not sure I’m too worried about the great unwashed leading us astray. “The majority does a really good job of getting important stuff wrong.” But the elites do/did an equally good job.

But, surely, the real question is why provide an in-house voting system for anyone? If what we’re worried about is a ranking or filter system that makes one paper more accessible or authoritative than another, just don’t rank or filter. But do allow some sort of brief association, maybe (“so and so flagged this as interesting”).

Or maybe not. I only care about the reviews of reviewers I trust. I usually get referred to papers/webposts via others’ bibliographies, references and hotlinks. My online community is fully capable of alerting me to online finds. (Yes. I’m sure I miss things, but that’s my fault, my risk.) If I want to promote a paper or post, I talk about it in my blog, on Facebook, through group emails, etc. to others in my community.

Will quality win out in a web-based market place of ideas? Probably not. But material and ideas will become available in ways they haven’t for a long, long time.

By: KF

KF — Sat, 21 Mar 2009 19:03:53 +0000

Again, I think that determining what constitutes genuine engagement, what makes a good review, and so forth — all of these things would have to be subject to community-determined standards, and assessed by the community at large. I’m not trying to avoid answering the question by deferring everything to “the community,” but rather to indicate that scholars, as a community, already have a series of processes through which we assess one another’s work and hold one another accountable; making those processes more transparent — and even more, making those processes themselves subject to the same kind of assessment and accountability — seems to me a necessary component of the future of peer review in digital environments.

By: undine

undine — Sat, 21 Mar 2009 17:53:04 +0000

I like the “instant review by peers” instead of “peer review” concept, but there are some issues to be worked out. The first is the amazon.com syndrome, when reviewers of academic books sometimes post what they do without much regard for the actual quality of the book. You mentioned readers who “genuinely engage” with a book: who decides what will constitute this genuine engagement?

By: KF

KF — Fri, 20 Mar 2009 21:54:31 +0000

Absolutely — but there’s a lot of space between the current mode of peer-selection, in which two to three people make the decision on behalf of all of the rest of us, and “just anyone.” And I don’t mean, in invoking crowd-sourcing, that judgment and responsibility go out the window. The problems with the kinds of systems you mention are precisely the reason that I argue (in a chapter I hope you’ll get to see soon) that in a peer-to-peer review system, the most important element is less the review of the texts involved, than the review of the peers, such that readers come to know which other readers’ judgments they trust. I don’t mean to oversimplify what’s undoubtedly going to be a complex, difficult process, but I do believe that the crux issue in creating genuinely functional digital scholarly publishing systems is going to be the degree to which they engage their communities in an ongoing process of self-creation. And making peer review everyone’s responsibility is one aspect of that process.

By: Alex H.

Alex H. — Fri, 20 Mar 2009 20:49:16 +0000

Why shouldn’t we allow any reader who genuinely engages with a text to become a “peer”? I think this is the crux, and I think it can't simply be overlooked. I think Slashdot and Kuro5hin and others give examples of why "just anyone" or even "most interested" are not great models. The majority does a really good job of getting important stuff wrong. Do I want a peer group that still believes there were WMD in Iraq deciding whether this works. Wikipedia is the counter example--it does pretty OK, but I think that's because there is a bit of a built in selection beyond self-interest.