Beckett, Chris and Simon Inger
Self-Archiving and Journal Subscriptions: Co-existence or Competition? An international Survey of Librarians' Preferences (pdf 59pp)
Publishing Research Consortium, 26 October 2006
Comment: Most publishers now support open access in some form; for established publishers and journals this is principally through hybrid OA and support for Romeo green self-archiving in repositories. The publishers' aim presumably is to moderate the pace of transition to new open access business models. These publishers appear to want to reserve judgement on these approaches to open access, however, should the speed of change threaten current models before new models are ready to take over. Unease is most evident in the response to funder mandates for self-archiving, and the favoured method of
countering the mandates is to suggest that journals may become unsustainable if self-archived OA copies of published papers replace paid subscriptions. So far there is no clear evidence to connect OA and self-archiving with journal cancellations by libraries, so any serious published studies that claim to shed light on the issue are seized upon. Thus, as the principal faultline between publishers, journals and OA it was predictable that once a publisher with a vested interest had promoted the Beckett and Inger report as evidence of the case for cancellations, that it would receive a barrage of comment and be picked over to identify truth, bias and all sorts of subjective interpretations. That's what happened, and below we report how the prosecution opened the case against Beckett and Inger's findings being used to predict journal cancellations, and how it was defended. The following edited contributions can be found in full in, unless otherwise indicated, the
SPARC Open Access Forum Archive from 10 November with the subject lines 'Study Identifies Factors That Could Lead to Cancelled Subscriptions' and 'Self-Archiving and Journal Subscriptions: Flawed Method'.
David Goodman, 10 November: "Dear Chris and Simon, I've just this afternoon heard you present your work at the Charleston Conference, where you were careful to emphasize that you were not studying cancellation nor pdf, but rather the preferences of librarians with respect to choice of journals in a context deliberately made totally free of such considerations--just what preference for content the librarians have.
"I see that the organization for which you have done the work considers it differently, for both their headline and their summary repeatedly assert that your work identified factors relevant to journal cancellation. This is of course contrary to your oral presetation, and also to your own summary of the report on the pdf report."
Simon Inger and Chris Beckett, 12 November (fwd by David Goodman): "The work was a study of some of the key parameters that librarians consider in the acqusition of different incarnations of content (not just journals). Of course cost was a major factor in their considerations and free was seen as the most appealing of the levels proposed. In Chris' introductory remarks he emphasised the importance of the results to publishers in the context of the potential risk of journal cancellations.
"In fact, the second paragraph of the summary in our report states "Overall the survey shows that a significant number of librarians are likely to substitute OA materials for subscribed resources given certain levels of reliability, peer-review and currency of the information available." We believe the
press release to which you refer is therefore a reasonable reflection of our conclusions."
Fred Friend, 10 November: "on a first quick reading - I had not found a clear connection between the results of the study questionnaire and the cancellation of library subscriptions. This study tells us that librarians prefer to acquire high-quality content recently-published and at as low a cost as possible. To me this shows that librarians are doing the job they are paid to do"
Stevan Harnad,
Self-Archiving and Journal Subscriptions: Critique of PRC Study, Open Access Archivangelism, 13 November: "The Publishing Research Consortium commissioned a study of acquisitions librarian preferences to see whether they could predict such cancellations in the future using a "Share of Preference model," but the study has a glaring methodological flaw that invalidates its conclusion (that self-archiving will cause cancellations)."
Simon Inger and Chris Beckett, 16 November (fwd by Peter Suber): "Stevan focuses his criticism on five main points:
1. The methodology deployed and the entire point of conducting a conjoint survey at all: "We decided to undertake a conjoint survey because we felt that other attitudinal surveys of what future intentions might be were highly prone to being bogged down exactly because surveyees were asked in absolute terms to what extent they would like one scenario, and then another, without ever asking them to choose between them."
Harnad response: "The conjoint method is no doubt a good method for estimating or ranking relative product property preferences in general. But in the particular case of library journal acquisitions/cancellations, OA and self-archiving, as noted, the method not only does not remedy the the possibility of bias, but it bypasses the question of cancellations altogether -- the question that I take it that (for lack of actual cancellation data) the survey was trying to answer."
2. Whether or not OA can be considered a product in any meaningful sense: Can articles in Open Access repositories be considered a product and one that librarians may select instead of journals? Absolutely they can. Is the issue here that they are free via OA, or that they are not organised and packaged?
3. The issue of bias: "The whole Open Access debate evokes an emotional response from publishers, librarians and researchers on both sides of the debate. At the same time, so does the word "cancellation". For that matter, so does the phrase "serials crisis". We wanted to avoid using all of these phrases in the research so as not to cloud people's judgement in favour of their beliefs alone. This is one way of avoiding one type of bias. Specifically the type of bias we sought to eliminate was an emotional bias, not a bias for or against OA per se."
Harnad response: "I think the attempt to avoid all of these emotional (and notional) biases was a commendable one, and it would have been successful too, if the conjoint-preference method had been amenable to analysing the anarchic phenomenon of author self-archiving and its likely effect on librarian acquisition/cancellation. But it is not, because anarchic, blanket self-archiving is simply not an acquisition/cancellation matter."
4. The statement of apparently obvious or banal findings: ""The fact that everyone would like something for free rather than paying for it", for example. In fact the survey shows that not everyone would prefer that. Even in a completely like for like situation. Possibly because people are suspicious of free things. Much more important, however, is how the decision becomes qualified by other factors -
and to what extent they are qualified.
5. The validity of inferring cancellation behaviour from the findings: "can we infer cancellation behaviour from the results? Yes, we can. Because it is unrealistic to expect that everyone that expresses a preference for acquiring a product that looks very much like content on OA repositories would still continue to acquire a paid-for version. Some will, of that we have very little doubt. But likewise some won't."
Mark Ware, 13 November: I conducted a survey of librarians' opinion on the same topic earlier this year which received responses from 340 librarians (report available on the ALPSP website;
free summary article). Although my methodology was much less sophisticated than this study's (and there was some criticism on one of the lists that some of my questionnaire begged a key question), there are nevertheless some puzzling differences between the findings in the two surveys:
(1) Librarians in this survey expressed no preference for the publisher's final version over the author's refereed post-print. I have no particular argument with the librarians on this but they appear to say something different in my survey. When we asked "What freely available versions would you consider an acceptable substitute for the journal?", 97% chose the final journal pdf but only 39% the author's post-print. The recent finding does seem anomalous, though: as the authors say, it is not concurrent with current observed behaviour.
(2) This survey finds that a 6 month embargo had little impact (on librarians' preference for (delayed) OA material rather than the paid-for version) ... the key point for many publishers is where the tipping point lies, and on this there appears to be a conflict with our earlier findings. The very different methodologies makes it hard to compare reliably, but my data showed only 18% of librarians regarded material embargoed for 4 or more months an acceptable substitute for a subscription."
David Goodman, 15 November: "such studies at the Beckett & Inger study and one by Ware are important as prototypes. The differences between them demonstrate that neither of them is likely to be actually modeling true opinion, let alone behavior."
Phil Davis,
liblicense post, 16 November: "If the PRC results were predictive of actual behavior, one would expect that subscription-based journals that provided delayed free content would see massive library cancellations. Are these publishers, some of whom provide free access after as little as 2-months committing subscription hari-kari? Seems not."
Heather Morrison, 14 November: "the primary factors determining collections decisions are not taken into account: research and educational priorities of the university, and faculty assessment of the importance of journals. When we take these factors into account, we can see why it makes sense that librarians continue to subscribe to physics journals, even when prices are considered high and virtually all of the articles are available for free in arXiv."
Chris Beckett,
liblicense post, 16 November: "Initially a number of factors were identified by SIS (Scholarly Information Strategies Ltd) that they thought library decision-makers were most likely to consider important when purchasing content for libraries. These were tested and validated ... Feedback from this process resulted in a reduction of the attributes to be tested in the conjoint analysis from an original eight (12) to six and to some minor rewording of the attitudinal survey.
"The footnote (12) indicates "The two attributes that were include in the original draft conjoint survey but subsequently excluded were "Archive and Permanence" and "Importance to your Collection"
"Importance to your Collection", which I think equates to Heather's point of "research and educational priorities of the university and faculty assessment of the importance of journals", was therefore considered and proposed for inclusion in the formulation of the conjoint survey. ... The logic for its exclusion was simply that, in the context of making a choice between different incarnations OF THE SAME CONTENT (i.e. the same article appearing in a journal, in a licensed database or on an OA Institutional or subject repository), the importance of that particular article is constant across the different incarnations. An article does not become more or less important to the collection needs of the institution because it appears in an OA archive rather than a licensed database or a journal."
Jan Velterop, 15 November: "What Beckett and Inger do amounts to market research. A sound practice in any line of business, and always flawed, because what is measured is intentions. What Harnad wants is an analysis afterwards, dismissing market research, saying that it is not the same as evidence (though it is, albeit of intentions). That is also flawed, in that it amounts to destructive testing."
Stevan Harnad, 15 November: "I think Jan has not understood my critique at all. The profound methodological flaw I pointed out was not that the PRC survey was asking about librarians' intentions, but that it was asking about the
wrong intentions, in the wrong way. As a result, the librarians' expressed intentions give us no indication whatsoever about the causal effect of self-archiving on cancellations: neither about (1) whether there will be such a cancellation effect at all, nor about (2) how much self-archiving would cause a cancellation effect, nor about (3) how big such a cancellation effect would be, nor about (4) how soon such a cancellation effect would happen, if it were to happen.
"All we have from this study is the confirmation of a banal fact that anyone could have stated in advance: All things being equal, everyone would prefer a free product to a priced product and would prefer the product immediately upon purchase, rather than after a delay. (I invite anyone who thinks otherwise to explain clearly to everyone how the PRC survey answers any of questions (1)-(4).
"The PRC survey, to repeat, did not ask acquisitions librarians about what they think they would
cancel, in exchange for what, under what conditions. It asked what product they would prefer to
acquire over what product (where among the properties of some of the "products" was the fact that they were free!).
David Goodman, 16 November: "The questions asked were as follows:
'In each question you will be presented with a table that:
Lists some of the key factors affecting renewal and collection development decisions.
Has three columns (A, B and C), each with different values for each of the key factors.:'
They do not mention the word 'acquire.'
One of the values for the cost factor was 'free'
Only 20% of respondents were acquisitions or collection development librarians.
...
The actual questions, possible responses, and demographics are in the Appendix."
