Survey: (Photo credit: Wikipedia)
Scientists, by nature, are a skeptical bunch. They demand that studies follow rigorous methodology, are written dispassionately, and subjected to, often, rounds of external critique and revision. No other truth-seeking institution exposes one’s writing to a group of peers, whose responsibility it is to look for faults. The end result is a new piece of knowledge that functions as a truth-claim. When enough consistent truth-claims are made and consensus is formed, progress is made in science. Western democracies that spend billions of taxpayer dollars to fund science do so largely because this form of truth-seeking helps to define standards of care, fuel commerce, and help guide government policy.
Or at least this is the way it is supposed to work.
In their opinion piece appearing in the July issue of Learned Publishing (“The Access Question“), Alice Meadows, Bob Cambell, and Keith Webster argue that when it comes to scientific publishing, we are operating with double standards.
They review five large surveys dealing with access to the scientific literature and point out gross inadequacies in their methodologies, including such fundamental problems such as the inability to specify the target population. They go deeper in exposing how question bias can radically alter how scientists respond to a question.
For example, consider the question, ”There is NO problem with access to scientific publication.” Does the emphasis on the capitalization of the word “no” change the meaning of the question? For many, it does.
Also consider the leading, double-barreled question posed in the SOAP Survey to gauge what scientists think about open access publishing:
Do you think your research field benefits, or would benefit from journals that publish Open Access articles?
It should not be surprising that different surveys, designed and conducted in different ways, result with radically different results, as any introductory text in survey methods will tell you. What should be surprising is that these surveys are used to guide national science policy. Meadows writes:
The poor methodology (including loaded questions) of most of the surveys described above indicates a motivation to achieve a certain result rather than sound evidence for informing policy. This is ironic as the ultimate objective is to develop a better system for access to the peer-reviewed outcome of research. Perhaps such surveys in the future should themselves be properly peer reviewed. Until then those that quote figures from these surveys should do so with care and understanding of how these findings were produced.
If we care that science is conducted with rigor, shouldn’t we demand the same with science policy?
It is not clear that science policymaking can or should be more scientific, just because science is the subject. Should business policymaking be more businesslike? Art policymaking more artistic? Policy is made by the political system, where advocacy is the method. There are advocacy polls and objective polls, and serious players know the difference. Scientists may despise advocacy as unscientific, but they cannot wish it away. Advocacy is central to democracy.
Posted by David Wojick | Jul 10, 2012, 6:49 amIt is pretty clear to me that policymaking can be more scientific. For a start, the introduction of randomised controlled trials would vastly improve upon current public policy: http://www.cabinetoffice.gov.uk/sites/default/files/resources/TLA-1906126.pdf. Whether we should displace the current system is dependent on the results a scientific approach produces.
Also, it’s not just scientific policymaking, but policymaking in general that should be made more scientific (including the arts). The author was simply pointing out that the tools of science, having been unparalleled in their success for certain domains of knowledge-generation, should be implemented on a wider scale (in this case, to help improve current research into science policy).
Posted by James Winters | Jul 12, 2012, 4:02 pmYou seem to have missed my point, which is that science policymaking is no different from policymaking in general, and not more scientific in any case. In fact you seem to agree. As for making policymaking in general more scientific, how do you propose to do that?
Posted by David Wojick | Jul 12, 2012, 9:38 pmNo, I don’t think I missed you point. But you also said “It is not clear that science policymaking can or should be more scientific”. I was responding to the point that policymaking can and should be more scientific (irrespective of the subject). Also, I already said how I would go about doing this (well, tentative first steps): see the link above and read up on randomised controlled trials.
Posted by James Winters | Jul 13, 2012, 3:54 amJames and David, I think you are arguing two very different (but related points). If I understand your positions, James is arguing that the EVIDENCE used to support a particular policy decision needs to be based on rigorous methods (e.g. randomized control trials). David is arguing that politics is the process of assembling RHETORICAL TOOLS that advance a particular policy outcome and that science is merely one–but not exclusively the only one–of those rhetorical tools.
Posted by Phil Davis | Jul 13, 2012, 6:40 amIt turns out that we are talking about two different levels of policymaking. RCTs can indeed be used to test specific interventions. The US Dept of Education is doing something like this, testing teaching systems, many of which do not work it seems. I am talking about policymaking at the Congressional level. Funding fusion research for example, where RCTs do not seem to be applicable.
There is also a fairness problem with RCTs. We could test a 4 month OA mandate by picking a random set of journals and hammering just them to see if they survived. Would that be fair?
Posted by David Wojick | Jul 13, 2012, 7:13 amA sophisticated analysis of the relationship between science and policymaking is offered by Ted Brown in this book: http://www.psupress.org/books/titles/978-0-271-03535-2.html. Are you the Philip J. Davis who reviewed it for SIAM?
Posted by Sandy Thatcher | Jul 10, 2012, 10:11 amPerhaps this is a good subject for NSF’s Science of Science and Innovation Policy program. http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=501084 Anyone want to submit a grant application?
Posted by David | Jul 10, 2012, 5:22 pmI have been an advisor to the SCISIP program, but NSF does not take apps from small businesses, only academics, for most programs. That is an unfortunate message.
Posted by David Wojick | Jul 10, 2012, 6:20 pm