ALL BLOG POSTS AND COMMENTS COPYRIGHT (C) 2003-2016 VOX DAY. ALL RIGHTS RESERVED. REPRODUCTION WITHOUT WRITTEN PERMISSION IS EXPRESSLY PROHIBITED.

Tuesday, June 28, 2016

The problem of peer review

Peer review simply isn't what it is advertised to be; it is not only little more than editing, most of the time it is not even competent editing:
As a scientist with a 15 year career behind me so far, I am afraid that my experiences reflect this. Peer review is excellent in theory but not in practice. Much of the time, the only vetting the papers get are two relatively junior people in a field (often grad students or postdocs) giving it a thumbs up or thumbs down. That is absolutely it. In theory, the editors should make the decisions with the recommendations of the reviewers, but the editors rarely have the time or the expertise to judge the papers and often automatically defer to reviewers. Also, the papers should be reviewed by luminaries of the field, but these folks rarely have the time, and either decline invitations or bounce the work to a student or another trainee. It's not just bad papers that get through, but also good, rigorous, papers that are bounced by this system.

Many if not most of the people in academic science today, at least in biology (my field), are overwhelmed with the need to publish in such high volumes, few people with the needed expertise can afford the time to go over the results in detail. All this while, at the same time and for the same reason, the volume of papers that needs to be reviewed goes up. I've heard of (and had myself) papers havve lingered for 4+ months before they even went out for review.

And, in our rush to publish, we often don't read this literature carefully ourselves but start citing papers anyway, which weaves these potentially weak or erroneous papers even more tightly into the fabric of their field.

It's difficult to care a lot about the quality of your work when you know the extra effort often doesn't help something go through this fickle review process, and when you know people will cite it without really reading it closely. There is little incentive to spend longer on a paper to make sure everything is right and the results are reproducible because there is very little accountability for errors and huge rewards for being prolific.
The ironic thing is that True Believers and the I Fucking Love Science crowd genuinely believe that "peer reviewed science" is the gold standard for evidence. But there is a reason scientific evidence is not automatically allowed in a court of law, let alone considered conclusive, and the more we learn about the defects of peer review, the better we understand that science's credibility is limited.

We have a word for science that is trustworthy, and that word is engineering. Until science can be applied, it cannot be fully trusted to be correct.

All peer review is really designed to do is to reassure the reader that the information presented fits safely within the confines of the consensus status quo.

Labels:

74 Comments:

Blogger Caladan June 28, 2016 5:09 AM  

Just check out the #real_peerreview hashtag on Twitter.

Exposes the absolute worst of the social sciences.

Anonymous Spinrad's Agent June 28, 2016 5:20 AM  

He could have also mentioned that academic journals often operate as old boy's clubs, with members favorably reviewing each other's work in an endless cycle of back scratching.

And he didn't even get onto fudging results. That's the bigger issue; outright fraud in the pursuit of grants, status, publications and access to nubile students.

Blogger guest June 28, 2016 5:24 AM  

http://retractionwatch.com/

Anonymous Spinrad's Agent June 28, 2016 5:26 AM  

Caladan wrote:Just check out the #real_peerreview hashtag on Twitter.

Twitter's thought police have been all over that. You can see the archive here: https://archive.is/lWsU3

Blogger guest June 28, 2016 5:28 AM  

http://www.icr.org/article/8533

Blogger L' Aristokrato June 28, 2016 5:30 AM  

Watch this video on the subject, it sums up everything:
https://www.youtube.com/watch?v=o3Sjp_hIrQ0

Blogger intuitivereason June 28, 2016 6:07 AM  

For the system to get better, a) Peer Reviewers must be identified, and b) peer review of an article must count as equivalent to a first authorship.

Blogger ValeriusMaximus June 28, 2016 6:16 AM  

Being in engineering, I've read many research articles, both old and new. The general quality of articles seems to be in decline. Many, particularly those written by non-Japanese Asians, appear to have been written by children. It is not necessarily that the data is not good or that their arguments are not sound, they just can't express it in a useful succinct way.

Here's a sample chosen randomly from an Abstract in the Journal Acta Materialia (Volume 96, 1 September 2015, Pages 301–310) by Seok Su Sohna, et al: "The reduction in weight of automotive vehicles has been devoted to improve fuel efficiency by reducing vehicles’ exhaust emission [1] and [2]. However, the traditional approach to obtain the weight reduction from down-gauged high-strength steels has many limitations of keeping the stiffness [3]. "

How did this get past the editors desk?!


Blogger Human Animal June 28, 2016 6:17 AM  

We have a word for science that is trustworthy, and that word is engineering.
And between engineers and actual problems stand the lawyers, the lobbyists and the "environmental scientists."

In the future, soft science students will be hunted seasonally, like Deer, just to keep their numbers down and protect the economy.

Blogger residentMoron June 28, 2016 6:20 AM  

"We have a word for science that is trustworthy, and that word is engineering. Until science can be applied, it cannot be fully trusted to be correct."

The late James P Hogan would be proud.

Just coincidentally, I am employed in a field where all my customers are either researchers and their employers (mostly universities but also a scattering of public and private research institutes) or the funders of same.

We are acutely aware of the problem not only of sloppy and incomplete work but also of deliberately falsified results, plagiarism, and other forms of manipulation of the system for pecuniary gain.

We are also acutely aware of how little anyone involved in this multi-billion euro industry wants to openly discuss these problems. The last thing anyone wants to discuss being their implications for the credibility of anything published today.

As we have seen so clearly in other spheres of human endeavour, tolerating the non-performance of the inept, the lazy, and the criminal, inevitably brings the entire profession into disrepute.

A big issue with peer review is that reviewers are often paid significantly more for a recommendation to publish than for meaningful criticism. But this is still only one symptom of a very sick industry. Only one example of the moral hazard that has corrupted the entire field, that everyone in it is aware of, but that nobody in it wants to bring out in the open, for fear that the public - on whose largesse this all depends - will revolt and stop paying the bills.

Blogger Shimshon June 28, 2016 6:23 AM  

@8 The editors are likely just as qualified as the writers today.

I was shocked at how bad the writing of fellow college students was in the 1980s (even in a supposedly competitive school like UCSD). I ran a typing service one year, and I got all sorts of dreck, with just a few students who could write. Today it's probably abominable.

Blogger Lobo Util June 28, 2016 6:49 AM  

It sounds like a Masters Thesis or PhD Dissertation is much more rigorously reviewed than a peer reviewed article in Nature or Lancet.

Anonymous Be Not Afraid June 28, 2016 7:05 AM  

Pretty much right. I had a paper where I had to badger the editor many times to get back a review. A trusted colleague told me the referee was a competitor, and had circulated copies to his own group for their use in their project, while sitting on the paper for 6+ months to slow me down.

The only solution for the largely useless peer review--it's good at catching elementary mistakes if the author happens to write them down--is free and open access to raw data and anything needed for calibration (codes in the case of models). As one of the Climategate "scientists" wrote "Why should I make the data available to you, when your aim is to try to find something wrong with it..."

Blogger synp June 28, 2016 7:07 AM  

intuitivereason wrote:For the system to get better, a) Peer Reviewers must be identified, and b) peer review of an article must count as equivalent to a first authorship.

Although then you need some quality control of the peer review itself. Who will review the reviewers?

Moreover, they need to encourage publishing negative results. Most research is bound to have negative results. Not publishing negative results leads to (a) pressure to fudge the data to show positive results, and (b) more researches doing the same research again thinking it is novel, which leads to (c) repeating the same experiment over and over again until (perhaps by chance) a positive result comes out.

But yes, if we could reward researchers for quality reviews and encourage the older researches to concentrate on this more, that could increase the quality significantly.

Anonymous fred June 28, 2016 7:17 AM  

@8: Well the lead author sounds Korean, I bet he's not a native English speaker. Asian languages have very different grammar than English, and besides, even English grammar is a train wreck in its own right, so I'm not surprised. Back when I studied Japanese, I found the grammar very elegant and beautiful, but also from Mars. My French was a little better, but I always found myself forming a sentence in my mind in English grammar, and then translating it into French aloud, and getting plenty of things wrong. How embarrassment!

@11: Well I bet it's partly so awful now because social media has taught people to write in this compressed, encoded style which throws out all kinds of elements which were there for a reason. If you rite w/ emoticons U R doin it rong LOL.

Still, I guess college students should have basic proficiency at least, but people take actually good writing for granted, when it's really an art. Some people can only play scales and arpeggios, some can play Martha My Dear, and some can play Schumann and Satie.

Anonymous fred June 28, 2016 7:25 AM  

@8: For years I carried around in my wallet a little piece of wax paper that had the instructions on it for a Magic Egg, a dime-store toy, made in China no doubt. It's a little ball of like putty and when you soak it in water it grows like 3x its size.

The instructions went something like "Put the thing for the purpose into the glass. It will be for the admiring of it the thing for size, and also the respect from all those who see."

Blogger Human Animal June 28, 2016 7:28 AM  

re: ValeriusMaximus
People who want to be understood speak simply.
Be Not Afraid is probably right: Being understood just makes you easier to criticize.

(Gonna build a Wall. It's gonna be Huge.)

Blogger Avalanche June 28, 2016 7:28 AM  

@13 " As one of the Climategate "scientists" wrote "Why should I make the data available to you, when your aim is to try to find something wrong with it...""

To which the answer is: because THAT Is how the scientific process works!! Sheesh!

Blogger Shimshon June 28, 2016 7:32 AM  

@15 From what I read, it sounds like we are nearly to the point where actual literacy is not expected or taught or required in any way. It sure is shocking.

There was still the expectation of literacy (in the spirit of readable and conveys information effectively). Entering UCSD students who couldn't pass a writing test had to take an intro course. And the standards were strict enough that many people who (IMHO) could write attended this class. I don't know for sure, but I wouldn't be surprised if inability to pass this course resulted in some sort of expulsion after some number of tries. My TA for my course was a ridiculously strict and arbitrary woman. She accused me of cheating because my writing quality had noticeably improved (something which I guess she was unaccustomed to). The girl we students had unanimously (having read and reviewed each other's work) considered A-worthy got a C+.

Blogger Avalanche June 28, 2016 7:33 AM  

If you want to be really horrified, look at what your doctors are using as 'references'! (Some study determined that the regular, patient-treating doctor takes an average of SEVENTEEN YEARS to get and use the "latest" medical research results.

http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/308269/

Also: http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124

Blogger The Kurgan June 28, 2016 7:42 AM  

agreed. And as an engineer, I assure you that any science theory will go through several iterations before it gets engineered so it actually works.
The one exception I am aware of might have been Tesla. The man. Not the car. Apparently he did his testing in his head.

Blogger Francis Parker Yockey June 28, 2016 7:56 AM  

On the other hand, if you look at how much information about new drugs (especially adverse effects) only comes out in the post-marketing phase, having a doctor who's too quick to jump on the latest fad is not so great, either. It's almost as if there's a financial incentive to suppress any negative information about new drugs during the trial phase, or something...

Blogger Francis Parker Yockey June 28, 2016 7:56 AM  

On the other hand, if you look at how much information about new drugs (especially adverse effects) only comes out in the post-marketing phase, having a doctor who's too quick to jump on the latest fad is not so great, either. It's almost as if there's a financial incentive to suppress any negative information about new drugs during the trial phase, or something...

Blogger KCFleming June 28, 2016 7:58 AM  

Around 2004, I had submitted a paper to JAMA which was meant as a rebuttal to the 2003 article "Proposal of the Physicians' Working Group for Single-Payer National Health Insurance."

I learned that one of the reviewers for my paper was a JAMA editor who was also an author on the paper I was criticizing.
I requested a different editor, and swiftly got a call from the editor-reviewer in question.

She started yelling at me on the phone, and I had to stop her and ask who was calling (she had not identified herself, just began yelling). I seriously though a crazy patient had called.

When I suggested that an author who I was criticizing ought not to review that paper because of the obvious conflict of interest, she erupted again. I told her she was being unethical and she said, "Your paper will never be published anywhere."

She was correct.
I only was able to get it in at the Heritage Foundation.
Not 'peer reviewed,' so it 'doesn't count' toward my publications.
The fix was in for the national healthcare 'debate.'

And my eyes were suddenly open.

Blogger szopen June 28, 2016 8:01 AM  

I'd say this is not entirely fair. I've been reviewing papers for years (computer science field), starting as a junior PhD student (that's part it true - I was being sent a lot of papers to review from my advisor). And yeah, sometimes I have no expertise in a particular topic and still I'm pressed to make a review. However, you have no idea how much bullshit is being written. For each mediocre paper you see in press, there are ten times as many BS papers which were vetted.

Also, while many of my papers were vetted because people who were reviewing it had no idea about what they were writing about, from time to time I had a great reviews which actually helped me to write better.

As for idea that reviewers should not be anonymous, that's absolutely awful idea, as taht would merely mean that no one would dare to stand against the known authorities, and number of harsh, honest reviews would go down to zero. Idea that reviewer should become co-author is also absurd.

I would say that something else would have to be changed:

full anonimisation of authors (though it often won't work for obvious reasons, but still could help just a bit)+ open access to data and all tools to allow replication + reducing number of PhD students and universities - that's just for starters.

Blogger YIH June 28, 2016 8:19 AM  

But there is a reason scientific evidence is not automatically allowed in a court of law, let alone considered conclusive, and the more we learn about the defects of peer review, the better we understand that science's credibility is limited.
Thanks to TV jurors are often predisposed to accept 'science' as a definitive answer. Trouble is, just like a lot of things ''as seen on TV'' are exaggerated, very exaggerated, or just plain BS.

Blogger ValeriusMaximus June 28, 2016 8:39 AM  

fred wrote:@8: Well the lead author sounds Korean, I bet he's not a native English speaker.

I don't necessarily blame the authors, they were all Korean, as you correctly surmised. Editors of an English language journal should do their jobs and, working with the authors, correct the language.

Blogger VD June 28, 2016 8:41 AM  

I'd say this is not entirely fair.

You're wrong. It is better than fair, it is accurate.

Anonymous Stickwick June 28, 2016 8:49 AM  

Peer review is a serious problem in many fields. However, some fields are still striving to maintain their scientific integrity. There is no such crisis in my field (astronomy and astrophysics) that I am aware of.

The point of bringing this up is to understand why corruption in the peer review process happens. Some scientific fields are corrupted, because there is a huge amount of money and/or power vested in supporting a certain paradigm (climate science, medicine); with others it's because the field is dominated by ideology (biology). Judging by the results, the worst corruptors by far are money and power. Remove government involvement in science (climate change, medicine), and this ceases to be a problem.

Blogger mpeirce June 28, 2016 8:59 AM  

Replication is important too. If an experiment isn't replicable no one should trust it.

Blogger YIH June 28, 2016 9:00 AM  

ValeriusMaximus wrote:Being in engineering, I've read many research articles, both old and new. The general quality of articles seems to be in decline. Many, particularly those written by non-Japanese Asians, appear to have been written by children. It is not necessarily that the data is not good or that their arguments are not sound, they just can't express it in a useful succinct way.

Here's a sample chosen randomly from an Abstract in the Journal Acta Materialia (Volume 96, 1 September 2015, Pages 301–310) by Seok Su Sohna, et al: "The reduction in weight of automotive vehicles has been devoted to improve fuel efficiency by reducing vehicles’ exhaust emission [1] and [2]. However, the traditional approach to obtain the weight reduction from down-gauged high-strength steels has many limitations of keeping the stiffness [3]. "

How did this get past the editors desk?!


Let me guess, the editor's name was Google Translate ;)

Anonymous Napoleon 12pdr June 28, 2016 9:03 AM  

"We have a word for science that is trustworthy, and that word is engineering."

I'm stealing that!

I'm an engineer (flight test, to be exact), and the backbiting in the pure science world never ceases to astound me. Yes, I've written a laundry list of professional papers, but the biggest problem has been getting them approved for publication, not the knife-in-the-back manipulation of the science crowd.

Blogger residentMoron June 28, 2016 9:04 AM  

The big name universities (in the USA, the big ten spend something like 1.5 billion per year on research projects) will eventually act in self defense and simply employ, train, and keep on their staff, a permanent force of reviewers who test against a set of standards for literacy, logic, clarity, and originality.

They will consult various academics within the university (or network if it becomes a formal cooperative effort) where they absolutely need specialist knowledge in depth, but for a lot of papers they will defend their reputations in-house.

Blogger synp June 28, 2016 9:04 AM  

Stickwick wrote:Peer review is a serious problem in many fields. However, some fields are still striving to maintain their scientific integrity. There is no such crisis in my field (astronomy and astrophysics) that I am aware of.

I know very little about astronomy, so maybe the following doesn't make sense. But the thing about astronomy is that it is expensive. Those telescopes are hugely expensive and funded by either governments or philanthropists. Either way, there is pressure to deliver positive results.

Consider the discovery of exoplanets. Even our best telescopes can't directly observe exoplanets. Instead astronomers deduce the existence of exoplanets from perturbations in the light coming from stars. But is this the only explanation for these perturbations? Studies claiming the find exoplanets bring in more budget and fame. Suppose an astronomer published a paper with an alternate explanation for the perturbations, essentially being the wet blanket who says we may have not yet found exoplanets, would this meet with some resistance?

Same question about gravity waves.

Blogger residentMoron June 28, 2016 9:07 AM  

@25

No, I have a very good idea exactly how much bullshit is being written. That is, however, irrelevant, as the protest is not at the amount of bullshit being written but at the amount of bullshit being published after peer review.

Anonymous sth_txs June 28, 2016 9:13 AM  

As a graduate engineering students in the late 90's, I could never understand who had time to read all of that crap produced in any field. My understanding there several thousand engineering journals worldwide. IEEE alone had least three dozen publications.

I used to joke with a professor about how is article is read if only by the peer review committee and published in the Journal Of Engineering Read By Nobody.

Little of it is any real breakthrough in any field.

Blogger darkdoc June 28, 2016 9:23 AM  

I've been in academic medicine for 30 years.

While I make no excuses for the level of corruptness and dishonesty, mixed with pride, position and arrogance, I think the dominant position behind scientific literature and peer review represents cowardice more than any other. A person willing to challenge the system from within is rare indeed. Bucking the system extracts a price.

But then I have always maintained that Doctors are just nerds with power. A sad mix for sure.

Blogger rho June 28, 2016 9:40 AM  

synp wrote:Instead astronomers deduce the existence of exoplanets from perturbations in the light coming from stars. But is this the only explanation for these perturbations?

Kepler has been up for 7 years. Regular periodic dimming may be something else entirely, but exoplanet is the simplest explanation.

I forgot which star it was, but they recently found one that had something very strange going on that some wags were suggesting a potential Dyson sphere. Anything that doesn't fit the expected model of regular, periodic dimming doesn't go into the exoplanet bucket.

Gravity waves, who knows? But as far as I've read, astrophysicists find the experimental results interesting and hopeful, but not definitive. I imagine Stickwick has the insider viewpoint, I'm just an enthusiast.

Anonymous Stickwick June 28, 2016 9:41 AM  

synp: I know very little about astronomy, so maybe the following doesn't make sense. But the thing about astronomy is that it is expensive. Those telescopes are hugely expensive and funded by either governments or philanthropists. Either way, there is pressure to deliver positive results.

I could see why someone would assume that; however, the truth is that there is no pressure to deliver positive results, at least not the way you’re imagining it. Big telescopes are designed and built to keep doing what we’re already doing, only faster, better, and in higher volume. The chances of not getting meaningful results with big telescopes are zilch.

Consider the discovery of exoplanets. Even our best telescopes can't directly observe exoplanets.

Not true.

Suppose an astronomer published a paper with an alternate explanation for the perturbations, essentially being the wet blanket who says we may have not yet found exoplanets, would this meet with some resistance? … Same question about gravity waves.

You think this doesn’t happen? There are alternate theories for just about everything in my field.

Blogger S1AL June 28, 2016 9:45 AM  

The best part is that the habits that lead to this system are developed and encouraged in undergraduate work. The system is flawed from root to branch.

Blogger szopen June 28, 2016 9:47 AM  

@36
You don't. There is a reason why you have a list of journals and conferences ranked by their reliability. There is a reason why "IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS" is much more trustworthy than "New Generation computing". Same with conferences: OPODIS is much better conference than some IEEE conference in China. The reasons is mostly the quality of peer review. When bad papers are starting to appear in a journal, then people start to stop reading it, quoting it, and its impact goes down.

Blogger Avalanche June 28, 2016 9:59 AM  

@22 "only comes out in the post-marketing phase, having a doctor who's too quick to jump on the latest fad is not so great, either."

If we are smart enough to not trust the govt, WHY do we trust doctors and Big Pharma?

Just look into the continuing danger of docs pushing statins -- esp. for women (for whom there is not a single study showing beneficial effects)!

OBVIOUSLY, don't decide to just up-and-quit... but fer shure educate yourself and make an educated decision, preferably with your doc, if he's amenable to getting educated himself! (Oh, and women with HIGHER cholesterol live longer than women with lower cholesterol... Does your doc know that? Probably not.)

A place to start:
https://proteinpower.com/drmike/2009/02/26/more-statin-madness/

and this, from here: https://proteinpower.com/drmike/2013/10/31/statins-everyone-maybe-anyone/
============
Here is the take home message on statins:

Statins do not decrease all-cause mortality in the vast majority of people. Long-term studies have never been able to demonstrate that women of any age or with any degree of heart disease live longer by taking statins. The same long-term studies show that men over the age of 65 live no longer by taking statins. Men under 65 who have never had heart disease – and we're talking actual heart disease here, not just an elevated cholesterol level – gain no longevity benefit from taking statins. The only small group of people who have been shown to benefit from statins are men under 65 who have had a heart attack. But unfortunately that benefit is small.

Multiple studies have shown that taking statins does reduce both the incidence and severity of heart attacks. But these same studies don’t show any increase in longevity for those taking statins (other than the small benefit for men under 65 who have had heart attacks). Why? Statins simply trade one risk for another. Take them and you reduce the risk of a heart attack but increase your risk for cancer, diabetes, kidney failure, and side effects related to the drugs themselves. Many people die each year from statin-induced side effects. Despite what anyone may tell you, statins are not benign drugs.
=============

and, from this: https://proteinpower.com/drmike/2012/01/16/statins-and-diabetes/
==========
...women studied as part of the Women’s Health Initiative who were on statin drugs during the study developed diabetes at greater rates than those who were not on these drugs. According to the statistical analysis of the authors, being on a statin increased the relative risk of developing diabetes by 48 percent!
==========

Blogger rho June 28, 2016 10:03 AM  

Peer review is like changing the oil in a car. It's necessary, but not sufficient. Peer review won't route around dishonesty, rent-seeking or insufferable navel gazing, just as two quarts of Castrol won't fix a blown head gasket.

Anonymous BGKB June 28, 2016 10:09 AM  

Twitter's thought police have been all over that. You can see the archive here

Dam it now I feel like I have to read the Black Feminist Calculus paper.

Anonymous JayDell June 28, 2016 10:10 AM  

I'm in science and can confirm peer review is not everything it is sold to be. The reviewers are completely dependent on the data the authors provide them. They almost never see the raw data, so they can't catch fraud or serious experimental errors.

In my experience, most papers are accepted or rejected primarily on the subject matter, and not the particular qualities of the paper itself. After the initial decision is made, they make the perfunctory critiques to justify the acceptance or rejection.

Blogger CM June 28, 2016 10:12 AM  

When I was a very young engineer with little experience and no knowledge of the system, I was tasked with "designing" an updated system based on old documentation.

It was "peer reviewed" by the more senior engineers who developed the original system and signed off on my incompetent work. Two years later, they started development using my design and showed how completely incompetent I was.

I have no issue sharing this because i have no pride in my limited abilities in software development.

Blogger praetorian June 28, 2016 10:22 AM  

Replication is important too. If an experiment isn't replicable no one should trust it.

My favorite are "meta-studies" where other studies aren't replicated, but just averaged over using high school-tier statistics.

Almost no work, and you get to confirm the biases of the existing field's research, thereby almost guaranteeing publication. I Fucking Love Science in action, folks.

Blogger synp June 28, 2016 10:29 AM  

Stickwick wrote:synp: I know very little about astronomy, so maybe the following doesn't make sense. But the thing about astronomy is that it is expensive. Those telescopes are hugely expensive and funded by either governments or philanthropists. Either way, there is pressure to deliver positive results.

I could see why someone would assume that; however, the truth is that there is no pressure to deliver positive results, at least not the way you’re imagining it. Big telescopes are designed and built to keep doing what we’re already doing, only faster, better, and in higher volume. The chances of not getting meaningful results with big telescopes are zilch.

Consider the discovery of exoplanets. Even our best telescopes can't directly observe exoplanets.

Not true.

Suppose an astronomer published a paper with an alternate explanation for the perturbations, essentially being the wet blanket who says we may have not yet found exoplanets, would this meet with some resistance? … Same question about gravity waves.

You think this doesn’t happen? There are alternate theories for just about everything in my field.

Thanks.

Blogger residentMoron June 28, 2016 10:35 AM  

BGKB wrote:Twitter's thought police have been all over that. You can see the archive here

Dam it now I feel like I have to read the Black Feminist Calculus paper.


Don't waste your time.

Read "The Racist Implications of Air Conditioning in Chicago School Buses" instead.

Blogger szopen June 28, 2016 10:35 AM  

@45 JayDell
(1) Not all scientific papers are experimental. In my field a lot is purely theoretical and for those, there is mostly no problem at all.

(2) Making experiments is time-consuming. Reviewers would have to repeat this process to check whether authors get their result right. Hence, some minimum trust is needed.

(3) IMO it would be great if the papers with merely confirmations or negative results would be published more often, and maybe even given priority AND taken into account when evaluating the scientist's work (by whomever would be doing the evaluation).

I am not sure what to do to heal the situation, but _IMO_ peer review is not part of the problem (per @43). It's just like reading books: you tend (or at least me, at least here in Poland) to be more eager to buy books/stories from established houses than from self-publishers, because you tend to trust that established publishing houses would filter most of the trash. So instead of reading thousands of mostly papers, you read hundreds of them, with much lower crap ratio.

Of course, it's not enough, as established houses may stop filtering, or the filtering may start to be dictated by ideology (not really a problem here in Poland with fantasy and sf, though this is what decreased a lot Polish "mainstream" readership).

Anonymous Mr. Rational June 28, 2016 10:39 AM  

Vox wrote:The ironic thing is that True Believers and the I Fucking Love Science crowd genuinely believe that "peer reviewed science" is the gold standard for evidence.
Just because it's pretty much the best we've got, doesn't mean it's perfect.  Nothing human is.

there is a reason scientific evidence is not automatically allowed in a court of law, let alone considered conclusive
IIUC, neither is anything else.  Eyewitness testimony is some of the worst, yet is accorded weight far out of proportion to its reliability.

One of the most effective improvements to the system might be to have a set of paid experts in areas like statistics who go over papers and make certain that the work has been done correctly.  None of this will catch expert fraud (such as fabricated data), but that's where people get their PhD's revoked and lose tenure, job and pension.

Human Animal wrote:In the future, soft science students will be hunted seasonally
So much simpler and cheaper to send them straight to barista training and skip the middleman.  Nobody who can't handle fractions and algebra should be able to get into university, ESPECIALLY ed school.

residentMoron wrote:Only one example of the moral hazard that has corrupted the entire field, that everyone in it is aware of, but that nobody in it wants to bring out in the open, for fear that the public - on whose largesse this all depends - will revolt and stop paying the bills.
If I make any lobbying trips to my state capital this year, one of the things I'm going to push is a cutoff of state aid for all students in degree programs that do not require math and science to complete AND passage of independently-proctored exams in both for graduation.

synp wrote:which leads to (c) repeating the same experiment over and over again until (perhaps by chance) a positive result comes out.
This has a name:  the File Drawer Effect.

The Kurgan wrote:The one exception I am aware of might have been Tesla. The man. Not the car. Apparently he did his testing in his head.
As an electrical engineer, I can see how Tesla did it.  He stood on the shoulders of James Clerk Maxwell.

Blogger VD June 28, 2016 10:45 AM  


IIUC, neither is anything else. Eyewitness testimony is some of the worst, yet is accorded weight far out of proportion to its reliability.


Despite its flaws, eyewitness testimony is statistically and empirically more reliable than scientific evidence.

Anonymous Napoleon 12pdr June 28, 2016 10:55 AM  

Tesla was reputed to have the ability to visualize the hardware in his head...an internal CAD/CAM system. Work out the bugs there, then go to working drawings and blueprints.

Blogger Quadko June 28, 2016 11:01 AM  

Peer review:
* Treated like it means "Reviewed (and probably duplicated) by Peers of the Realm and the House of Lords"
* Best case means "Read by a jury of the author's peers" - juniors review juniors.
* But mostly it means "Scanned by a peon"

Blogger The Other Robot June 28, 2016 11:29 AM  

@49: A quick search dud not turn that paper up.

Anonymous Jack Amok June 28, 2016 11:32 AM  

...overwhelmed with the need to publish in such high volumes...

Cranking out tons of cheap, crappy science like some Chinese sweatshop churning out Happy Meal toys. And we're supposed to respect the intellect of people in this profession?

Blogger exfarmkid June 28, 2016 11:34 AM  

VD: "We have a word for science that is trustworthy, and that word is engineering. Until science can be applied, it cannot be fully trusted to be correct."

Technology has a tendency to time-lead scientific understanding.

OTOH, let us also recall how aggressively inventors/investors compete over finding marketable applications from scientific findings which look juicy. Maybe journals should be required to note which of their research papers get cited in patents?

Anonymous Quartermaster June 28, 2016 11:39 AM  

@50
Of course, given what has been seen with Tor, you can't trust the publishing houses not to print sewage.

Anonymous Be Not Afraid June 28, 2016 11:44 AM  

>But the thing about astronomy is that it is expensive.

It is, but there's little money to be made *doing* astronomy, so as a field, it's in good shape. NASA, in fact, has a laudable practice of making raw and calibration data public in many cases (Hubble data archive, etc.). This is not agency-wide; there's NASA's GISS (the climatology folks) which IIRC keeps their methods hidden.

The corruption of money comes up in fields like medicine, where a successful drug trial is worth billions; or climate "science," where control of the entire world economy (and of the the lives of the peasants) is at stake.

Blogger clk June 28, 2016 11:53 AM  

VD says "We have a word for science that is trustworthy, and that word is engineering. Until science can be applied, it cannot be fully trusted to be correct"

While those of us in the engineering field appreciate this high regard you hold for us (and I realize its a exclusive/limited club of those disciplines that you respect :) ) .. if you limit your science to that which only can be applied you have pretty much limited yourself to the macroscopic world existing on a thin life supporting layer of the earth, and science studied only up to the 1940's or so ... the really interesting science of things really small -- (quantum, particle physics...) and the things on scales really big (like astrophysics) and theoretical areas like advanced maths are hardly likely to be applied within any readers lifetime... so such a test for validity is practically useless other than a rhetorical bat to hit the practitioners of biology on the head .. which I am fine with ...

I have always wanted to ask this and i don't think you have ever elaborated on this over the years ... with a Dad as a engineer (and an MIT educated one at that..which is top 1%) statistically there should have been a higher chance of you being one as well (most engineers I know are children of former engineers and/or scientists) and being high IQ it would have been a natural path -- so why not you ? ...

Blogger residentMoron June 28, 2016 11:53 AM  

The Other Robot wrote:@49: A quick search dud not turn that paper up.

Sorry, mate. I am 80% thru Castalia House's new book - "The Missionaries" - and that's the title of a paper in the book. But I'm lying in a hospital bed cranked full of opiates (me, not the bed) so my memory is a bit hazy. I know that's not the right title but I can't recall exactly what it was.

Anonymous Mr. Rational June 28, 2016 12:02 PM  

VD wrote:Despite its flaws, eyewitness testimony is statistically and empirically more reliable than scientific evidence.
Assertion without evidence.  How can a mistaken eyewitness identification be re-run for confirmation?

Jack Amok wrote:Cranking out tons of cheap, crappy science like some Chinese sweatshop churning out Happy Meal toys. And we're supposed to respect the intellect of people in this profession?
Victims of a broken system, much like the biologists whose findings support HBD but dare not put them into language accessible to the shrieking hate mobs of the academic left.

If more papers were tossed for methodological errors, there would be fewer but better papers.  The system has routes to reform.

clk wrote:the really interesting science of things really small -- (quantum, particle physics...) and the things on scales really big (like astrophysics) and theoretical areas like advanced maths are hardly likely to be applied within any readers lifetime...
You obviously haven't done much study of technology.  Modern electronics exploits the heck out of the QM properties of electrons, and that's just one thing.  Advanced maths are the basis of public-key cryptography.

@61 I don't know whether to offer sympathies or be jealous.

Blogger RobertT June 28, 2016 12:04 PM  

I'm not entirely certain what peer is supposed to entail, but at my firm all our output goes through peer review before going to the client. This was also the case at AA & Andersen Consulting/Accenture. It doesn't matter how good the intentions are, people simply reviewing the work miss all kinds of things. And the kinds of things generally the more tedious, involved or complex. Every firm deals with this in their own way. So as far as I'm concerned, scientific peer review is meaningless. I certainly wouldn't stake my life or reputation on perfect peer review.

Anonymous Jack Amok June 28, 2016 12:25 PM  

If more papers were tossed for methodological errors, there would be fewer but better papers. The system has routes to reform.

If if and buts were fruits and nuts...

The system is broken. Nobody is making any effort to reform it, or even replace it. It's going to die of it's own. What will take it's place?

Blogger exfarmkid June 28, 2016 12:52 PM  

63. RobertT

If I was the only person on staff who could solve a particular problem and if constrained from subcontracting external technical review, I had to schedule extra time in my proposals so that I could leave the report for a week or two, look at it again with fresh eyes, and catch my own stupid mistakes. And said mistakes always exist.

And not to put too fine a point on it, my personal experience taught me that the average reviewer is on the lazy side. Anyone with diligence and reasonable intelligence should be able to go outside their comfort zone at bit and at least be able to ask things like "what did the customer want?" and "can you show me how you delivered?"

Blogger Lobo Util June 28, 2016 12:55 PM  

I would imagine the pure math papers don't have this problem.

Blogger residentMoron June 28, 2016 1:12 PM  

"@61 I don't know whether to offer sympathies or be jealous."

Both is OK, too. ;)

Legal highs of this quality don't come around too often. That's probably a good thing considering what I had to go through to get here.

Anonymous Stickwick June 28, 2016 1:48 PM  

rho: I forgot which star it was, but they recently found one that had something very strange going on that some wags were suggesting a potential Dyson sphere. Anything that doesn't fit the expected model of regular, periodic dimming doesn't go into the exoplanet bucket.

That’s correct. It has to fit a specific pattern to be considered an exoplanet candidate. The oddly-behaving star that behaves as though it has a lot of debris around it, nobody has a clue what it is, and they admit as much.

Gravity waves, who knows? But as far as I've read, astrophysicists find the experimental results interesting and hopeful, but not definitive. I imagine Stickwick has the insider viewpoint, I'm just an enthusiast.

The physics community is confident of the gravity wave results. This will likely earn the key scientists on the LIGO team a Nobel prize.

clk: the really interesting science of things really small -- (quantum, particle physics...) and the things on scales really big (like astrophysics) and theoretical areas like advanced maths are hardly likely to be applied within any readers lifetime

Not true. As someone else already pointed out, without quantum physics, we would not have the explosion of high tech, nor would we have the many applications of spectroscopy.

And you can thank large-scale astrophysics for GPS. Einstein’s general relativity, which is largely the product of deductive reasoning, predicts changes in the flow of time due to relative gravity. GR has been confirmed many times over by astronomical observations. Without this very specific knowledge and application to satellite technology, GPS would be useless.

What many people don’t realize is that knowledge gained from astrophysics has practical applications. This is because the universe is a laboratory in which “experiments” can be conducted in extreme conditions that cannot currently, and may never, be replicated on earth.

This doesn’t include what’s called technology transfer, in which technology developed specifically for application to astronomical research turns out to have practical applications for things like medicine, communications, and aerospace.

Blogger Locke Demosthenes June 28, 2016 2:37 PM  

Mods removed the thread. Here's a link to it archived: https://r.go1dfish.me/r/AskReddit/comments/4q6q8s/serious_scientists_of_reddit_whats_craziest_or/d4qninl

Blogger Alexandros June 28, 2016 6:22 PM  

If you think that's bad, look up the story of a study called "cuckoo for coco puffs?" being accepted into a research journal.

Anonymous Bukulu June 29, 2016 3:09 AM  

"All peer review is really designed to do is to reassure the reader that the information presented fits safely within the confines of the consensus status quo."

I like to put it this way -- the question that the peer reviewer is trying to answer for the publisher is this: Will We Look Dumb If We Publish This?

That's it.

Blogger Pteronarcyd June 29, 2016 3:05 PM  

Peer review is not part of the scientific method. Publishing one's findings is so one's peers may read it and weigh in is, but "peer review" is simply gate keeping for the editor too lazy to read all submittals to his journal.

Blogger Pteronarcyd June 29, 2016 3:06 PM  

Peer review is not part of the scientific method. Publishing one's findings is so one's peers may read it and weigh in is, but "peer review" is simply gate keeping for the editor too lazy to read all submittals to his journal.

Blogger Jehu July 06, 2016 1:09 PM  

John Q Public thinks a lot of things that aren't so, but would be a hell of a lot better if they were. In the case of peer review, he thinks that a bunch of guys with similar expertise review everything about the paper and replicate any experiments done in it. In the case of Social Security, he thinks its like a 401k with a separate account for him being kept with forced contributions. If you pressed him, he'd say it was probably in T-bills.

Post a Comment

Rules of the blog
Please do not comment as "Anonymous". Comments by "Anonymous" will be spammed.

<< Home

Newer Posts Older Posts