Meyer revisited
This is the write-up of a talk I gave at the SciCAR conference in Dortmund on September 7, 2017.
The organizers of the conference asked me to prepare a talk called “Meyer Revisited”. Philip Meyer is well-known in datajournalism circles. He was among the first ones to use the methods of social science for journalistic purposes - and it turned out to be a great idea. That was close to fifty years ago, so it might be time to take a modern look at the ground-breaking book Meyer wrote in the late 1960’s and revamped in a new edition in 1991, Precision Journalism.
To prepare this talk, I re-read Meyer’s book. Long story short: the book’s still good, but some of the assumptions Meyer made about the impact of datajournalism were wrong, and we need to address them.
It’s still fresh
OK, some chapters of the book clearly are outdated. When Meyer lists spreadsheet software, for instance (“Lotus, SuperCalc, PC-Calc”) or when he explains how data storage on magnetic tapes works1, the book does feel antiquated.
Much of the non-technical parts are still useful. The workflow Meyer describes in his first chapter (“Collect, Store, Retrieve, Analyze, Reduce and Communicate data”) is the one we teach in journalism schools today, though we now put much less emphasis on data storage, retrieval and reduction because computers let us store and sort data fairly easily. Meyer’s concept of “models” remains very current, too. In a nutshell, he argues that the underlying assumptions about an issue define how data will be collected and, ultimately, what analyses we’ll be able to do with the data.
The importance of models was shown once again last month when Reuters published an investigation on people who died following a taser strike, to take just one example.2 Axon, the company that manufactures the stun guns, measured the issue in a certain way, which led to a tally of 24 deaths since the early 2000’s. By changing the model, or the way the issue was measured, Reuters came to a total of 1005 deaths. The difference lies in how one decides that the taser caused the death. It turns out that Axios tends to sue the coroners who write down “taser” as the cause of death.3 As a result, a tally based on causes of death will be very low. Reuters changed the model and came closer to the truth.
Precision journalism meets academia
What I found most interesting in Meyer’s book wasn’t his advice on how to do datajournalism - that’s still very good. More interesting are the assumptions he made when he wrote. Meyer expected precision journalism to be widely adopted because it allows journalists to advance towards the truth much more efficiently than anything else.4 In that, he is right. The scientific method is, by far, the most efficient way to produce factual truth. When it is replicable and falsifiable, a result obtained from datajournalism is much stronger and useful than a story that was obtained in another way. Compare that with a work that relies on anonymous sources or on a limited number of testimonies. Let’s take a piece by Seymour Hersh, an American journalist and Pulitzer prize winner, as an example. Because it does not follow any scientific method, the article in which he claims that Osama Bin Laden was offered to the Americans by the Pakistani secret service5 can hardly be verified or replicated. As a result, whether Hersh is right or wrong is a matter of personal conviction. This kind of journalism might be good by many standards, but it does not bring us any closer to the truth. However, it helps us define new hypotheses or, as Meyer would say, find new models.
Meyer argued that precision journalism, because it was interested in factual truth and followed the scientific method, was bound to grow close to academia. In the 1991 version of the book, he pointed at the spread of classes in computer-assisted reporting in American journalism schools as proof that the alliance between the two was working.6
At Journalism++, an agency for data-driven journalism that I co-founded and ran between 2011 and 2017, we tried several times to work with academics. It wasn’t easy. The first roadblock was a different relation to time. As journalists, we try to publish rapidly, or to choose a publication date that will maximize impact (you don’t publish a piece on snow-making in June). Academics have different time scales. They, too, need to be fast, in order to be the first ones to put their names on a keyword, thereby ensuring a lot of citations later on and an increased h-index. (The h index is a measure of their importance among their peers which can have a crucial role in their careers.) However, although an academic might rush to publish, the rush will last more than a year, during which she can refuse that the journalists publish anything on their joint research. What if another academic reads the articles and publishes a paper under her own name first?
Another roadblock when working with academics is the relationship to data. Because an academic needs to obtain a p-value under 0.01, she has a tendency to test lots of hypotheses under a single project. This, in turns, leads to the collection of many data points for each entity in a sample. As journalists, we are interested in hypothesis testing, too, but we’re more interested in user happiness. A happy user is a user who shares the project on Facebook, and this is, for many of us, the single biggest measure of success. When a user has to give 30 pieces of information about herself before seeing the product, she is definitely not happy. If you want to feel the difference between the two approaches, you can compare a dialect quiz we made for the Swiss public service broadcaster in 2015, the Parlomètre, with a similar project done at the same time by a Swiss university.7
No love affair
I’m sure many of those who worked in joint projects between journalists and academics could come up with more stories. Of course, we could dismiss these as mere anecdotes, or we could consider such mishaps teething troubles that will be overcome as soon as both fields learn to collaborate.
Maybe that’s what will happen. But still. Journalism and academia have lots of reasons to work together. The clout of a media brand could help academics recruit participants in a study. An academic could benefit from a journalist’s skills to communicate a finding to the general public. Journalists working closely with academics would benefit from the possible scoops they might have when an experiment is completed. When it comes to looking for and communicating the truth, there are many objective reasons why journalists and academics should team up.
This has not happened since Meyer published Precision Journalism fifty years ago. I know of very few peer-reviewed articles co-authored by a journalist and a scientist, for instance. Some individuals clearly stand out. Many science journalists, such as Ben Goldacre in the United Kingdom or Susan E. McGregor in the United States, provide an interface between both worlds,8 but they are few and far between.
In the same vein, it takes time to think of joint experiments where newsrooms played an active role. Some projects bring together newsrooms and researchers, such as ContentCheck, a joint project between research labs and Le Monde, a French daily,9 or InVid, a browser plugin developed by Agence France Presse and a consortium of universities.10 Such projects are funded by national or European grants, with all the caveats that it implies. I could not find a joint experiment where organizations taking part were not subsidized.
Until today, we didn’t have a joint conference for both professions. Let’s hope that the efforts of the Volkswagen Stiftung, a German non-profit (no relation to the criminal car maker11), who organized it, will bear fruits and act as a catalyst to bring the two worlds closer together.
But in the current environment, where journalists and academics would do anything for a few thousand euros, grants are not catalysts. They are subsidies.
Changing the model
Maybe we could see clearer by changing the model, as Meyer would say. Maybe the assumption he made about the nature of academia and newsrooms were wrong. Let’s look at what goes well in the interactions between the two. When a university publishes a press release which is then taken up by thousands of media outlets without any substantial modification, the process is well-oiled and efficient. The university (sometimes the individual academic) wants exposure for the paper that was just published (or just written) and the newsroom is happy to receive content that cost nothing to produce and that is widely shared among readers.
When a university publishes a press release explaining that a single brand of chocolate drink helps recover from concussion, many newsrooms are happy to spread the word (research does not back that fact, by the way).12 Because the chocolate drink example already made the rounds last year, I set out to look for another one. It took me approximately fifteen seconds to browse through the “Science” section of a major French news magazine to find an article on a yet-to-be-published piece of research by two statisticians who make grandiloquent claims in biology (men cannot live more than 114.1 years, they contend).13 Of course, this nonsense spread after the University of Tilburg published a press release.14
Maybe, just maybe, we should consider the hypothesis that neither newsrooms nor academia are very much interested in finding the truth. This would explain why they rarely collaborate on truth-finding projects and cheerfully get together when it comes to spreading falsehoods. After all, for all the good vibes European journalists put in their declarations of principles, such as the Munich Declaration of the Duties and Rights of Journalists15, where truth comes in first place, I’ve yet to see newsrooms enforce any of it. In some European countries, egregious cases of falsification will get you fired, but this is by no means the case everywhere.16 Short of this, smaller cases of lying or republishing lies won’t hurt your career as a journalist.
Similarly, the incentives currently in place in academia do little to foster truth-seeking. Instead, researchers who want to succeed professionally need to publish a lot and fulfill grant contracts. Those who do best are the ones who will not hesitate to kill a result if it shows that the assumptions of the organization that funded the project were false. A successful researcher today is a researcher who will not let the truth come in the way of a good funding opportunity.17
Few journalists make finding factual truths their main goal and a dwindling number of academics do. Hopefully, the programs of the Volkswagen Stiftung will help these minorities find each other and conduct interesting projects. But we need to review our expectations: As long as the incentives to produce and communicate untruths remain in place in academia and in the newsroom, Philip Meyer’s vision of academics and journalists working hand in hand will remain a mirage.
The questions and answers session brought to the fore several other examples of successful collaborations between academics and journalists. The Jupyter Notebook, a way to share code in an easy-to-read format, is a good example of a project steered by both academics and practitioners (not only journalists). The collaboration between investigative journalist Sam Roe and Lee Newman of the University of Colorado is also remarkable. But again, I don’t deny that there are hundreds of scientists and journalists doing great things, I just hypothesize that the incentives in place for the thousands of others are such that they probably won’t work together on truth-seeking projects.
Another participant argued that we should not call “journalism” the work of a newsroom that is not dedicated to finding or communicating the truth (similarly, we should not call “science” anything that is not aimed at coming closer to the truth). While I agree for “science” and prefer the term “academia”, I’d argue that “journalism” refers to a medium and not to a practice. Beyond the irrelevant lexicographical debate lies a strategic choice: Should one try to reform a newsroom to steer it towards making truth-seeking its core mission, or should one work outside of it? I don’t have an answer.
Notes
1. both examples are in Chapter 4, Computers.
2. See A 911 plea for help, a Taser shot, a death - and the mounting toll of stun guns.
3. See an example from Arizona: Judge rules for Taser in cause-of-death decisions.
4. Meyer never writes this directly, but, in Chapter 1, he equates journalism with truth-finding many times and then goes on to argue that precision journalism is more powerful to achieve that goal.
5. Read The Killing of Osama bin Laden.
6. Meyer writes in Chapter 1 that ‘The ready acceptance of [precision journalism] in academe was due in part to its contribution to the healing of the breach between the green eyeshade [the cap telegraphers wore] and chi-square [a statistical method] factions.’
7. Note that there was no collaboration between the two.
8. See the list of publications by Ben Goldacre and by Susan McGregor.
9. See ContentCheck.
10. See the InVID website.
11. The Volkswagen Stiftung is independent, but receive the dividends of the VW shares owned by the state of Lower-Saxony. As for the adjective, VW pleaded guilty to criminal charges, you can actually write this without fear of libel.
12. Read The University of Maryland just released a report on its incredibly irresponsible chocolate milk research.
13. The piece is here: Des chercheurs fixent les limites de la vie humaine entre 114 et 115 ans.
14. Find it here: The oldest human does not get any older.
16. German newsrooms have cases when reporters were fired after falsifications, such as the Hitler Diaries (see Ein Tiefpunkt des deutschen Journalismus: Die ‘Hitler-Tagebücher’ at DW) or the fake interviews in Neon (see Die Butter verriet ihn in taz). On the other hand, a French news anchor faked an interview with Fidel Castro in 1991 and remained in place 20 more years.
17. I wrote about this at length in The Collapse of Academia and Data-driven journalism in the post-truth public sphere. For better articles, read for instance Vox’s investigation, The 7 biggest problems facing science, according to 270 scientists.