What Twitter May Be Able to Tell Us (in Advance) about Citations

7 June 2013 by Matt Shipman, posted in Uncategorized

Image: Roger Winstead

Social media platforms allow people to exchange information, including scientific information. That’s one reason many scientists are active on social media. I just read a paper (not new, but new to me) that suggests social media – particularly Twitter – may actually also serve as something of a crystal ball for predicting the scientific impact of journal articles.

I read a recent post by entomology researcher Cameron Webb on whether social media can increase the exposure of newly-published research. (It’s a great post, which I encourage you to read). Early in his piece, Webb mentions a 2011 article published in the Journal of Medical Internet Research (JMIR). I clicked through to the article itself, and it’s pretty interesting.

Here’s the key point: “Tweets can predict highly cited articles within the first 3 days of article publication. Social media activity either increases citations or reflects the underlying qualities of the article that also predict citations.” However, the study author, Dr. Gunther Eysenbach, does note that “one should neither expect nor hope for perfect correlation.”

What the paper looked at

Eysenbach collected all tweets that linked directly to articles in JMIR from July 2008 to November 2011. He then pulled out a subset of those tweets. Specifically, he pulled out the 1,573 tweets that included links to any of the 55 articles published in JMIR between July 22, 2009, and June 30, 2010. Those 55 articles got an average of 13.9 “tweetations” (tweets that included a direct link to the journal article) within 7 days of publication. 49.8 percent of those tweetations were sent in the first two days of an article’s publication.

Eysenbach also got the citation counts for those 55 articles, as of November 2011, from both Scopus and Google Scholar. The articles got an average of 7 citations on Scopus, and 13 on Google Scholar.

Eysenbach also came up with several new metrics for measuring an article’s activity on Twitter, including one called the “twimpact factor.” (I know what you’re thinking – the names sound a little cutesy to me too.) Twimpact factor (twn) “is defined as the cumulative number of tweetations within n days after publication.” So, for example, tw7 would be the total number of tweetations seven days after publication.

What the paper found

In looking at tweetations, citations, twimpact factor and his other new metrics, Eysenbach found that there was a correlation between a journal article’s Twitter metrics and its citations. Specifically, Eysenbach states that highly-tweeted articles are 11 times more likely to end up being highly cited. [If you want to see the charts and read the statistical analyses, you can read the paper.]

Here Eysenbach (wisely, in my opinion) sounds a note of caution. “Correlation is not causation, and it harder [sic] to decide whether extra citations are a result of the social media buzz, or whether it is the underlying quality of the article of an article or newsworthiness that drives both the buzz and the citations.” [Note: this is the so-called “earmark hypothesis,” which I’ve written about before in the context of whether news stories boost citations.]

But Eysenbach is also quick to point out that “it is not inconceivable that exposure on Twitter leads to a few extra citations….many scientists see the value of Twitter in being a constant live literature alert service crowdsourced from peers.”

Questions

I have a couple questions about these results. First, I’d like to see whether these trends hold true for other journals. As Eysenbach says himself in the paper, “as a journal about the internet and social media [JMIR]…has a sophisticated readership that is generally ahead of the curve in adopting Web 2.0 tools.” So, would “tweetations” be a good predictor of citations for articles published in journals that focus on a less social media-savvy audience? I have no idea, but I’m curious.

Second, this paper looks at data from 2008 through 2011. Given the dynamic nature of social media, I’d be curious to see what has changed since then. Are more scientists using Twitter now than were using Twitter in November 2011? I’m guessing that the answer is yes, but I’m only guessing. And, if there are more scientists on Twitter, has that strengthened or weakened any correlation between tweetations and citations?

Journal publishers and science communication scholars – there’s a paper there for somebody.

Update (7:08 a.m. EST, June 7): Also, please read this post from The Scholarly Kitchen critiquing the Eysenbach study. (Worth noting that the version of the paper I worked from is the *corrected* version.) Excellent observations here from Phil Davis. And check out the comments on that post too. Given the lag between the paper’s publication (Dec. 2011) and when I wrote the post, I looked for other posts on the paper – but couldn't find them. Thanks to Tom Webb for sending this my way!


2 Responses to “What Twitter May Be Able to Tell Us (in Advance) about Citations”

  1. Paige Brown Jarreau Reply | Permalink

    I saw this post featured on SciLogs' Facebook page... and knew it was your post before I even saw the author. :)

    Cool stuff!

Leave a Reply


nine − 3 =