non elite journals PDF

Nom original: non elite journals.pdf

Titre: broad-non-elite-total.eps

Auteur: acha

Ce fichier au format PDF 1.4 a été généré par gnuplot 4.4 patchlevel 3 / dvips + GPL Ghostscript GIT PRERELEASE 9.08, et a été envoyé sur Fichier-PDF le 25/10/2014 à 19:03, depuis l'adresse IP 212.195.x.x. La présente page de téléchargement du fichier a été vue 1288 fois.
Taille du document: 105 Ko (11 pages).
Confidentialité: fichier public

Aperçu du document

Rise of the Rest: The Growing Impact of Non-Elite Journals
Anurag Acharya, Alex Verstak, Helder Suzuki, Sean Henderson,
Mikhail Iakhiaev, Cliff Chiung Yu Lin, Namit Shetty

arXiv:1410.2217v1 [cs.DL] 8 Oct 2014

Google Inc.
October 9, 2014
Abstract
In this paper, we examine the evolution of the impact of non-elite journals. We attempt to
answer two questions. First, what fraction of the top-cited articles are published in non-elite
journals and how has this changed over time. Second, what fraction of the total citations are
to non-elite journals and how has this changed over time.
To answer these questions, we studied citations to articles published in 1995-2013. We
computed the 10 most-cited journals and the 1000 most-cited articles each year for all the 261
subject categories included in Scholar Metrics. We considered the 10 most-cited journals in a
category as the elite journals for the category and all other journals in the category as non-elite.
There are two main conclusions from our study. First, the fraction of highly-cited articles
published in non-elite journals increased steadily over 1995-2013. While the elite journals still
publish a substantial fraction of high-impact articles, many more authors of well-regarded papers
in a diverse array of research fields are choosing other venues.
Our analysis indicates that the number of top-1000 papers published in non-elite journals
for the representative subject category went from 149 in 1995 to 245 in 2013, a growth of 64%.
Looking at broad research areas, 4 out of 9 broad areas saw at least one-third of the top-cited
articles published in non-elite journals in 2013. All broad areas of research saw a growth in
the fraction of top-cited articles published in non-elite journals over 1995-2013. For 6 out of 9
broad areas, the fraction of top-cited papers published in non-elite journals for the representative
subject category grew by 45% or more.
Second, now that finding and reading relevant articles in non-elite journals is about as easy as
finding and reading articles in elite journals, researchers are increasingly building on and citing
work published everywhere. Considering citations to all articles, the percentage of citations to
articles in non-elite journals went from 27% of all citations in 1995 to 47% in 2013. Six out
of nine broad areas had at least 50% of total citations going to articles published in non-elite
journals in 2013.

1

Introduction

Several factors have driven the sustained impact of elite journals. First, these journals have traditionally been available in many more academic libraries worldwide. The costs of physical distri1

bution and storage required due to print publication meant only selected journals would be widely
available. Articles published in these journals had a much higher likelihood of being read, built
upon and cited.
Second, success metrics in scholarly communication had largely been computed at the journallevel. Those of us who have been on the academic job market well know that our resumes were
likely to be summarized as “X articles in the top journals”. As a result, researchers usually target
elite journals for their best work.
Third, literature research approaches had primarily been either browsing journal issues or scanning reverse chronological search results (most-recent-first). These approaches present researchers
with a large number of articles to scan and require substantial effort to track down relevant articles.
As a result, researchers had been more likely to limit the scope of their literature search to elite
journals.
There have been several dramatic changes in scholarly communication over the last two decades
that have the potential to significantly influence these factors. First, scholarly journals have largely
moved from physical distribution of print issues to online availability of individual articles. A
large number of journals have also digitized older articles and made them available online. Many
publishers and aggregators provide large collections as a part of Big Deal licenses. As a result, it
is easier for many more libraries to provide access to publications beyond a core collection of elite
journals.
Second, success metrics for researchers have expanded to include article-level metrics. These
include per-article citation counts as well as aggregate metrics such as the h-index [6]. Furthermore,
these metrics are widely available to all users without a subscription – which makes it easier for
both researchers and those considering their resumes to view these metrics. This allows researchers
to highlight the success of their impactful articles no matter where they are published.
Third, search services now cover all available journals, instead of a selected subset. Furthermore,
they index the entire text of articles instead of just abstracts and keywords. The common ranking
approach has moved from reverse chronological to relevance ranking (most-relevant-first). Finding
and reading relevant articles in non-elite journals is now about as easy as finding and reading
articles in elite journals.
To understand the influence of these changes on the impact of non-elite journals, we studied
citations to articles published in 1995-2013. We attempted to answer two questions. First, what
fraction of the top-cited articles are published in non-elite journals and how has this changed over
time. This covers the impact of the most visible papers, which are often the ones that make key
contributions. Second, what fraction of the total citations to all articles are to non-elite journals
and how has this changed over time. This covers the impact of all papers.
We computed the 10 most-cited journals and the 1000 most-cited articles each year for all the
261 subject categories included in Scholar Metrics [10]. We considered the 10 most-cited journals in
a category as the elite journals for the category and all other journals in the category as non-elite.
There are two main conclusions from our study. First, the fraction of highly-cited articles

2

published in non-elite journals increased steadily over 1995-2013. While the elite journals still
publish a substantial fraction of high-impact articles, many more authors of well-regarded papers
in a diverse array of research fields are choosing other venues.
Our analysis indicates that the number of top-1000 papers published in non-elite journals for the
representative subject category went from 149 in 1995 to 245 in 2013, a growth of 64%. Looking at
broad research areas, 4 out of 9 broad areas saw at least one-third of the top-cited articles published
in non-elite journals in 2013. All broad areas of research saw a growth in the fraction of top-cited
articles published in non-elite journals over 1995-2013. For 6 out of 9 broad areas, the fraction
of top-cited papers published in non-elite journals for the representative subject category grew by
45% or more.
Second, now that finding and reading relevant articles in non-elite journals is about as easy
as finding and reading articles in elite journals, researchers are increasingly building on and citing
work published everywhere. Considering citations to all articles, the percentage of citations to
articles in non-elite journals went from 27% of all citations in 1995 to 47% in 2013. Six out of nine
broad areas had at least 50% of total citations going to articles published in non-elite journals in
2013.

2

Methods

For this study, we included all journals and conferences that were assigned to one or more subject
category in the 2014 release of Scholar Metrics. The Scholar Metrics inclusion criteria for publication
venues were [4]: (1) publish 100 or more articles over 2009-2013, (2) at least one article must receive
at least one citation over 2009-2013, (3) follow Google Scholar indexing guidelines. Scholar Metrics
limits categorization into subject categories to English publications. Accordingly, this study covers
all the English language journals and conferences included in Scholar Metrics. Scholar Metrics also
includes selected preprint repositories. Preprint repositories are not included in this study.
We used the subject categories from the 2014 release of Scholar Metrics. We created a group of
articles for each subject-category-year combination, such as Immunology for the year 2000. Each
category-year group included all articles published in the given year in all publications in the given
category.
For each publication, we included all articles with a publication date within 1995-2013, both
inclusive. Note that each journal or conference can be associated with more than one subject
category. Such publications are included in the computation for each category they are a part of.
We identified the 10 most-cited journals for each category-year group. For this, we used the
ordering mechanism used in Scholar Metrics [4]: journals are sorted by their h5-index with ties
being broken by their h5-median.1 These journals were considered as the elite journals for the
1

The h5-index of a publication is the largest number h such that at least h articles published in the last five
complete calendar years in that publication were cited at least h times each. The h5-core of a publication is a set
of top cited h articles from the publication that were published in the last five complete calendar years. These are
the articles that the h5-index is based on. The h5-median of a publication is the median of the citation counts in its

3

group. All other journals in the group were considered non-elite. Note that the list of elite journals
was recomputed each year which allowed the analysis to capture changes in the focus of subject
categories as well as newly successful journals.
Next, we computed the list of the 1000 most-cited articles in each category-year group. These
were considered the top-cited articles in the category-year group. We determined how many of
these top-cited articles were published in non-elite journals.
Given the large number of subject categories under study (261), we grouped subject categories
into broad research areas. We used the broad areas from Scholar Metrics for this, with one change
— we separated Engineering and Computer Science. The citation patterns in these two areas are
significantly different and this separation allowed us to explore the differences. We also added All
articles as the union of all broad areas.
For each year, we sorted the subject categories in each broad area by the number of top-cited
articles published in non-elite journals. We then picked the median subject category in each broad
area as the representative for the area — roughly half the subject categories in the area would
have more top-cited articles published in non-elite journals than the representative and roughly
half would have fewer such articles. Note that we recomputed the representative category for each
year to ensure that we picked the middle point in the list of subject categories at all times.
We picked the median subject category as the representative instead of computing an average
across all categories to limit distortions due to outliers. In addition to selecting a representative
category, we also computed the 25th and the 75th percentile categories for each broad area in
each year. These are the categories whose number of non-elite top-cited articles was larger than or
equal to that for 25% and 75% of all the categories in the area, respectively. The results for these
categories help us get an idea of how the number of top-cited articles in non-elite journals varied
across the entire set of subject categories in an area.
Finally, we computed the number of citations to all articles in a category-year group as well as
the number of citations to all articles in the group that were published in non-elite journals.

3

Results

Figure 1 presents the trend in publication of top-cited papers. It shows that the fraction of topcited papers published in non-elite journals has grown steadily over 1995-2013. The graphs for
the representative subject category as well as those for the 25th and 75th percentile categories
are similar. This indicates that the trend of growth in fraction of top-cited papers published in
non-elite journals holds across a wide range of subject categories.
Figure 2 presents the trend in publication of top-cited papers for individual broad areas. It
shows that all areas saw growth in the fraction of top-cited papers published in non-elite journals;
6 out of 9 broad areas seeing a substantial increase.
Table 1 presents the number of 2013 top-cited papers published in non-elite journals for the
h5-core. The h5-median is a measure of the distribution of citations to the articles in the h5-core.

4

0.4
75%
median
25%
0.35

Fraction of top-cited articles

0.3

0.25

0.2

0.15

0.1

0.05
1995

2000

2005
Publication year

2010

2013

The line marked median presents the data for the representative subject category in each year. Lines
marked 25% and 75% represent results for the 25th and 75th percentile categories, respectively.

Figure 1: Fraction of top-cited papers published in non-elite journals over 1995-2013.
representative subject category in each broad area. It also presents the change since 1995. The
change over 1995-2013 is computed as a percentage:
(num non elite in 2013 − num non elite in 1995)/num non elite in 1995 ∗ 100
It shows that the number of top-1000 papers published in non-elite journals for the overall
representative subject category went from 149 in 1995 to 245 in 2013, a growth of 64%. Four out
of nine broad areas saw at least one-third of the top-cited articles published in non-elite journals
in 2013. All broad areas of research saw a growth in the fraction of top-cited articles published
in non-elite journals over 1995-2013. For 6 out of 9 broad areas, the fraction of top-cited papers
published in non-elite journals for the representative subject category grew by 45% or more.
Figure 3 presents the fraction of citations to non-elite journals over 1995-2013. Note that this
is computed by dividing the sum of citations to all articles in non-elite journals by the sum of
citations to articles in all journals. The fraction of non-elite citations went from 27% in 1995 to
47% in 2013. The graph shows growth over the entire period, the growth rate being lower in the
first third of the period under study (1995-2000) and higher over the rest of the period (2001-2013).
Figure 4 presents the evolution of the fraction of non-elite citations for all broad areas. It shows
that all broad areas saw a significant increase in the fraction of non-elite citations over 1995-2013.

5

0.55

0.3
75%
median
25%

75%
median
25%

0.25

0.45

Fraction of top-cited articles

Fraction of top-cited articles

0.5

0.4
0.35
0.3
0.25
0.2
0.15

0.2

0.15

0.1

0.05

0.1
0.05
1995

2000

2005
Publication year

2010

0
1995

2013

(a) Physics & Mathematics

2005
Publication year

2010

2013

(b) Health & Medical Sciences

0.3

0.5
75%
median
25%

0.45
Fraction of top-cited articles

0.25
Fraction of top-cited articles

2000

0.2

0.15

0.1

0.05

75%
median
25%

0.4
0.35
0.3
0.25
0.2
0.15

0
1995

2000

2005
Publication year

2010

0.1
1995

2013

(c) Chemical & Material Sciences

2000

2005
Publication year

2010

2013

2010

2013

(d) Computer Science

0.55
75%
median
25%

0.3

0.45

Fraction of top-cited articles

Fraction of top-cited articles

0.5

0.4
0.35
0.3
0.25
0.2

75%
median
25%

0.25
0.2
0.15
0.1
0.05

0.15
0.1
1995

2000

2005
Publication year

2010

0
1995

2013

(e) Business, Economics & Management

2000

2005
Publication year

(f) Engineering

Figure 2: Per-area changes in the fraction of top-cited papers published in non-elite journals over
1995-2013.

6

0.5

0.4
0.35
0.3
0.25

0.25

0.2

0.15

0.1

0.2
0.15
1995

75%
median
25%

0.3
Fraction of top-cited articles

Fraction of top-cited articles

0.45

0.35
75%
median
25%

2000

2005
Publication year

2010

0.05
1995

2013

(g) Social Sciences

2005
Publication year

2010

2013

(h) Life Sciences & Earth Sciences
75%
median
25%

0.55
Fraction of top-cited articles

2000

0.5

0.45

0.4

0.35

0.3
1995

2000

2005
Publication year

2010

2013

(i) Humanities, Literature & Arts

Figure 2: Per-area changes in the fraction of top-cited papers published in non-elite journals over
1995-2013.

7

Broad area

Top-cited non-elite in 2013

Change since 1995

289
192
108
345
174
333
349
177
414
245

204%
98%
80%
72%
63%
45%
18%
18%
6%
64%

Physics & Mathematics
Health & Medical Sciences
Chemical & Material Sciences
Computer Science
Engineering
Business, Economics & Management
Social Sciences
Life Sciences & Earth Sciences
Humanities, Literature & Arts
All articles

The numbers for each broad area are from the representative subject category for the area.

Table 1: Change in the number of top-1000 papers published in non-elite journals over 1995-2013.
Table 2 presents the per-area growth in numerical form, for ease of comparison. It shows that
six out of nine broad areas had at least 50% of total citations going to articles published in non-elite
journals in 2013. Furthermore, 8 out of 9 broad areas saw an increase of 40% or more in the fraction
of citations to non-elite journals over 1995-2013.

4

Related Work

The idea that a small core set of journals covers most of the key papers in a discipline has long
been prevalent in the study of scholarly communication. For example, see [1, 2, 3] or do a query
for “core journals” in article titles on Google Scholar [5].
Larivi`ere et al [7] examined the concentration of citations at the article level over 1900-2007
and concluded that while distributions of citations remained highly skewed, the fraction of citations
to highly cited articles has been decreasing. Looking at broad research areas, they found that for
Natural Sciences & Engineering and Medicine, the decrease in concentration started around 1990.
Lozano et al [9] studied the relationship between impact factor and article citations. They
examined three broad categories, Natural Sciences & Medicine, Physics and Social Sciences, over
1902-2009 and found that the correlation between impact factor of the journal and the citations
its articles receive has been weakening since around 1990. In another part of their study, they
computed the fraction of 10% most-cited articles that appear in the 10% highest impact-factor
journals. For this, they focused on Natural Sciences & Medicine. They found that a decreasing
fraction of most-cited articles is being published in the most-cited journals.
In a follow-up study [8], they took a closer look at 13 journals, seven traditionally top-ranked
journals and six upcoming journals. They found that since around 1990, the fraction of most-cited
papers published in the traditionally top-ranked journals has been dropping and the fraction of

8

0.5

Fraction of citations to non-elite journals

0.45

0.4

0.35

0.3

0.25
1995

2000

2005
Publication year

2010

2013

Figure 3: Fraction of citations to non-elite journals over 1995-2013.
0.6

Fraction of citations to non-elite journals

0.55

hum
bus
soc
phy
bio

chm
eng
med
cs

0.5

0.45

0.4

0.35

0.3

0.25

0.2

0.15
1995

2000

2005
Publication year

2010

2013

bio: Life Sciences & Earth Sciences; bus: Business, Economics & Management; cs: Computer Science;
chm: Chemical & Material Sciences; eng: Engineering; hum: Humanities, Literature & Arts; med:
Health & Medical Sciences; phy: Physics & Mathematics; soc: Social Sciences

Figure 4: Fraction of citations to non-elite journals for broad areas of research.

9

Broad area

Non-elite citations in 2013

Change since 1995

50%
43%
51%
30%
50%
53%
41%
53%
56%
47%

133%
95%
83%
67%
52%
51%
46%
42%
22%
74%

Computer Science
Life Sciences & Earth Sciences
Health & Medical Sciences
Physics & Mathematics
Engineering
Business, Economics & Management
Chemical & Material Sciences
Humanities, Literature & Arts
Social Sciences
All articles

Table 2: Change in the fraction of citations to non-elite journals over 1995-2013.
such papers published in the upcoming journals has been increasing.
The goal of our study is similar to those of Larivi`ere et al [8, 9]. We have examined the impact
of non-elite journals for a large number of specific research fields. Structuring such a study to
consider individual research fields separately is necessary since: (1) usually only the journals in
a specific field are the possible venues for the highly-cited articles in the field, (2) there are large
differences in the citation frequency in different fields and grouping a large number of research fields
usually results in the fields with higher citation frequency dominating the results. That said, our
results are complementary to theirs and point in the same direction.

5

Conclusions

There are two main conclusions from our study. First, the fraction of highly-cited articles published
in non-elite journals increased steadily over 1995-2013. While the elite journals still publish a
substantial fraction of high-impact articles, many more authors of well-regarded papers in a diverse
array of research fields are choosing other venues.
Our analysis indicates that the number of top-1000 papers published in non-elite journals for the
representative subject category went from 149 in 1995 to 245 in 2013, a growth of 64%. Looking at
broad research areas, 4 out of 9 broad areas saw at least one-third of the top-cited articles published
in non-elite journals in 2013. All broad areas of research saw a growth in the fraction of top-cited
articles published in non-elite journals over 1995-2013. For 6 out of 9 broad areas, the fraction
of top-cited papers published in non-elite journals for the representative subject category grew by
45% or more.
Second, the fraction of citations to articles published in non-elite journals has grown substantially over most research areas. The percentage of citations to articles in non-elite journals went
from 27% of all citations in 1995 to 47% in 2013. Six out of nine categories had at least 50% of

10

total citations going to articles published in non-elite journals in 2013.
Now that finding and reading relevant articles in non-elite journals is about as easy as finding
and reading articles in elite journals, researchers are increasingly building on and citing work
published everywhere.

References
[1] Samuel C Bradford. Sources of information on specific subjects. Engineering, 137:85–86, 1934.
[2] Eugene Garfield. Citation analysis as a tool in journal evaluation. Science, 178:471, 1972.
[3] Eugene Garfield. Significant scientific literature appears in a small core of journals. The
Scientist, 10:13, 1996.
[4] Google Scholar Metrics help page. http://scholar.google.com/intl/en/scholar/metrics.html,
2014.
[5] Google
Scholar
query
for
“core
journals”
http://scholar.google.com/scholar?q=intitle:“core+journals”, 2014.

in

article

titles.

[6] Jorge E Hirsch. An index to quantify an individual’s scientific research output. Proceedings of
the National academy of Sciences of the United States of America, 102(46):16569–16572, 2005.
´
[7] Vincent Larivi`ere, Yves Gingras, and Eric
Archambault. The decline in the concentration of
citations, 1900–2007. Journal of the American Society for Information Science and Technology,
60(4):858–862, 2009.
[8] Vincent Larivi`ere, George A Lozano, and Yves Gingras. Are elite journals declining? Journal
of the Association for Information Science and Technology, 65(4):649–655, 2014.
[9] George A Lozano, Vincent Larivi`ere, and Yves Gingras. The weakening relationship between
the impact factor and papers’ citations in the digital age. Journal of the American Society for
Information Science and Technology, 63(11):2140–2145, 2012.
[10] Helder Suzuki. 2014 Scholar Metrics Released. http://googlescholar.blogspot.com/2014/06/2014scholar-metrics-released.html, 2014.

11