Numerous critiques have been made regarding the use of impact factors, both in terms of their statistical validity and also of their implications for how science is carried out and assessed. Criticism of impact factors also extends to its impact on researcher behavior. While the emphasis on high-impact journals may lead to strategic publishing practices that prioritize journal prestige over the quality and relevance of research, it's important to acknowledge the "privilege paradox". Younger researchers, particularly those from under-represented regions, often lack the established reputation or networks to secure recognition outside of these metrics.
Inapplicability of impact factor to individuals and between-discipline differences It has been stated that impact factors in particular and citation analysis in general are affected by field-dependent factors which invalidate comparisons not only across disciplines but even within different fields of research of one discipline. The percentage of total citations occurring in the first two years after publication also varies highly among disciplines from 1–3% in the mathematical and physical sciences to 5–8% in the biological sciences. Thus impact factors cannot be used to compare journals across disciplines. Impact factors are sometimes used to evaluate not only the journals but the papers therein, thereby devaluing papers in certain subjects. In 2004, the
Higher Education Funding Council for England was urged by the
House of Commons Science and Technology Select Committee to remind
Research Assessment Exercise panels that they are obliged to assess the quality of the content of individual articles, not the reputation of the journal in which they are published. Other studies have repeatedly stated that impact factor is a metric for journals and should not be used to assess individual researchers or institutions.
Questionable editorial policies that affect the impact factor Because impact factor is commonly accepted as a proxy for research quality, some journals adopt editorial policies and practices, some acceptable and some of dubious purpose, to increase their impact factor. For example, journals may publish a larger percentage of
review articles which generally are cited more than research reports. Research undertaken in 2020 on dentistry journals concluded that the publication of "systematic reviews have significant effect on the Journal Impact Factor ... while papers publishing clinical trials bear no influence on this factor. Greater yearly average of published papers ... means a higher impact factor." Journals may also attempt to limit the number of "citable items"—i.e., the denominator of the impact factor equation—either by declining to publish articles that are unlikely to be cited (such as case reports in medical journals) or by altering articles (e.g., by not allowing an
abstract or
bibliography in hopes that Journal Citation Reports will not deem it a "citable item"). As a result of negotiations over whether items are "citable", impact factor variations of more than 300% have been observed. Items considered to be uncitable—and thus are not incorporated in impact factor calculations—can, if cited, still enter into the numerator part of the equation despite the ease with which such citations could be excluded. This effect is hard to evaluate, for the distinction between editorial comment and short original articles is not always obvious. For example, letters to the editor may be part of either class. Another less insidious tactic journals employ is to publish a large portion of its papers, or at least the papers expected to be highly cited, early in the calendar year. This gives those papers more time to gather citations. Several methods, not necessarily with nefarious intent, exist for a journal to cite articles in the same journal which will increase the journal's impact factor. Beyond editorial policies that may skew the impact factor, journals can take overt steps to
game the system. For example, in 2007, the specialist journal
Folia Phoniatrica et Logopaedica, with an impact factor of 0.66, published an editorial that cited all its articles from 2005 to 2006 in a protest against the "absurd scientific situation in some countries" related to use of the impact factor. The large number of citations meant that the impact factor for that journal increased to 1.44. As a result of the increase, the journal was not included in the 2008 and 2009
Journal Citation Reports.
Coercive citation is a practice in which an editor forces an author to add extraneous citations to an article before the journal will agree to publish it, in order to inflate the journal's impact factor. A survey published in 2012 indicates that coercive citation has been experienced by one in five researchers working in economics, sociology, psychology, and multiple business disciplines, and it is more common in business and in journals with a lower impact factor. Editors of leading business journals banded together to disavow the practice. However, cases of coercive citation have occasionally been reported for other disciplines.
Assumed correlation between impact factor and quality The journal impact factor was originally designed by Eugene Garfield as a metric to help librarians make decisions about which journals were worth indexing, as the JIF aggregates the number of citations to articles published in each journal. Since then, the JIF has become associated as a mark of journal "quality", and gained widespread use for evaluation of research and researchers instead, even at the institutional level. It thus has significant impact on steering research practices and behaviours. By 2010, national and international research funding institutions were already starting to point out that numerical indicators such as the JIF should not be considered as a measure of quality. In fact, research was indicating that the JIF is a highly manipulated metric, and the justification for its continued widespread use beyond its original narrow purpose seems due to its simplicity (easily calculable and comparable number), rather than any actual relationship to research quality. Empirical evidence shows that the misuse of the JIF—and journal ranking metrics in general—has a number of negative consequences for the scholarly communication system. These include gaps between the reach of a journal and the quality of its individual papers and insufficient coverage of social sciences and humanities as well as research outputs from across Latin America, Africa, and South-East Asia. Additional drawbacks include the marginalization of research in
vernacular languages and on locally relevant topics and inducement to unethical authorship and citation practices. More generally, the impact factor fosters a reputation economy, where scientific success is based on publishing in prestigious journals ahead of actual research qualities such as rigorous methods, replicability and social impact. Using journal prestige and the JIF to cultivate a competition regime in academia has been shown to have deleterious effects on research quality. A number of regional and international initiatives are now providing and suggesting alternative research assessment systems, including key documents such as the
Leiden Manifesto and the
San Francisco Declaration on Research Assessment (DORA).
Plan S calls for a broader adoption and implementation of such initiatives alongside fundamental changes in the scholarly communication system. As appropriate measures of quality for authors and research, concepts of research excellence should be remodelled around transparent workflows and accessible research results.
Negotiated values Results of an impact factor can change dramatically depending on which items are considered as "citable" and therefore included in the denominator. One notorious example of this occurred in 1988 when it was decided that meeting abstracts published in
FASEB Journal would no longer be included in the denominator. The journal's impact factor jumped from 0.24 in 1988 to 18.3 in 1989. Publishers routinely discuss with Clarivate how to improve the "accuracy" of their journals' impact factor and therefore get higher scores.
Distribution skewness [blue] and
PLOS ONE [orange]) are shown to represent a highly cited and less cited journal, respectively. The high citation impact of
Nature is derived from relatively few highly cited papers. Modified after Callaway 2016. Because citation counts have highly
skewed distributions, For example, about 90% of
Natures 2004 impact factor was based on only a quarter of its publications. Thus the actual number of citations for a single article in the journal is in most cases much lower than the mean number of citations across articles. Furthermore, the strength of the relationship between impact factors of journals and the citation rates of the papers therein has been steadily decreasing since articles began to be available digitally. The effect of outliers can be seen in the case of the article "A short history of SHELX", which included this sentence: "This paper could serve as a general literature citation when one or more of the open-source SHELX programs (and the Bruker AXS version SHELXTL) are employed in the course of a crystal-structure determination". This article received more than 6,600 citations. As a consequence, the impact factor of the journal
Acta Crystallographica Section A rose from 2.051 in 2008 to 49.926 in 2009, more than
Nature (at 31.434) and
Science (at 28.103). The second-most cited article in
Acta Crystallographica Section A in 2008 had only 28 citations. Critics of the JIF state that use of the
arithmetic mean in its calculation is problematic because the pattern of citation distribution is skewed and citation distributions metrics have been proposed as an alternative to impact factors. However, there have also been pleas to take a more nuanced approach to judging the distribution skewness of the impact factor. Ludo Waltman and Vincent Antonio Traag, in their 2021 paper, ran numerous simulations and concluded that "statistical objections against the use of the IF at the level of individual articles are not convincing", and that "the IF may be a more accurate indicator of the value of an article than the number of citations of the article". A 2020 research paper went further. It indicated that by querying open access or partly open-access databases, like Google Scholar, ResearchGate, and Scopus, it is possible to calculate approximate impact factors without the need to purchase Web of Science / JCR.
Broader negative impact on science Just as the impact factor has attracted criticism for various immediate problems associated with its application, so has there also been criticism that its application undermines the broader process of science. Research has indicated that bibliometrics figures, particularly the impact factor, decrease the quality of peer review an article receives, cause a reluctance to share data, and a reduce the scope of publishable research. "For many researchers the only research questions and projects that appear viable are those that can meet the demand of scoring well in terms of metric performance indicators – and chiefly the journal impact factor." Furthermore, the process of publication and science is slowed down – authors automatically try and publish with the journals with the highest impact factor – "as editors and reviewers are tasked with reviewing papers that are not submitted to the most appropriate venues". ==Institutional responses to criticism of the impact factor==