The standards for promotion and tenure are rising, and increasingly academic researchers are required to justify the quality of their research. However, academics come from a wide variety of disciplines, and it is difficult to understand what constitutes “quality” in various disciplines. In addition, within my specific discipline of instructional design technology, there are literally hundreds of potential journals that I could publish in. Increasingly, my colleague Peter Rich and I saw a need to lay out clear criteria for understanding what matters in scholarship and which journals/articles are more valuable than others. In addition, this framework needs to be flexible enough to be adapted to various disciplines.
Rigor, Impact, Prestige
We argue that research journals can be evaluated based on their rigor, impact, and prestige. Data representing these three criteria will vary by discipline. For example, for some disciplines the ISI Impact Factor is a decent indicator. However, in our articles we show that for educational research, this is not the case. In addition, each data point (acceptance rate, impact statistics, etc.) are easily manipulated. But, by collecting data on all three criteria and evaluating the data holistically, we can better evaluate the quality of academic publications.
In 2012, this framework was adopted by the BYU McKay School of Education as their guiding framework for evaluating scholarship within the college. The college changed “impact” to “influence” to de-emphasize the role of impact factors. I like this change, and would use influence if we were to revise the original article. The framework was also adopted in 2014 by AECT as part of its recommendations for tenure and promotion.
Articles on the Rigor, Impact, & Prestige Framework
Rigor, Impact and Prestige: A Proposed Framework for Evaluating Scholarly Publications, published in Innovative Higher Education (2012). This article received the 2015 Outstanding Journal Article award from AECT.
This article presents the framework, along with examples about how to apply it to different kinds of publications.
New technologies, new approaches to evaluating academic productivity, published in Educational Technology (2012).
This article explains how emerging technologies could assist in collecting data for the three criteria.
Articles Related to Journal and Research Quality
Mendeley: Creating communities of scholarly inquiry through research collaboration, published in TechTrends (2011).
This article discussed the potential utility of Mendeley to improving scholarly practices.
Beyond transcription: technology, change, and refinement of method, published in Forum: Qualitative Social Research (2011).
This article discusses reasons for using emerging technologies to improve qualitative research, particularly in the area of transcription.
Alternate (Article-format) Dissertations
Benefits, Challenges, and Perceptions of the Multiple Article Dissertation Format in Instructional Technology, published in Australasian Journal of Educational Technology (2014).
This article is a report of research into perceptions in instructional technology departments about the effectiveness of article-format dissertations.
Educational Technology Journal Series
In 2011, we began publishing a new series of articles in Educational Technology. Each article reported on trends within a major journal in the field of Educational Technology, reporting the key authors, methodologies, and topics central to each journal over the previous decade. This series of articles have proven helpful to many researchers in better understanding the unique discourses of each journal, and how to fit their research to a publication outlet.
The articles in this series are available on Academia.edu.
A meta-synthesis of the first 10 of these articles, showing trends across all journals was published in British Journal of Educational Technology.