• Keine Ergebnisse gefunden

Investigating the active learning research landscape through a bibliometric analysis of an influential meta‑analysis on active learning

N/A
N/A
Protected

Academic year: 2022

Aktie "Investigating the active learning research landscape through a bibliometric analysis of an influential meta‑analysis on active learning"

Copied!
24
0
0

Wird geladen.... (Jetzt Volltext ansehen)

Volltext

(1)

Vol.:(0123456789) ORIGINAL PAPER

Investigating the active learning research landscape through a bibliometric analysis of an influential meta‑analysis on active learning

Amedee Marchand Martella1 · Jane Kinkus Yatcilla2 · Helen Park3 · Nancy E. Marchand‑Martella4  · Ronald C. Martella3

Received: 15 February 2021 / Accepted: 2 August 2021 / Published online: 1 September 2021

© The Author(s), under exclusive licence to Springer Nature Switzerland AG 2021

Abstract

To gain a better understanding of the landscape of active learning research, we conducted a bibliometric analysis of 1671 scholarly sources that cited the influen- tial meta-analysis on active learning conducted by Freeman et al. We investigated eight research questions including yearly publication trends; authorships; country/

region affiliations, organizations, and funding entities; Web of Science Core Collec- tion (WOSCC) subject categories; document types; and publication outlets. Results showed an increasing number of sources from 2014 to 2019, 17% of sources sup- ported by the National Science Foundation, approximately 75% of sources published in journals, the majority of sources published in Science, Technology, Engineer- ing, and Mathematics (STEM) journals, and most of the publication outlets cate- gorized in education- or science-related disciplines in WOSCC. In addition, there were 5 countries/regions and 8 universities that tended to be the most highly rep- resented within sources citing the meta-analysis. These results highlight the impact of a prominent meta-analysis and the widespread reach it has had around the world.

Implications for how this study may impact research consumers and producers are discussed.

Keywords Active learning · Bibliometric analysis · Bibliometrics · STEM education · Course

* Amedee Marchand Martella martella@purdue.edu

1 Department of Psychology, College of Health and Human Sciences, Purdue University, 610 Purdue Mall, West Lafayette, IN 47907, USA

2 Libraries and School of Information Studies, Purdue University, West Lafayette, USA

3 Department of Educational Studies, Purdue University, West Lafayette, USA

4 College of Education, Purdue University, West Lafayette, USA

(2)

Abbreviations

ACS American Chemical Society Symposium Series ARRA American Reinvestment and Recovery Act

AAC&U Association of American Colleges and Universities AAU Association of American Universities

APLU Association of Public Land-Grant Universities ICAP Interactive Constructive Active Passive DBER Discipline-based education research E&ER Education and Educational Research ERIC Education Resources Information Center HHMI Howard Hughes Medical Institute NCES National Center for Education Statistics NIH National Institutes of Health

NSF National Science Foundation

NSF DUE National Science Foundation Department of Undergraduate Education

Non-STEM Non-Science, Technology, Engineering, and Mathematics OECD Organisation for Economic Co-operation and Development PISA Programme for International Student Assessment

STEM Science, Technology, Engineering, and Mathematics JY Second author

ED U.S. Department of Education

HHS U.S. Department of Health and Human Services WOSCC Web of Science Core Collection

UK United Kingdom U.S. United States

Introduction

When we envision a college course, one of the first images that comes to mind is of a professor standing at the front of the room lecturing to students. In fact, the tradi- tional lecture method is the prominent mode of instruction in college courses (Free- man et al. 2014; Stains et al. 2018) and is considered an efficient means of transfer- ring knowledge to students, particularly to those in larger classroom environments (Smith and Cardaciotto 2011). However, this method of teaching is often thought to promote passive learning and lead to lower student performance than more engag- ing methods (Deslauriers et al. 2019). One way to address the science, technology, engineering, and math (STEM) fatigue (i.e., lacking motivation) (Williams 2014) and attrition (i.e., dropping out of STEM courses/majors) (Chen 2013) students may experience in their college STEM courses is to adopt methods that are more active and engaging than traditional lecture (Reimer et al. 2016).

When it comes to college course transformations to promote student success in STEM in recent years, the inclusion of active learning has gained momentum (see Martella et al. 2021a; Stains et al. 2018; and Theobald et al. 2020 for details on active learning), even in large enrollment, introductory courses (Swap and Walter

(3)

2015). Active learning has been defined as “instructional activities involving stu- dents in doing things and thinking about what they are doing” (Bonwell and Eison 1991, p. iii). This definition incorporates a myriad of instructional approaches and interpretations on what should be included in college courses to ensure student success. In the research literature, active learning is “currently applied to a spec- trum of approaches ranging from traditional lecture with a few clicker questions to student-centered learning with group work using little to no instructor guidance or instruction” (Martella and Demmig-Adams 2018, p. 138). Despite the need for clearer definitions of active learning, it remains “a useful construct” (Auerbach and Andrews 2018, p. 4) and is supported by analysis from Chi and Wylie (2014) in their Interactive Constructive Active Passive (ICAP) theoretical framework. The ICAP framework theorizes four progressive levels of learner engagement: passive (P = lis- tening and receiving information), active (A = making overt actions such as taking notes), constructive (C = generating new outputs such as the solution to a problem or explanation of a concept), and interactive  (I = having group members contrib- ute jointly to constructive outputs). In this framework, learning is hypothesized to increase as the level of engagement increases (from P to A to C to I).

To investigate the effectiveness of active learning compared to traditional lecture, Freeman et al. (2014) conducted the largest and most comprehensive meta-analysis on active learning. They were interested in whether we should ask or tell when it comes to STEM instruction. The results of the meta-analysis provided evidence that active learning leads to higher student performance (almost half a standard deviation higher) and lower student failure rates (1.5 times lower) than traditional lecture in STEM college courses.

Although there was a great deal of research examining active learning versus traditional lecture in STEM college courses prior to this study, the Freeman et al.

(2014) meta-analysis synthesized data across studies to determine the overall impact of active learning compared to traditional lecture. Thus, this synthesis makes a pow- erful case for adopting active learning as an instructional method and for conducting further research on active learning (Wieman 2014). Given the importance of reduc- ing STEM fatigue and attrition, coupled with the evidence of the success of active learning in STEM found by Freeman et al. (2014), we might expect the meta-analy- sis to influence subsequent education research. Indeed, by mid-2019 Freeman et al.

(2014) had almost 1000 more citations than the next most highly cited “active learn- ing” article from 2014 in Web of Science Core Collection (WOSCC), and exceeded the next most highly cited “active learning” article in Scopus by 1200 citations (Martella et al. 2021b)–clear evidence that many other researchers had taken notice of the paper.

Based upon this article’s prominence, an analysis of a collection of sources that cited Freeman et al. (2014) would lend insight into the landscape of active learning research and could reveal important characteristics of the scholarship of active learn- ing in STEM (and non-STEM) education. The type of analysis that would allow for this “landscape view” of this meaningful research is a bibliometric analysis.

While the prefix biblio- usually refers to books, here it is useful to understand the prefix as referring to any unit of scholarly output, such as journal articles, technical reports, conference papers, etc., while the suffix -metric refers to measurement, as

(4)

usual. The most basic purpose of a bibliometric analysis is to establish the produc- tivity of people and institutions based on scholarly output (Ball 2018). Thus, the primary indicators of bibliometric analyses are numbers of papers (e.g., of a particu- lar author, a research program, or an institution) and their reception by the commu- nity based on the number of times they are cited in subsequent articles (Ball 2018).

Additional bibliometric analyses are based on other quantifiable characteristics of publications including, for example, which journals (a) published the most articles on a topic, (b) were heavily referenced by a selected group of publications, and (c) more frequently cited the articles in a selected group of publications (Gingras 2016).

Over time, the term bibliometrics has come to include the analysis of any facet of article meta-data—that is, any information about publications, including all ele- ments of a citation, the text of article titles or abstracts, author keywords or database indexing terms, information about research funding, or any other descriptive terms a database provider may attach to an article to characterize or organize it. Therefore, bibliometric analyses are differentiated from evidence synthesis methodologies, such as systematic review and meta-analysis, in that these latter types of articles examine and synthesize information from the full text of articles, whereas bibliomet- ric analyses quantify only information about the published works. Researchers use a systematic review or meta-analysis approach to dig deep into the content of and data provided by research studies, including assessing the studies’ methodologies, risk of bias, and overall quality. Bibliometric analysis zooms out to take a more panoramic look at the superficial features of research publications such as author affiliations, journal information and publication dates, and keywords from titles, abstracts, or author keywords, and generally does not comment on the content of individual stud- ies. While a team approach is required on an evidence synthesis project to avoid bias on the part of the researchers, bibliometric studies have been conducted by individ- ual authors as well as by groups. Systematic review/meta-analysis researchers select individual studies from their search results according to explicitly stated inclusion and exclusion criteria driven by the research question, while bibliometric research- ers generally aggregate a data set based on their research question, which is often exploratory in nature, and conduct summary analyses on the entire data set.

Another major difference between a bibliometric analysis and a systematic review or meta-analysis is the search strategy employed to retrieve articles for examination. Organizations such as the Campbell Collaboration, a social sci- ences-focused organization that publishes systematic reviews (Campbell Collab- oration 2014), and Cochrane, an organization that publishes health-related sys- tematic reviews (Lefebvre et al. 2019), report that a systematic review must be based on comprehensive and explicitly defined search strategies and must include avoiding publication bias by also searching for unpublished studies. In contrast, there are no oversight organizations or protocol guidelines for bibliometric stud- ies. Bibliometric analyses, which tend to be used descriptively, and which do not synthesize new information based on previous studies as do systematic reviews or meta-analyses, are often based on data obtained by searching for a topic in one multidisciplinary database like Web of Science: Core Collection (WOSCC) or Scopus, or by focusing on the output of a single journal title or a select group of journals. Evidence synthesis projects generally collect a large group of research

(5)

reports that examine similar types of interventions, for similar types of problems or in similar environments, as the Freeman et al. (2014) meta-analysis collected studies about active learning interventions in STEM college courses. In contrast, bibliometric analyses can be applied to any group of articles regardless of their degree of heterogeneity.

Bibliometric analyses are not a new methodology for educational researchers. A recent search of the Education Resources Information Center (ERIC) database on the keywords “bibliometric OR bibliometrics” returned over 1000 results, while a search in WOSCC on “bibliometric OR bibliometrics” and limited to the Web of Science research category Education and Educational Research (E&ER) returned over 500 results. Ivanovic and Ho (2019) studied publication and citation patterns for articles in the E&ER category that had been cited at least 100 times. From these data they determined the top journals, authors, institutions, and countries that pro- duced these articles, as well as the extent to which this literature is published by single or multiple authors and whether multiple authors come from the same insti- tution. Bibliometric techniques established that both the publication rates and cita- tion rates of E&ER articles have been increasing steadily over time. Further, these researchers were able to determine that while education institutions in the U.S. dom- inated the E&ER category, the rate of international collaborations among these pub- lications was lower than they expected, given the general growing emphasis placed on international collaboration in research in general.

Researchers can use bibliometric analyses with additional computational metrics to support a hypothesis or to describe a research landscape. For example, Martí-Par- reño et al. (2016) used bibliometric analyses to identify publication trends in the lit- erature of game-based learning while also using two computational methods, social network analysis, and cluster analysis, to discover author collaboration patterns and to identify themes around which the articles tended to aggregate. Findings indicated an increasing interest in educational games based on number of articles published per year. They also indicated that while the journals publishing these articles were predominantly from WOSCC’s Education & Educational Research category, vari- ous other biomedical and social sciences categories outside E&ER were also rep- resented. Based on bibliometric measures, they realized that researchers must look further afield than one discipline’s list of journals or search more than one database for pertinent information on game-based learning.

In another example, Assefa and Rorissa (2013) combined bibliometric analyses with co-word analysis (or word co-occurrence) to quantify the extent to which words or phrases appeared in articles along with “science education,” “technology educa- tion,” “engineering education,” or “mathematics education.” They used this informa- tion to describe, and therefore better understand, the underlying structure of STEM education research. For example, even though STEM education has been researched at the elementary and secondary levels, it became evident that “mathematics educa- tion” was represented across education levels from early childhood through college levels, while “engineering education” was limited to college or university level. By examining the meta-data of research articles, rather than by synthesizing the con- tent of the articles, these researchers were able to identify gaps in either educational practices or research efforts.

(6)

In the realm of active learning research, two articles present bibliometric analyses of the literature on flipped classrooms. First, Yang et al. (2017) searched Science Citation Index and Social Science Citation Index (subsets of WOSCC) for articles on flipped or inverted classrooms published between 2000 and 2015. They used biblio- metric analyses to identify publication rates, top countries and institutions, and top journals and WOSCC subject categories of this literature. Their analysis was based on 149 papers, and the number of publications per year was low from 2000 through 2011, but then the rate suddenly skyrocketed. This article indicated that the United States (U.S.) was by far the largest producer of the articles with 68.45% of the pub- lications; Australia was a distant second place (6.04%), and Saudi Arabia in seventh place (1.34%).

By contrast, Al-Shabibi and Al-Ayasra (2019) searched for articles on the flipped classroom in multiple databases that did not include WOSCC or its subsets. Their search covered the years 2012–2019 and included articles written in Arabic or Eng- lish. They found 7650 studies and conducted analyses on a random sample of 233 papers. In this analysis, the U.S. produced 32.2% of the articles with Saudi Ara- bia a much closer second with 26.2%. Nevertheless, each of the studies on flipped classrooms described a portion of the research on flipped classrooms to answer the researchers’ questions.

In addition to looking at countries, Yang et al. (2017) looked for top institutions producing the literature, top publishing journals, and author keywords used in the papers. Meanwhile, Al-Shabibi and Al-Ayasra (2019) were concerned with looking for evidence of the effectiveness of flipped classroom, and their analyses addressed words describing study types, study subjects (i.e., students versus teachers or fac- ulty), and the frequency with which study variables such as achievement or motiva- tion occurred. The fact that two bibliometric studies of articles about flipped class- rooms used very different search strategies and quantified different characteristics about the collected articles is further illustrative of the methodological difference between bibliometric studies and systematic reviews/meta-analyses.

The examples given above establish that bibliometric analyses are useful for quantifying and describing various facets of educational research. To date, no bib- liometric analysis has been conducted on the sources that cite the Freeman et al.

(2014) meta-analysis. Given the prominence of the meta-analysis in the research lit- erature, the purpose of the present study was to provide a better understanding of the landscape of active learning research through a bibliometric analysis of those sources citing the Freeman et al. (2014) meta-analysis. This landscape helps pro- vide direction for those who are interested in reading about (i.e., active learning con- sumers) or conducting research on (i.e., active learning producers) active learning.

Although one could conduct a bibliometric analysis of the literature on active learn- ing in general, narrowing the focus to the research literature containing citations to Freeman et al. (2014) allows two insights. First, rather than simply assessing how often Freeman et al. (2014) was cited, a bibliometric analysis of this type can pro- vide relevant insight into the range of its impact (e.g., by going deeper into by when, by whom, and where the meta-analysis was cited). Second, a bibliometric analysis can describe the landscape of sources referring to active learning in the context of its effectiveness relative to lecture. In other words, there is any number of reasons

(7)

why the term “active learning” could show up in a paper; however, by focusing on those that actually cited Freeman et al. (2014) in their published research, we have constrained the reasons to be related to a meta-analysis that focuses on the question of relative effectiveness of active learning versus lecture.

Research questions

We were interested in addressing eight research questions similar to those noted by Li et al. (2020a,b) related to our examination of the active learning research land- scape to better inform consumers and researchers.

1. What were the yearly publication trends of sources containing Freeman et al.

(2014) citations, starting May 12, 2014 (publication date of Freeman et al. 2014) until July 10, 2020 (date of data collection)? Determining publication trends allows active learning consumers and producers to gauge changes in the interest level of active learning, in general, and the Freeman et al. (2014) meta-analysis, in particular.

2. Who were the most common authors (either first author or any author order) of sources containing Freeman et al. (2014) citations? Examining the most common authors allows active learning consumers and producers to pinpoint researchers who are writing about active learning and/or studying its efficacy. Pinpointing these authors may provide opportunities for future collaboration or more in-depth examination of their publication or presentation efforts.

3. Which country or region affiliations, based on author addresses, were most fre- quently associated with sources containing Freeman et al. (2014) citations? Deter- mining where authors are located provides insight into the interest and popularity of the approach across the globe. This insight may prove beneficial in searching for academic positions and collaborations and can also point to national interest in pedagogy and educational reform at the college level.

4. Which organizational affiliations, based on author addresses, were most frequently associated with sources containing Freeman et al. (2014) citations? Examining where authors are employed provides insight into the interest and popularity of the approach across universities/colleges and other organizations. This insight may prove beneficial in searching for academic positions and collaborations and can also point to university/college and other organizational interest in pedagogy and educational reform at the college level.

5. Which funding entities were most frequently listed by authors in sources contain- ing Freeman et al. (2014) citations? Determining where financial support has been obtained allows active learning producers to identify potential funding sources for present or future work related to active learning.

6. Which WOSCC subject categories were most commonly associated with sources containing Freeman et al. (2014) citations? Examining which subject categories tend to contain papers with citations to Freeman et al. can help discipline-specific consumers and producers gauge the level of interest in active learning within their particular disciplines.

(8)

7. What were the most common document types of sources containing Freeman et al. (2014) citations? Determining document sources helps consumers set the parameters for potential outlets of information and producers to determine how they may showcase their own work.

8. What publication outlets (e.g., journal titles, proceedings titles) were most com- monly associated with sources containing Freeman et al. (2014) citations? Exam- ining publication outlets helps consumers search for specific active learning docu- ments and producers to target potential places to publish their work.

Method Search process

For the purpose of this study, we use the term “source” to refer to any individual item of scholarship, including primary research articles, review articles, proceed- ings papers, editorials, letters, etc. To answer our research questions, we devel- oped methods based on bibliometric analyses, i.e., quantitative analyses of the meta-data for a collection of sources that cited Freeman et al. (2014).

Our first step was to identify an appropriate source of article information for our analysis. There are currently three main options for obtaining citation data:

Web of Science Core Collection (WOSCC), Scopus, and Google Scholar. We chose WOSCC because this database provides broad, multidisciplinary coverage of peer-reviewed research publications, and therefore would provide coverage of both education-focused publications as well as STEM disciplinary publica- tions. In addition, WOSCC is the longest-established database that tracks cita- tions information, it provides a well-recognized subject classification for research journals, and it enables easy download of the citing sources for further analysis (compared with Google Scholar, for example, which also tracks numbers of cita- tions but does not provide an efficient means of downloading the meta-data of large numbers of citing sources).

Second, we had to decide which document types would be appropriate for inclusion in our analyses. While WOSCC indexes peer-reviewed journals and proceedings, these publications also publish numerous types of sources that may not be peer-reviewed, such as editorials, news items, and book reviews. How- ever, precisely because they are not subjected to the peer-review process, these types of sources also tend to be published much more quickly after submission than do papers submitted for peer-review. Because one of our research questions addresses yearly publication patters of citing sources, we decided to include these types of sources in our analysis to capture the earliest appearance of Freeman et al. (2014) as a cited reference. Therefore, no document type was excluded.

To generate data for this bibliometric analysis, the second author (JY) searched the WOSCC database on July 10, 2020 and downloaded the citations and abstracts for all of the 1671 sources that cited Freeman et al. (2014) as of that date.

(9)

Data analysis

The answers to some of our research questions came directly from WOSCC prescribed bibliometric analyses, including research questions #1 (What were the yearly publica- tion trends of sources containing Freeman et al. (2014) citations), #3 (Which country or region affiliations, based on author addresses, were most frequently associated with sources containing Freeman et al. (2014) citations?), #6 (Which WOSCC subject cat- egories were most commonly associated with sources containing Freeman et al. (2014) citations?), #7 (What were the most common document types of sources containing Freeman et al. (2014) citations?), and #8 (What publication outlets (e.g., journal titles, proceedings titles) were most commonly associated with sources containing Freeman et al. (2014) citations?).

For the other research questions (#2, #4, and #5), the WOSCC prescribed analyses did not provide the level of detail in which we were interested. To answer these ques- tions, we used VantagePoint text analytics software (www. vpins titute. org) to conduct data clean-up, such as combining name variations, and used a combination of Vantage- Point and Microsoft Excel to further analyze the 1671 sources that cited Freeman et al.

(2014). For example, with regard to question #2 (Who were the most common authors (either first author or any author order) of sources containing Freeman et al. (2014) citations?), the WOSCC analysis “Authors” did not reflect author order, it simply indi- cated the number of times an author’s name appears, in any order, nor did it combine multiple variations of an author’s name. VantagePoint can differentiate between first and non-first author order of WOSCC data and has a data clean-up functionality that facilitates quickly combining name variations, which led to different “top” authors than the WOSCC “Author” analysis.

For question #4 (Which organizational affiliations, based on author addresses, were most frequently associated with sources containing Freeman et al. (2014) citations?) the WOSCC “Organizations” analysis again did not differentiate between first author’s affiliation and other author’s affiliation; this is a differentiation included in the Van- tagePoint analysis of WOSCC data. For question #5 (Which funding entities were most frequently listed by authors in sources containing Freeman et al. (2014) citations?), the WOSCC analysis on “Funding Agencies” did not combine name variants such as

“National Science Foundation Department of Undergraduate Education,” “National Science Foundation DUE,” and “NSF DUE.” Because of variations in how authors listed funding acknowledgements, we used a combination of data clean-up in Van- tagePoint, followed by manual name checking and combining in Microsoft Excel, to summarize all of the funding acknowledgements to one agency. For instance, in the example given above with variations of the NSF DUE program, these acknowledge- ments were combined with acknowledgements to all other National Science Founda- tion (NSF) programs to calculate NSF’s total number of acknowledgements.

Results

In the following sections, we report findings corresponding to each of the eight research questions.

(10)

Yearly publication trends

The Freeman et  al. (2014) meta-analysis was first published on May 12, 2014.

Our analysis of the 1671 sources that cited this influential meta-analysis between that date and July 10, 2020 showed a positive trend. As shown in Table 1, the number of sources citing Freeman et  al. (2014) increased each year with 30 (1.80%) in 2014, 103 (6.16%) in 2015, 245 (14.66%) in 2016, 313 (18.73%) in 2017, 397 (23.76%) in 2018, 425 (25.43%) in 2019, and 158 (9.46%) for the first half of 2020. The largest increase from one year to the next was between 2015 and 2016 with a gain of 142 citations.

Authorships

There were 5073 distinct authors across the 1671 sources, after merging name variations in VantagePoint software.

First author only

As shown in Table 1, the three most frequent first author names (including ties) included MM Cooper (n = 6 sources [0.36%]); KM Cooper (n = 5 sources [0.30%]); and CJ Ballen, D Drummond, SL Eddy, MT Hora, and NJ Pienta (n = 4 sources [0.24%] each).

Any author order

As shown in Table 1, the five most frequent author names in any authorship order (including ties) included SL Eddy (n = 14 sources [0.84%]); MK Smith (n = 10 sources [0.60%]); SE Brownell (n = 9 sources [0.54%]); E Berger, J DeBoer and C Wieman (n = 8 sources [0.48%] each); and E Brewe, MM Cooper, and EL Dolan (n = 7 sources [0.42%] each).

Country/region affiliations

Authors who cited Freeman et al. (2014) had affiliations in 90 countries across the 1671 sources as indicated by author address or reprint address.

First author only

As shown in Table 1, the most frequently listed country/region affiliations of first authors included the United States (U.S.) (n = 1012 sources [60.56%]), the United

(11)

Table 1 Bibliometric results across categorical variables

Variable Results

Publication year 1671 total sources

Number of sources citing Freeman et al. (2014) by year

∙ 2014: 30 sources (1.80%)

∙ 2015: 103 sources (6.16%)

∙ 2016: 245 sources (14.66%)

∙ 2017: 313 sources (18.73%)

∙ 2018: 397 sources (23.76%)

∙ 2019: 425 sources (25.43%)

∙ 2020: 158 sources (9.46%)

Authorship 5073 distinct authors across the 1671 sources First authors Most frequently listed author names

∙ MM Cooper: 6 sources (0.36%)

∙ KM Cooper: 5 sources (0.30%)

∙ CJ Ballen: 4 sources (0.24%)

∙ D Drummond: 4 sources (0.24%)

∙ SL Eddy: 4 sources (0.24%)

∙ MT Hora: 4 sources (0.24%)

∙ NJ Pienta: 4 sources (0.24%) Any author order ∙ SL Eddy: 14 sources (0.84%)

∙ MK Smith: 10 sources (0.60%)

∙ SE Brownell: 9 sources (0.54%)

∙ E Berger: 8 sources (0.48%)

∙ J DeBoer: 8 sources (0.48%)

∙ C Wieman: 8 sources (0.48%)

∙ E Brewe: 7 sources (0.42%)

∙ MM Cooper: 7 sources (0.42%)

∙ EL Dolan: 7 sources (0.42%)

Country/Region 90 countries/regions across the 1671 sources Affiliation Most frequently listed country/region affiliations:

First authors ∙ U.S.: 1012 sources (60.56%)

∙ UK: 70 sources (4.19%)

∙ Spain: 65 sources (3.89%)

∙ Australia: 56 sources (3.35%)

∙ Canada: 54 sources (3.23%)

All authors ∙ U.S.: 1061 sources (63.49%), 2801 total author affiliations

∙ UK: 98 sources (5.86%), 154 total author affiliations

∙ Australia: 81 sources (4.85%), total 151 author affiliations

∙ Spain: 73 sources (4.37%), 121 total author affiliations

∙ Canada: 71 sources (4.25%), 111 total author affiliations Organization affiliation 1613 organizations across the 1671 sources

Most frequently listed organization affiliations First authors only ∙ University of Georgia: 18 sources (1.08%)

∙ University of Washington: 18 sources (1.08%)

∙ Michigan State University: 15 sources (0.90%)

∙ University of Central Florida: 15 sources (0.90%)

∙ University of Colorado: 15 sources (0.90%)

(12)

U.S. = United States; UK = United Kingdom; NSF = National Science Foundation; HHS = U.S. Depart- ment of Health and Human Services; NIH = National Institutes of Health; HHMI = Howard Hughes Med- ical Institute; ED = U.S. Department of Education

Table 1 (continued)

Variable Results

All authors ∙ University of Washington: 34 sources (2.03%);

52 total author affiliations

∙ Michigan State University: 32 sources (1.92%);

65 total author affiliations

∙ Purdue University: 32 sources (1.92%);

73 total author affiliations

∙ Arizona State University: 31 sources (1.86%);

65 total author affiliations

∙ University of Colorado: 30 sources (1.80%);

46 total author affiliations

Funding support 925 funding entities across the 1671 sources Most frequently listed funding entities:

∙ NSF: 286 sources (17.12%)

∙ NIH: 60 sources (3.59%)

∙ HHS: 34 sources (2.03%)

∙ HHMI: 34 sources (2.03%)

∙ ED: 11 sources (0.66%)

Web of science category 161 categories across the 1671 sources Most common categories:

∙ Education, Scientific Disciplines: 724 sources (43.33%)

∙ Education & Educational Research: 629 sources (37.64%)

∙ Chemistry, Multidisciplinary: 107 sources (6.40%)

∙ Engineering, Multidisciplinary: 79 sources (4.73%)

∙ Computer Science, Interdisciplinary Applications: 62 sources (3.71%) Document type 11 document types across the 1671 sources

Most common document types

∙ Journal articles: 1245 sources (74.51%)

∙ Proceedings papers: 256 sources (15.32%)

∙ Editorial material: 97 sources (5.80%)

∙ Book chapters: 67 sources (4.01%)

∙ Reviews: 65 sources (3.89%)

Publication outlet 753 different publication outlets across the 1671 sources Most common sources

∙ CBE Life Sciences Education: 109 sources (6.52%)

∙ Journal of Chemical Education: 69 sources (4.13%)

∙ Advances in Physiology Education: 42 sources (2.51%)

∙ Journal of Microbiology and Biology Education: 40 sources (2.39%)

∙ American Chemical Society Symposium Series: 33 sources (1.97%)

(13)

Kingdom (UK) (n = 70 sources [4.19%]), Spain (n = 65 sources [3.89%]), Aus- tralia (n = 56 sources [3.35%]), and Canada (n = 54 sources [3.23%]).

All authors

The most frequently listed country/region affiliations of all authors included the U.S. (n = 1061 sources [63.49%], 2801 total author affiliations), UK (n = 98 sources [5.86%] 154 total author affiliations), Australia (n = 81 sources [4.85%], 151 total author affiliations), Spain (n = 73 sources [4.37%], 121 total author affiliations), and Canada (n = 71 sources [4.25%], 111 total author affiliations) (see Table 1).

Organization affiliations

Authors who cited Freeman et al. (2014) were from 1613 organizations across the 1671 sources as indicated by author or reprint address.

First author only

As shown in Table 1, the five most frequently listed organization affiliations of first authors included University of Georgia and University of Washington (n = 18 sources [1.08%] each); and Michigan State University, University of Central Florida, and University of Colorado (n = 15 sources [0.90%] each).

All authors

The five most frequently listed organization affiliations of all authors are shown in Table 1. These included the University of Washington (n = 34 sources [2.03%], 52 total author affiliations), Michigan State University (n = 32 sources [1.92%], 65 total author affiliations), Purdue University (n = 32 sources [1.92%], 73 total author affiliations), Arizona State University (n = 31 sources [1.86%], 65 total author affili- ations), and University of Colorado (30 sources [1.80%], 46 total author affiliations).

Funding entities

Of the 1671 sources that contained citations to Freeman et al. (2014), 705 of these sources listed funding support. Overall, there were 925 separate funding entities including academic departments, universities, private organizations, and national funding agencies. As shown in Table 1, the five most frequently listed funding enti- ties included the National Science Foundation (NSF) (n = 286 sources [17.12%]);

the National Institutes of Health (NIH) (n = 60 sources [3.59%]); U.S. Department of Health and Human Services (HHS) and the Howard Hughes Medical Institute (HHMI) (n = 34 sources [2.03%] each); and the U.S. Department of Education (ED) (n = 11 sources [0.66%]). Three of these five funding agencies were related to health/

medicine which is an area outside of the STEM disciplines discussed in Freeman et al. (2014).

(14)

WOSCC subject categories

WOSCC assigns each journal, book, or proceedings it indexes at least one subject category from its list of 255 subjects across the sciences, social sciences, and arts and humanities; articles in WOSCC are assigned the same category(ies) as their par- ent publication, even if their subject is different. Of the 1671 sources citing Free- man et al. (2014), 161 out of 255 WOSCC categories were represented. As shown in Table 1, the five most common categories were Education, Scientific Disciplines (n = 724 sources [43.33%]); Education & Educational Research (n = 629 sources [37.64%]); Chemistry, Multidisciplinary (n = 107 sources [6.40%]); Engineering, Multidisciplinary (n = 79 sources [4.73%]); and Computer Science, Interdisciplinary Applications (n = 62 sources [3.71%]). Although the Freeman et  al. (2014) meta- analysis included studies that addressed active learning only in STEM disciplines, many sources citing the meta-analysis were found in journals and books categorized in (a) non-STEM disciplines such as communication, law, philosophy, religion, business, and art, and (b) health disciplines such as nursing, surgery, and medical informatics. More specifically, there were 135 sources [8.08%] that came from pub- lications assigned to categories in the social sciences or arts and humanities and an additional 289 sources [17.30%] that came from publication outlets assigned to at least one of 50 WOSCC categories pertaining to biomedical topics (i.e., categories that straddle both the life sciences and the social sciences).

Document types

There were 11 document types of the 1671 sources citing Freeman et al. (2014) and were categorized in WOSCC as journal articles, proceedings papers, editorial mate- rial, book chapters, reviews, early access papers, letters, meeting abstracts, books, news items, or reprints. As shown in Table 1, the five most common document types included journal articles (n = 1245 sources [74.51%]), proceedings papers (n = 256 sources [15.32%]), editorial material (n = 97 sources [5.80%]), book chapters (n = 67 sources [4.01%]), and reviews (n = 65 sources [3.89%]).

Publication outlets

Of the 1671 sources citing Freeman et al. (2014), there were 753 different outlets including journals, conferences, and books. As shown in Table 1, the top publica- tion outlets based on number of sources were CBE Life Sciences Education (n = 109 sources [6.52%]), the Journal of Chemical Education (n = 69 sources [4.13%]), Advances in Physiology Education (n = 42 sources [2.51%]), Journal of Micro- biology and Biology Education (n = 40 sources [2.39%]), and American Chemical Society Symposium Series (n = 33 sources [1.97%]). The majority of the publica- tion outlets were assigned WOSCC STEM categories or education-related catego- ries. However, journals from WOSCC non-STEM categories were also represented.

Examples include American Journal of Pharmaceutical Education (n = 13 sources

(15)

[0.78%]), Advances in Medical Education and Practice (n = 8 sources [ 0.48%]), Journal of Legal Education (n = 3 sources [0.18%]), and American Speech (n = 2 sources [0.12%]).

Discussion

Freeman et al. (2014) is a highly cited meta-analysis on active learning and remains the largest and most comprehensive analysis on active learning in the research lit- erature (Martella et al. 2021b). Due to its prominence, we conducted a bibliomet- ric analysis of sources indexed in WOSCC that cited Freeman et al. (2014) to gain insight into the landscape of research and education-related scholarship that might benefit consumers and producers of active learning research. Through our break- down of the sources citing the meta-analysis using eight research questions includ- ing an examination of (1) yearly publication trends, (2) authorships, (3) country/

region affiliations, (4) organization affiliations, (5) funding entities, (6) WOSCC categories, (7) document types, and (8) publication outlets, we gained insight into the impact of Freeman et al. (2014) and the representation of different publication outlets and affiliations in the publishing of sources that cited the meta-analysis. This information may prove helpful to consumers and producers alike in locating infor- mation, publishing works, or determining funding agencies, for example.

Yearly publication trends

There has been an increasing number of sources citing Freeman et al. (2014) each year since its publication in mid-2014, beginning with 30 sources published in 2014 (including 16 articles, 2 proceedings papers, and one book chapter). The largest increase in sources from one year to the next was from 2015 to 2016. Given that the meta-analysis was published in mid-2014 and given the time it takes to develop and conduct research projects and publish manuscripts, publishing a paper by 2015 (0.5–1.5 years after Freeman et al. [2014] published) may have been too short of a timeline for many projects/papers to be conducted and ultimately published. How- ever, 2016 would have given authors 1.5–2.5 years to complete and publish their works and for a greater number of people to be exposed to the meta-analysis.

The overall positive trend in sources could also be attributed to national and uni- versity-wide initiatives and recommendations that targeted STEM education reform efforts with a focus on alternative instructional practices such as active learning.

For example, in 2015, the White House Office of Science and Technology Policy (2015) made a national call to action to improve STEM education with a focus on effective teaching approaches. In 2017, the Association of American Universi- ties (AAU) completed a status report based on 5 years of implementing reforms to improve foundational STEM courses (AAU 2017). The AAU also dedicated 2018 and 2019 to provide strategy recommendations and to maximize networks (Kezar 2018) as well as to organize a large STEM education workshop (Miller et al. 2019).

Key members from a collection of universities along with the Association of Public

(16)

Land-Grant Universities (APLU) also wrote a letter to President Obama on Octo- ber 11, 2016 to “applaud your establishment of Active Learning Day to recognize the importance of improving teaching in STEM fields” (AAU & APLU 2016). The Association of American Colleges and Universities (AAC&U) (2018) developed a strategic plan for 2018–2022 for using evidence-based STEM reform practices and promoting an “active learning week” which was enacted in October 2018 to have faculty pledge to try active learning in the classroom for a week. It should be noted that Freeman et al. (2014) was often cited in various reports and papers that came out of these initiatives and recommendations (e.g., see AAU 2017).

Outside of the U.S., other countries have also had initiatives and recommenda- tions for improving STEM Education. For example, in 2015, the Canadian Science, Technology, and Innovation Council (2015) released a report entitled State of the Nation where one of the recommendations to address the performance gap between Canada and the top 5 performing countries was to develop better curricula that could integrate STEM with business and innovation-related skills. Based on these recommendations, Canada has had a number of initiatives related to STEM edu- cation (DeCoito 2016). In Australia, the STEM crisis (e.g., students not pursuing STEM subjects) has led to the conclusion that STEM education should be a national priority (Bagshaw 2015). The Australian Government has taken the STEM crisis seriously and has provided significant funding for initiatives that relate to improv- ing STEM teaching and learning (Department of Education, Skills and Employment 2020). With calls to improve STEM education and initiatives in place to support these calls, the increase in sources containing citations to a meta-analysis on the relative effectiveness of active learning in STEM disciplines is understandable.

Authorships

There were many authors whose names were on multiple sources that cited Free- man et al. (2014). A Google search on the most represented authors revealed that the majority are from STEM disciplines and conduct discipline-based education research (DBER). Our search results revealed: S. L. Eddy is in the Department of Biological Sciences at Florida International University, M. K. Smith is in the School of Biology and Ecology at the University of Maine, S. E. Brownell is in the School of Life Sciences at Arizona State University, K. M. Cooper is in the School of Life Sciences at Arizona State University, E. Berger is in the School of Engineering Edu- cation at Purdue University, J. DeBoer is in the School of Engineering Education at Purdue University, C. Wieman is in the Physics Department and the Graduate School of Education at Stanford University, E. Brewe is in the School of Educa- tion (physics education) at Drexel University, M. M. Cooper is in the Department of Chemistry at Michigan State University, E. L. Dolan is in the Department of Bio- chemistry and Molecular Biology at the University of Georgia-Athens, N. J. Pienta was in the Department of Chemistry at the University of Georgia-Athens, C. J. Bal- len in the Department of Biological Sciences at Auburn University, D. Drummond is in the Ilumens Simulation Department at Paris Descartes University, and M. T.

(17)

Hora is in the Department of Liberal Arts and Applied Studies at the University of Wisconsin-Madison.

Country/region affiliations

The five most common countries/regions with which authors of sources that cited Freeman et  al. (2014) were affiliated included the U.S., UK, Spain, Australia, and Canada. It should be noted that WOSCC is U.S.-centric in its journal cover- age (J. Figueroa, personal communication) and thus authors from the U.S. may be more represented as they may be more likely to know about and publish in publi- cation outlets that are from the U.S.. The U.S. may also be highly represented as its researchers struggle to catch up with other countries behind which they have lagged in the preparation of students in math and the sciences. For example, on the Organisation for Economic Co-operation and Development (OECD) Programme for International Student Assessment (PISA) exam (most comprehensive international assessment of student learning), U.S. high school students ranked 31st in mathemat- ics and 13th in science out of 36 countries that are members of the OECD (National Center for Education Statistics [NCES] 2018). Further, “fewer than four in 10 col- lege students who intend to major in a math- or science-based field actually stick with it” (Williams 2014, para. 2). As previously discussed, the U.S. has had large- scale initiatives and calls to improve STEM education with a focus on instruction.

Given the lower performance of U.S. students in both K-12 and college, delving into alternative instructional approaches such as active learning has become a popular area of research.

When also examining the other four countries on the PISA, the UK ranked 13th in math and 9th in science; Spain ranked 28th in math and 25th in science; Australia ranked 24th in math and 12th in science; and Canada ranked 7th in math and 5th in science (NCES 2018). Each of these countries has expressed a need for STEM edu- cational improvement and/or specific ways they are addressing STEM educational reform (for example, see Morgan and Kirby, 2016, for the UK, STEM PD Net 2019, for Spain; Department of Education, Skills and Employment in the Australian Gov- ernment, 2020, for Australia; and the Science, Technology and Innovation Council, 2015, for Canada) and has, based on the number of sources citing Freeman et al.

(2014), been interested in the topic of active learning.

Organization affiliations

The most common organizations with which authors of sources citing Freeman et al. (2014) were affiliated included University of Washington, Michigan State University, University of Central Florida, University of Colorado, Purdue Uni- versity, Arizona State University, and University of Georgia. These organiza- tions were all located in the U.S., and as previously presented, the U.S. was the most affiliated country with sources citing Freeman et al. (2014). Further, four of the most represented organizations presented in this section (i.e., University of Georgia, Purdue University, Michigan State University, and Arizona State

(18)

University) employ(ed) seven of the most represented authors (see section above labeled “Authorship”). Upon further investigation of these institutions, we found each of them to have (a) a center(s) related to STEM education and/or excellence in teaching and learning (e.g., the Center for STEM Learning at the University of Colorado), (b) DBER researchers (e.g., Carl Wieman at Stanford University), and (c) facilities/other resources to support educational/instructional reform and research (e.g., the Wilmeth Active Learning Center at Purdue University).

Funding entities

The five most listed funding entities included the NSF, NIH, HHS, the HHMI, and ED. These funding agencies are not surprisingly affiliated with the U.S.

given that the majority of sources that cited Freeman et  al. (2014) were affili- ated with this same country. It is also not surprising that approximately 17% of sources citing the meta-analysis were supported by NSF as this foundation has funding opportunities that support educational research in a number of STEM disciplines and not just in “science,” for example. NSF also has a large number of fellowships/grants and programs aimed at STEM education and STEM diver- sity and inclusion that allow for many funding opportunities—as stated on its website: “The National Science Foundation plays a leadership role in developing and implementing efforts to enhance and improve STEM education in the United States (NSF 2020, p.1). ED often collaborates with NSF to improve STEM educa- tion and broaden participation and also has several offices tasked with supporting STEM (U.S. Department of Education 2020). ED has supported STEM education research over the years as illustrated by Li et al. (2020b) in a recent systematic analysis of publicly funded projects related to STEM education. These authors identified 127 funded projects by the Institute of Education Sciences (from 2003 to 2019). Research questions addressed funding levels, trends, types of awardees, participant populations, goals noted in STEM education, disciplinary foci, and research methods used to conduct STEM research. Findings noted the number of funded STEM education projects and funding amounts were variable over the years (e.g., more funded projects occurred in 2007, fewer were funded in 2014);

typically funded projects were 3–4 years in duration, budgeted for approximately

$1.1 to $3.4 million, involved multiple principal investigators, were awarded to universities, were focused on development and innovation efforts, and involved experimental research.

Although there were three major non-STEM funding agencies, these agencies have funds targeting STEM education research and educational reform. For exam- ple, NIH has STEM Education Challenge Grants funded through the American Reinvestment and Recovery Act (ARRA) for STEM education research (NIH 2020); HHMI has a science education program to improve science education in K-12, college, graduate school, and beyond (Brown University, 2020); and HHS contains the agency “NIH” which, as stated above, has funding for STEM educa- tion research (NIH 2020).

(19)

WOSCC categories

The five most represented WOSCC categories included Education, Scientific Dis- ciplines; Education & Educational Research; Chemistry, Multidisciplinary; Engi- neering, Multidisciplinary; and Computer Science, Interdisciplinary Applications.

Given that research on active learning or STEM teaching and learning is in the domain of education and is therefore likely to be published in an education-related publication outlet, the two most prevalent WOSCC categories are related specifically to education. Chemistry, Multidisciplinary was the third most prevalent category;

chemistry education has been discussed since the 1920s and has had a long history in the topic of chemistry education (Singer et al. 2012). Engineering, Multidiscipli- nary was the fourth most prevalent category; early accounts of engineering educa- tion interest were stimulated by the establishment in 1910 of Bulletin of the Society for the Promotion of Engineering Education which later became known as the Jour- nal of Engineering Education (Singer et al. 2012) and thus this discipline has had a long history in the topic of engineering education. Computer Science, Interdiscipli- nary Applications was the fifth most prevalent category; computer science education has a shorter history than the other DBER disciplines (Tedre et al. 2018) but has become a fruitful and important area of research given high enrollments in computer science and the emergence of online courses (Hambrusch and Guzdial 2015).

Document types

It is important to consider that although the Freeman et  al. (2014) meta-analysis involved studies on STEM courses, a large number of sources citing the meta-anal- ysis were found in journals and books categorized in non-STEM and health disci- plines. Perhaps the popularity of the meta-analysis in these non-STEM sources is indicative of a gap in the research literature for systematic reviews or meta-analyses on active learning pertaining to non-STEM or STEM + non-STEM disciplines.

Publication outlets

The majority of sources citing Freeman et al. (2014) were journal articles. The five most common journals in which these articles were published included CBE Life Sciences Education, Journal of Chemical Education, Advances in Physiology Edu- cation, Journal of Microbiology Biology Education, and American Chemical Society Symposium Series. CBE Life Sciences Education is a prominent DBER journal and is sponsored by the American Society for Cell Biology with support from the How- ard Hughes Medical Institute (Singer et al. 2012). In fact, the Editor-in-Chief is Erin L. Dolan from the University of Georgia (this author and organization were in the top five most represented authors and organizations for sources citing Freeman et al.

[2014]). The Journal of Chemical Education is another prominent DBER journal and is published by the Division of Chemical Education of the American Chemical Society (Singer et al. 2012). Advances in Physiology Education publishes research

(20)

in physiology, neuroscience, and pathophysiology education and is from the Amer- ican Physiological Society (Advances in Physiology Education 2020). It was also listed as a journal that often publishes data on student learning and attitudes in col- lege biology courses (Singer et al. 2012). The Journal of Microbiology and Biology Education is a journal that publishes articles on biology education and is sponsored by the American Society for Microbiology (Journal of Microbiology and Biology Education, 2020). Finally, the American Chemical Society Symposium Series (ACS) is part of the American Chemical Society eBooks where each chapter is written by an expert in fields such as agricultural and food chemistry and organic chemistry, among many others (ACS Publications, 2020).

Notably, three of these five publication outlets are related to biology education, and two of these sources are related to chemistry education. Although biology was not listed as a top five WOSCC category, there were several other biology-related categories in the Web of Science such as Biochemistry Molecular Biology and Cell Biology that, when combined, make biology a highly represented discipline for publishing sources that cite Freeman et al. (2014). Biology, like chemistry, has a long history in DBER and graduate training programs in biological and education research have become quite popular (Stains et al. 2014).

Conclusions and implications

The Freeman et al. (2014) meta-analysis on active learning is a highly cited arti- cle that has been cited in sources affiliated with a wide array of authors, countries, organizations, funding agencies, and publication outlets. It is evident that consumers and producers in both STEM and non-STEM disciplines have discovered the Free- man et al. meta-analysis and found its contents to be important to discuss and cite in their papers. A conversation surrounding active learning and STEM instruction has thus found its way to thousands of consumers and producers in a range of disci- plines and from many countries and organizations. For consumers of active learning research, our results provide insight into (a) the increasing popularity of this topic over time and (b) which authors, countries, organizations, documents, and outlets to search when seeking information related to active learning with the largest per- centage of key authors being from the United States and working at large research intensive universities, the majority of findings published through article outlets and in STEM versus non-STEM journals, and most funding provided by the National Science Foundation in the U.S.

For producers of active learning research, our results provide important insight into several key factors. These include (a) the increasing interest in active learn- ing research illustrating the importance of the topic to new researchers and those immersed in conducting research that has a direct impact on higher education; (b) opportunities for collaboration as it relates to examination of specific authors’ pub- lication or presentation efforts, particularly at research intensive universities; (c) direction in searching for academic positions and collaborations as they relate to which researchers and universities are active in this research area; (d) details related to potential funding sources for present or future work related to active learning and

(21)

where funding is likely to be provided; (e) information related to the level of inter- est in active learning within particular disciplines with a focus on life/biological sciences, chemical education, and physiology, along with non-STEM areas such as pharmaceutical education, medicine, and legal education; and (f) specific details on document sources and publication outlets for publishing findings, again with a focus on journals as publication outlets.

In addition, several implications arise from the results of this bibliometric anal- ysis. First, STEM fatigue and attrition are likely affecting student performance in other countries beyond the U.S. and Australia. Thus, we would expect to see an increase in interest and research activity in other countries as their focus turns to the use of active learning pedagogy to improve math and science performance. Second, the top funding agencies are from the U.S. likely due to the more intense focus on improving STEM education as compared to other countries. However, much of this funding is limited to a handful of universities. There may be a wider impact of active learning in the U.S. if this funding was distributed across more U.S. institutions.

Relatedly, research funding comes primarily from STEM-focused entities such as NSF. A greater impact of this research funding may occur if more widespread fund- ing is offered to non-STEM fields by non-STEM funding agencies. Third, given that much of the research is published in STEM-focused journals in the U.S., it seems reasonable that researchers could target mainstream educational journals to have a greater impact in other academic areas. Also, publishing in international journals could have a wider impact as opposed to focusing on U.S.-based journals. Finally, the majority of researchers are affiliated with U.S. institutions. Research in the area of active learning may provide opportunities for collaboration across the globe and allow for an international focus on STEM education.

Author contributions All authors contributed to the eventual publication of this project including article gathering, data analysis, and manuscript preparation and submission.

Funding The first author acknowledges support from the National Science Foundation Graduate Research Fellowship Program under grant number DGE-1842166. Any opinions, findings, and conclusions or rec- ommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Data availability Materials can be obtained by contacting the first author for further information on the articles used in this analysis.

Declarations

Conflit of interests There are no financial or nonfinancial competing interests.

References

Advances in Physiology Education (2020) About advances in physiology education. American Physi- ological Society. https:// journ als. physi ology. org/ advan ces/ about.

Al-Shabibi TS, Al-Ayasra MAK (2019) Effectiveness of the flipped classroom strategy in learning out- comes (Bibliometric study). Int J Learn Teach Educ Res 18(3):96–127. https:// doi. org/ 10. 26803/

ijlter. 18.3.6

(22)

American Chemical Society (2020) ACS symposium series ebooks. ACS Publications. https:// pubs. acs.

org/ series/ sympo sium

Assefa SG, Rorissa A (2013) A bibliometric mapping of the structure of STEM education using co-word analysis. J Am Soc Inform Sci Technol 64(12):2513–2536. https:// doi. org/ 10. 1002/ asi. 22917 Association of American Colleges and Universities (AAC&U) (2018) 2018–2022 strategic plan: educat-

ing for democracy. https:// www. aacu. org/ sites/ defau lt/ files/ files/ about/ AACU_ Strat egicP lan_ 2018- 22. pdf

Association of American Universities (AAU) (2017) Progress toward achieving systemic Change: a five- year status report on the AAU undergraduate STEM education initiative. https:// www. aau. edu/ sites/

defau lt/ files/ AAU- Files/ STEM- Educa tion- Initi ative/ STEM- Status- Report. pdf

Association of American Universities and Association of Public Land-grant Universities (2016) Letter to President on active learning day. https:// newsc enter. sdsu. edu/ educa tion/ crmse/ files/ 07179- Letter_

to_ Presi dent_ on_ Active_ Learn ing_ Day_ 101216. pdf

Auerbach AJJ, Andrews TC (2018) Pedagogical knowledge for active-learning instruction in large under- graduate biology courses: a large-scale qualitative investigation of instructor thinking. Int J STEM Educ 5(1):1–25. https:// doi. org/ 10. 1186/ s40594- 018- 0112-9

Bagshaw E (2015) STEM crisis: NSW Education Minister Adrian Piccoli to focus on STEM’s image problem. The Sydney Morning Herald. https:// www. smh. com. au/ educa tion/ stem- crisis- nsw- educa tion- minis ter- adrian- picco li- to- focus- on- stems- image- probl em- 20151 203- glefg6. html

Ball R (2017) Bibliometric methods: basic principles and indicators. In: Ball R (ed) An introduction to bibliometrics. Chandos, pp 15–56. https:// doi. org/ 10. 1016/ B978-0- 08- 102150- 7. 00003-7.

Bonwell CC, Eison JA (1991) Active learning: creating excitement in the classroom. ASHE-ERIC Higher Education Report No. 1. http:// files. eric. ed. gov/ fullt ext/ ED336 049. pdf

Brown University (2020) Howard Hughes Medical Institute (HHMI). https:// www. brown. edu/ initi atives/

stem- educa tion/ hhmi

Collaboration C (2014) Campbell Collaboration systematic reviews: policies and guidelines. Campbell Collab. https:// doi. org/ 10. 4073/ cpg. 2016.1

Chen X (2013) STEM attrition: College students’ paths into and out of STEM fields (NCES 2014–001).

National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Edu- cation. https:// nces. ed. gov/ pubs2 014/ 20140 01rev. pdf

Chi MTH, Wylie R (2014) The ICAP framework: Linking cognitive engagement to active learning out- comes. Educ Psychol 49(4):219–243. https:// doi. org/ 10. 1080/ 00461 520. 2014. 965823

DeCoito I (2016) STEM education in Canada: a knowledge synthesis. Can J Sci Math Technol Educ 16(2):114–128. https:// doi. org/ 10. 1080/ 14926 156. 2016. 11662 97

Department of Education Skills and Employment (2020) Support for science, technology, engineering and mathematics (STEM). Australian Government. https:// www. educa tion. gov. au/ suppo rt- scien ce- techn ology- engin eering- and- mathe matics

Deslauriers L, McCarty LS, Miller K, Callaghan K, Kestin G (2019) Measuring actual learning ver- sus feeling of learning in response to being actively engaged in the classroom. Proc Natl Acad Sci 116(39):19251–19257. https:// doi. org/ 10. 1073/ pnas. 18219 36116

Freeman S, Eddy SL, McDonough M, Smith MK, Okoroafor N, Jordt H, Wenderoth MP (2014) Active learning increases student performance in science, engineering, and mathematics. Proc Natl Acad Sci 111(23):8410–8415. https:// doi. org/ 10. 1073/ pnas. 13190 30111

Gingras Y (2016) The origins of bibliometrics. In: Gingras Y (ed) Bibliometrics and research evaluation:

uses and abuses. MIT Press, pp 1–10

Hambrusch S, Guzdial M (2015) Computing education research becomes a research area in CISE’s CAREER proposals. Comput Res News 27(5):61–64

Ivanovic L, Ho Y-S (2019) Highly cited articles in the education and educational research category in the social science citation index: a bibliometric analysis. Educ Rev 71(3):277–286. https:// doi. org/ 10.

1080/ 00131 911. 2017. 14152 97

Journal of Microbiology and Biology Education (2020) Journal home. American Society for Microbiol- ogy. https:// www. asmsc ience. org/ conte nt/ journ al/ jmbe

Kezar A (2018) Scaling improvement in STEM learning environments: The strategic role of a national organization. American Association of American Universities and Pullias Center for Higher Educa- tion. https:// pulli as. usc. edu/ downl oad/ scali ng- impro vement- in- stem- learn ing- envir onmen ts- the- strat egic- role- of-a- natio nal- organ izati on/

Lefebvre C, Glanville J, Briscoe S, Littlewood A, Marshall C, Metzendorf M-I, Noel-Storr A, Rader T, Shokraneh F, Thomas J, Wieland LS (2019) Searching for and selecting studies. In: Cochrane

Referenzen

ÄHNLICHE DOKUMENTE

The integration work takes place at 2 levels: the level of knowledge representation, where features associated with various modalities (image, text/speech transcripts,

Therefore, the present paper is an attempt to examine the existence of business cycles in Iran's economy and analyze the causes and roots of its development in the

Quite apart from such a rigorous modeling of non-standard labor values, Kliman (2007) is a strong proponent of what is known as the Temporal Single-System Interpre- tation of

Does your MSCA, your organisation or you as an expert support the opinion of the German CA that REACH registered substances which are emitted into the environment and which have

The approach combines state-of-the-art ontology learning paradigms to extract first- order features from the above sources and uses supervised machine-learning techniques to derive

University of Economics in Bratislava, Faculty of Business Management, Department of

Maybe Matthew, lacking a list of Armenian kings and their dates, knew that a king called Ashot (that is, Ashot III) occupied the throne in this year, and at some later point in

The paper deals with a problem of reconstruction of extremal boundary disturbances in a parabolic system.. The reconstruction is perfomed on the basis of inaccurate observations