literacy review 2021

New Curriculum Review Gives Failing Marks to Two Popular Reading Programs

• Share article

Corrected : A previous version of this story cited an incorrect figure for the share of K-2 early reading and special education teachers who said they use Fountas and Pinnell’s Leveled Literacy Intervention.

Two of the nation’s most popular early literacy programs that have been at the center of a debate over how to best teach reading both faced more new critiques in the past few weeks, receiving bottom marks on an outside evaluation of their materials.

EdReports—a nonprofit organization that reviews K-12 instructional materials in English/language arts, math, and science— published its evaluation of Fountas and Pinnell Classroom Tuesday, finding that the program didn’t meet expectations for text quality or alignment to standards. The release comes on the heels of the group’s negative evaluation last month of the Units of Study from the Teachers College Reading and Writing Project, another popular early reading program.

Together, the two reports received the lowest ratings EdReports has given for K-2 curricula in English/language arts, and they’re among the three lowest for ELA in grades 3-8.

“The materials don’t reflect the shifts—text quality and complexity—especially in K-2,” said Stephanie Stephens, EdReports’ ELA content specialist for early literacy, referencing key components of the Common Core State Standards—a big part of the organization’s review criteria.

These two literacy brands, both published by Heinemann, command large shares of the early reading market.

In 2019, a nationally representative EdWeek Research Center survey found that 43 percent of K-2 early reading and special education teachers use Fountas and Pinnell’s Leveled Literacy Intervention, the intervention companion to Fountas and Pinnell Classroom. The same survey found that 16 percent of teachers used the Units of Study for Teaching Reading.

Recently, these programs have faced criticism from educators and researchers that the instructional methods they use don’t align with, or in some cases contradict, the research on how to develop strong readers. Fountas and Pinnell has pushed back against these characterizations. Lucy Calkins, the director of the Teachers College Reading and Writing Project, has announced an upcoming revision to the Units of Study , set to be released in summer 2022. (EdReports reviewed the current version of the materials.)

How these programs attend to foundational skills—teaching students to recognize and manipulate the sounds in words, and then matching those sounds to written letters—is one of the main focuses of the critique. It’s also something that EdReports turned a renewed attention toward.

Fountas and Pinnell Classroom and Units of Study are two of the three K-2 reading programs to have gone through EdReports’ updated review tools for English/language arts, which “dig deeper” into the sequencing of foundational skills teaching. These new evaluation criteria also look for what EdReports calls “bloat,” whether all of the content in a set of materials can be taught in one year. Open Court, the third program evaluated with these new tools, partially met expectations.

In its two responses to the reviews on the EdReports website, Heinemann wrote that the EdReports’ rubrics aren’t a good fit for programs like Fountas and Pinnell Classroom and Units of Study.

“FPC greatly values the importance of responsive teaching and the teacher agency required to adjust, extend, and enrich learning based on individual student needs,” reads one response. “The EdReports rubric provides no way to measure these deeply valuable components of an effective literacy system.”

This ethos of teacher agency is one of the reasons that Fountas and Pinnell and the Units of Study are so popular with educators. Both give teachers, and oftentimes students, choice over materials and activities. Still, the EdReports reviews could affect whether schools continue to use them, said Morgan Polikoff, an associate professor of education at USC Rossier who studies K-12 curriculum and standards.

The reviews may influence state-level recommendations, or district leaders might reference them the next time they have to choose curriculum, Polikoff said. It’s also possible that parent advocates, like those in Minnesota who have petitioned their school board for better reading instruction , could use these reviews. “A bad EdReports rating could be another piece of evidence that those parents could potentially bring to bear” in attempts to jettison these programs, he said.

Kareem Weaver, a member of the Oakland NAACP Education Committee and the co-founder of FULCRUM, an Oakland group that advocates for evidence-based literacy instruction, said that the reviews could provide the impetus for school districts to reconsider the use of programs that he says don’t work for all kids.

“I’m really hoping it will make people do a double take,” he said.

Reviews critique text complexity, foundational skills

Since its launch in 2015, EdReports has recruited educators—teachers and other instructional leaders—to conduct its reviews, and to develop the rubrics used to judge materials. These rubrics measure alignment to the Common Core State Standards, usability in a classroom setting, and other indicators of quality, such as text complexity.

The company is one of the few organizations that provides external evaluations of curricula, and its reviews have a wide reach: As of 2020, EdReports said that at least 1,084 districts use its reviews, including 89 of the 200 largest districts in the country. (There are about 13,400 school districts in the United States.)

Still, not everyone agrees with EdReports’ conclusions. Publishers have critiqued the group’s methodology and rating system in the past, claiming that reviewers failed to consider supplemental materials and taking issue with the organization’s “gateway” system , which requires that a program meet the standards set for alignment before it can be evaluated on other features. EdReports made a few changes to its process after publishers pushed back on its first set of math reviews, though the gateway system remains.

Fountas and Pinnell Classroom failed to pass the first gateway. In K-2, reviewers said that core texts didn’t meet standards for quality or complexity, and that speaking and writing assignments didn’t require students to use evidence from the texts they read. EdReports also critiqued Fountas and Pinnell’s text leveling system, which it said was “not accompanied by an accurate text complexity analysis and a rationale for educational purpose and placement in the grade level.” The group gave a similar evaluation for the program in grades 3-8.

While the K-2 program’s word study lessons teach phonics, “the program does not present a research-based or evidence-based explanation for the sequence” of instruction, reviewers found. The report also claims that the program doesn’t consistently devote enough time to systematic instruction in phonological awareness, phonics, and fluency.

Units of Study also didn’t pass EdReports’ first gateway, which measures alignment to the common core. For grades K-2, reviewers said that texts featured in the materials “are not appropriately complex for the grade level and do not build in complexity over the course of the year.” They also noted that the program focused mostly on reading skills instruction, rather than “questions and tasks aligned to grade-level standards,” like asking students to use information from the text to support opinions.

Instruction in foundational reading skills like phonological awareness and phonics, they said, “lacks a cohesive and intentional scope and sequence.” The review also notes that the materials rely on cueing strategies for word identification: prompting students to draw on pictures, context, and sentence structure—along with letters—to figure out what words say. But research has shown that pulling students ’ attention away from the letters can lower the chances that they’ll use their knowledge of letter-sound correspondences to read through a word, making it less likely that they’ll be able to map the spelling to the spoken word in their memory.

Reviewers found text complexity lacking in grades 3-8, as well, and they said that the program lacks “a variety of regular, standards-aligned, text-based listening and speaking opportunities,” as well as opportunities for on-demand writing and systematic vocabulary development.

Not every program reviewed against EdReports’ new rubric received low marks. Open Court, the third program reviewed with the new tools, fared better. It partially met expectations at the first gateway, and also at the second gateway, which measures knowledge building. In grades K-2, reviewers reported a research-based approach to foundational skills instruction, but noted that there wasn’t enough practice with encoding—hearing sounds and converting them into written language.

Reviewers said that only some texts were “appropriately complex for the grade level,” and also said that the program missed opportunities for standards-aligned activities. They had the same critique for the grades 3-5 materials.

Publishers claim that EdReports tool is mismatched to their approach

As part of the review process, EdReports solicits publisher responses to its evaluations, posted publicly on its website. McGraw Hill, which publishes Open Court, and Heinemann both critiqued the review process in their responses.

The McGraw Hill response claimed that EdReports had overlooked end-of-unit opportunities for students to demonstrate knowledge, citing the curriculum’s unit-long “Inquiry” process.

Heinemann criticized the EdReports review process for omitting texts that students read outside of whole-group instruction.

In Units of Study, students only spend limited time in a whole-group “minilesson,” before moving on to the reading workshop, during which they apply the skills taught in the minilesson to independent reading, reading with a partner, or working with the teacher one-on-one or in small groups. FPC is structured similarly, with whole-class minilessons but also guided reading, independent reading, and student book clubs.

Heinemann’s responses argue that EdReports’ review design prioritizes textbook-style reading curricula, and fails to capture the quality of texts that students might read on their own or in small groups. The publisher did not respond to EdReports’ critiques of foundational skills instruction.

Stephens, of EdReports, said the group is not discriminating based on design and approach, but rather evaluating whether students have guaranteed access to grade-level text. “If they’re using independent reading at their level, there’s not a guarantee that’s at grade level,” she said.

Separately, Irene Fountas and Gay Su Pinnell, the program’s namesakes and founding authors, have begun to publish a 10-part blog series rebutting claims that their program is not aligned to reading science.

In the series , the authors defend their program’s use of cueing and other strategies that are central to their materials but which studies have shows are ineffective, like leveled reading groups .

“If a reader says ‘pony’ for ‘horse’ because of information from the pictures, that tells the teacher that the reader is using meaning information from the pictures, as well as the structure of the language, but is neglecting to use the visual information of the print,” one of the blogs reads. “His response is partially correct, but the teacher needs to guide him to stop and work for accuracy.” This idea is in direct contrast to what most cognitive scientists say about how strong readers process new words.

The Teachers College Reading and Writing Project, which writes the Units of Study, has also separately responded to the EdReports reviews. A post on the group’s website argues that the program has a different approach to meeting common-core standards than EdReports does. “At a fundamental level, ours is a paradigm where choice matters, where agency matters. EdReports uses a rubric that does not value those things.” TCRWP cited, for example, that when teachers were provided with a choice to assign on-demand writing, EdReports didn’t award full marks because the writing was not a requirement.

“This is always the challenge of applying a rubric to things that differ in a lot of ways. It’s an imperfect science,” said Polikoff, of USC Rossier. “The question is, is it better than not having it? And to me, the answer is yes.”

EdReports is working with one set of criteria, and can give teachers information about how programs line up according to that criteria—information that is often hard to come by, Polikoff said. There aren’t many avenues for teachers to find third-party evaluations of materials, he added.

Matthew Alexander, the director of elementary literacy and numeracy for Hall County Schools in Gainesville, Ga., said his district relies both on outside evaluation and internal data in making decisions about what programs to use.

Hall County uses one piece of Fountas and Pinnell Classroom—the Phonics, Spelling, and Word Study component—across its 20 elementary schools. The district also use its Benchmark Assessment System.

Alexander plans to discuss the review with other leaders in the school system, as it relates to their phonics instruction. But he’s hesitant to make any quick changes, because Hall County only started using Phonics, Spelling, and Word Study in the 2019-20 school year, right before the pandemic hit.

“If we were seeing that in our schools, that our kids were not making gains as readers, we would certainly look to see if we would shift our resources in a different direction. But with just three years of non-typical data, it’s hard to make that statement,” Alexander said.

Review tool changes address foundational skills, program ‘bloat’

The low ratings on some indicators in these reviews stem from changes to EdReports’ review tools.

In 2020, EdReports announced its first revision to its criteria and its evidence guides—a sort of handbook for reviewers that helps them identify evidence that programs meet, or don’t meet, the criteria. Part of this update are two key changes to how reviewers evaluate English/language arts materials.

One has to do with how reviewers approach foundational skills instruction in K-5. Criteria and evidence guides are more specific about when and how these skills should be taught.

For example, criteria that require systematic and explicit teaching in the alphabetic principle, phonemic awareness, phonics, and other skills has now been split into four subcategories, each with its own grade-by-grade breakdown of what students should be able to do in the evidence guide.

EdReports has also cut guidance that says programs “should instruct the teacher to employ syntactic or semantic cueing systems when the phonics patterns do not work or to confirm a word choice.” These changes have come as reporting and the work of reading researchers have turned increased public scrutiny toward cueing over the past few years.

What’s Changed in EdReports’ New Review Criteria?

Enlarge PDF

The revision brings the comprehensive ELA reviews more in line with the stricter criteria in stand-alone reviews of foundational skills, which EdReports launched in 2019, said Stephens. This way, she said, comprehensive reading programs will be judged as rigorously on their foundational skills components.

Still, Stephens thinks that the programs reviewed under the revised tools would have fared similarly under the originals. The revision provided “clarity,” she said, rather than an entire new scoring system.

The other change to the review process concerns what EdReports calls program “bloat.”

If a program says, for example, take 15 minutes a day for reading and 20 minutes for foundational skills, is that actually doable with the materials provided? Or is there too much content to feasibly get through? The program should offer a “clear and concise” pathway through the standards, Stephens said.

EdReports has also made some changes to its math review process, and has updated its criteria for gateway 3, which measures usability, across all subjects.

Louisa Moats, an early literacy expert and the lead writer of LETRS, a professional development program for reading teachers, has critiqued EdReports’ criteria in the past . She said that the new review tools are more closely aligned with research-based practice in reading instruction.

“These standards are much better for identifying practices in programs that are wildly off base. They’re a pretty good firewall in recognizing the programs that are wildly misaligned with reading science and with practices that have been shown to be ineffective with most kids. … That’s really good,” she said.

Still, she said, even if a program passes the review, its success or failure is going to come down to how the skills are taught in the classroom.

Weaver, in Oakland, said that the field needs more information about the effectiveness of popular reading materials. “What [EdReports] doesn’t do is it doesn’t talk about student achievement results. It doesn’t talk about how kids do with the program. And that’s fine, because they don’t claim to do that. But a lot of districts think they do,” he said.

“Alignment with the standards is the bare minimum that we should be able to expect from the curriculum,” Weaver said.

A version of this article appeared in the December 01, 2021 edition of Education Week as Popular Reading Programs Get Failing Marks

Sign Up for EdWeek Update

Edweek top school jobs.

Sign Up & Sign In

• Open access
• Published: 08 June 2022

A systematic review on digital literacy

• Hasan Tinmaz   ORCID: orcid.org/0000-0003-4310-0848 1 ,
• Yoo-Taek Lee   ORCID: orcid.org/0000-0002-1913-9059 2 ,
• Mina Fanea-Ivanovici   ORCID: orcid.org/0000-0003-2921-2990 3 &
• Hasnan Baber   ORCID: orcid.org/0000-0002-8951-3501 4  

Smart Learning Environments volume  9 , Article number:  21 ( 2022 ) Cite this article

55k Accesses

43 Citations

10 Altmetric

Metrics details

The purpose of this study is to discover the main themes and categories of the research studies regarding digital literacy. To serve this purpose, the databases of WoS/Clarivate Analytics, Proquest Central, Emerald Management Journals, Jstor Business College Collections and Scopus/Elsevier were searched with four keyword-combinations and final forty-three articles were included in the dataset. The researchers applied a systematic literature review method to the dataset. The preliminary findings demonstrated that there is a growing prevalence of digital literacy articles starting from the year 2013. The dominant research methodology of the reviewed articles is qualitative. The four major themes revealed from the qualitative content analysis are: digital literacy, digital competencies, digital skills and digital thinking. Under each theme, the categories and their frequencies are analysed. Recommendations for further research and for real life implementations are generated.

Introduction

The extant literature on digital literacy, skills and competencies is rich in definitions and classifications, but there is still no consensus on the larger themes and subsumed themes categories. (Heitin, 2016 ). To exemplify, existing inventories of Internet skills suffer from ‘incompleteness and over-simplification, conceptual ambiguity’ (van Deursen et al., 2015 ), and Internet skills are only a part of digital skills. While there is already a plethora of research in this field, this research paper hereby aims to provide a general framework of digital areas and themes that can best describe digital (cap)abilities in the novel context of Industry 4.0 and the accelerated pandemic-triggered digitalisation. The areas and themes can represent the starting point for drafting a contemporary digital literacy framework.

Sousa and Rocha ( 2019 ) explained that there is a stake of digital skills for disruptive digital business, and they connect it to the latest developments, such as the Internet of Things (IoT), cloud technology, big data, artificial intelligence, and robotics. The topic is even more important given the large disparities in digital literacy across regions (Tinmaz et al., 2022 ). More precisely, digital inequalities encompass skills, along with access, usage and self-perceptions. These inequalities need to be addressed, as they are credited with a ‘potential to shape life chances in multiple ways’ (Robinson et al., 2015 ), e.g., academic performance, labour market competitiveness, health, civic and political participation. Steps have been successfully taken to address physical access gaps, but skills gaps are still looming (Van Deursen & Van Dijk, 2010a ). Moreover, digital inequalities have grown larger due to the COVID-19 pandemic, and they influenced the very state of health of the most vulnerable categories of population or their employability in a time when digital skills are required (Baber et al., 2022 ; Beaunoyer, Dupéré & Guitton, 2020 ).

The systematic review the researchers propose is a useful updated instrument of classification and inventory for digital literacy. Considering the latest developments in the economy and in line with current digitalisation needs, digitally literate population may assist policymakers in various fields, e.g., education, administration, healthcare system, and managers of companies and other concerned organisations that need to stay competitive and to employ competitive workforce. Therefore, it is indispensably vital to comprehend the big picture of digital literacy related research.

Literature review

Since the advent of Digital Literacy, scholars have been concerned with identifying and classifying the various (cap)abilities related to its operation. Using the most cited academic papers in this stream of research, several classifications of digital-related literacies, competencies, and skills emerged.

Digital literacies

Digital literacy, which is one of the challenges of integration of technology in academic courses (Blau, Shamir-Inbal & Avdiel, 2020 ), has been defined in the current literature as the competencies and skills required for navigating a fragmented and complex information ecosystem (Eshet, 2004 ). A ‘Digital Literacy Framework’ was designed by Eshet-Alkalai ( 2012 ), comprising six categories: (a) photo-visual thinking (understanding and using visual information); (b) real-time thinking (simultaneously processing a variety of stimuli); (c) information thinking (evaluating and combining information from multiple digital sources); (d) branching thinking (navigating in non-linear hyper-media environments); (e) reproduction thinking (creating outcomes using technological tools by designing new content or remixing existing digital content); (f) social-emotional thinking (understanding and applying cyberspace rules). According to Heitin ( 2016 ), digital literacy groups the following clusters: (a) finding and consuming digital content; (b) creating digital content; (c) communicating or sharing digital content. Hence, the literature describes the digital literacy in many ways by associating a set of various technical and non-technical elements.

• Digital competencies

The Digital Competence Framework for Citizens (DigComp 2.1.), the most recent framework proposed by the European Union, which is currently under review and undergoing an updating process, contains five competency areas: (a) information and data literacy, (b) communication and collaboration, (c) digital content creation, (d) safety, and (e) problem solving (Carretero, Vuorikari & Punie, 2017 ). Digital competency had previously been described in a technical fashion by Ferrari ( 2012 ) as a set comprising information skills, communication skills, content creation skills, safety skills, and problem-solving skills, which later outlined the areas of competence in DigComp 2.1, too.

• Digital skills

Ng ( 2012 ) pointed out the following three categories of digital skills: (a) technological (using technological tools); (b) cognitive (thinking critically when managing information); (c) social (communicating and socialising). A set of Internet skill was suggested by Van Deursen and Van Dijk ( 2009 , 2010b ), which contains: (a) operational skills (basic skills in using internet technology), (b) formal Internet skills (navigation and orientation skills); (c) information Internet skills (fulfilling information needs), and (d) strategic Internet skills (using the internet to reach goals). In 2014, the same authors added communication and content creation skills to the initial framework (van Dijk & van Deursen). Similarly, Helsper and Eynon ( 2013 ) put forward a set of four digital skills: technical, social, critical, and creative skills. Furthermore, van Deursen et al. ( 2015 ) built a set of items and factors to measure Internet skills: operational, information navigation, social, creative, mobile. More recent literature (vaan Laar et al., 2017 ) divides digital skills into seven core categories: technical, information management, communication, collaboration, creativity, critical thinking, and problem solving.

It is worth mentioning that the various methodologies used to classify digital literacy are overlapping or non-exhaustive, which confirms the conceptual ambiguity mentioned by van Deursen et al. ( 2015 ).

• Digital thinking

Thinking skills (along with digital skills) have been acknowledged to be a significant element of digital literacy in the educational process context (Ferrari, 2012 ). In fact, critical thinking, creativity, and innovation are at the very core of DigComp. Information and Communication Technology as a support for thinking is a learning objective in any school curriculum. In the same vein, analytical thinking and interdisciplinary thinking, which help solve problems, are yet other concerns of educators in the Industry 4.0 (Ozkan-Ozen & Kazancoglu, 2021 ).

However, we have recently witnessed a shift of focus from learning how to use information and communication technologies to using it while staying safe in the cyber-environment and being aware of alternative facts. Digital thinking would encompass identifying fake news, misinformation, and echo chambers (Sulzer, 2018 ). Not least important, concern about cybersecurity has grown especially in times of political, social or economic turmoil, such as the elections or the Covid-19 crisis (Sulzer, 2018 ; Puig, Blanco-Anaya & Perez-Maceira, 2021 ).

Ultimately, this systematic review paper focuses on the following major research questions as follows:

Research question 1: What is the yearly distribution of digital literacy related papers?

Research question 2: What are the research methods for digital literacy related papers?

Research question 3: What are the main themes in digital literacy related papers?

Research question 4: What are the concentrated categories (under revealed main themes) in digital literacy related papers?

This study employed the systematic review method where the authors scrutinized the existing literature around the major research question of digital literacy. As Uman ( 2011 ) pointed, in systematic literature review, the findings of the earlier research are examined for the identification of consistent and repetitive themes. The systematic review method differs from literature review with its well managed and highly organized qualitative scrutiny processes where researchers tend to cover less materials from fewer number of databases to write their literature review (Kowalczyk & Truluck, 2013 ; Robinson & Lowe, 2015 ).

Data collection

To address major research objectives, the following five important databases are selected due to their digital literacy focused research dominance: 1. WoS/Clarivate Analytics, 2. Proquest Central; 3. Emerald Management Journals; 4. Jstor Business College Collections; 5. Scopus/Elsevier.

The search was made in the second half of June 2021, in abstract and key words written in English language. We only kept research articles and book chapters (herein referred to as papers). Our purpose was to identify a set of digital literacy areas, or an inventory of such areas and topics. To serve that purpose, systematic review was utilized with the following synonym key words for the search: ‘digital literacy’, ‘digital skills’, ‘digital competence’ and ‘digital fluency’, to find the mainstream literature dealing with the topic. These key words were unfolded as a result of the consultation with the subject matter experts (two board members from Korean Digital Literacy Association and two professors from technology studies department). Below are the four key word combinations used in the search: “Digital literacy AND systematic review”, “Digital skills AND systematic review”, “Digital competence AND systematic review”, and “Digital fluency AND systematic review”.

A sequential systematic search was made in the five databases mentioned above. Thus, from one database to another, duplicate papers were manually excluded in a cascade manner to extract only unique results and to make the research smoother to conduct. At this stage, we kept 47 papers. Further exclusion criteria were applied. Thus, only full-text items written in English were selected, and in doing so, three papers were excluded (no full text available), and one other paper was excluded because it was not written in English, but in Spanish. Therefore, we investigated a total number of 43 papers, as shown in Table 1 . “ Appendix A ” shows the list of these papers with full references.

Data analysis

The 43 papers selected after the application of the inclusion and exclusion criteria, respectively, were reviewed the materials independently by two researchers who were from two different countries. The researchers identified all topics pertaining to digital literacy, as they appeared in the papers. Next, a third researcher independently analysed these findings by excluded duplicates A qualitative content analysis was manually performed by calculating the frequency of major themes in all papers, where the raw data was compared and contrasted (Fraenkel et al., 2012 ). All three reviewers independently list the words and how the context in which they appeared and then the three reviewers collectively decided for how it should be categorized. Lastly, it is vital to remind that literature review of this article was written after the identification of the themes appeared as a result of our qualitative analyses. Therefore, the authors decided to shape the literature review structure based on the themes.

As an answer to the first research question (the yearly distribution of digital literacy related papers), Fig.  1 demonstrates the yearly distribution of digital literacy related papers. It is seen that there is an increasing trend about the digital literacy papers.

Yearly distribution of digital literacy related papers

Research question number two (The research methods for digital literacy related papers) concentrates on what research methods are employed for these digital literacy related papers. As Fig.  2 shows, most of the papers were using the qualitative method. Not stated refers to book chapters.

Research methods used in the reviewed articles

When forty-three articles were analysed for the main themes as in research question number three (The main themes in digital literacy related papers), the overall findings were categorized around four major themes: (i) literacies, (ii) competencies, (iii) skills, and (iv) thinking. Under every major theme, the categories were listed and explained as in research question number four (The concentrated categories (under revealed main themes) in digital literacy related papers).

The authors utilized an overt categorization for the depiction of these major themes. For example, when the ‘creativity’ was labelled as a skill, the authors also categorized it under the ‘skills’ theme. Similarly, when ‘creativity’ was mentioned as a competency, the authors listed it under the ‘competencies’ theme. Therefore, it is possible to recognize the same finding under different major themes.

Major theme 1: literacies

Digital literacy being the major concern of this paper was observed to be blatantly mentioned in five papers out forty-three. One of these articles described digital literacy as the human proficiencies to live, learn and work in the current digital society. In addition to these five articles, two additional papers used the same term as ‘critical digital literacy’ by describing it as a person’s or a society’s accessibility and assessment level interaction with digital technologies to utilize and/or create information. Table 2 summarizes the major categories under ‘Literacies’ major theme.

Computer literacy, media literacy and cultural literacy were the second most common literacy (n = 5). One of the article branches computer literacy as tool (detailing with software and hardware uses) and resource (focusing on information processing capacity of a computer) literacies. Cultural literacy was emphasized as a vital element for functioning in an intercultural team on a digital project.

Disciplinary literacy (n = 4) was referring to utilizing different computer programs (n = 2) or technical gadgets (n = 2) with a specific emphasis on required cognitive, affective and psychomotor skills to be able to work in any digital context (n = 3), serving for the using (n = 2), creating and applying (n = 2) digital literacy in real life.

Data literacy, technology literacy and multiliteracy were the third frequent categories (n = 3). The ‘multiliteracy’ was referring to the innate nature of digital technologies, which have been infused into many aspects of human lives.

Last but not least, Internet literacy, mobile literacy, web literacy, new literacy, personal literacy and research literacy were discussed in forty-three article findings. Web literacy was focusing on being able to connect with people on the web (n = 2), discover the web content (especially the navigation on a hyper-textual platform), and learn web related skills through practical web experiences. Personal literacy was highlighting digital identity management. Research literacy was not only concentrating on conducting scientific research ability but also finding available scholarship online.

Twenty-four other categories are unfolded from the results sections of forty-three articles. Table 3 presents the list of these other literacies where the authors sorted the categories in an ascending alphabetical order without any other sorting criterion. Primarily, search, tagging, filtering and attention literacies were mainly underlining their roles in information processing. Furthermore, social-structural literacy was indicated as the recognition of the social circumstances and generation of information. Another information-related literacy was pointed as publishing literacy, which is the ability to disseminate information via different digital channels.

While above listed personal literacy was referring to digital identity management, network literacy was explained as someone’s social networking ability to manage the digital relationship with other people. Additionally, participatory literacy was defined as the necessary abilities to join an online team working on online content production.

Emerging technology literacy was stipulated as an essential ability to recognize and appreciate the most recent and innovative technologies in along with smart choices related to these technologies. Additionally, the critical literacy was added as an ability to make smart judgements on the cost benefit analysis of these recent technologies.

Last of all, basic, intermediate, and advanced digital assessment literacies were specified for educational institutions that are planning to integrate various digital tools to conduct instructional assessments in their bodies.

Major theme 2: competencies

The second major theme was revealed as competencies. The authors directly categorized the findings that are specified with the word of competency. Table 4 summarizes the entire category set for the competencies major theme.

The most common category was the ‘digital competence’ (n = 14) where one of the articles points to that category as ‘generic digital competence’ referring to someone’s creativity for multimedia development (video editing was emphasized). Under this broad category, the following sub-categories were associated:

Problem solving (n = 10)

Safety (n = 7)

Information processing (n = 5)

Content creation (n = 5)

Communication (n = 2)

Digital rights (n = 1)

Digital emotional intelligence (n = 1)

Digital teamwork (n = 1)

Big data utilization (n = 1)

Artificial Intelligence utilization (n = 1)

Virtual leadership (n = 1)

Self-disruption (in along with the pace of digitalization) (n = 1)

Like ‘digital competency’, five additional articles especially coined the term as ‘digital competence as a life skill’. Deeper analysis demonstrated the following points: social competences (n = 4), communication in mother tongue (n = 3) and foreign language (n = 2), entrepreneurship (n = 3), civic competence (n = 2), fundamental science (n = 1), technology (n = 1) and mathematics (n = 1) competences, learning to learn (n = 1) and self-initiative (n = 1).

Moreover, competencies were linked to workplace digital competencies in three articles and highlighted as significant for employability (n = 3) and ‘economic engagement’ (n = 3). Digital competencies were also detailed for leisure (n = 2) and communication (n = 2). Furthermore, two articles pointed digital competencies as an inter-cultural competency and one as a cross-cultural competency. Lastly, the ‘digital nativity’ (n = 1) was clarified as someone’s innate competency of being able to feel contented and satisfied with digital technologies.

Major theme 3: skills

The third major observed theme was ‘skills’, which was dominantly gathered around information literacy skills (n = 19) and information and communication technologies skills (n = 18). Table 5 demonstrates the categories with more than one occurrence.

Table 6 summarizes the sub-categories of the two most frequent categories of ‘skills’ major theme. The information literacy skills noticeably concentrate on the steps of information processing; evaluation (n = 6), utilization (n = 4), finding (n = 3), locating (n = 2) information. Moreover, the importance of trial/error process, being a lifelong learner, feeling a need for information and so forth were evidently listed under this sub-category. On the other hand, ICT skills were grouped around cognitive and affective domains. For instance, while technical skills in general and use of social media, coding, multimedia, chat or emailing in specific were reported in cognitive domain, attitude, intention, and belief towards ICT were mentioned as the elements of affective domain.

Communication skills (n = 9) were multi-dimensional for different societies, cultures, and globalized contexts, requiring linguistic skills. Collaboration skills (n = 9) are also recurrently cited with an explicit emphasis for virtual platforms.

‘Ethics for digital environment’ encapsulated ethical use of information (n = 4) and different technologies (n = 2), knowing digital laws (n = 2) and responsibilities (n = 2) in along with digital rights and obligations (n = 1), having digital awareness (n = 1), following digital etiquettes (n = 1), treating other people with respect (n = 1) including no cyber-bullying (n = 1) and no stealing or damaging other people (n = 1).

‘Digital fluency’ involved digital access (n = 2) by using different software and hardware (n = 2) in online platforms (n = 1) or communication tools (n = 1) or within programming environments (n = 1). Digital fluency also underlined following recent technological advancements (n = 1) and knowledge (n = 1) including digital health and wellness (n = 1) dimension.

‘Social intelligence’ related to understanding digital culture (n = 1), the concept of digital exclusion (n = 1) and digital divide (n = 3). ‘Research skills’ were detailed with searching academic information (n = 3) on databases such as Web of Science and Scopus (n = 2) and their citation, summarization, and quotation (n = 2).

‘Digital teaching’ was described as a skill (n = 2) in Table 4 whereas it was also labelled as a competence (n = 1) as shown in Table 3 . Similarly, while learning to learn (n = 1) was coined under competencies in Table 3 , digital learning (n = 2, Table 4 ) and life-long learning (n = 1, Table 5 ) were stated as learning related skills. Moreover, learning was used with the following three terms: learning readiness (n = 1), self-paced learning (n = 1) and learning flexibility (n = 1).

Table 7 shows other categories listed below the ‘skills’ major theme. The list covers not only the software such as GIS, text mining, mapping, or bibliometric analysis programs but also the conceptual skills such as the fourth industrial revolution and information management.

Major theme 4: thinking

The last identified major theme was the different types of ‘thinking’. As Table 8 shows, ‘critical thinking’ was the most frequent thinking category (n = 4). Except computational thinking, the other categories were not detailed.

Computational thinking (n = 3) was associated with the general logic of how a computer works and sub-categorized into the following steps; construction of the problem (n = 3), abstraction (n = 1), disintegration of the problem (n = 2), data collection, (n = 2), data analysis (n = 2), algorithmic design (n = 2), parallelization & iteration (n = 1), automation (n = 1), generalization (n = 1), and evaluation (n = 2).

A transversal analysis of digital literacy categories reveals the following fields of digital literacy application:

Technological advancement (IT, ICT, Industry 4.0, IoT, text mining, GIS, bibliometric analysis, mapping data, technology, AI, big data)

Networking (Internet, web, connectivity, network, safety)

Information (media, news, communication)

Creative-cultural industries (culture, publishing, film, TV, leisure, content creation)

Academia (research, documentation, library)

Citizenship (participation, society, social intelligence, awareness, politics, rights, legal use, ethics)

Education (life skills, problem solving, teaching, learning, education, lifelong learning)

Professional life (work, teamwork, collaboration, economy, commerce, leadership, decision making)

Personal level (critical thinking, evaluation, analytical thinking, innovative thinking)

This systematic review on digital literacy concentrated on forty-three articles from the databases of WoS/Clarivate Analytics, Proquest Central, Emerald Management Journals, Jstor Business College Collections and Scopus/Elsevier. The initial results revealed that there is an increasing trend on digital literacy focused academic papers. Research work in digital literacy is critical in a context of disruptive digital business, and more recently, the pandemic-triggered accelerated digitalisation (Beaunoyer, Dupéré & Guitton, 2020 ; Sousa & Rocha 2019 ). Moreover, most of these papers were employing qualitative research methods. The raw data of these articles were analysed qualitatively using systematic literature review to reveal major themes and categories. Four major themes that appeared are: digital literacy, digital competencies, digital skills and thinking.

Whereas the mainstream literature describes digital literacy as a set of photo-visual, real-time, information, branching, reproduction and social-emotional thinking (Eshet-Alkalai, 2012 ) or as a set of precise specific operations, i.e., finding, consuming, creating, communicating and sharing digital content (Heitin, 2016 ), this study reveals that digital literacy revolves around and is in connection with the concepts of computer literacy, media literacy, cultural literacy or disciplinary literacy. In other words, the present systematic review indicates that digital literacy is far broader than specific tasks, englobing the entire sphere of computer operation and media use in a cultural context.

The digital competence yardstick, DigComp (Carretero, Vuorikari & Punie, 2017 ) suggests that the main digital competencies cover information and data literacy, communication and collaboration, digital content creation, safety, and problem solving. Similarly, the findings of this research place digital competencies in relation to problem solving, safety, information processing, content creation and communication. Therefore, the findings of the systematic literature review are, to a large extent, in line with the existing framework used in the European Union.

The investigation of the main keywords associated with digital skills has revealed that information literacy, ICT, communication, collaboration, digital content creation, research and decision-making skill are the most representative. In a structured way, the existing literature groups these skills in technological, cognitive, and social (Ng, 2012 ) or, more extensively, into operational, formal, information Internet, strategic, communication and content creation (van Dijk & van Deursen, 2014 ). In time, the literature has become richer in frameworks, and prolific authors have improved their results. As such, more recent research (vaan Laar et al., 2017 ) use the following categories: technical, information management, communication, collaboration, creativity, critical thinking, and problem solving.

Whereas digital thinking was observed to be mostly related with critical thinking and computational thinking, DigComp connects it with critical thinking, creativity, and innovation, on the one hand, and researchers highlight fake news, misinformation, cybersecurity, and echo chambers as exponents of digital thinking, on the other hand (Sulzer, 2018 ; Puig, Blanco-Anaya & Perez-Maceira, 2021 ).

This systematic review research study looks ahead to offer an initial step and guideline for the development of a more contemporary digital literacy framework including digital literacy major themes and factors. The researchers provide the following recommendations for both researchers and practitioners.

Recommendations for prospective research

By considering the major qualitative research trend, it seems apparent that more quantitative research-oriented studies are needed. Although it requires more effort and time, mixed method studies will help understand digital literacy holistically.

As digital literacy is an umbrella term for many different technologies, specific case studies need be designed, such as digital literacy for artificial intelligence or digital literacy for drones’ usage.

Digital literacy affects different areas of human lives, such as education, business, health, governance, and so forth. Therefore, different case studies could be carried out for each of these unique dimensions of our lives. For instance, it is worth investigating the role of digital literacy on lifelong learning in particular, and on education in general, as well as the digital upskilling effects on the labour market flexibility.

Further experimental studies on digital literacy are necessary to realize how certain variables (for instance, age, gender, socioeconomic status, cognitive abilities, etc.) affect this concept overtly or covertly. Moreover, the digital divide issue needs to be analysed through the lens of its main determinants.

New bibliometric analysis method can be implemented on digital literacy documents to reveal more information on how these works are related or centred on what major topic. This visual approach will assist to realize the big picture within the digital literacy framework.

Recommendations for practitioners

The digital literacy stakeholders, policymakers in education and managers in private organizations, need to be aware that there are many dimensions and variables regarding the implementation of digital literacy. In that case, stakeholders must comprehend their beneficiaries or the participants more deeply to increase the effect of digital literacy related activities. For example, critical thinking and problem-solving skills and abilities are mentioned to affect digital literacy. Hence, stakeholders have to initially understand whether the participants have enough entry level critical thinking and problem solving.

Development of digital literacy for different groups of people requires more energy, since each group might require a different set of skills, abilities, or competencies. Hence, different subject matter experts, such as technologists, instructional designers, content experts, should join the team.

It is indispensably vital to develop different digital frameworks for different technologies (basic or advanced) or different contexts (different levels of schooling or various industries).

These frameworks should be updated regularly as digital fields are evolving rapidly. Every year, committees should gather around to understand new technological trends and decide whether they should address the changes into their frameworks.

Understanding digital literacy in a thorough manner can enable decision makers to correctly implement and apply policies addressing the digital divide that is reflected onto various aspects of life, e.g., health, employment, education, especially in turbulent times such as the COVID-19 pandemic is.

Lastly, it is also essential to state the study limitations. This study is limited to the analysis of a certain number of papers, obtained from using the selected keywords and databases. Therefore, an extension can be made by adding other keywords and searching other databases.

Availability of data and materials

The authors present the articles used for the study in “ Appendix A ”.

Baber, H., Fanea-Ivanovici, M., Lee, Y. T., & Tinmaz, H. (2022). A bibliometric analysis of digital literacy research and emerging themes pre-during COVID-19 pandemic. Information and Learning Sciences . https://doi.org/10.1108/ILS-10-2021-0090 .

Article   Google Scholar  

Beaunoyer, E., Dupéré, S., & Guitton, M. J. (2020). COVID-19 and digital inequalities: Reciprocal impacts and mitigation strategies. Computers in Human Behavior, 111 , 10642. https://doi.org/10.1016/j.chb.2020.106424

Blau, I., Shamir-Inbal, T., & Avdiel, O. (2020). How does the pedagogical design of a technology-enhanced collaborative academic course promote digital literacies, self-regulation, and perceived learning of students? The Internet and Higher Education, 45 , 100722. https://doi.org/10.1016/j.iheduc.2019.100722

Carretero, S., Vuorikari, R., & Punie, Y. (2017). DigComp 2.1: The Digital Competence Framework for Citizens with eight proficiency levels and examples of use (No. JRC106281). Joint Research Centre, https://publications.jrc.ec.europa.eu/repository/handle/JRC106281

Eshet, Y. (2004). Digital literacy: A conceptual framework for survival skills in the digital era. Journal of Educational Multimedia and Hypermedia , 13 (1), 93–106, https://www.learntechlib.org/primary/p/4793/

Eshet-Alkalai, Y. (2012). Thinking in the digital era: A revised model for digital literacy. Issues in Informing Science and Information Technology, 9 (2), 267–276. https://doi.org/10.28945/1621

Ferrari, A. (2012). Digital competence in practice: An analysis of frameworks. JCR IPTS, Sevilla. https://ifap.ru/library/book522.pdf

Fraenkel, J. R., Wallen, N. E., & Hyun, H. H. (2012). How to design and evaluate research in education (8th ed.). Mc Graw Hill.

Google Scholar  

Heitin, L. (2016). What is digital literacy? Education Week, https://www.edweek.org/teaching-learning/what-is-digital-literacy/2016/11

Helsper, E. J., & Eynon, R. (2013). Distinct skill pathways to digital engagement. European Journal of Communication, 28 (6), 696–713. https://doi.org/10.1177/0267323113499113

Kowalczyk, N., & Truluck, C. (2013). Literature reviews and systematic reviews: What is the difference ? . Radiologic Technology, 85 (2), 219–222.

Ng, W. (2012). Can we teach digital natives digital literacy? Computers & Education, 59 (3), 1065–1078. https://doi.org/10.1016/j.compedu.2012.04.016

Ozkan-Ozen, Y. D., & Kazancoglu, Y. (2021). Analysing workforce development challenges in the Industry 4.0. International Journal of Manpower . https://doi.org/10.1108/IJM-03-2021-0167

Puig, B., Blanco-Anaya, P., & Perez-Maceira, J. J. (2021). “Fake News” or Real Science? Critical thinking to assess information on COVID-19. Frontiers in Education, 6 , 646909. https://doi.org/10.3389/feduc.2021.646909

Robinson, L., Cotten, S. R., Ono, H., Quan-Haase, A., Mesch, G., Chen, W., Schulz, J., Hale, T. M., & Stern, M. J. (2015). Digital inequalities and why they matter. Information, Communication & Society, 18 (5), 569–582. https://doi.org/10.1080/1369118X.2015.1012532

Robinson, P., & Lowe, J. (2015). Literature reviews vs systematic reviews. Australian and New Zealand Journal of Public Health, 39 (2), 103. https://doi.org/10.1111/1753-6405.12393

Sousa, M. J., & Rocha, A. (2019). Skills for disruptive digital business. Journal of Business Research, 94 , 257–263. https://doi.org/10.1016/j.jbusres.2017.12.051

Sulzer, A. (2018). (Re)conceptualizing digital literacies before and after the election of Trump. English Teaching: Practice & Critique, 17 (2), 58–71. https://doi.org/10.1108/ETPC-06-2017-0098

Tinmaz, H., Fanea-Ivanovici, M., & Baber, H. (2022). A snapshot of digital literacy. Library Hi Tech News , (ahead-of-print).

Uman, L. S. (2011). Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychiatry, 20 (1), 57–59.

Van Deursen, A. J. A. M., Helsper, E. J., & Eynon, R. (2015). Development and validation of the Internet Skills Scale (ISS). Information, Communication & Society, 19 (6), 804–823. https://doi.org/10.1080/1369118X.2015.1078834

Van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2009). Using the internet: Skills related problems in users’ online behaviour. Interacting with Computers, 21 , 393–402. https://doi.org/10.1016/j.intcom.2009.06.005

Van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2010a). Measuring internet skills. International Journal of Human-Computer Interaction, 26 (10), 891–916. https://doi.org/10.1080/10447318.2010.496338

Van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2010b). Internet skills and the digital divide. New Media & Society, 13 (6), 893–911. https://doi.org/10.1177/1461444810386774

van Dijk, J. A. G. M., & Van Deursen, A. J. A. M. (2014). Digital skills, unlocking the information society . Palgrave MacMillan.

van Laar, E., van Deursen, A. J. A. M., van Dijk, J. A. G. M., & de Haan, J. (2017). The relation between 21st-century skills and digital skills: A systematic literature review. Computer in Human Behavior, 72 , 577–588. https://doi.org/10.1016/j.chb.2017.03.010

Download references

This research is funded by Woosong University Academic Research in 2022.

Author information

Authors and affiliations.

AI & Big Data Department, Endicott College of International Studies, Woosong University, Daejeon, South Korea

Hasan Tinmaz

Endicott College of International Studies, Woosong University, Daejeon, South Korea

Yoo-Taek Lee

Department of Economics and Economic Policies, Bucharest University of Economic Studies, Bucharest, Romania

Mina Fanea-Ivanovici

Abu Dhabi School of Management, Abu Dhabi, United Arab Emirates

Hasnan Baber

You can also search for this author in PubMed   Google Scholar

Contributions

The authors worked together on the manuscript equally. All authors have read and approved the final manuscript.

Corresponding author

Correspondence to Hasnan Baber .

Ethics declarations

Competing of interests.

The authors declare that they have no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Tinmaz, H., Lee, YT., Fanea-Ivanovici, M. et al. A systematic review on digital literacy. Smart Learn. Environ. 9 , 21 (2022). https://doi.org/10.1186/s40561-022-00204-y

Download citation

Received : 23 February 2022

Accepted : 01 June 2022

Published : 08 June 2022

DOI : https://doi.org/10.1186/s40561-022-00204-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Digital literacy
• Systematic review
• Qualitative research

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Open access
• Published: 07 July 2023

Digital transformation and digital literacy in the context of complexity within higher education institutions: a systematic literature review

• Silvia Farias-Gaytan   ORCID: orcid.org/0000-0001-5858-5900 1 ,
• Ignacio Aguaded 2 &
• Maria-Soledad Ramirez-Montoya 1  

Humanities and Social Sciences Communications volume  10 , Article number:  386 ( 2023 ) Cite this article

8761 Accesses

11 Citations

3 Altmetric

Metrics details

• Science, technology and society

The incessant changes in technology generate new products and services, presenting multiple opportunities for the complex educational environment. Consequently, higher education institutions must be attentive to these changes to ensure that students have the knowledge and skills necessary for the work environment. This research aimed to identify studies related to digital transformation and digital literacy in higher education institutions through a systematic study of literature. The search resulted in 830 articles published in the Scopus and Web of Science databases from 2015 to 2022. Quality questions, inclusion and exclusion criteria were applied where 202 articles were selected for the study. The results show (a) interest of educational institutions in empirical studies where technologies are incorporated for didactic purposes, (b) challenges of opportunity in training programs to develop digital competences of teachers and students, (c) little interest in the development of media literacy, (d) the methodological aspects of the studies allow exploring new perspectives of digital transformation in higher education. This article may be of interest to academics, decision-makers and trainers of future professionals to introduce educational technology into learning processes in line with the complex demands of the world of work and society.

Similar content being viewed by others

Education reform and change driven by digital technology: a bibliometric study from a global perspective

Exploring factors influencing pre-service teacher’s digital teaching competence and the mediating effects of data literacy: empirical evidence from China

A bibliometric analysis of knowledge mapping in Chinese education digitalization research from 2012 to 2022

Introduction.

At the end of the twentieth century, the emergence of the internet led to organizations’ digital transformation from analogous to digital information (“digitization”), followed by the incorporation of information technologies into business processes (“digitalization”) (Verhoef et al., 2019 ). Several authors make no distinction between digitalization and digital transformation (Hess et al., 2016 ; Tratkowska, 2020 ; Xiao, 2020 ). Verhoef et al. (2021) propose that digital transformation goes further; its impact generates new business models and value creation. Organizations’ various areas are influenced and committed to change to remain relevant (Anderson and Ellerby, 2018 ). For this study, the term “digital transformation” (DT) was used.

Digital transformation goes beyond just incorporating technologies. An example of this is to consider that digital technologies and automation demand that the workforce develop digital skills and human-centered skills (Digital Transformation Expert Panel, 2021 ), which impacts aspects such as culture, processes, as well as the strategy of the organization (Fischer et al., 2020 ) consequently the organization must make the necessary adjustments for its effective implementation. These impacts reach all business lines including higher education.

Higher education institutions, in particular, must be attentive to the changes in the environment and society to ensure that students have the knowledge and skills demanded. Morin ( 2019 , 2020 ) invites us to think of complexity as a challenge of contemporary thinking, which requires a reform of our way of thinking, since classical scientific thinking was previously built on three foundations: order, separability and reason, but developments in science have undermined these foundations. In this sense, high-level competences such as reasoning for complexity become indispensable in the formation of critical, systemic, scientific and innovative thinking (Ramírez-Montoya et al., 2022 ; Vázquez-Parra et al., 2022 ). Complex environments require active (Patiño et al., 2023 ), collaborative (Romero-Rodríguez et al., 2022 ), open education (Suárez-Brito et al., 2022 ) and digital technology systems (George-Reyes et al., 2023 ; Ponce et al., 2022 ). Because of this, education systems around the world have made various efforts to address the influence of digital technologies and DT, such as UNESCO’s ‘Working Group on Education on Digital Skills and Work’ (UNESCO, 2017 ), the “Bologna Digital 2020” report in Europe (Rampelt et al., 2019 ), the “Outline of China’s National Plan for Medium and Long-term Educational Reform and Development (2010–2020)” of the Chinese government (Xiao, 2020 ), and the Digital Educational Agenda ADE.mx in Mexico (SEP, 2019 ). Likewise, this transformation has triggered the development of topics of interest that intertwine education with technology as proposed by González-Pérez et al. ( 2019 ) (Table 1 ):

Currently, skills performed in digital environments have been added to the basic skills performed in analog environments. Digital literacy involves mastering software and hardware, development, analysis, and interaction with digital content (Chetty et al., 2018 ). Skills such as problem-solving and applying technology were derived from digital technologies (UNESCO, 2017 ), and are considered essential for workers to adapt to digital transformation (Digital Transformation Expert Panel, 2021 ). As new technology becomes available to users, it demands from them continuous learning to remain relevant.

Due to the above, it is worthwhile to research the use and impact of technologies in the educational field on the delivery of content, pedagogical practices, and evaluation and management of learning (Williamson and Hogan, 2020 ), as well as its impact on users, teachers and students. Systematic studies of related literature are scarce, during this investigation, we found four reviews ranging from 2020 to 2021; they focused on the development of digital skills of students (Starkey, 2020 ), or university professors (Bilbao Aiastui et al., 2021 ), on digital competence assessment processes and methods in higher education (Sillat et al., 2021 ), and one focused on media literacy (Manca et al., 2021 ). This study contributes to the subject by integrating digital transformation practices in education, as well as studies on digital competencies of students and teachers, which are key roles of higher education institutions.

This article aims to identify recent studies (2015–2022) related to the issues of digital transformation and digital literacy in higher education institutions through a systematic study of literature. The study seeks to answer what educational trends higher education institutions are using, as well as what studies they have carried out in this regard, and the opportunities they have identified to advance in digital transformation and digital literacy. This study can serve as a basis for higher education institutions interested in exploring educational innovations to identify these implementations and their outcomes and seek inter-institutional collaborations with common interests.

Methodology

The study was conducted through a systematic literature review (SLR) based on the guidelines proposed by Kitchenham and Charters ( 2007 , p. 11), “a means to identify, evaluate and interpret relevant research on a particular topic". The phases to carry out the study were adapted from Kitchenham et al. ( 2010 ) and are described as follows:

Phase 1 Planning: The research starts from the objective of analyzing studies related to the topics of digital transformation and digital literacy in higher education institutions. A series of questions were defined to guide the review; these questions were derived from the integration of elements that would contribute to identify trends in digital transformation, research methods and instruments used in assessing such practices, as well as opportunities for future research; such findings would be useful to other researchers interested in the subject (Kitchenham and Charters, 2007 ) (Table 2 ).

Phase 2 Execution: The articles were selected using inclusion criteria such as the publication period between 2015 and 2022, studies in higher education institutions, focus on students and professors, and empirical research or mixed studies. Articles not arbitrated or published in languages other than Spanish and English were excluded (Table 3 ).

The search was conducted based on the above criteria in the Scopus and WoS databases (Table 4 ). 202 studies met the specified criteria (Fig. 1 ).

The flowchart presents the process of classifying the articles based on inclusion and exclusion criteria and the resulting number of articles. The flowchart was adapted from Moher et al. ( 2009 ).

Phase 3 Results: The results of each research question were analyzed to determine the educational trends higher education institutions incorporate, the studies they have carried out in this regard, and the opportunities to advance in digital transformation.

Results are presented based on the research questions. For data analysis, Excel and Power BI were used. The database is available at the following link: https://doi.org/10.6084/m9.figshare.21972170.v2 .

RQ1 What are the trends and topics addressed in the articles?

The trends identified were determined based on the emerging themes of educational technology by González-Pérez et al. ( 2019 ), highlighting digital pedagogies (166 articles), which “link pedagogical and technological supports to adapt to each area of knowledge” (González-Pérez et al., 2019 , p. 189). Examples include implementing the “blended learning” strategy (Power and Kannara, 2016 ; Tang and Chaw, 2016 ; Wang et al., 2022 ) and studies on digital skills (Ting, 2015 ; Tømte et al., 2015 ; Torres-Gastelú et al., 2019 ) and media competencies (Koc and Barut, 2016 ; Jormand et al., 2022 ) Second place went to adaptive technologies (21 articles) that “introduce systems that adapt to the needs of society and encourage learning” (González-Pérez et al., 2019 , p. 189). Examples are the use of Web 2.0 tools (Sichel et al., 2019 ), e-portfolio (Carl and Strydom, 2017 ), e-Learning (Divya and Mohamed Haneefa, 2018 ; Feriady et al., 2020 ), adaptive systems (Murray and Pérez, 2015 ), and social networks (Amaro-Jiménez et al., 2016 ; Robles Moral and Fernández Díaz, 2021 ).

To a lesser extent, the rest of the trends were found in 6 articles on technological models (Andrew et al., 2018 ; Bond et al., 2018 ; Kör et al., 2017 ) and open technologies (Cronin, 2017 ; Paskevicius and Irvine, 2019 ; Spieler et al., 2020 ). Finally, there were articles on disruptive technologies that use extended reality resources (Astudillo Torres, 2019 ; Bucea-Manea-Ţoniş et al., 2020 ) and smart technologies for mobile learning (Pinto Molina et al., 2019 ) (Fig. 2 ).

The rectangles show the proportion and number of published articles classified according to specific emerging issues in the use of educational technology as proposed by González-Pérez et al. ( 2019 ).

The analysis of the author’s keywords highlighted the issue of digital competence and digital literacy (de Ovando Calderón and Jara, 2019 ; Liu et al., 2020 ; Oria, 2020 ) and, to a lesser extent, digital teaching and media literacy (Tetep and Suparman, 2019 ; Sánchez-Caballé and Esteve-Mon, 2022 ) Also notable were keywords regarding technology in these topics (Roa Banquez et al., 2021 ; Rodríguez-Hoyos et al., 2021 ) (Fig. 3 ).

Main keywords identified in the reviewed articles.

RQ2 What are the trends in research methods observed in the articles?

Studies on digital literacy and digital transformation increased in the last three years; in 2022, it rose 53% compared to the previous year. The most commonly used research method (56%) was quantitative (Guillén-Gámez and Peña, 2020 ; Kim et al., 2018 ; Miguel-Revilla et al., 2020 ). Qualitative methods were found in similar proportions (Kajee, 2018 ; Önger and Çetin, 2018 ), and mixed methods (Pozos Pérez and Tejada Fernández, 2018 ; Techataweewan and Prasertsin, 2018 ) (Fig. 4 ).

Number of published articles during 2015–2022 classified by research method, qualitative, quantitative or mixed method.

Also, the highest number of articles was found in Spain, which represents 32% of the total, and shows an interest in digital transformation and digital literacy issues in higher education institutions; followed by Turkey with ten, the United States with nine, and Chile, China and Mexico with seven research papers each (Fig. 5 ).

Proportion of published articles distributed by country.

RQ3 What are the main findings in digital transformation and digital literacy?

The principal findings center on studies on the level of digital skills, and use of educational technology (Fig. 6 ). The most significant number of articles (121) focuses on digital competency (Blayone, 2018 ; Hong and Kim, 2018 ; Torres-Coronas and Vidal-Blasco, 2015 ; Zhao et al., 2021 ). The use of educational technology involves 2.0 technologies (Novakovich, 2016 ), virtual communities (Robin Sullivan et al., 2018 ), online education, or e-Learning (Aznar Díaz et al., 2019 ; Hamutoğlu et al., 2019 ; Gumede and Badriparsad, 2022 ). Regarding media literacy, it was found in eight articles (Altamirano Galván, 2021 ; Brown et al., 2016 ; Koc and Barut, 2016 ; Jormand et al., 2022 ; Leier and Gruber, 2021 ; Olivia-Dumitrina et al., 2019 ; Reyna and Meier, 2018 ; Robles Moral and Fernández Díaz, 2021 ). Two additional issues identified were environmental protection (Amador-Alarcón et al., 2022 ) and educational process (Makarova et al., 2021 ) both of interest to today’s situation faced by higher education institutions.

Trends, topics and main findings from the reviewed articles.

RQ4 What are the authors’ recommendations for future studies? And RQ5 What are the opportunities identified in the studies?

By correlating these two questions, we identified four opportunities regarding digital literacy and digital transformation (Fig. 7 ); first, that higher education institutions have training programs for both students and teachers to help them develop digital skills (Igbo and Imo, 2020 ; Martzoukou et al., 2020 ; Sandí Delgado, 2020 ), media skills (López-Meneses et al., 2020 ; Reyna and Meier, 2018 ; Romero-Rodriguez et al., 2016 ), and critical thinking (Kocak et al., 2021 ; Nagel et al., 2022 ; Vetter and Sarraf, 2020 ). Second, that the development of skills requires to enhance learning design by incorporating new didactic strategies, and educational technologies in academic programs (Boulton, 2020 ; del Prete and Almenara, 2020 ; Foster, 2020 ; Liesa-Orús et al., 2020 ; McGrew et al., 2019 ), and that the impact of these changes improves learning (Castellanos et al., 2017 ; Dafonte-Gómez et al., 2018 ; Sosa Díaz and Palau Martín, 2018 ).

Frequency of recommendations and opportunities for future studies.

On the other hand, methodological recommendations for future studies included incorporating new instruments and variables to collect more information (Kamardeen and Samaratunga, 2020 ; Khalil and Srinivasan, 2019 ; Varga-Atkins, 2020 ; Vetter and Sarraf, 2020 ). Others pointed to increasing the sample size (Amhag et al., 2019 ; Kolodziejczyk et al., 2020 ; Munoz-Repiso and del Pozo, 2016 ; Pozo-Sánchez et al., 2020 ). To a lesser extent, longitudinal studies were recommended to test the models used (He et al., 2018 ; Johnston, 2020 ). In addition, we found that 28% of the studies did not include recommendations, and 31% did not include opportunities for future studies.

RQ6 What are the stated limitations in digital literacy studies involving digital transformation?

The limitations indicated in the studies refer primarily to the small sample size (45%) (Arango et al., 2020 ; Romero-Tena et al., 2020 ; Tugtekin and Koc, 2020 ). To a lesser extent, limitations were found with the instrument used to carry out the study (Heuling et al., 2021 ; Nikou and Aavakare, 2021 ; Sánchez-Caballé and Esteve-Mon, 2022 ). Problems with the technology used was another limitation highlighted in eight studies (Castellano, 2016 ; Pozo-Sánchez et al., 2020 ). Finally, seven studies reported limitation regarding its feasibility (Dafonte-Gómez et al., 2018 ; Fázik and Steinerová, 2020 ; Kerr et al., 2019 ) and one on the low response obtained (Myyry et al., 2022 ); 36% of the studies did not include limitations (Fig. 8 ).

Frequency of limitations found in the reviewed articles. The figure does not include data from articles that did not specify the limitations (36%).

Incorporating educational trends and new technologies in the educational environment has highlighted the need to continue developing skills that allow their adoption by teachers and students. The interest in digital pedagogies and the study of digital competencies were relevant trends among higher education institutions aiming to use adaptive, intelligent, open, or disruptive technologies and technological models (Fig. 2 ). The transition from the analog to the digital world in both processes and products of organizations is part of their journey towards digital transformation (Hess et al., 2016 ; Tratkowska, 2020 ). It also includes organizational and cultural changes among users and operators (Anderson and Ellerby, 2018 ). However, we must point out that technology is not the end in itself but should be a means to facilitate learning.

Therefore, studies employing the scientific method where the benefit can be determined are relevant, and those that examine areas of opportunity by adopting technologies in the learning process. In the last three years, empirical studies on incorporating educational innovations in teaching practice in higher education institutions increased, most applying mainly quantitative methods (Figs. 4 and 5 ). Spain is the country that stands out with the most studies (64). In some cases, the impetus for these efforts has come from the establishment of educational strategies at the national (SEP, 2019 ; Xiao, 2020 ) and regional level (Rampelt et al., 2019 ). These studies denote international interest in the influence of digital transformation, and digital literacy on the educational process.

Digital technology skills and knowledge are hallmarks of the twenty-first-century generations. Digital literacy and educational technology accounted for 95% of the study findings, and only 4% focused on media literacy. Required job competencies include software and hardware skills, critical thinking, information analysis, and the ability to create and communicate content (Chetty et al., 2018 ; Silva et al., 2021 ; UNESCO, 2017 ). “Workers who can combine ‘human’ skills like empathy, cooperation and negotiation with cognitive skills such as problem-solving, will thrive in an economy that increasingly relies on both types of skill” (Digital Transformation Expert Panel, 2021 ). As the work environment and education continue to evolve along new technologies.

In addition to the conceptual components, the methodological aspects of the studies allow exploring new perspectives of digital transformation in higher education. In the studies reviewed, 44% of the recommendations concerned using new instruments, and exploring new variables, while 56% were about sample size increase and longitudinal studies (Fig. 7 ). Although they have not been conceived or designed for the educational field, the technologies are embedded today in the learning process (González-Pérez et al., 2019 ). Studies on their adoption allow testing and validating methodologies and instruments to have reliable data for their implementation (García-Ruiz et al., 2014 ). Though used simultaneously by teachers and students, the adoption of technology may require the implementation of different strategies or approaches to meet the needs of each group.

The ability to learn and unlearn is being tested by constantly introducing technologies into human activities. The opportunities reported by the studies coincide with the need for institutions to have training programs to develop skills for larger groups (27%). In the case of students, other topics of interest are the use of technology, enriched learning experiences, and security and privacy issues (Fig. 7 ). Organizations’ digital transformation strategy must consider the training of their members and their users because the skills required for the job become increasingly specialized (Anderson and Ellerby, 2018 ; Hess et al., 2016 ; Verhoef et al., 2019 ). In order to get the best out of educational technology, users are required to have a minimum level of digital literacy (Kerr et al., 2019 ). Higher education institutions are a fertile place to continue studies on digital transformation and the development of digital literacy of their members.

Therefore, empirical studies on the experiences and challenges faced by higher education institutions in adopting technologies in the learning process and strategies implemented to train teachers and students are relevant. The limitations reported in the studies focused on methodological issues, with the sample size being the most crucial aspect to consider (45%). These studies were carried out in groups managed by the researcher, making it difficult to project the results. The systematic literature review methodology emphasizes the analysis of variables to answer research questions so that similarities and differences among studies can be identified (Kitchenham et al., 2010 ; Kitchenham and Charters, 2007 ). Inter-institutional collaboration can contribute to achieving results that help find joint strategies to promote the adoption of educational innovations and the development of competencies in both teachers and students.

Limitations

This study was limited to trends in higher education institutions in a specific period of time (2015–2022). Another limitation was the selection of two databases, Scopus and Web of Science, which although they include high-impact journals, articles from other databases were not considered; future research can continue the timeline and include other systems and databases.

Conclusions

The digital transformation of higher education institutions goes beyond its impact on administrative and operational processes. The study showed that teachers have incorporated educational trends, new pedagogies and technologies for didactic purposes, and this has highlighted the need to develop the level of digital literacy of both teachers and students. Higher education institutions, as trainers of future professionals, must acknowledge the need for digital transformation and act upon to develop strategies so students and teachers are prepared for the demands of the workplace.

The pandemic spurred the urgency of developing digital skills for both teachers and students. Technologies they used for socializing and leisure became necessary tools for study and work. Higher education institutions are conducting studies on their experiences of adopting educational technologies and the impact on their users. Although related empirical studies on media literacy were scarce, since it is linked to the use of technology, future studies have an opportunity to assess how it develops in the following years. These should examine teachers’ and students’ performance, their critical capacity as media users, and content creators.

The development of teachers’ digital competencies involves not only the mastery of technology but also the improvement of their teaching practice with the appropriate pedagogical use of technology to contribute to student learning. There are opportunities for higher education institutions in measuring digital competencies to find strengths and weaknesses to focus their training programs. The same applies to students, who should be provided with the relevant training for the development of digital skills and prevent the lack of these from becoming an obstacle to their performance in the classroom.

This study aimed to identify the state of digital transformation and digital literacy in higher education institutions and their impact on students and teachers. Digital transformation and new technologies are generating complex environments that demand the development of digital and high-level skills. Technological progress provides opportunities to enhance the learning process. Research must continue to assess the performance and students’ learning gains. This study can serve as a basis for higher education institutions interested in exploring educational innovations to identify these implementations and their outcomes and seek inter-institutional collaborations with common interests.

Data availability

The datasets generated during and/or analyzed in the current study are available in Figshare repository: https://doi.org/10.6084/m9.figshare.21972170 .

Altamirano Galván SG (2021) Perfil de alfabetización mediática de estudiantes y docentes de educación superior. CPU-e, Revista de Investigación Educativa 32:88–110. https://doi.org/10.25009/cpue.v0i32.2735

Article   Google Scholar  

Amador-Alarcón MP, Torres-Gastelú CA, Lagunes-Domínguez A, Medina-Cruz H, Argüello-Rosales CA (2022) Perceptions of environmental protection of university students: a look through digital competences in Mexico. Sustainability 14(18):11141. https://doi.org/10.3390/su141811141

Amaro-Jiménez C, Hungerford-Kresser H, Pole K (2016) Teaching with a technological twist: exit tickets via Twitter in literacy classrooms. J Adolesc Adult Lit 60(3):305–313. https://doi.org/10.1002/jaal.572

Amhag L, Hellström L, Stigmar M (2019) Teacher educators’ use of digital tools and needs for digital competence in higher education. J Digit Learn Teach Educ 35(4):203–220. https://doi.org/10.1080/21532974.2019.1646169

Anderson C, Ellerby W (2018) Digital Maturity Model. Achieving digital maturity to drive growth. In: Deloitte (Issue February). https://www2.deloitte.com/content/dam/Deloitte/global/Documents/Technology-Media-Telecommunications/deloitte-digital-maturity-model.pdf

Andrew M, Taylorson J, Langille DJ, Grange A, Williams N (2018) Student attitudes towards technology and their preferences for learning tools/devices at two universities in the UAE. J Inf Technol Educ Res 17:309–344. https://doi.org/10.28945/4111

Arango DAG, Fernández JEV, Carrillo JAO, Rojas ÓAC, Villa CFH (2020) Dimensions of digital competence in university teachers: Relational analysis based on components | Dimensiones de competencia digital en docentes universitarios: Análisis relacional basado en componentes. RISTI-Revista Iberica de Sistemas e Tecnologias de Informacao 2020(E28):945–960

Google Scholar  

Astudillo Torres MP (2019) Aplicación de la realidad aumentada en las prácticas educativas universitarias. RELATEC: Revista Latinoamericana de Tecnología Educativa 18(2):203–218. https://doi.org/10.17398/1695-288X.18.2.203

Aznar Díaz I, Cáceres Reche MP, Romero Rodríguez JM (2019) Digital competence of an E-learning tutor: an emerging model of good teaching practices in ICT. Texto Livre 12(3):49–68. https://doi.org/10.17851/1983-3652.12.3.49-68

Bilbao Aiastui E, Arruti Gómez A, Carballedo Morillo R (2021) A systematic literature review about the level of digital competences defined by DigCompEdu in higher education. Aula Abierta 50(4):841–850. https://doi.org/10.17811/rifie.50.4.2021.841-850

Blayone T (2018) Reexamining digital-learning readiness in higher education: Positioning digital competencies as key factors and a profile application as a readiness tool. Int J E-Learn Corp Gov Healthcare High Educ 17(4):425–451

Bond M, Marín VI, Dolch, Bedenlier S, Zawacki-Richter O (2018) Digital transformation in German higher education: student and teacher perceptions and usage of digital media. Int J Educ Technol High Educ 15(1). https://doi.org/10.1186/s41239-018-0130-1

Boulton P (2020) Digitally proficient but disconnected from the outdoor world? A reflection on pedagogies used in an Early Years degree in higher education. J Appl Res High Educ 13(1):195–210. https://doi.org/10.1108/JARHE-03-2019-0066

Brown C, Czerniewicz L, Noakes T (2016) Online content creation: looking at students’ social media practices through a Connected Learning lens. Learn Media Technol 41(1):140–159. https://doi.org/10.1080/17439884.2015.1107097

Bucea-Manea-Ţoniş R, Bucea-Manea-Ţoniş R, Simion VE, Ilic D, Braicu C, Manea N (2020) Sustainability in higher education: The relationship between work-life balance and XR e-learning facilities. Sustainability (Switzerland), 12(14). https://doi.org/10.3390/su12145872

Carl A, Strydom S (2017) e-Portfolio as reflection tool during teaching practice: The interplay between contextual and dispositional variables. South Afr J Educ 37(1). https://doi.org/10.15700/saje.v37n1a1250

Castellano J (2016) Advanced media english–a modern PrOCALL course. CALL-EJ 17(1):52–66

Castellanos A, Sánchez C, Calderero JF (2017) Nuevos modelos tecnopedagógicos. Competencia digital de los alumnos universitarios. Revista Electrónica de Investigación Educativa 19(1):1–9. https://doi.org/10.24320/redie.2017.19.1.1148

Chetty K, Qigui L, Gcora N, Josie J, Wenwei L, Fang C (2018). Bridging the digital divide: measuring digital literacy. Economics. https://doi.org/10.5018/economics-ejournal.ja.2018-23

Cronin C (2017) Openness and praxis: exploring the use of open educational practices in higher education. Int Rev Res Open Dist Learn 18(5):15–34. https://doi.org/10.19173/irrodl.v18i5.3096

Dafonte-Gómez A, García-Crespo O, Ramahi-García D (2018) ‘Flipped learning’ y competencia digital: diseño tecnopedagógico y percepción del alumnado universitario. Index Comunicación 8(2):275–294. http://plataformarevistascomunicacion.org/2018/11/articulo-flipped-learning-competencia-digital-diseno-tecnopedagogico-percepcion-del-alumnado-universitario/

de Ovando Calderón JS, Jara VJ (2019) Digital competence of health sciences teachers of a Chilean university Pixel-Bit, Revista de Medios y Educacion 56:193–211. https://doi.org/10.12795/pixelbit.2019.i56.10

del Prete A, Almenara JC (2020) Use of the virtual learning environment among higher education teaching staff: a gender analysis | El uso del Ambiente Virtual de Aprendizaje entre el profesorado de educación superior: Un análisis de género. Revista de Educacion a Distancia, 20(62). https://doi.org/10.6018/RED.400061

Digital Transformation Expert Panel. (2021) The Learning Country. Digital Transformation Skills Strategy. https://www.digitalskillsformation.org.au/wp-content/uploads/2021/05/Digital-Transformation-Skills-Strategy-010521.pdf?v=2

Divya P, Mohamed Haneefa K (2018) Digital reading competency of students: A study in universities in Kerala. DESIDOC J Libr Inf Technol 38(2):88–94. https://doi.org/10.14429/djlit.38.2.12233

Fázik J, Steinerová J (2020) Technologies, knowledge and truth: the three dimensions of information literacy of university students in Slovakia. J Doc https://doi.org/10.1108/JD-05-2020-0086

Feriady M, Nurkhin A, Mahmud N, Setiani R, Astuti DP (2020) Influence of organizational suport and digital literacy on lecturer acceptance of e-learning in indonesia: a modification of technologi acceptance model. Int J Sci Technol Res 9(1):2229–2233

Fischer M, Imgrund F, Janiesch C, Winkelmann A (2020) Strategy archetypes for digital transformation: defining meta objectives using business process management. Inf Manage. https://doi.org/10.1016/j.im.2019.103262

Foster B (2020) Information literacy beyond librarians: a data/methods triangulation approach to investigating disciplinary IL teaching practices. Evid Based Libr Inf Pract 15(1):20–37. https://doi.org/10.18438/EBLIP29635

García-Ruiz R, Ramírez-García A, Rodríguez-Rosell M (2014) Educación en alfabetización mediática para una nueva ciudadanía prosumidora. Comunicar XXII(43):15–23. https://doi.org/10.3916/C43-2014-01

George-Reyes CE, Ramírez-Montoya MS, López-Caudana EO (2023) Imbrication of the Metaverse in the complexity of education 4.0: Approach from an analysis of the literatura [Imbricación del Metaverso en la complejidad de la educación 4.0: Aproximación desde un análisis de la literatura]. Pixel-Bit. Revista de Medios y Educación 66:199–237. https://doi.org/10.12795/pixelbit.97337

González-Pérez LI, Ramírez-Montoya MS, García-Peñalvo FJ (2019) Innovación educativa en estudios sobre el desarrollo y uso de la tecnología: un mapeo sistemático. In: Ramírez Montoya MS, & Valenzuela González JR (eds.), Innovación educativa: tendencias globales de investigación e implicaciones prácticas (Primera, pp. 137–160). Ediciones OCTAEDRO, S. L. https://octaedro.com/libro/innovacion-educativa-tendencias-globales-de-investigacion-e-implicaciones-practicas/

Guillén-Gámez FD, Peña MP (2020) Univariate analysis of digital competence in physical education: an empirical study | Análisis Univariante de la Competencia Digital en Educación Física: Un estudio empírico. Retos 37:326–332

Gumede L, Badriparsad N (2022) Online teaching and learning through the students’ eyes –Uncertainty through the COVID-19 lockdown: a qualitative case study in Gauteng province, South Africa. Radiography 28(1):193–198. https://doi.org/10.1016/j.radi.2021.10.018

Article   CAS   PubMed   Google Scholar  

Hamutoğlu NB, Savaşçı M, Sezen-Gültekin G (2019) Digital literacy skills and attitudes towards e-learning. J Educ Fut, August, 93–107. https://doi.org/10.30786/jef.509293

He T, Zhu C, Questier F (2018) Predicting digital informal learning: an empirical study among Chinese University students. Asia Pac Educ Rev 19(1):79–90. https://doi.org/10.1007/s12564-018-9517-x

Hess T, Matt C, Benlian A, Wiesböck F (2016) Options for formulating a digital transformation strategy. MIS Q Executive 15(2):103–119. https://www.researchgate.net/publication/291349362

Heuling LS, Wild S, Vest A (2021) Digital competences of prospective engineers and science teachers: a latent profile and correspondence analysis. Int J Educ Math Sci Technol 9(4):760–782. https://doi.org/10.46328/ijemst.1831

Hong AJ, Kim HJ (2018) College students’ Digital Readiness for Academic Engagement (DRAE) Scale: scale development and validation. Asia-Pac Educ Res 27(4):303–312. https://doi.org/10.1007/s40299-018-0387-0

Igbo HU, Imo NT (2020) Digital libraries and access to information in Nigerian Federal Universities: The impact of technology variables. J Inf Knowl Manage 19(2). https://doi.org/10.1142/S0219649220500136

Johnston N (2020) The shift towards digital literacy in Australian University libraries: developing a digital literacy framework. J Aust Libr Inf Asso 69(1):93–101. https://doi.org/10.1080/24750158.2020.1712638

Article   MathSciNet   Google Scholar  

Jormand H, Bashirian S, Barati M, Rezapur-Shahkolai F, Babamiri M (2022) Evaluation of a web-based randomized controlled trial educational intervention based on media literacy on preventing substance abuse among college students, applying the integrated social marketing approach: a study protocol. Trials 23(1):1006. https://doi.org/10.1186/s13063-022-06913-6

Article   PubMed   PubMed Central   Google Scholar  

Kajee L (2018) Teacher education students engaging with digital identity narratives. S Afr J Educ 38(2). https://doi.org/10.15700/saje.v38n2a1501

Kamardeen I, Samaratunga M (2020) Digiexplanation driven assignments for personalising learning in construction education. Constr Econ Build 20(3):103–123. https://doi.org/10.5130/AJCEB.v20i3.7000

Kerr J, Dale VH, Gyurko F (2019) Evaluation of a MOOC Design Mapping Framework (MDMF): Experiences of academics and learning technologists. Electron J E-Learn 17(1):38–51. http://www.ejel.org/volume17/issue1/p38

Khalil R, Srinivasan V (2019) Massive open online courses/ MOOCS: a gateway to enrich e-learning in management education. Int J Innov Creat Change 7(5):112–124

Kim HJ, Hong AJ, Song H-D (2018) The relationships of family, perceived digital competence and attitude, and learning agility in sustainable student engagement in higher education. Sustainability (Switzerland), 10(12). https://doi.org/10.3390/su10124635

Kitchenham B, Pretorius R, Budgen D, Brereton OP, Turner M, Niazi M, Linkman S (2010) Systematic literature reviews in software engineering-a tertiary study. Inf Softw Technol 52(8):792–805. https://doi.org/10.1016/j.infsof.2010.03.006

Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. https://doi.org/10.1145/1134285.1134500

Koc M, Barut E (2016) Development and validation of New Media Literacy Scale (NMLS) for university students. Comput Hum Behav 63:834–843. https://doi.org/10.1016/j.chb.2016.06.035

Kocak O, Coban M, Aydin A, Cakmak N (2021) The mediating role of critical thinking and cooperativity in the 21st century skills of higher education students. Think Ski Creat 42:100967. https://doi.org/10.1016/j.tsc.2021.100967

Kolodziejczyk I, Gibbs P, Nembou C, Sagrista MR (2020) Digital skills at divine word university, Papua New Guinea. IAFOR J Educ 8(2):107–124. https://doi.org/10.22492/ije.8.2.06

Kör H, Erbay H, Engin M, Dünder E (2017) An examination of the correlation between science and technology attitudes scale, frequency of smartphone usage scale and lifelong learning scale scores using the structural equation model. J Baltic Sci Educ 16(1):86–99

Leier V, Gruber A (2021) Team New Zealand-Sweden-Germany: a joint venture exploring language learning in digital spaces. JALT CALL J 17(3):298–324. https://doi.org/10.29140/jaltcall.v17n3.410

Liesa-Orús M, Latorre-Cosculluela C, Vázquez-Toledo S, Sierra-Sánchez V (2020) The technological challenge facing higher education professors: perceptions of ICT tools for developing 21st Century skills. Sustainability (Switzerland), 12(13). https://doi.org/10.3390/su12135339

Liu ZJ, Tretyakova N, Fedorov V, Kharakhordina M (2020) Digital literacy and digital didactics as the basis for new learning models development. Int J Emerg Technol Learn 15(14):4–18. https://doi.org/10.3991/ijet.v15i14.14669

Article   CAS   Google Scholar  

López-Meneses E, Sirignano FM, Vázquez-Cano E, Ramírez-Hurtado JM (2020) University students’ digital competence in three areas of the DigCom 2.1 model: a comparative study at three European universities. Australas J Educ Technol 36(3):69–88. https://doi.org/10.14742/AJET.5583

Makarova O, Ldokova G, Egorova R (2021) Analysis of students’ views of the quality of pedagogical education in Russia. Int J Educ Math Sci Technol 9(3):462–481. https://doi.org/10.46328/ijemst.1624

Manca S, Bocconi S, Gleason B (2021) “Think globally, act locally”: a glocal approach to the development of social media literacy. Comput Educ 160:104025. https://doi.org/10.1016/j.compedu.2020.104025

Martzoukou K, Fulton C, Kostagiolas P, Lavranos C (2020) A study of higher education students’ self-perceived digital competences for learning and everyday life online participation. J Doc 76(6):1413–1458. https://doi.org/10.1108/JD-03-2020-0041

McGrew S, Smith M, Breakstone J, Ortega T, Wineburg S (2019) Improving university students’ web savvy: an intervention study. Br J Educ Psychol 89(3):485–500. https://doi.org/10.1111/bjep.12279

Article   PubMed   Google Scholar  

Miguel-Revilla D, Martínez-Ferreira JM, Sánchez-Agustí M (2020) Assessing the digital competence of educators in social studies: an analysis in initial teacher training using the TPACK-21 model. Australas J Educ Technol 36(2):1–12. https://doi.org/10.14742/ajet.5281

Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med 6(7):1–6. https://doi.org/10.1371/journal.pmed.1000097

Morin E (2019) Pensar la Complejidad. Crisis ymetamorfosis. Universitat de Valencia

Morin E (2020) Cambiemos de Via. Lecciones de la pandemia. Paidos

Munoz-Repiso A, del Pozo M (2016) Analysis of the digital competences of graduates of university degrees to be a teacher. Revista Latinoamericana de Tecnologia Educativa-RELATEC 15(2):155–168. https://doi.org/10.17398/1695-288X.15.2.155

Murray MC, Pérez J (2015) Informing and performing: a study comparing adaptive learning to traditional learning. Inform Sci 18(1):111–125. https://doi.org/10.28945/2165

Myyry L, Kallunki V, Katajavuori N, Repo S, Tuononen T, Anttila H, Kinnunen P, Haarala-Muhonen A, Pyörälä E (2022) COVID-19 accelerating academic teachers’ digital competence in distance teaching. Front Educ 7. https://doi.org/10.3389/feduc.2022.770094

Nagel M-T, Zlatkin-Troitschanskaia O, Fischer J (2022) Validation of newly developed tasks for the assessment of generic Critical Online Reasoning (COR) of university students and graduates. Front Educ 7. https://doi.org/10.3389/feduc.2022.914857

Nikou S, Aavakare M (2021) An assessment of the interplay between literacy and digital technology in higher education. Educ Inf Technol 26(4):3893–3915. https://doi.org/10.1007/s10639-021-10451-0

Novakovich J (2016) Fostering critical thinking and reflection through blog-mediated peer feedback. J Comput Assist Learn 32(1):16–30. https://doi.org/10.1111/jcal.12114

Olivia-Dumitrina N, Casanovas M, Capdevila Y (2019) Academic writing and the internet: Cyber-plagiarism amongst university students. J New Approach Educ Res 8(2):112–125. https://doi.org/10.7821/naer.2019.7.407

Önger S, Çetin T (2018) An investigation into digital literacy views of social studies preservice teachers in the context of authentic learning. Rev Int Geogr Educ Online 8(1):109–124

Oria B (2020) Edmodo como herramienta de aprendizaje telecolaborativo online en el aula de inglés. Encuentro. 28:49–70

Paskevicius M, Irvine V (2019) Open education and learning design: Open pedagogy in praxis. J Interact Media Educ 2019(1). https://doi.org/10.5334/jime.512

Patiño A, Ramírez-Montoya MS, Buenestado-Fernández M (2023) Active learning and education 4.0 for complex thinking training: analysis of two case studies in open education. Smart Learn Environ 10(8). https://doi.org/10.1186/s40561-023-00229-x

Pinto Molina M, Gómez-Hernández J-A, Sales D, Cuevas-Cerveró A, Fernández Pascual R, Caballero Mariscal D, Guerrero-Quesada DJ, Navalón C (2019) Aprender y enseñar competencias digitales en un entorno móvil: avances de una investigación aplicada a profesorado y alumnado universitario de Ciencias Sociales. Revista Ibero-Americana de Ciencia Da Informacao 12(2):585–596. https://doi.org/10.26512/10.26512/rici.v

Ponce P, Ramirez R, Ramirez-Montoya MS, Molina A, MacCleery B, Ascanio M (2022) From understanding a simple DC motor to developing an electric vehicle AI controller rapid prototype using MATLAB-Simulink, real-time simulation and complex thinking. Front Educ 7:941972. https://doi.org/10.3389/feduc.2022.941972

Power J, Kannara V (2016) Best-practice model for technology enhanced learning in the creative arts. Res Learn Technol 24. https://doi.org/10.3402/rlt.v24.30231

Pozos Pérez KV, Tejada Fernández J (2018) Competencias digitales en docentes de educación superior: niveles de dominio y necesidades formativas. Revista Digital de Investigación En Docencia Universitaria 12(2):59–87. https://doi.org/10.19083/ridu.2018.712

Pozo-Sánchez S, López-Belmonte J, Fuentes-Cabrera A, Moreno-Guerrero A-J (2020) Incidence of retro-innovation in higher education. Radio and television as complementary tools when using the educational model known as flipped learning. Form Univ 13(3):139–146. https://doi.org/10.4067/S0718-50062020000300139

Ramírez-Montoya MS, Castillo-Martínez IM, Sanabria-Zepeda JC, Miranda J (2022) Complex thinking in the framework of education 4.0 and open innovation—a systematic literature review. J Open Innov Technol Market Complex 8(4). https://doi.org/10.3390/joitmc8010004

Rampelt F, Orr D, Knoth A (2019) Bologna Digital 2020–White Paper on Digitalisation in the European Higher Education Area. Publisher: Geschäftsstelle Hochschulforum Digitalisierung beim Stifterverband für die Deutsche Wissenschaft e.V

Reyna J, Meier P (2018). Using the Learner-Generated Digital Media (LGDM) framework in tertiary science education: a pilot study. Educ Sci 8(3). https://doi.org/10.3390/educsci8030106

Roa Banquez K, Viviana Rojas Torres CG, González Rincón LJ, Ortiz Ortiz EG (2021) El docente en la era 4.0: una propuesta de formación digital que fortalezca el proceso de enseñanza y aprendizaje. Revista Virtual Universidad Católica Del Norte 63:126–160. https://doi.org/10.35575/rvucn.n63a6

Robin Sullivan R, Neu V, Yang F (2018) Faculty development to promote effective instructional technology integration: a qualitative examination of reflections in an online community. Online Learn J 22(4):341–359. https://doi.org/10.24059/olj.v22i4.1373

Robles Moral FJ, Fernández Díaz M (2021) Future primary school teachers’ digital competence in teaching science through the use of social media. Sustainability 13(5):2816. https://doi.org/10.3390/su13052816

Rodríguez-Hoyos C, Fueyo Gutiérrez A, Hevia Artime I (2021) Competencias digitales del profesorado para innovar en la docencia universitaria. Analizando el uso de los dispositivos móviles. Pixel-Bit, Revista de Medios y Educación 61:71–97. https://doi.org/10.12795/pixelbit.86305

Romero-Rodriguez L, Torres-Toukoumidis D, Pérez-Rodríguez M, Aguaded I (2016) Analfanauts and fourth screen: lack of infodiets and media and information literacy in Latin American University Students. Fonseca-J Commun 12:11–25. https://core.ac.uk/download/pdf/60674744.pdf

Romero-Rodríguez JM, Ramirez-Montoya, MS, Glasserman-Morales LD, Ramos Navas-Parejo M (2022) Collaborative online international learning between Spain and Mexico: a microlearning experience to enhance creativity in complexity. Educ+Train. https://doi.org/10.1108/ET-07-2022-0259

Romero-Tena R, Barragán-Sánchez R, Llorente-Cejudo C, Palacios-Rodríguez A (2020) The challenge of initial training for early childhood teachers. A cross sectional study of their digital competences. Sustainability (Switzerland), 12(11). https://doi.org/10.3390/su12114782

Sánchez-Caballé A, Esteve-Mon FM (2022) Digital teaching competence of university teachers: a comparative study at two European universities. Australas J Educ Technol 50–61. https://doi.org/10.14742/ajet.7408

Sandí Delgado JC (2020) Desarrollo de competencias digitales en el profesorado a través de juegos serios: un estudio de caso aplicado en la Universidad de Costa Rica (UCR). E-Ciencias de La Información. https://doi.org/10.15517/eci.v10i2.38946

SEP (2019) Agenda Digital Educativa ADE.mx. In: Secretaría de Educación Pública. https://infosen.senado.gob.mx/sgsp/gaceta/64/2/2020-02-05-1/assets/documentos/Agenda_Digital_Educacion.pdf

Sichel CE, Javdani S, Ueberall S, Liggett R (2019) Leveraging youths’ digital literacies: the E-Responder social media violence interruption model and pilot evaluation. J Prevent Intervent Community 47(2):76–89. https://doi.org/10.1080/10852352.2019.1582145

Sillat LH, Tammets K, Laanpere M (2021) Digital competence assessment methods in higher education: a systematic literature review. Educ Scie 11(8):402. https://doi.org/10.3390/educsci11080402

Silva MB, Borges G, Fantin M, Almeida M, Aguaded I (2021) Media competence in children aged 9 to 12 in Brazilian settings. Revista Brasileira de Ciências Da Comunicação 44(1):21–45. http://portcom.intercom.org.br/revistas/index.php/revistaintercom/article/view/3487/2499

Sosa Díaz MJ, Palau Martín RF (2018) Flipped classroom para adquirir la competencia digital docente: una experiencia didáctica en la Educación Superior. Pixel-Bit Revista de Medios y Educación 52:37–54. https://doi.org/10.12795/pixelbit.2018.i52.03

Spieler B, Grandl M, Ebner M, Slany W (2020) Bridging the gap: a computer science Pre-MOOC for first semester students making with kids view project. Electron J E-Learn. https://doi.org/10.34190/EJEL.20.18.3.004

Starkey L (2020) A review of research exploring teacher preparation for the digital age. Camb J Educ 50(1):37–56. https://doi.org/10.1080/0305764X.2019.1625867

Suárez-Brito P, López-Caudana EO, Baena-Rojas JJ, Ramírez-Montoya MS (2022) Eliciting complex thinking through open educational resource projects. J Soc Stud Educ Res 13(4):56–77. https://jsser.org/index.php/jsser/article/view/4472

Tang CM, Chaw LY (2016) Digital literacy: A prerequisite for effective learning in a blended learning environment? Electron J E-Learn 14(1):54–65

Techataweewan W, Prasertsin U (2018) Development of digital literacy indicators for Thai undergraduate students using mixed method research. Kasetsart J Soc Sci 39(2):215–221. https://doi.org/10.1016/j.kjss.2017.07.001

Tetep, Suparman A (2019) Students’ digital media literacy: effects on social character. Int J Recent Technol Eng 8(2 Special Issue 9):394–399. https://doi.org/10.35940/ijrte.B1091.0982S919

Ting Y-L (2015) Tapping into students’ digital literacy and designing negotiated learning to promote learner autonomy. Internet High Educ 26:25–32. https://doi.org/10.1016/j.iheduc.2015.04.004

Tømte C, Enochsson A-B, Buskqvist U, Kårstein A (2015) Educating online student teachers to master professional digital competence: The TPACK-framework goes online. Comput Educat 84:26–35. https://doi.org/10.1016/j.compedu.2015.01.005

Torres-Coronas T, Vidal-Blasco M-A (2015) Students and employers perception about the development of digital skills in higher education | Percepción de estudiantes y empleadores sobre el desarrollo de competencias digitales en la educación superior. Revista de Educacion 367:63–89. https://doi.org/10.4438/1988-592X-RE-2015-367-283

Torres-Gastelú CA, Cordero-Guzmán DM, Soto-Ortíz JL, Mory-Alvarado A (2019) Influence of factors about the manifestation of digital citizenship. Prisma Soc 26:27–49

Tratkowska K (2020) Digital transformation: theoretical backgrounds of digital change. Manage Sci 24(4):32–37. https://doi.org/10.15611/ms.2019.4.05

Tugtekin EB, Koc M (2020) Understanding the relationship between new media literacy, communication skills, and democratic tendency: Model development and testing. New Media Soc 22(10):1922–1941. https://doi.org/10.1177/1461444819887705

UNESCO (2017) Working group on education: Digital skills for life and work. In: Broadband commission for sustainable development

Varga-Atkins T (2020) Beyond description: In search of disciplinary digital capabilities through signature pedagogies. Res Learn Technol 28:1–19. https://doi.org/10.25304/rlt.v28.2467

Vázquez-Parra JC, Cruz-Sandoval M, Carlos-Arroyo M (2022) Social entrepreneurship and complex thinking: a bibliometric study. Sustainability 14(20). https://doi.org/10.3390/su142013187

Verhoef PC, Broekhuizen T, Bart Y, Bhattacharya A, Qi Dong J, Fabian N, Haenlein M (2019) Digital transformation: a multidisciplinary reflection and research agenda. J Bus Res https://doi.org/10.1016/j.jbusres.2019.09.022

Vetter MA, Sarraf KS (2020) Assessing the Art + feminism Edit-a-thon for Wikipedia literacy, learning outcomes, and critical thinking. Interact Learn Environ https://doi.org/10.1080/10494820.2020.1805772

Wang HL, Almeida S, Frino B, Wijayawardena K, Rauf A, Hardie G (2022) Dialogue matters a lot: autoethnographic reflections of an Australian teaching team managing first-year undergraduate students. Int J Manage Educ 20(3):100699. https://doi.org/10.1016/j.ijme.2022.100699

Williamson B, Hogan A (2020). Commercialisation and privatisation in / of education in the context of Covid-19. In: Educ Int Res (Issue July). Education International. https://codeactsineducation.wordpress.com/2020/07/14/evolution-global-education-industry-during-pandemic/

Xiao J (2020) Digital transformation in higher education: critiquing the five-year development plans (2016-2020) of 75 Chinese Universities. Dist Educ 40(4):515–533

Zhao Y, Sánchez Gómez MC, Pinto Llorente AM, Zhao L (2021) Digital competence in higher education: students’ perception and personal factors. Sustainability 13(21):12184. https://doi.org/10.3390/su132112184

Download references

Acknowledgements

The authors would like to thank the support from Tecnologico de Monterrey through the “Challenge-Based Research Funding Program 2022”. Project ID # I003-IFE001-C2-T3–T. Also, academic support from Writing Lab, Institute for the Future of Education, Tecnologico de Monterrey, México.

Author information

Authors and affiliations.

Tecnologico de Monterrey, Monterrey, Mexico

Silvia Farias-Gaytan & Maria-Soledad Ramirez-Montoya

University of Huelva, Huelva, Spain

Ignacio Aguaded

You can also search for this author in PubMed   Google Scholar

Contributions

Three authors contributed to the content of the article, conceptualizing the approach: IA, M-SR-M; supporting the study theoretically and methodologically: SF-G, M-SR-M; and discussing the data: SF-G, IA.

Corresponding author

Correspondence to Silvia Farias-Gaytan .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

This article does not contain any studies with human participants performed by any of the authors.

Informed consent

Additional information.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Farias-Gaytan, S., Aguaded, I. & Ramirez-Montoya, MS. Digital transformation and digital literacy in the context of complexity within higher education institutions: a systematic literature review. Humanit Soc Sci Commun 10 , 386 (2023). https://doi.org/10.1057/s41599-023-01875-9

Download citation

Received : 31 August 2022

Accepted : 20 June 2023

Published : 07 July 2023

DOI : https://doi.org/10.1057/s41599-023-01875-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Digital proficiency: assessing knowledge, attitudes, and skills in digital transformation, health literacy, and artificial intelligence among university nursing students.

• Ebtsam Aly Abou Hashish
• Hend Alnajjar

BMC Medical Education (2024)

Digital Literacy, Sustainable Development and Radiation Regulation: Policy and Information Systems Implications

• Farid Gasmi
• Paul Noumba Um

Information Systems Frontiers (2024)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

SYSTEMATIC REVIEW article

Digital literacy in the university setting: a literature review of empirical studies between 2010 and 2021.

• 1 Departamento de Psicología, Área de Psicología Evolutiva y de la Educación, Universidad de Almería, Almeria, Spain
• 2 Departamento de Psicología, Sociología y Filosofía, Universidad de León, Leon, Spain
• 3 Departamento de Psicología Evolutiva y de la Educación, Universidad de Salamanca, Salamanca, Spain
• 4 Instituto Politécnico de Coímbra, Coimbra, Portugal
• 5 Coimbra Education School, Research Group in Social and Human Sciences Núcleo de Investigação em Ciências Sociais e Humanas da ESEC (NICSH), Coimbra, Portugal

The impact of digital devices and the Internet has generated various changes at social, political, and economic levels, the repercussion of which is a great challenge characterized by the changing and globalized nature of today's society. This demands the development of new skills and new learning models in relation to information and communication technologies. Universities must respond to these social demands in the training of their future professionals. This paper aims to analyze the empirical evidence provided by international studies in the last eleven years, related to the digital literacy of university students, including those pursuing degrees related to the field of education. Our findings highlight the fact that the digital literacy that is offered in universities to graduate/postgraduate students, in addition to treating digital literacy as a central theme, also focuses on perceived and developed self-efficacy. This is done by strengthening competencies related to digital writing and reading, the use of databases, the digital design of content and materials, and the skills to edit, publish or share them on the web, or applications aimed at treating digital literacy as emerging pedagogies and educational innovation. Secondly, we found studies related to digital competencies and use of the Internet, social networks, web 2.0, or the treatment of digital risks and their relationship with digital literacy. Thirdly, we found works that, in addition to focusing on digital literacy, also focused on different psychological constructs such as motivation, commitment, attitudes, or satisfaction.

Systematic review registration: https://www.scopus.com/home.uri ; https://www.recursoscientificos.fecyt.es/ .

Introduction

The concept of digital literacy (DL) appears for the first time in the works of Zurkowski (1974) , for whom it is an ability to identify, locate, and examine information. However, despite its novelty, the conceptions it encompasses have been changing ( Lim and Newby, 2021) . Proof of this are the contributions of Gilster (1997) who combines the idea that DL is also closely linked to skills such as access, evaluation, and management of information used in learning processes. Digital learning is understood as the set of technical-procedural, cognitive, and socio-emotional skills necessary to live, learn, and work in a digital society ( Eshet-Alkalai, 2012 ; European Commission, 2018 ). It is related to reading, writing, calculation skills, and effective use of technology in personal, social, and professional areas. It is also considered inseparable from the social and educational needs of the society in which we live ( Larraz, 2013 ; Brata et al., 2022 ). Therefore, we refer to a concept that has several aspects including the technological aspect, the informative and multimedia aspect, and the communicative aspect. It involves a complete process and multiple literacies ( Gisbert and Esteve, 2011 ; Lázaro, 2015 ; Valverde et al., 2022 ). It requires mastery of certain competencies related to the identification of training needs, access to information in digital environments, the use of ICT tools to manage information, interpretation, and representation of information, and the evaluation of information and the transmission of information ( Covello and Lei, 2010 ; Walsh et al., 2022 ).

Digital literacy in university students

In recent years, society has undergone enormous changes with the digitalization of many of its spheres at the information level, the communication level, the level of knowledge acquisition, the level of the establishment of social relations, and even the level of leisure. Thus, our habits and means of accessing, managing, and transforming information have also changed ( European Union, 2013 ; Cantabrana and Cervera, 2015 ; Allen et al., 2020 ; López-Meneses et al., 2020 ).

These developments have also had a great impact on the educational field, in which we have to rethink firstly what kind of students we are training in terms of the skills they need in today's society, and secondly, whether we are training a profile of future teachers capable of training a student body that uses information and communication technologies as something inherent to their own personal and social development. In short, digital communication has changed practices related to literacy and has gained great relevance in the development of knowledge in the twenty-first century ( Comisión Europea, 2012 , 2013 ; European Commission, 2012 ; OECD, 2012 ; Unión Europea, 2013 ; Instituto Nacional de Tecnologías Educativas y Formación del Profesorado, 2017 ; Gudmundsdottir and Hatlevik, 2018 ; Pérez and Nagata, 2019 ; Fernández-de-la-Iglesia et al., 2020 ).

The European Commission (2013 ) indicates that initial teacher training (IDT) should integrate teachers' digital literacy, betting on the pedagogical use of digital tools, enabling them to use them in an effective, appropriate, and contextualized manner. This teaching competence should be characterized by having a holistic, contextualized, performance-, function-, and development-oriented character. In short, it is about incorporating and adequately using ICT as a didactic resource ( Cantabrana and Cervera, 2015 ; Castañeda et al., 2018 ; Tourón et al., 2018 ; Chow and Wong, 2020 ; Vodá et al., 2022 ).

In this sense, according to the work of Krumsvik (2009) , the CDD ( competencia digital docente de los profesores –digital competency training for teachers) is composed of four components: basic digital skills ( Bawden, 2008 ), didactic competence with ICT ( Koehler and Mishra, 2008 ; Gisbert and Esteve, 2011 ), learning strategies, and digital training or training.

While at the Spanish level, the Common Framework of Digital Teaching Competence of the National Institute of Educational Technologies and Teacher Training ( INTEF, 2017 ) standardizes it in five areas: information and information literacy, communication and collaboration, digital content creation, security, and problem solving ( López-Meneses et al., 2020 ). Recently, they have been consolidated as competencies that must be acquired by any university student, along with the knowledge, skills, and attitude that make up a digitally competent citizen ( Recio et al., 2020 ; Indah et al., 2022 ).

Digital literacy in future teachers

Several efforts have been made to equip future teachers with these competencies through different standards and frameworks to the level of learning acquired ( Fraser et al., 2013 ; INTEF, 2017 ; UNESCO, 2018 ). However, how to work these competencies in initial training is still a hotly debated topic, in which special attention is paid to the promotion of experiences of a pedagogical and innovative nature to transform teaching practices, involving the integration of technologies in the classroom, as stated in the Horizon Report 2019 for the Higher Education ( Educause, 2019 ; Le et al., 2022 ).

Universities are in a moment of transformation, from a teacher-focused teaching model to a model based on active learning through the use of digital technologies, giving rise to a new type of education in which the use of digital devices is intrinsic ( Area, 2018 ; Aarsand, 2019 ). If digital resources and devices are an inescapable part of current and future teaching practice, digital competency training for future teachers becomes extremely relevant, given that teachers need to acquire these competencies in their initial training to integrate them into their practices as future teachers. That is, the digital competence (DC) acquired during their initial training significantly predicts the integration of technologies in future teaching practice ( Nikou and Aavakare, 2021 ), which could range from basic digital literacy to the integration of technologies in their daily teaching practice ( Gisbert et al., 2016 ; Alanoglu et al., 2022 ). Several studies have defined the different indicators that make up DC ( Siddiq et al., 2017 ; González et al., 2018 ; Rodríguez-García et al., 2019 ; Cabero-Almenara and Palacios-Rodríguez, 2020 ).

This calls for a new paradigm, in which future teachers must be digitally literate, in terms of the application of active methodologies, digital competencies, and the use of innovative strategies, styles, and approaches ( Garcia-Martin and Garcia-Sanchez, 2017 ; Gómez-García et al., 2021 ).

Currently, literacy workshops for future professionals are being carried out in a timely and precise manner from customized short training capsules to specific semester-long subjects in undergraduate or postgraduate studies. The training is focused on several specific aspects of digital literacy, but there is a lack of experience in imparting comprehensive digital training. In addition, there are just a few interactions with professional experts in such literacy ( Ata and Yildirim, 2019 ; Campbell and Kapp, 2020 ; Domingo-Coscolla et al., 2020 ; Tomczyk et al., 2020 ; Vinokurova et al., 2021 ).

The present study

For the present study, we based our approach on quality and current education, in which DC was postulated as a key element for the development of students. The educational system was tasked with preparing them for their full development and participation in society ( OECD, 2011 ). For this reason, digital literacy is understood as an essential requirement for development in the society in which we live, based on the promotion of strategies related to searching, obtaining, processing, and communicating information. All these aspects have been consolidated as the dimensions of literacy in the twenty-first century ( Piscitelli, 2009 ; Martín and Tyner, 2012 ). It is, therefore, necessary to understand the reality of this subject and to investigate how these practices are being developed in the context of work. And secondly, it is equally necessary to implement new interventions and lines of research that respond to this urgent need for literacy required by today's society. Therefore, we posed the following research questions: What psychoeducational and learning variables are key in digital literacy? What is the current situation internationally regarding digital literacy in all disciplines in pre-service teacher education? What are the differences in digital literacy requirements pre and post pandemic?

The objective of this study is to analyze the empirical evidence provided by international studies from 2010 to 2021 related to the digital literacy of university students, including those who are pursuing careers related to the educational field.

Relevant differences will be observed in the contributions in empirical evidence from international studies pre-post-pandemic; and drawn from diverse cultural backgrounds (Spanish-Latin, Portuguese, Finnish, etc.,), gender, and personal digital resources.

Materials and methods

The systematic review is composed of four phases, following the model of Miller et al. (2016) and Scott et al. (2018) .

PHASE 1: Search terms: In this phase, we developed a schematic of search terms from Web of Science and Scopus databases. We also accessed the databases to locate specific studies that were referenced in the publications that we found in the databases during our initial search. The schematic of terms and thematic axes that were used as a starting point for scanning both databases for anything related to the descriptor “digital” and the descriptor “literacy” is presented in Figure 1 .

Figure 1 . Diagram of search terms used in the systematic review.

PHASE 2: Selection process based on inclusion and exclusion criteria. The following selection criteria were applied: year of publication between 2010 and 2021, availability of full text, and language of publication in English, Portuguese, or Spanish. Once the first results were obtained, they were selected based on title, abstract, and the use of standardized instruments in their methodology. We rejected the studies that used “ ad hoc ” instruments to measure digital competence.

In addition, the selection indicators provided by Cooper and Hedges (1994) and Cooper (2009) were used, such as peer-reviewed journals, referenced databases, and citation indexes.

PHASE 3: Analysis of methodological quality and indicators based on scientific evidence. Following Torgerson (2007) and Risko et al. (2008) and taking into consideration the MQQn ( Risko et al., 2008 ), we used seven indicators to analyze the quality and effectiveness of the studies ( Acosta and Garza, 2011 ). These were: alignment of theory, findings, reliability and validity, descriptive details of participants and the study, sample, and consistency of findings and conclusions with the data ( Risko et al., 2008 ). Alternatively, evidence-based indicators were also used along with study effect sizes ( Díaz and García, 2016 ; Canedo-García et al., 2017 ).

PHASE 4: Reliability and outcomes. Reliability was established for both the selection criteria and the coding criteria during each phase, to evidence the replicability of the results. In addition, the results entailed a qualitative analysis of the selected studies, the central arguments, and the evidence provided in a modulated way to address the research questions.

Therefore, the procedure to be followed was documented and charted according to the PRISMA statement ( Moher et al., 2009 ; Page et al., 2021 ) (see Figure 2 ). Likewise, an analysis was undertaken of the key foci in the various studies to highlight the relevant findings and evidence they provided in this regard. The key focus of our work was: first, to analyze the documents related to the digital literacy of university students; second, to identify which variables affect digital literacy; and third, to undertake a comparative analysis between the different variables that were analyzed.

Figure 2 . Flowchart of search results of empirical studies in databases applying the criteria of Moher et al. (2009 ) and Page et al. (2021) .

All the selected studies had as samples university students who were pursuing some type of degree or postgraduate degree related to education, and therefore, studying to become future teachers. An intervention design was presented that corresponds to a pre-intervention, the intervention itself, and a post-intervention using techniques such as the activation of prior knowledge, instructions, emulation, and subsequent tests. We also found studies that had an experimental design assessing control groups and experimental groups ( Kajee and Balfour, 2011 ; Kuhn, 2017 ; Pequeño et al., 2017 ; Sharp, 2018 ; Lerdpornkulrat et al., 2019 ).

In the case of those responsible for the intervention, practically in all cases, the teacher acts as such, with one or two of them taking the lead. Although the presence of specialized personnel should also be highlighted, as is the case of the work elaborated by Alfonzo and Batson (2014) and Elliott et al. (2018) in which a professional librarian also intervened. Or, in the work detailed by Ball (2019) , where a consultant who is not a teacher but a professional expert in the use of digital devices and trained for such an occasion by a responsible brand (Apple) carried out the training at the center.

If we examine the constructs or competencies covered by the works selected in our search, we find that all of them, in addition to dealing with digital literacy, also focus on self-efficacy perceived and developed through digital literacy.

The results of our study could be understood under different themes.

First, we found studies that referred to digital competence and other educational issues. Within them, we found a series of competencies that are emphasized such as digital writing and reading. Research developed from digital media, such as databases, web, or applications aimed at the treatment of digital literacy was noted as emerging pedagogies and educational innovation. The digital design of content and materials and the skills to edit, publish or share them, and competencies related to mathematics and its digital literacy, formed part of digital literacy.

Second, we found studies related to digital competence and the use and employment of the Internet, social networks, web 2.0, and the treatment of digital risks and their relationship with digital literacy.

Third, we found works that in addition to focusing on digital literacy, also focused on different psychological constructs such as motivation, commitment, attitudes, or satisfaction ( Tables 1 , 2 ).

Table 1 . Summary of the results found.

Table 2 . Summary of the interventions found.

Regarding instructional literature, we found a large number of results on mass training programs or courses in which digital literacy was the focus. Examples include a course offered in which students could sign up to, or modules taught during the teaching of a subject. We also found investigations on interventions that had been carried out through different subjects in the study program from where the sample was taken. In this case, the samples were taken on an ad hoc basis from a specific student body which the researcher intentionally decided based on a previous intervention experience with them ( Ata and Yildirim, 2019 ; Ball, 2019 ; Campbell and Kapp, 2020 ; Domingo-Coscolla et al., 2020 ; Tomczyk et al., 2020 ; Vinokurova et al., 2021 ).

In terms of material resources, all the studies used some type of documentation (digital or not) with instructions on the development of the activities, in which the students were provided with what to do and the steps to follow. In this case, the development scenario was both online and face-to-face, based on different activities given through workshops or seminars for their development.

It should also be noted that in those investigations in which the intervention itself required a specific application or program, the same was used, specifically, and even the intervention had a specific scenario since it was carried out in person in specialized laboratories where experts and specific material was available for this purpose. As an example of these specific materials, in our results, we found the use of the Photo Story 3, Dashboard, and Wikipedia, as well as the EMODO program or the SELI platform ( Kajee and Balfour, 2011 ; Robertson et al., 2012 ; Ball, 2019 ; Hamutoglu et al., 2019 ; Tomczyk et al., 2020 ).

Regardless of the setting and the program or application employed, we can classify the duration of these interventions into two broad groups: those that had a duration of <1 semester, and those that had an intervention whose duration ranged from one semester to one academic year.

Regarding the instruments used, it should be noted that most of them used survey forms as an evaluation instrument, either by the researcher or by the students. In addition, it is usually used as a resource to collect information of a personal nature and about one's own experience throughout the intervention. We must also highlight the fact that in many of the results found, this form was used digitally or virtually, abandoning the old paper forms ( Kajee and Balfour, 2011 ; Robertson et al., 2012 ; Carl and Strydom, 2017 ; Elliott et al., 2018 ; Ball, 2019 ; Lerdpornkulrat et al., 2019 ; Campbell and Kapp, 2020 ).

Regarding the use of questionnaires, scales or self-reports, we found several works that used participants' digital literacy histories as instruments. Through them, the researcher could learn first-hand about the sample's personal experience of digital literacy, the previous knowledge they possess, the digital skills they had mastered, those they lack, or those they consider they should improve. It also included the sample's vision regarding the use and employment of digital resources in teaching practice ( Kajee and Balfour, 2011 ; Robertson et al., 2012 ; Pequeño et al., 2017 ; Elliott et al., 2018 ).

In the case of scales, we found two papers that employed a Likert-scale elaborated ad hoc . We also found studies that employed standardized scales like the Information Literacy Assessment Scale for Education (ILAS-ED), the Digital Literacy Scale, or the E-Learning Attitudes Scale.

Some of the studies we reviewed used semi-structured interviews as a means of monitoring and providing feedback to the students Table 3 ; ( Kajee and Balfour, 2011 ; Alfonzo and Batson, 2014 ; Gill et al., 2015 ; Carl and Strydom, 2017 ; Elliott et al., 2018 ; Elphick, 2018 ; Ata and Yildirim, 2019 ; Campbell and Kapp, 2020 ).

Table 3 . Assessment intervention in the reviewed studies.

As for the sequence through which the different interventions were developed, we found two types—first, those that divided the contents in time, as is the case of the work of Kajee and Balfour (2011) , who covered a first semester digital writing from online classes, self-instructions and face-to-face classes in a specific laboratory, and in a second semester was exposed to different digital research techniques, following the same methodology. In contrast, we spotted the second type, where the same technique was followed throughout the study, as is the case of Robertson et al. (2012) . They applied digital stories as a tool for the development of the activity, but also the evaluation of the competency. In the research carried out by Lerdpornkulrat et al. (2019) , it is apparent that with the use of the rubric, the teacher gave them an example of the work and asked them all to practice evaluating and grading this work. In this way, they could check if they understood how to use a rubric. They then used the rubric to self-assess their work. After receiving feedback, both groups of students revised and resubmitted their completed projects again.

In the investigation by Elliott et al. (2018) , the intervention was structured in work modules with the following sequence of sessions: they were introduced in the first session with opportunities for group discussions and questions. Essential module reading was provided in weekly online study units and module workshops integrated academic reading and writing activities, such as paraphrasing and referencing, with module content.

In the study by Ball (2019) , in the first year, the students took modules on publishing history, culture, markets, and media. In the second year, the intervention was based on their publishing skills, reading for writing development, and grammar and general literacy.

Hamutoglu et al. (2019) organized their intervention in different weeks, such that during the first week of the 14-week semester, the instructor oriented the students for the course and administered pre-tests. In the following week, students were provided with a session on the Edmodo platform and orientation training on the course content.

In the work of Gabriele et al. (2019) , the experimental research plan (i.e., activities to be performed, methodology to be adopted) was established over 4 months followed by the organization of the reading material (power point presentations, introductory videos of the software, handouts, ad hoc created applications as examples).

We also found interventions that had very short time durations, but provide daily detail of the contents and interventions. Similarly, Alfonzo and Batson (2014) dedicate 1 day to the search and orientation in digital resources, 1 day to the APA standards, and 3 days to develop and use a specific application.

In the research by Istenic et al. (2016) , the intervention was based on six different types of tasks related to a variety of mathematical problems, including problems with redundant data, problems with multiple solutions, problems with multiple paths to the solution, problems with no solution, mathematical problems in logic, and problems with insufficient information.

In some interventions, the sequence through which they are developed is the very development of the subject of the degree course from which they are implemented, as is the case of the work of Gill et al. (2015) .

In the work of Carl and Strydom (2017) , students were first familiarized with the devices and then introduced to electronic portfolios, which helped them to create blogs that serve as platforms for electronic portfolios, and guided them on how to collect artifacts and how to reflect and share content.

In one work we found narrative was used as a technique so that the students could later present their work, analyze it in groups, rework it and present it again to their classmates. Kuhn (2017) , Pequeño et al. (2017 ), and Elphick (2018) followed this model.

Adopting a novel consultative approach, Botturi (2019) co-designed the intervention with his students in two steps: they were surveyed 4 weeks before the start of the course and asked to choose between two options: an overview of different topics/methods/experiences, or an in-depth exploration of one or two topics/methods/experiences. All respondents indicated a preference for the first option and provided indications of the topics they wished to cover (see Tables 4 , 5 ).

Table 4 . Assessment instruments used in the instructional intervention in the reviewed studies.

Table 5 . Treatment fidelity.

The limitations of our search are listed in Table 6 . At the theoretical level, we encountered studies that were not very current, missing research questions or hypotheses, or even missing objectives. At the statistical level, we found several studies had a small or unrepresentative sample.

Table 6 . Limitations of the instructional interventions described in the empirical studies reviewed.

Analyzing the interventions themselves, we identified a few limitations, especially in those studies that neither indicates the tasks, record the entire process, or lack key information to replicate the intervention. In some studies, key information relating to the person carrying out the intervention was missing, particularly on whether they had the specific training for this purpose. Another limitation that was identified was that very few evaluation strategies were in place to evaluate the interventions (see Table 7 ).

Table 7 . Treatment fidelity.

Similarly, gaps were found regarding ethical controls, where in some studies the main limitation was that ethical controls were non-existent or not specified ( Robertson et al., 2012 ; Istenic et al., 2016 ; Kuhn, 2017 ; Elphick, 2018 ; Ata and Yildirim, 2019 ; Tomczyk et al., 2020 ).

Figure 3 shows the evolution over the years of the samples used in each of the studies from 2011 to 2020.

Figure 3 . Evolution over years of the samples used in the studies from 2010 to 2021.

Figure 4 shows the evolution over the years of the controls used in each of the studies from 2011 to 2021.

Figure 4 . Evolution over years of the controls used in studies from 2010 to 2021.

This work aimed to analyze the empirical evidence found in international studies between 2011 to 2021 related to the digital literacy of university students, including those pursuing degrees in education. This objective has been met.

Regarding the first focus related to literacy, this paper highlighted the fact that studies from the West are the most prevalent in this field ( Çoklar et al., 2017 ; Ata and Yildirim, 2019 ; Hamutoglu et al., 2019 ; Sujarwo et al., 2022 ), which correspond to cross-sectional studies, mostly employing instruments such as “the Digital Literacy Scale” developed by Ng (2012) , and “the information literacy self-efficacy scale (ILS)” developed by Kurbanoglu et al. (2006) . Regarding the level of mastery, the results showed an upper intermediate level of competence in information and digital literacy, communication, and collaboration, but a low intermediate level in terms of digital content creation, particularly in the creation and dissemination of multimedia content using different tools ( López-Meneses et al., 2020 ; Moreno et al., 2020 ).

Regarding the second focus, digital literacy in university students, this study reviewed the various contributions of other works and found the presence of a competent group in this field, which makes efficient use of both the Internet and digital media ( Çoklar et al., 2016 ; Ata and Yildirim, 2019 ; Lim and Newby, 2021 ). However, differences were also found in this collective relating to gender, where women were more competent than men in digital literacy, information literacy, technological literacy, and communicative literacy ( Hamutoglu et al., 2019 ; López-Meneses et al., 2020 ; Navarro, 2020 ). However, on the other hand, we lso found studies that revealed particular gender gaps where men showed a higher propensity for DL, while women outperform men in the overall digital literacy test ( Ata and Yildirim, 2019 ). Ata and Yildirim (2019) also found differences in DL between students where university students studying science or mathematics-related majors had higher levels of digital literacy than students majoring in social sciences or psychology fields ( Ata and Yildirim, 2019 ; Chow and Wong, 2020 ).

And as for the third focus, digital literacy in future teachers, we found a dual use of digital literacy, in its social and leisure aspect (searching or maintaining friendships through social networks, sharing digital content, downloading content, or playing online games), and in its academic aspect (searching in search engines, working through online documents, organizing or synthesizing information from different processors, using computer programs to make presentations, edit images or content, or create audiovisual content ( López-Meneses et al., 2020 ).

The main contribution of this review lies in its comparison between pre/post-pandemic studies, which show a great increase in the use of technologies in the educational world (across the curriculum), and research work focused on measuring the competencies of these devices ( Baber et al., 2022 ). These new investigations have not only followed the line of previous ones but focused on the measurement of digital literacy and its influence on it by variables such as the degree of origin, gender, age, or being a digital native or immigrant ( Castañeda-Peña et al., 2015 ; Çoklar et al., 2016 ; Castañeda et al., 2018 ; Ata and Yildirim, 2019 ; Gür et al., 2019 ; Hamutoglu et al., 2019 ; Lerdpornkulrat et al., 2019 ; González et al., 2020 ; Navarro, 2020 ; De Sixte et al., 2021 ). But there has been an expansion of the topics and variables that are studied in conjunction with digital literacy, among which we find as a novelty, the study of psycho-educational variables such as academic motivation ( Chow and Wong, 2020 ), self-efficacy and motivation ( Lerdpornkulrat et al., 2019 ), effort expectations ( Nikou and Aavakare, 2021 ), and self-concept as a student and as a teacher ( Yeşilyurt et al., 2016 ). The importance attached to the educational field, the identification of different roles or behaviors within the concept of digital literacy that is delimited, or even the types of uses within the concept of digital literacy ( López-Meneses et al., 2020 ; Moreno et al., 2020 ; Navarro, 2020 ; Lim and Newby, 2021 ) are new trends.

Therefore, we can affirm that in this study the research predictions are fulfilled, in that the results found show relevant differences from international studies pre-post pandemic; and by different cultural backgrounds (Spanish Latin, Portuguese, Finnish...), gender, and personal digital resources. In terms of applications for educational practice, these results do not indicate that university students are competent in terms of digital literacy, although they demonstrate some competencies like online information search, information evaluation, information processing, information communication, and dissemination skills ( Çoklar et al., 2016 ; Lerdpornkulrat et al., 2019 ). Therefore, there is the risk of training an incomplete student body in digital competence. For complete and comprehensive digital literacy for university students, especially future teachers, there is an urgent need to invest in digital literacy programs. This will ensure that the comprehensive digital competence of students corresponds to the use and employment of the Internet and digital devices in their teaching tasks ( Gisbert et al., 2016 ), and be a guarantee of their integration into teaching practice ( Aslan and Zhu, 2016 ; Nikou and Aavakare, 2021 ).

As for the limitations of this work, they are closely related to the seven indicators for analyzing study quality and effectiveness (Acosta and Garza, 2011), which are: alignment of theory, findings, reliability and validity, descriptive details of participants, and the study, sample, and consistency of findings and conclusions with the data ( Risko et al., 2008 ). Along with evidence-based indicators, and effect sizes of studies ( Díaz and García, 2016 ; Canedo-García et al., 2017 ). So future lines of research or work, should take into account overcoming these limitations, and embrace them in the face of their development.

The number of studies found in the systematic review is comparable to what is usual in this type of study and even higher. For example, in the exemplary systematic review by Scott et al. (2018) , they identified only 29 studies that met the quality criteria, reviewing 50 years of studies published in the US, and of these, only four were quantitative. In the study by Borgi et al. (2020) , they only found ten studies that fit the criteria in a very good analysis. Other systematic reviews go along the same lines, and in the same journal and section Frontiers in Psychology . For example, Dickson and Schubert (2020) and Liu et al. (2022) found only six studies in a review of great interest; the study by Nguyen et al. (2021) identified 18 eligible articles; Shou et al. (2022) with 12 studies included; or Tarchi et al. (2021) ; Huang (2022) found seven studies for quantitative analysis and eight for indirect evidence; Coxen et al. (2021) with 21 articles included in the focal analyzes of the systematic review. The number of studies to be representative is not defined by the number but by the existence of such studies. In a systematic review, all studies are reviewed, thus the population of published studies that fit the indicated criteria. With these studies, it was possible to do an analysis of objective indicators in a general comparison between studies; assessing the instruments used; examining the characteristics of the interventions such as strategies, instructional procedure, and psychological variables considered; comparing the fidelity controls of the treatments, which guarantees their rigor and their application in the terms prescribed by the empirical validation of the interventions; and reviewing the limitations of the studies and their contributions by years. These contributions were based on objective data from the studies and have been represented in tables and figures. In addition, a qualitative analysis is provided that highlights the value of intervention studies in relation to digital competence, and the key psychological variables that have been used. It is true that the studies published since 2010 were used, and that there could have been more studies before, but considering the evolution of this type of focus in relation to digital competence and the psychological variables involved, it is evident that the most interesting thing is to consider the recent years which is when its need and use has been generalized throughout the population.

Conclusions

In general, the results show that university students are digitally literate and make efficient use of both the Internet and digital media. In this sense, we found an intermediate or higher level in skills related to communication and collaboration, such as through different chat rooms, platforms, and communication applications. But an intermediate-low level in terms of digital content creation, especially in the creation and dissemination of multimedia content. So, this should be one of the future competencies to increase in this group. Although there are differences according to gender, age, or degree of origin.

We have to invest in comprehensive digital literacy programs for teachers in initial training, which appears implicit in the training plans of their official studies. Digital literacy needs to be a part of the official curriculum, and be developed rather quickly as a separate subject but in an interdisciplinary manner throughout their training. In this way, they become digitally literate people capable of creating and generating digital content and possessing the necessary competencies and skills to use and share such content.

We must also invest in assessing teachers' self-perception. Only by knowing their opinion, skills, and shortcomings, can digital training programs be designed. Digital literacy is a predictor of good digital use and a predictor of the good use and employment of digital devices and the Internet in the future when they would be teaching.

The findings of this study compel us to consider the following: first, we need to rethink the form and manner in which future teachers are capacitated in digital literacy, if we are doing it in the best way, or if on the contrary there are gaps that should be solved. Second, we should take into account the contributions of the results found and their consequences to formulate effective intervention designs and strategies to effectively capacitate pre-service teachers in digital literacy.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Author contributions

J-NS-G, NG-Á, IM-R, JG-M, and SB-C: conceptualization, methodology, software, writing—review and editing, visualization, supervision, and validation. NG-A: formal analysis, investigation, and resources: UAL, ULE, USAL, IPC, data curation, writing—original draft preparation, and funding acquisition. J-NS-G and NG-A: project administration. All authors contributed to the article and approved the submitted version.

The generalx operating funds of the universities have been used Universidad de León (Spain), Universidad de Almería (Spain), Universidad de Salamanca (Spain), Instituto Politécnico de Coimbra and NICSH (Portugal).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Aarsand, P. (2019). Categorization activities in norwegian preschools: digital tools in identifying, articulating, and assessing. Front. Psychol. 10, 973. doi: 10.3389/fpsyg.2019.00973

PubMed Abstract | CrossRef Full Text | Google Scholar

Acosta, S., and Garza, T. (2011). The Podcasting Playbook: A Typology of Evidence-Based Podagogy for PreK-12 Classrooms with English Language Learners .

Google Scholar

Alanoglu, M., Aslan, S., and Karabatak, S. (2022). Do teachers' educational philosophies affect their digital literacy? The mediating effect of resistance to change. Educ. Inf. Technol . 27, 3447–3466. doi: 10.1007/s10639-021-10753-3

* Alfonzo, P. M., and Batson, J. (2014). Utilizing a co-teaching model to enhance digital literacy instruction for doctoral students. Int. J. Doc. Stud. 9, 61–71. doi: 10.28945/1973

CrossRef Full Text | Google Scholar

Allen, J., Belfi, B., and Borghans, L. (2020). Is there a rise in the importance of socioemotional skills in the labor market? Evidence from a trend study among college graduates. Front. Psychol. 11, 1710. doi: 10.3389/fpsyg.2020.01710

Area, M. (2018). Hacia la universidad digital: dónde estamos y a dónde vamos? Rev. Iberoam. Educ. Dist. 21, 25–30. doi: 10.5944/ried.21.2.21801

Aslan, A., and Zhu, C. (2016). Investigating variables predicting turkish pre-service teachers' integration of ICT into teaching practices. Br. J. Educ. Technol. 48, 552–570. doi: 10.1111/bjet.12437

* Ata, R., and Yildirim, K. (2019). Exploring Turkish pre-service teachers' perceptions and views of digital literacy. Educ. Sci. 9, 40–56. doi: 10.3390/educsci9010040

Baber, H., Fanea-Ivanovici, M., Lee, Y.-T., and Tinmaz, H. (2022). A bibliometric analysis of digital literacy research and emerging themes pre-during COVID-19 pandemic. Inform. Learn. Sci. 123, 214–232. doi: 10.1108/ILS-10-2021-0090

* Ball, C. (2019). WikiLiteracy: enhancing students' digital literacy with wikipedia. J. Inform. Liter. 13, 253–271. doi: 10.11645/13.2.2669

Bawden, D. (2008). Revisión de los conceptos de alfabetización informacional y alfabetización digital: traducción. Anal. Document. 5, 361–408. Available online at: https://revistas.um.es/analesdoc/article/view/2261

Borgi, M., Collacchi, B., Giuliani, A., and Cirulli, F. (2020). Dog visiting programs for managing depressive symptoms in older adults: a meta-analysis. Gerontologist 60, e66–e75. doi: 10.1093/geront/gny149

* Botturi, L. (2019). Digital and media literacy in pre-service teacher education: a case study from Switzerland. Nordic J. Dig. Liter. 14, 147–163. doi: 10.18261/issn.1891-943x-2019-03-04-05

Brata, W., Padang, R., Suriani, C., Prasetya, E., and Pratiwi, N. (2022). Student's digital literacy based on students' interest in digital technology, internet costs, gender, and learning outcomes. Int. J. Emerg. Technol. Learn. 17,138–151. doi: 10.3991/ijet.v17i03.27151

Cabero-Almenara, J., and Palacios-Rodríguez, A. (2020). Marco europeo de competencia digital docente “DigCompEdu”. Traducción y adaptación del cuestionario “DigCompEdu Check-In”. Edmetic 9, 213–234. doi: 10.21071/edmetic.v9i1.12462

* Campbell, E., and Kapp, R. (2020). Developing an integrated, situated model for digital literacy in pre-service teacher education. J. Educ. 79, 18–30. doi: 10.17159/2520-9868/i79a02

Canedo-García, A., García-Sánchez, J. N., and Pacheco-Sanz, D. I. (2017). a systematic review of the effectiveness of intergenerational programs. Front. Psychol. 8, 1882. doi: 10.3389/fpsyg.2017.01882

Cantabrana, J. L. L., and Cervera, M. G. (2015). El desarrollo de la competencia digital docente a partir de una experiencia piloto de formación en alternancia en el Grado de Educación. Educar 51, 321–348. doi: 10.5565/rev/educar.725

* Carl, A., and Strydom, S. (2017). e-Portfolio as reflection tool during teaching practice: the interplay between contextual and dispositional variables. S. Afr. J. Educ. 37, 1–10. doi: 10.15700/saje.v37n1a1250

Castañeda, L., Esteve, F., and Adell, J. (2018). Por qué es necesario repensar la competencia docente para el mundo digital? Rev. Educ. Distan. 56, 2–20. doi: 10.6018/red/56/6

Castañeda-Peña, H., Barbosa-Chacón, J. W., Marciales, G., and Barreto, I. (2015). Profiling information literacy in higher education: traces of a local longitudinal study. Univer. Psychol. 14, 445–458. doi: 10.11144/Javeriana.upsy14-2.pilh

Chow, S. K. Y., and Wong, J. L. K. (2020). Supporting academic self-efficacy, academic motivation, and information literacy for students in tertiary institutions. Educ. Sci. 10, 361. doi: 10.3390/educsci10120361

Çoklar, A., Efilti, E., Şahin, Y., and AkÇay, A. (2016). Determining the reasons of technostress experienced by teachers: A qualitative study. Turkish Online J. Qual. Inq . 7, 71–96. doi: 10.17569/tojqi.96082

Çoklar, A. N., Yaman, N. D., and Yurdakul, I. K. (2017). Information literacy and digital nativity as determinants of online information search strategies. Comput. Hum. Behav. 70, 1–9. doi: 10.1016/j.chb.2016.12.050

Comisión Europea (2012). Informe Conjunto de 2012 del Consejo y de la Comisión sobre la Aplicación del Marco Estratégico Para la Cooperación Europea en el Ámbito de la Educación y la Formación (ET 2020). Comisión Europea . Available online at: https://eur-lex.europa.eu/legal-content/ES/TXT/PDF/?uri=CELEX:52012XG0308(01)andfrom=EN

Comisión Europea (2013). Monitor Education and Training 2013. Comisión Europea . Available online at: https://op.europa.eu/es/publication-detail/-/publication/25626e01-1bb8-403c-95da-718c3cfcdf19/language-en

Cooper, H. (2009). Research Synthesis and Meta-Analysis: A Step-By-Step Approach . London: Sage.

Cooper, H., and Hedges, L. V. (1994). The Handbook of Research Synthesis. New York: Russell Sage.

Covello, S., and Lei, J. (2010). A Review of Digital Literacy Assessment Instruments. IDE-712 Front-End Analysis Research . Syracuse: Analysis for Human Performance Technology Decisions.

Coxen, L., van der Vaart, L., Van den Broeck, A., and Rothmann, S (2021). Basic psychological needs in the work context: a systematic literature review of diary studies. Front. Psychol . 12:698526. doi: 10.3389/fpsyg.2021.698526

De Sixte, R., Fajardo, I., Mañá, A., Jáñez, Á., Ramos, M., García-Serrano, M., et al. (2021). Beyond the educational context: relevance of intrinsic reading motivation during COVID-19 confinement in Spain. Front. Psychol. 12, 703251. doi: 10.3389/fpsyg.2021.703251

Díaz, C., and García, J. N. (2016). Identification of relevant elements for promoting eficient interventions in older people. J. Psychodidact. 21, 157–173. doi: 10.1387/RevPsicodidact.13854

Dickson, G. T., and Schubert, E. (2020). Music on prescription to aid sleep quality: a literature review. Front. Psychol. 11, 1695. doi: 10.3389/fpsyg.2020.01695

* Domingo-Coscolla, M., Bosco, A., Carrasco Segovia, S., and Sánchez Valero, J. A. (2020). Fomentando la competencia digital docente en la universidad: percepción de estudiantes y docentes. Rev. Invest. Educ. 38, 167–782. doi: 10.6018/rie.340551

Educause (2019). Horizon Report 2019—Higher Education Edition . Boulder: Educause.

* Elliott, S., Hendry, H., Ayres, C., Blackman, K., Browning, F., Colebrook, D., et al. (2018). 'On the outside I'm smiling but inside I'm crying.' Communication successes and challenges for undergraduate academic writing. J. Furth. High. Educ. 45, 758–770. doi: 10.1080/0309877X.2018.1455077

* Elphick, M. (2018). The impact of embedded iPad use on student perceptions of their digital capabilities. Educ. Sci. 8, 102–125. doi: 10.3390/educsci8030102

Eshet-Alkalai, Y. (2012). Thinking in the digital era: a revised model for digital literacy. Iss. Inform. Sci. Inform. Technol. 9, 267–276. doi: 10.28945/1621

European Comission. (2012). The European Commission and bureaucratic autonomy: Europe's custodians . Cambridge: Cambridge University Press.

European Comission. (2013). The European Commission of the Twenty-First Century . Oxford: Oxford University Press.

European Commission (Ed) (2018). Proposal for a Council Recommendation on Key Competences for Lifelong Learning. New York: European Commission.

European Union (2013). Digital protest skills and online activism against copyright reform in France and the European Union. Policy Int. 5, 27–55.

Fernández-de-la-Iglesia, J. C., Fernández-Morante, M. C., Cebreiro, B., Soto-Carballo, J., Martínez-Santos, A. E., and Casal-Otero, L. (2020). Competencias y actitudes para el uso de las TIC de los estudiantes del grado de maestro de Galicia. Publicaciones 50, 103–120. doi: 10.30827/publicaciones.v50i1.11526

Fraser, J., Atkins, L., and Richard, H. (2013). Digilit Leicester. Supporting Teachers, Promoting Digital Literacy, Transforming Learning . Leicester: Leicester City Council.

* Gabriele, L., Bertacchini, F., Tavernise, A., Vaca-Cárdenas, L., Pantano, P., and Bilotta, E. (2019). Lesson planning by computational thinking skills in Italian pre-service teachers. Inform. Educ. 18, 69–104. doi: 10.15388/infedu.2019.04

Garcia-Martin, J., and Garcia-Sanchez, J. N. (2017). Pre-service teachers' perceptions of the competence dimensions of digital literacy and of psychological and educational measures. Comp. Educ. 107, 54–67. doi: 10.1016/j.compedu.2016.12.010

* Gill, L., Dalgarno, B., and Carlson, L. (2015). How does pre-service teacher preparedness to use ICTs for learning and teaching develop through their degree program? Austral. J. Teach. Educ. 40, 36–59. Available online at: https://www.scopus.com/record/display.uri?eid=2-s2.0-84870756234andorigin=inwardandtxGid=927e8280a577067fddaa8df02fa0b665andfeatureToggles=FEATURE_NEW_DOC_DETAILS_EXPORT:1

Gilster, P. (1997). Digital Literacy (p. 1) . New York, NY: Wiley Computer Pub.

Gisbert, M., and Esteve, F. (2011). Digital Leaners: la competencia digital de los estudiantes universitarios. La Cuestión Univer. 7, 48–59.

Gisbert, M., Esteve, F., and Lázaro, J. (2016). La competencia digital de los futuros docentes: “cómo se ven los actuales estudiantes de educación” Perspect. Educ. 55, 34–52. doi: 10.4151/07189729-Vol.55-Iss.2-Art.412

Gómez-García, G., Hinojo-Lucena, F. J., Fernández-Martín, F. D., and Romero-Rodríguez, J. M. (2021). Educational challenges of higher education: validation of the information competence scale for future teachers (ICS-FT). Educ. Sci. 12, 14. doi: 10.3390/educsci12010014

González, M. J. M., Rivoir, A., Lázaro-Cantabrana, J. L., and Gisbert-Cervera, M. (2020). “Cuánto importa la competencia digital docente? Análisis de los programas de formación inicial docente en Uruguay. Int. J. Technol. Educ. Innov. 6, 128–140. doi: 10.24310/innoeduca.2020.v6i2.5601

González, V., Román, M., and Prendes, M. P. (2018). Formación en competenciasdigitales para estudiantes universitarios basada en el modelo DigComp. Rev. Electró. Tecnol. Educ. 65, 1–15.

Gudmundsdottir, G. B., and Hatlevik, O. E. (2018). Newly qualified teachers' professional digital competence: implications for teacher education. Euro. J. Teach. Educ. 41, 214–231. doi: 10.1080/02619768.2017.1416085

Gür, D., Canan, Ö., Hamutoglu, N. B., Kaya, G., and Demirtas, T. (2019). The Relationship between lifelong learning trends, digital literacy levels and usage of web 2.0 tools with social entrepreneurship characteristics. Croat. J. Educ. 21, 45–76. Available online at: https://hrcak.srce.hr/220701

* Hamutoglu, N. B., Savaşçi, M., and Sezen-Gültekin, G. (2019). Digital literacy skills and attitudes towards e-learning. J. Educ. Fut. 16, 93–107. doi: 10.30786/jef.509293

Huang, C. (2022). Self-Regulation of learning and EFL learners' hope and joy: a review of literature. Front. Psychol. 13, 833279. doi: 10.3389/fpsyg.2022.833279

Indah, R. N., Budhiningrum, A. S., and Afifi, N. (2022). The research competence, critical thinking skills and digital literacy of Indonesian EFL students. J. Lang. Teach. Res. 13, 315–324. doi: 10.17507/jltr.1302.11

Instituto Nacional de Tecnologías Educativas y Formación del Profesorado (2017). Marco Común de Competencia Digital Docente Octubre 2017. Instituto Nacional de Tecnologías Educativas y Formación del Profesorado . Available online at: https://aprende.intef.es/sites/default/files/2018-05/2017_1020_Marco-Com%C3%BAn-de-Competencia-Digital-Docente.pdf

INTEF (Ed.) (2017). Marco Común de Competencia Digital Docente. INTEF . Available online at: https://aprende.intef.es/sites/default/files/2018-05/2017_1020_Marco-Com%C3%BAn-de-Competencia-Digital-Docente.pdf

* Istenic, A., Cotic, M., Solomonides, I., and Volk, M. (2016). Engaging preservice primary and preprimary school teachers in digital storytelling for the teaching and learning of mathematics. Br. J. Educ. Technol. 47, 29–50. doi: 10.1111/bjet.12253

* Kajee, L., and Balfour, R. (2011). Students' access to digital literacy at a South African university: privilege and marginalisation. S. Afr. Linguist. Appl. Lang. Stud. 29, 187–196. doi: 10.2989/16073614.2011.633365

Koehler, M. J., and Mishra, P. (2008). “Introducing tpck,” in The Handbook of Technological Pedagogical Content Knowledge (tpck) for Educators , ed AACTE Committee on Innovation and Technology (Mahwah, NJ: Lawrence Erlbaum Associates), 3–29.

Krumsvik, R. J. (2009). Situated learning in the network society and the digitised school. Euro. J. Teach. Educ. 32, 167–185. doi: 10.1080/02619760802457224

* Kuhn, C. (2017). Are students ready to (re)-design their personal learning environment? The case of the e-dynamic space. J. N. Approach. Educ. Res. 6, 11–19. doi: 10.7821/naer.2017.1.185

Kurbanoglu, S. S., Akkoyunlu, B., and Umay, A. (2006). Developing the information literacy scale. J. Doc. 62, 730–743. doi: 10.1108/00220410610714949

Larraz, E. F. (2013). Competencia digital en la educación superior: instrumentos de evaluación y nuevos entornos. Enl@ ce: Revista Venezolana de Información. Tecnología y Conocimiento . 10, 29–43.

Lázaro, J. (2015). La Competència Digital Docent Com a Eina per Garantir la Qualitat en l'ús de les TIC en un Centre Escolar (Tesis doctoral), Tarragona: Universitat Rovira i Virgili.

Le, B., Lawrie, G. A., and Wang, J. T. (2022). Student self-perception on digital literacy in STEM blended learning environments. J. Sci. Educ. Technol. 31, 303–321. doi: 10.1007/s10956-022-09956-1

* Lerdpornkulrat, T., Poondej, C., Koul, R., Khiawrod, G., and Prasertsirikul, P. (2019). The positive effect of intrinsic feedback on motivational engagement and self-efficacy in information literacy. J. Psychoeduc. Assess. 37, 421–434. doi: 10.1177/0734282917747423

Lim, J., and Newby, T. J. (2021). Preservice teachers' attitudes toward Web 2.0 personal learning environments (PLEs): considering the impact of self-regulation and digital literacy. Educ. Inform. Technol. 26, 3699–3720. doi: 10.1007/s10639-021-10432-3

Liu, W., Xu, Y., Xu, T., Ye, Z., Yang, J., and Wang, Y (2022). Integration of neuroscience and entrepreneurship: a systematic review and bibliometric analysis. Front. Psychol . 13, 810550. doi: 10.3389/fpsyg.2022.810550

López-Meneses, E., Sirignano, F. M., Vázquez-Cano, E., and Ramírez-Hurtado, J. M. (2020). University students' digital competence in three areas of the DigCom 2.1 model: a comparative study at three European universities. Austral. J. Educ. Technol. 36, 69–88. doi: 10.14742/ajet.5583

Martín, A., and Tyner, K. (2012). Media education, media literacy and digital competence. Co-municar 19, 31–39. doi: 10.3916/C38-2012-02-03

Miller, D. M., Scott, C. E., and McTigue, E. M. (2016). Writing in the secondary-level disciplines: a systematic review of context, cognition and content. Educ. Psychol. Rev. 30, 83–120. doi: 10.1007/s10648-016-9393-z

Moher D. Liberati A. Tetzlaff J. Altman D.G. The PRISMA Group. (2009). Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med . 6, e1000097. doi: 10.1371/journal.pmed.1000097

Moreno, A. J., Fernández, M. A., and Godino, A. L. (2020). Información y alfabetización digital docente: influencia de la rama formativa. J. Educ. Teach. Train. 10, 140–150. Available online at: https://digibug.ugr.es/handle/10481/59591

Navarro, J. A. M. (2020). La competencia digital de los estudiantes universitarios latinoamericanos. Int. J. Educ. Res. Innov. 14, 276–289. doi: 10.46661/ijeri.4387

Ng, W. (2012). Can we teach digital natives digital literacy? Comput. Educ. 59, 1065–1078. doi: 10.1016/j.compedu.2012.04.016

Nguyen, H. T. T., Hoang, A. P., Do, L. T. K., Schiffer, S., and Nguyen, H. T. H. (2021). The rate and risk factors of postpartum depression in vietnam from 2010 to 2020: a literature review. Front. Psychol . 12, 731306. doi: 10.3389/fpsyg.2021.731306

Nikou, S., and Aavakare, M. (2021). An assessment of the interplay between literacy and digital technology in higher education. Educ. Inform. Technol. 26, 3893–3915. doi: 10.1007/s10639-021-10451-0

OECD (2011). Informe Habilidades y Competencias Del Siglo xxi Para Los Aprendices Del Nuevo Milenio En los Paí ses de la . OECD . Available online at: http://recursostic.educacion.es/blogs/europa/media/blogs/europa/informes/Habilidades_y_competencias_siglo21_OCDE.pdf

OECD (2012). Education at a Glance 2012. OECD Indicators. Panorama de la educación. Indicadores de la OCDE 2014. Informe español . Available online at: https://www.oecd-ilibrary.org/education/education-at-a-glance-2012_eag-2012-en

Page, M. J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 372, n71. doi: 10.1136/bmj.n71

* Paige, K., Bentley, B., and Dobson, S. (2016). Slowmation: an innovative twenty-first century teaching and learning tool for science and mathematics pre-service teachers. Austral. J. Teach. Educ. 2, 1–15. doi: 10.14221/ajte.2016v41n2.1

* Pequeño, J. M. G., Rodríguez, E. F., and de la Iglesia Atienza, L. (2017). Narratives transmèdia amb joves universitaris. Una etnografia digital en la societat hiperconnectada. Anàlisi 57, 81–95. Available online at: https://raco.cat/index.php/Analisi/article/view/330248

Pérez, T. A., and Nagata, J. J. (2019). The digital culture of students of pedagogy specialising in the humanities in Santiago de Chile. Comput. Educ. 133, 1–12. doi: 10.1016/j.compedu.2019.01.002

Piscitelli, A. (2009). Nativos Digitales . Mexico: Santillana. doi: 10.26439/contratexto2008.n016.782

Recio, F., Silva, J., and Marchant, N. A. (2020). Análisis de la competencia digital en la formación inicial de estudiantes universitarios: un estudio de meta-análisis en la web of science. Rev. Med. Educ. 59, 125–146. doi: 10.12795/pixelbit.77759

Risko, V. J., Roller, C. M., Cummins, C., Bean, R. M., Block, C. C., Anders, P. L., et al. (2008). A critical analysis of research on reading teacher education. Read. Res. Q. 43, 252–288. doi: 10.1598/RRQ.43.3.3

* Robertson, L., Hughes, J., and Smith, S. (2012). “Thanks for the assignment!”: digital stories as a form of reflective practice. Lang. Liter. 14, 78–90. doi: 10.20360/G2S88D

Rodríguez-García, A. M., Raso, F., and Ruiz-Palmero, J. (2019). Competencia digital, educación superior y formación del profesorado: un estudio de meta-análisis en la web of Science. Pixel Bit 54, 65–81. doi: 10.12795/pixelbit.2019.i54.04

Scott, C. E., McTigue, E. M., Miller, D. M., and Washburn, E. K. (2018). The what, when, and how of preservice teachers and literacy across the disciplines: a systematic literature review of nearly 50 years of research. Teach. Teach. Educ. 73, 1–13. doi: 10.1016/j.tate.2018.03.010

* Sharp, L. A. (2018). Collaborative digital literacy practices among adult learners: levels of confidence and perceptions of importance. Int. J. Instruct. 11, 153–166. doi: 10.12973/iji.2018.11111a

Shou, S., Li, Y., Fan, G., Zhang, Q., Yan, Y., Lv, T., et al. (2022). The efficacy of cognitive behavioral therapy for tic disorder: a meta-analysis and a literature review. Front. Psychol . 13, 851250. doi: 10.3389/fpsyg.2022.851250

Siddiq, F., Gochyyev, P., and Wilson, M. (2017). Learning in digital networks – ICT literacy: a novel assessment of students' 21st century skills. Comput. Educ. 109, 11–37. doi: 10.1016/j.compedu.2017.01.014

Sujarwo, S., Tristanti, T., and Kusumawardani, E. (2022). Digital literacy model to empower women using community-based education approach. World J. Educ. Technol. Curr. Issues 14, 175–188. doi: 10.18844/wjet.v14i1.6714

Tarchi, C., Ruffini, C., and Pecini, C (2021). The contribution of executive functions when reading multiple texts: a systematic literature review. Front. Psychol . 12:, 716463. doi: 10.3389/fpsyg.2021.716463

* Tomczyk, Ł., Potyrała, K., Włoch, A., Wnek-Gozdek, J., and Demeshkant, N. (2020). Evaluation of the functionality of a new e-learning platform vs. previous experiences in e-learning and the self-assessment of own digital literacy. Sustainability 12, 10219. doi: 10.3390/su122310219

Torgerson, K. (2007). A genetic study of the acute anxious response to carbon dioxide stimulation in man. J. Psychiatric Res . 41, 906–917. doi: 10.1016/j.jpsychires.2006.12.002

Tourón, J., Martín, D., Navarro, E., Pradas, S., and Íñigo, V. (2018). Validación de constructo de un instrumento para medir la competencia digital docente de los profesores (CDD). Rev. Española Pedag. 76, 25–54. doi: 10.22550/REP76-1-2018-02

UNESCO (2018). ICT Competency Framework for Teachers. UNESCO . Available online at: https://en.unesco.org/themes/ict-education/competency-framework-teachers

Unión Europea (2013). Comprender las Polí ticas de la Unión Europea: Una Nueva Revolución Industrial . Unión Europea . Available online at: https://www.europarl.europa.eu/factsheets/es/sheet/61/los-principios-generales-de-la-politica-industrial-de-la-union

Valverde, J., González, A., and Acevedo, J. (2022). Desinformación y multialfabetización: Una revisión sistemática de la literatura. Comunicar 7, 97–110. doi: 10.3916/C70-2022-08

* Vinokurova, N., Mazurenko, O., Prikhodchenko, T., and Ulanova, S. (2021). Digital transformation of educational content in the pedagogical higher educational institution. Rev. Invest. Apuntes Univer. 11, 1–20. doi: 10.17162/au.v11i3.713

Vodá, A. I., Cautisanu, C., Grádinaru, C., Tánásescu, C., and de Moraes, G. H. S. M. (2022). Exploring digital literacy skills in social sciences and humanities students. Sustainability 14, 2483. doi: 10.3390/su14052483

Walsh, K., Pink, E., Ayling, N., Sondergeld, A., Dallaston, E., Tournas, P., et al. (2022). Best practice framework for online safety education: results from a rapid review of the international literature, expert review, and stakeholder consultation. Int. J. Child Comput. Interact. 33, 100474. doi: 10.1016/j.ijcci.2022.100474

Yeşilyurt, E., Ulaş, A. H., and Akan, D. (2016). Teacher self-efficacy, academic self-efficacy, and computer self-efficacy as predictors of attitude toward applying computer-supported education. Comput. Hum. Behav. 64, 591–601. doi: 10.1016/j.chb.2016.07.038

Zurkowski, P.G. (1974). The information service environment relationships and priorities. Relat. Pap. 5, 27.

* ^ The asterisk of focal references are APA standards.

Keywords: digital literacy, pre-service & teacher education, higher education, teachers', transversal competences

Citation: Gutiérrez-Ángel N, Sánchez-García J-N, Mercader-Rubio I, García-Martín J and Brito-Costa S (2022) Digital literacy in the university setting: A literature review of empirical studies between 2010 and 2021. Front. Psychol. 13:896800. doi: 10.3389/fpsyg.2022.896800

Received: 15 March 2022; Accepted: 23 May 2022; Published: 06 September 2022.

Reviewed by:

Copyright © 2022 Gutiérrez-Ángel, Sánchez-García, Mercader-Rubio, García-Martín and Brito-Costa. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

*Correspondence: Nieves Gutiérrez-Ángel, nga212@ual.es ; Jesús-Nicasio Sánchez-García, jn.garcia@unileon.es

Disclaimer: All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.

• Open access
• Published: 19 April 2023

AI literacy in K-12: a systematic literature review

• Lorena Casal-Otero   ORCID: orcid.org/0000-0002-0906-4321 1 ,
• Alejandro Catala   ORCID: orcid.org/0000-0002-3677-672X 2 , 3 ,
• Carmen Fernández-Morante   ORCID: orcid.org/0000-0003-4398-3361 1 ,
• Maria Taboada   ORCID: orcid.org/0000-0002-2353-596X 2 ,
• Beatriz Cebreiro   ORCID: orcid.org/0000-0003-2064-915X 1 &
• Senén Barro   ORCID: orcid.org/0000-0001-6035-540X 3  

International Journal of STEM Education volume  10 , Article number:  29 ( 2023 ) Cite this article

41k Accesses

52 Citations

28 Altmetric

Metrics details

The successful irruption of AI-based technology in our daily lives has led to a growing educational, social, and political interest in training citizens in AI. Education systems now need to train students at the K-12 level to live in a society where they must interact with AI. Thus, AI literacy is a pedagogical and cognitive challenge at the K-12 level. This study aimed to understand how AI is being integrated into K-12 education worldwide. We conducted a search process following the systematic literature review method using Scopus. 179 documents were reviewed, and two broad groups of AI literacy approaches were identified, namely learning experience and theoretical perspective. The first group covered experiences in learning technical, conceptual and applied skills in a particular domain of interest. The second group revealed that significant efforts are being made to design models that frame AI literacy proposals. There were hardly any experiences that assessed whether students understood AI concepts after the learning experience. Little attention has been paid to the undesirable consequences of an indiscriminate and insufficiently thought-out application of AI. A competency framework is required to guide the didactic proposals designed by educational institutions and define a curriculum reflecting the sequence and academic continuity, which should be modular, personalized and adjusted to the conditions of the schools. Finally, AI literacy can be leveraged to enhance the learning of disciplinary core subjects by integrating AI into the teaching process of those subjects, provided the curriculum is co-designed with teachers.

Introduction

In recent years, the convergence of huge computing power, massive amounts of data and improved machine learning algorithms have led to remarkable advances in Artificial Intelligence (AI) based technologies, which are set to be the most socially and economically disruptive technologies ever developed (Russell, 2021 ). The irruption of AI-based technology in our daily lives (e.g., robot vacuum cleaners, real-time location and search systems, virtual assistants, etc.) has generated a growing social and political interest in educating citizens about AI. The scientific community has also begun to engage in this education after detecting a significant gap in the understanding of AI, based on comments and fears expressed by citizens about this technology (West & Allen, 2018 ). Therefore, integrating AI into curricula is necessary to train citizens who must increasingly live and act in a world with a significant presence of AI.

It is worth noting that AI education addresses not only the learning of the scientific and technological foundations of AI, but also the knowledge and critical reflection on how a trustworthy AI should be developed and the consequences of not doing so. Hence, it is crucial to incorporate AI teaching from the earliest stages of education (Heintz, 2021 ). However, although some countries are making significant efforts to promote AI teaching in K-12 (Touretzky et al., 2019a ), this is being implemented through highly varied AI training experiences, such as data-driven design (Vartiainen et al., 2021 ), interactive data visualizations (Chittora & Baynes, 2020 ; von Wangenheim et al., 2021 ), virtual reality and robotics (Narahara & Kobayashi, 2018 ), games (Giannakos et al., 2020 ), or even based on combined workshop series (Lee et al., 2021 ). To date, there are very few methodological proposals on how to introduce the AI curriculum in K-12 education (Lee et al., 2020 ).

Since the development of a field requires prior research, we propose in this paper to identify and examine the way in which AI literacy is developing in K-12 around the world, to draw conclusions and guide teaching proposals for AI literacy in K-12. By highlighting and discussing the pros and cons of the different approaches and experiences in the literature, we aim to inspire new initiatives and guide the actors involved, from decisions-makers, for example in education policy, to teachers involved in their conception, design and implementation. We also hope to raise awareness of the importance of learning about AI from an early age, emphasizing the key aspects of this training and, hopefully, fueling the debate that needs to be fostered in our research community.

Integration of AI into the K12 curriculum

As a scientific-technological field, AI is just a few decades old. The name was coined in 1956, and since then different disciplines (such as computer science, mathematics, philosophy, neuroscience, or psychology) have contributed to its development from an interdisciplinary focus. AI is oriented to comprehend, model, and replicate human intelligence and cognitive processes into artificial systems. Currently, it covers a wide range of subfields such as machine learning, perception, natural language processing, knowledge representation and reasoning, computer vision, among many others (Russell & Norvig, 2021 ).

Starting in the 1970s, AI began to emerge in educational contexts through tools specifically designed to support learning, teaching, and the management of educational institutions. Since many jobs are now AI-related and will continue to increase in the coming years, some researchers believe that AI education should be considered as important as literacy in reading and writing (Kandlhofer et al., 2016 ). The highly interdisciplinary character is also another factor to consider. AI literacy can be defined as a set of skills that enable a solid understanding of AI through three priority axes: learning about AI, learning about how AI works, and learning for life with AI (Long & Magerko, 2020 ; Miao et al., 2021 ). The first axis focuses on understanding AI concepts and techniques to enable the recognition of which artifacts/platforms use AI and which do not. The second axis addresses the understanding of how AI works, to effectively interact with it. The third axis seeks to understand how AI can affect our lives, allowing us to critically evaluate its technology. Thus, AI literacy goes beyond the use of AI applications in education, such as Intelligent Tutoring Systems (ITS) (du Boulay, 2016 ).

The teaching of knowledge in AI has traditionally been carried out at the university level, focused on students who study disciplines closely related to computing and ICT in general. In recent years, AI learning has also started to be relevant both in university programs with diverse study backgrounds (Kong et al., 2021 ), as well as at the K-12 level (Kandlhofer & Steinbauer, 2021 ; Tedre et al., 2021 ). However, teaching AI at the K-12 level is not yet prevalent in formal settings and is considered challenging. Experts believe it is important to have some thought on what AI education should look like at the K-12 level so that future generations can become informed citizens who understand the technologies they interact with in their daily lives (Touretzky et al., 2019a ). Training children and teenagers will allow them to understand the basics of the science and technology that underpins AI, its possibilities, its limits and its potential social and economic impact. It also stimulates and better prepares them to pursue further studies related to AI or even to become creators and developers of AI themselves (Heintz, 2021 ).

Nowadays, research on AI teaching is still scarce (Chai et al., 2020a , 2020b ; Lee et al., 2020 ). The acquisition of knowledge in AI represents a great pedagogical challenge for both experts and teachers, and a cognitive challenge for students (Micheuz, 2020 ). Some countries are making significant efforts to promote AI education in K-12 (Touretzky et al., 2019b ), by developing relatively comprehensive curriculum guidelines (Yue et al., 2021 ). Through interviews with practitioners and policy makers from three different continents (America, Asia and Europe), some studies report on continuing works to introduce AI in K-12 education (He et al., 2020 ). Some other work focuses on examining and comparing AI curricula in several countries (Yue et al., 2021 ). In addition, there are a growing number of AI training experiences that explore pathways to optimize AI learning for K-12 students. However, most of them are somehow thematically limited as they do not adequately address key areas of AI, such as planning, knowledge representation and automated reasoning (Nisheva-Pavlova, 2021 ). Additionally, due to the rapid growth of AI, there is a need to understand how educators can best leverage AI techniques for the academic success of their students. Zhai et al. ( 2021 ) recommend that educators work together with AI experts to bridge the gap between technique and pedagogy.

Using a systematic review method, our research aims to present an overview of current approaches to understand how AI is taught worldwide. Several studies have conducted systematic reviews concerning applications of AI in education. Zhai et al. ( 2021 ) analyzed how AI was applied to the education domain from 2010 to 2020. Their review covers research on AI-based learning environments, from their construction to their application and integration in the educational environment. Guan et al. ( 2020 ) reviewed the main themes and trends in AI research in education over the past two decades. The authors found that research on the use of AI techniques to support teaching or learning has stood the test of time and that learner profiling models and learning analytics have proliferated in the last two decades. Ng et al. ( 2022 ) examined learner types, teaching tools and pedagogical approaches in AI teaching and learning, mainly in university computer science education. Chen et al. ( 2020 ) covered education enhanced by AI techniques aimed to back up teaching and learning. All these studies have focused on the main role that AI has played in educational applications over the last decades. However, in light of the recent need to consider how AI education should be approached at the K-12 level (Kandlhofer et al., 2016 ; Long & Magerko, 2020 ; Miao et al., 2021 ; Touretzky et al., 2019b ), it would be of great value to structure and characterize the different approaches used so far to develop AI literacy in K-12, as well as to identify research gaps to be explored. Recently, Yue et al. ( 2022 ) analyzed the main components of the pedagogical design in 32 empirical studies in K-12 AI education and Su et al. ( 2022 ) examined 14 learning experiences carried out in the Asian-Pacific region. These components included target audience, setting, duration, contents, pedagogical approaches to teaching, and assessment methods. Sanusi et al. ( 2022 ) reviewed research on teaching machine learning in K-12 from four perspectives: curriculum development, technology development, pedagogical development, and teacher training development. The findings of the study revealed that more studies are needed on how to integrate machine learning into subjects other than computer science. Crompton et al. ( 2022 ) carried out a systematic review on the use of AI as a supporting tool in K-12 teaching, which entails an interesting but narrower scope. Our study extends previous reviews on K-12 AI research by emphasizing how the current approaches are integrating AI literacy in K-12 education worldwide.

Research question

To begin the systematic review, a single research question (RQ) was formulated.

RQ: How are current approaches integrating AI literacy into K-12 education worldwide?

In essence, the RQ aims to investigate the characterization of the different approaches being employed to incorporate AI education in K-12. The following subsections in the methodology describe the search and the data collection process in such a way that an answer to the RQ can be provided in a replicable and objective fashion.

The research method chosen to conduct this research was the systematic literature review (SLR), following the guidelines posed by Kitchenham ( 2004 ). Accordingly, the following subsections summarize and document the key steps implemented in this research method.

Search process

We used Scopus to implement the search process. Scopus provides an integrated search facility to find relevant papers in its database based on curated metadata. It includes primary bibliographic sources published by Elsevier , Springer , ACM , and IEEE , among others. It provides a comprehensive coverage of journals and top-ranked conferences within fields of interest. We did not limit our search to specific journals or regular conference proceedings, as there is not yet a clearly established body of literature on the subject. All searches were performed based on title, keywords and abstract, and conducted between 21 October 2021 and 9 March 2023.

To decide the search string, we ran an initial search and found only a few papers focused on ‘literacy’ whereas the vast majority referred to the broader term ‘education’. Therefore, we decided to use both search terms (key issue 1 in Table 1 ). As some recent works combine the terms ‘Artificial Intelligence’ and ‘education’/’literacy’ into single terms such as ‘AI literacy’ or ‘AI education’, these were added to the search string (key issue 2 in Table 1 ). The educational stage was also included in the search string (key issue 3 in Table 1 ). As the search term ‘education’ also returns AI-based learning environments which are outside the scope of our review, we explicitly considered negated terms to leave out both computer-based learning and intelligent tutoring systems (key issue 4 in Table 1 ). A final decision was whether to use the term ‘Artificial Intelligence’ as a single umbrella term or to add narrower terms related to AI subfields (e.g., machine learning). After a preliminary inspection of a few relevant papers, we observed that such additional specific terms usually co-occur with the string ‘Artificial Intelligence’ in education, and they were therefore regarded as unnecessary. Thus, to capture the essence of our RQ and to build up the complete search string, we considered the search terms as shown in Table 1 . Eventually, this resulted in the following complete search string in Scopus:

TITLE-ABS-KEY ( ( ( ( literacy OR education) AND ( ( artificial AND intelligence))) OR ( "AI literacy" OR "AI education")) AND ( "primary school" OR "secondary school" OR k-12 OR "middle school"

OR "high school") AND NOT ( "computer-based learning") AND NOT ( "intelligent tutoring system")).

We included peer-reviewed papers published on topics related to literacy and education on AI at school. Then we excluded papers whose usage of AI was limited to 1) supporting computer-based learning only, with no focus on learning about AI; 2) supporting assessment/tutoring based on AI. We also excluded papers that targeted college students and those that were limited to K-12 programming/CS concepts as a prerequisite for learning about AI in the future. Following these inclusion and exclusion criteria, our search in Scopus returned an initial list of 750 documents. After we inspected the title, abstract, keywords and full-text screening, we obtained a final list of 179 documents.

Data collection extraction and synthesis strategy

Data collection extraction was performed, discussed, and coordinated through regular meetings. After inspecting and discussing 10% of the papers over multiple meetings, the authors agreed on the annotations presented in Table 2 . This process is important as it allowed us to build a data annotation scheme empirically emerging from the sampled papers. A copy of the papers was also kept for easy review in case of doubts or disagreements.

The data resulted in a spreadsheet with the metadata of the papers which passed the inclusion and exclusion criteria, and a document with the list of paper IDs together with the rest of annotations. Some Python scripts were used to further process metadata (e.g., counting participating countries, frequencies, etc.) and produce a more complete bibliographic report with histograms and overview counting. A more qualitative analysis was carried out to answer the research question based on paper reading and annotations.

The results were organized into two subsections. The first subsection is a bibliometric analysis of the reviewed studies, which is based on the metadata provided by Scopus. The second subsection provides a qualitative analysis of the studies, which is based on the extracted data annotations (see Table 2 ). Both analyses are complementary and together deliver a better understanding of the research articles retrieved.

Bibliometric analysis

Figure  1 shows that the annual scientific production has been modest. It gained traction in 2016 and increased sharply in 2020.

Annual scientific production: number of papers by year

Most of the contributions are conference publications (126 papers), while 52 are journal articles and one is a book chapter (Fig.  2 ).

Type of contributions: number of papers by type

Eighty out of 179 papers have at least a citation in Scopus. There are 13 papers that have 10 or more citations, and the most cited papers are Long and Magerko ( 2020 ) and Touretzky et al. ( 2019b ). Figure  3 summarizes the number of contributions by publishers, where Springer, IEEE and ACM stand out, followed by Elsevier. As for journals, there are no single journals concentrating the publication of articles. Nevertheless, there are some journals that are especially relevant and well-known by the community such as the International Journal of Child-Computer Interaction, Computers and Education: Artificial Intelligence, International Journal of Artificial Intelligence in Education, or IEEE Transactions on Education.

Frequency of publishers: number of papers by publisher

As for conferences, Fig.  4 summarizes the main conference events where papers are published. It includes flagship conferences Footnote 1 such as CHI and AAAI, top-ranked conferences such as HRI or SIGCSE and several noteworthy events (IDC, ICALT, ITiCSE, VL/HCC, to name a few). It is worth mentioning that AAAI is receiving contributions from recent years, which confirms the interest in the field in broadening the discussion to education. There are some additional publications associated with satellite AAAI events, such as workshops in CEUR-WS that deal with the issue under study. Although such contributions may sometimes be short, we decided to include them as they were relevant. For instance, the works published in (Herrero et al., 2020 ) and (Micheuz, 2020 ) include the German countrywide proposal for educating about AI, through a 6-module course focusing on explaining how AI works, the social discourse on AI and reducing existing misconceptions. On the other hand, Aguar et al. ( 2016 ) talk about teaching AI via an optional course which does not contribute to the final grades.

Main conference events: number of papers by conference

The analysis did not reveal particularly outstanding institutions (see Table 3 for a summary). Among the 299 affiliated institutions, we mostly find universities and research centers along with a few collaboration associations. The most active institutions are the Chinese University of Hong Kong, University of Eastern Finland and MIT, whose authors participated in a total of 19, 11 and 10 contributions, respectively.

Finally, the retrieved papers were co-authored by 643 different authors affiliated to research institutions from 42 countries. Figure  5 shows the histogram of participation by country. Of the 179 papers reviewed, most papers were written by authors affiliated with institutions in the same country. Only 32 papers involved authors from several countries. It is remarkable that in these cases at least one author is from the US, Hong Kong or China.

Country participation: number of papers by country

Literature analysis

By analyzing the data extracted, the papers were classified into two broad thematic categories according to the type of educational approach, namely, learning experience and theoretical perspective. The first category covers AI learning experiences focused on understanding a particular AI concept/technique or using specific tools/platforms to illustrate some AI concepts. The second category involves initiatives for the implementation of AI education for K-12 through the development of guidelines, curriculum design or teacher training, among others. Each main category was further subdivided into other subcategories to structure the field and characterize the different approaches used in developing AI literacy in K-12. Figure  6 shows all the identified categories and subcategories.

Taxonomy of approaches to AI learning in K-12

Learning experiences focused on understanding AI

This category covers learning experiences aimed at experimenting and becoming familiar with AI concepts and techniques. Based on the priority axes in AI literacy (Long & Magerko, 2020 ; Miao et al., 2021 ), we identified experiences aimed at acquiring basic AI knowledge to recognize artifacts using AI, learning how AI works, learning tools for AI and learning to live with AI.

Learning to recognize artifacts using AI

This subcategory refers to experiences that aim to understand AI concepts and techniques enabling the recognition of which artifacts/platforms use AI and which do not. Four studies were found in this subcategory. They are proposals aimed at helping young people to understand and demystify AI through different types of activities. These activities included conducting discussions after watching AI-related movies (Tims et al., 2012 ), carrying out computer-based simulations of human-like behaviors (Ho et al., 2019 ), experimenting as active users of social robots (Gonzalez et al., 2017 ) and programming AI-based conversational agents (Van Brummelen et al., 2021b ).

Learning about how AI works

This topic covers proposals designed to understand how AI works to make user interaction with AI easier and more effective. In this type of proposal, the focus is on methodology and learning is achieved through technology (Kim et al., 2023 ). The objective is to provide a better understanding of a particular aspect of reality in order to carry out a project or solve a problem (Lenoir & Hasni, 2016 ). The activities are supported by active experiences based on building and creating intelligent devices to achieve the understanding of AI concepts following the idea of Papert’s constructionism.

These experiences are mainly focused on teaching AI subfields such as ML or AI algorithms applied to robotics. Understanding the principles of ML, its workflows and its role in everyday practices to solve real-life problems has been the main objective of some studies (Burgsteiner et al., 2016 ; Evangelista et al., 2019 ; Lee et al., 2020 ; Sakulkueakulsuk et al., 2019 ; Vartiainen et al., 2021 ). In addition, there are also experiences focused on unplugged activities that simulate AI algorithms. For example, through classic games such as Mystery Hunt, one can learn how to traverse a graph without being able to see beyond the next path to be traversed (blind search) (Kandlhofer et al., 2016 ). Similarly, the AI4K12 initiative (Touretzky et al., 2019b ) collects a large set of activities and resources to simulate AI algorithms.

Learning tools for AI

This topic includes approaches that involve learning about AI support tools. The development of intelligent devices in the context of teaching AI requires specific programming languages or age-appropriate tools. Many of the tools currently available are focused on ML, with the aim of demystifying this learning in K-12 education (Wan et al., 2020 ). Some of them are integrated into block-based programming languages (such as Scratch or App Inventor) (Toivonen et al., 2020 ; von Wangenheim et al., 2021 ), enabling the deployment of the ML models built into games or mobile applications. Other approaches use data visualization and concepts of gamification to engage the student in the learning process (Reyes et al., 2020 ; Wan et al., 2020 ) or combine traditional programming activities with ML model building (Rodríguez-García et al., 2020 ).

This type of proposal aims to introduce AI through tools that enable the use of AI techniques. It is therefore an approach focused on learning by using AI-oriented tools. In this vein, different experiences have focused on learning programming tools for applications based on Machine Learning (Reyes et al., 2020 ; Toivonen et al., 2020 ; von Wangenheim et al., 2021 ; Wan et al., 2020 ), robotics (Chen et al., 2017 ; Eguchi, 2021 ; Eguchi & Okada, 2020 ; Holowka, 2020 ; Narahara & Kobayashi, 2018 ; Nurbekova et al., 2018 ; Verner et al., 2021 ), programming and the creation of applications (Chittora & Baynes, 2020 ; Giannakos et al., 2020 ; Kahn et al., 2018 ; Kelly et al., 2008 ; Park et al., 2021 ). Some of these tools use Scratch-based coding platforms to make AI-based programming attractive to children. In (Kahn et al., 2018 ), students play around with machine learning to classify self-captured images, using a block-based coding platform.

There are also experiences in which other types of environments are used to facilitate learning (Aung et al., 2022 ). In (Holowka, 2020 ; Verner et al., 2021 ), students can learn reinforcement learning through online simulation. In (Narahara & Kobayashi, 2018 ), a virtual environment helps students generate data in a playful setting, which is then used to train a neural network for the autonomous driving of a toy car-lab. In (Avanzato, 2009 ; Croxell et al., 2007 ), students experiment with different AI-based tasks through robotics-oriented competitions.

Learning for life with AI

This subcategory covers experiences aimed at understanding how AI can affect our lives thus providing us with skills to critically assess its technology. In (Vachovsky et al., 2016 ), technically rigorous AI concepts are contextualized through the impact on society. There are also experiences where students explore how a robot equipped with AI components can be used in society (Eguchi & Okada, 2018 ), program conversational agents (Van Brummelen et al., 2021b ), or learn to recognize credible but fake media products (video, photos), which have been generated using AI-based techniques ( 2021b ; Ali et al., 2021a ).

The ethical and philosophical implications of AI have also been addressed in some experiences ( 2021b ; Ali et al., 2021a ; Ellis et al., 2005 ), whereas others focus on training students to participate in present-day society and become critical consumers of AI (Alexandre et al., 2021 ; Cummings et al., 2021 ; Díaz et al., 2015 ; Kaspersen et al., 2022 ; Lee et al., 2021 ; Vartiainen et al., 2020 ).

Proposals for implementation of AI learning at the K-12 level

Some countries are making efforts to promote AI education in K-12. In the U.S., intense work is being carried out on the integration of AI in schools and among these schemes, AI4K12 stands out (Heintz, 2021 ). This scheme is especially interesting since it defines the national guidelines for future curricula, highlighting the essential collaborative work between developers, teachers and students (Touretzky et al., 2019a ). This idea of co-creation is also stressed in other schemes (Chiu, 2021 ). In the U.S. we can also mention the proposal made by the Massachusetts Institute of Technology, which is an AI curriculum that aims to engage students with its social and ethical implications (Touretzky et al., 2019a ). Although the United States is working intensively on the design of integrating this knowledge into the curriculum, so far AI is not widely offered in most K-12 schools (Heintz, 2021 ).

In China, the Ministry of Education has integrated AI into the compulsory secondary school curriculum (Ottenbreit-Leftwich et al., 2021 ; Xiao & Song, 2021 ). Among their schemes we can reference the AI4Future initiative of the Chinese University of Hong Kong (CUHK), which promotes the co-creation process to implement AI education (Chiu et al., 2021 ). In Singapore, a program for AI learning in schools has also been developed, where K-12 children learn AI interactively. However, the program is hindered by a lack of professionals (teachers) with adequate training (Heintz, 2021 ). In Germany, there are also several initiatives to pilot AI-related projects and studies (Micheuz, 2020 ), including the launch of a national initiative to teach a holistic view of AI. This initiative consists of a 6-module course aimed at explaining how AI works, stimulating a social discourse on AI and clarifying the abundant existing misconceptions (Micheuz, 2020 ). Canada has also designed an AI course for high schools. The course is intended to empower students with knowledge about AI, covering both its philosophical and conceptual underpinnings as well as its practical aspects. The latter are achieved by building AI projects that solve real-life problems (Nisheva-Pavlova, 2021 ).

The literature also highlights the different approaches that AI literacy should focus on: curriculum design, AI subject design, student perspective, teacher training, resource design and gender diversity. All these approaches are described in depth below.

AI literacy curriculum design

Approaches to curriculum development differ widely, ranging from the product-centered model (technical-scientific perspective) to the process-centered model (learner perspective) (Yue et al., 2021 ). AI literacy can be launched in primary and secondary education depending on the age and computer literacy of the students. To do this, it is necessary to define the core competencies for AI literacy according to three dimensions: AI concepts, AI applications and AI ethics and security (Long & Magerko, 2020 ; Wong et al., 2020 ). Research has focused on the understanding of the concepts, the functional roles of AI, and the development of problem-solving skills (Woo et al., 2020 ). This has led to proposing a redefinition of the curriculum (Han et al., 2019 ; Malach & Vicherková, 2020 ; Zhang et al., 2020 ) supported by different ideas that K-12 students should know (Chiu et al., 2021 ; Sabuncuoglu, 2020 ; Touretzky et al., 2019b ). Several countries have already made different curricular proposals (Alexandre et al., 2021 ; Micheuz, 2020 ; Nisheva-Pavlova, 2021 ; Ottenbreit-Leftwich et al., 2021 ; Touretzky et al., 2019b ; Xiao & Song, 2021 ), where they argue that the curricular design must include different elements such as content, product, process and praxis (Chiu, 2021 ). It is also convenient for learning in AI to follow the computational thinking model (Shin, 2021 ), contextualizing the proposed curriculum (Eguchi et al., 2021 ; Wang et al., 2020 ) and providing it with the necessary resources for teachers (Eguchi et al., 2021 ). In this sense, emerging initiatives highlight the need to involve teachers in the process of co-creating a curriculum associated to their context (Barlex et al., 2020 ; Chiu et al., 2021 ; Dai et al., 2023 ; Lin & Brummelen, 2021 ; Yau et al., 2022 ).

AI as a subject in K-12 education

Traditionally, including computer science or new technologies in the educational system has been carried out through a specific subject integrated into the curriculum or through the offer of extracurricular activities. In this sense, different proposals have suggested the integration of AI as a subject in K-12 education (Ellis et al., 2009 ; Knijnenburg et al., 2021 ; Micheuz, 2020 ; Sperling & Lickerman, 2012 ), in short-term courses (around 15 h) and divided into learning modules focused on classical and modern AI (Wong, 2020 ) or through MOOCs (Alexandre et al., 2021 ).

Student perspective on AI Literacy

Student-focused studies explore and analyze attitudes and previous knowledge to make didactic proposals adapted to the learner. Some of them measure their intention and interest in learning AI (Bollin et al., 2020 ; Chai et al., 2021 , 2020a , 2020b ; Gao & Wang, 2019 ; Harris et al., 2004 ; Sing, et al., 2022 ; Suh & Ahn, 2022 ), whereas others discuss their views on the integration of technologies in the education system (Sorensen & Koefoed, 2018 ) and on teaching–learning support tools in AI (Holstein et al., 2019 ).

Teacher training in AI

Teachers are key players for the integration of AI literacy in K-12, as proven by the numerous studies that examine this issue (An et al., 2022 ; Bai & Yang, 2019 ; Chiu & Chai, 2020 ; Chiu et al., 2021 ; Chounta et al., 2021 ; Judd, 2020 ; Kandlhofer et al., 2019 , 2021 ; Kim et al., 2021 ; Korenova, 2016 ; Lin et al., 2022 ; Lindner & Berges, 2020 ; Oh, 2020 ; Summers et al., 1995 ; Wei et al., 2020 ; Wu et al., 2020 ; Xia & Zheng, 2020 ). This approach places teachers at the center, bearing in mind what they need to know so as to integrate AI into K-12 (Itmazi & Khlaif, 2022 ; Kim et al., 2021 ). The literature analyzed reports on the factors that influence the knowledge of novice teachers (Wei, 2021 ) and focuses on teacher training in AI (Lindner & Berges, 2020 ; Olari & Romeike, 2021 ). Thus, AI training proposals can be found aimed at both teachers in training (Xia & Zheng, 2020 ) and practicing educators. Training schemes focus on their knowledge in technologies to facilitate their professional development (Wei et al., 2020 ) through the TPACK (Technological, Pedagogical and Content Knowledge) teaching knowledge model (Gutiérrez-Fallas & Henriques, 2020 ). Studies focusing on teachers’ opinions on curriculum development in AI are relevant (Chiu & Chai, 2020 ), as are their self-efficacy in relation to ICT (Wu et al., 2020 ), their opinions on the tools that support the teaching–learning process in AI (Holstein et al., 2019 ) and their teacher training in technologies (Cheung et al, 2018 ; Jaskie et al., 2021 ). These elements are central to the design of an AI literacy strategy in K-12. Both the co-design of ML curricula between AI researchers and K-12 teachers, and the assessment of the impact of these educational interventions on K-12 are important issues today. At present, there is a shortage of teachers with training in AI and working with teachers in training (Xia & Zheng, 2020 ) or with teachers in schools (Chiu et al., 2021 ) is proposed as an effective solution. One of the most interesting analyses of teacher competency proposes the acquisition of this skill for the teaching of AI in K-12, through the analysis of the curricula and resources of AI using TPACK. This model was formulated by (Mishra & Koehler, 2006 ) and aims to define the different types of knowledge that teachers need to integrate ICT effectively in the classroom. In this regard, it is suggested that teachers imparting AI to K-12 students require TPACK to build an environment and facilitate project-based classes that solve problems using AI technologies (Kim et al., 2021 ).

AI literacy support resources

Research using this approach focuses on presenting resources that support AI literacy (Kandlhofer & Steinbauer, 2021 ), considering that the creation of resources and repositories is a priority in supporting this teaching–learning process (Matarić et al., 2007 ; Mongan & Regli, 2008 ). However, these resources largely do not meet an interdisciplinary approach and do not embody a general approach to AI development (Sabuncuoglu, 2020 ).

Gender diversity in AI literacy

AI education, as a broad branch of computer science, also needs to address the issue of gender diversity. Lack of gender diversity can impact the lives of the people for whom AI-based systems are developed. The literature highlights the existence of proposals designed with a perspective toward gender, where the activities designed are specifically aimed at girls (Ellis et al., 2009 ; Jagannathan & Komives, 2019 ; Perlin et al., 2005 ; Summers et al., 1995 ; Vachovsky et al., 2016 ; Xia et al., 2022 ).

The huge impact that AI is having on our lives, at work and in every type of organization and business sector is easily recognizable today. No one doubts that AI is one of the most disruptive technologies in history, if not the most. In recent years, the expectations generated by AI, far from being deflated, have only grown. We are still a long way from general-purpose AI, but the application of AI to solve real problems has already taken hold for a wide range of purposes. It is therefore necessary for young people to know how AI works, as this learning will make it easier for them to use these technologies in their daily lives, both to learn and to interact with others.

Like any other technology, the potential uses and abuses of AI go hand in hand with its disruptive capacity. Many social groups and governments are expressing concern about the possible negative consequences of AI misuse. Although it is crucial to adequately regulate the use of AI, education is as important, if not more important, than regulation. Everything, whether good or bad, stems from the education received. Thus, education systems must prepare students for a society in which they will have to live and interact with AI. AI education will enable young people to discover how these tools work and, consequently, to act responsibly and critically. Therefore, AI literacy has become a relevant and strategic issue (Chiu & Chai, 2020 ).

This systematic review has focused on analyzing AI teaching–learning proposals in K-12 globally. The results confirm that the teaching of basic AI- related concepts and techniques at the K-12 level is scarce (Kandlhofer et al., 2016 ). Our work shows that there have been, on the one hand, different AI learning experiences and, on the other hand, proposals for the implementation of AI literacy, made at the political level and by different experts. The learning experiences described show that AI literacy in schools has focused on technical, conceptual, and applied skills in some domains of interest. Proposals for AI implementation, especially those defined by the US and China, reveal that significant efforts are being made to design models that frame AI literacy proposals.

We also found that there are hardly any AI learning experiences that have analyzed learning outcomes, e.g., through assessments of learners’ understanding of AI concepts. Obviously, this is a result of the infancy of these AI learning experiences at the K-12 level. However, it is important for learning experiences to be based on clearly defined competencies in a particular AI literacy framework, such as those proposed in the literature (Alexandre et al., 2021 ; Han et al., 2019 ; Long & Magerko, 2020 ; Malach & Vicherková, 2020 ; Micheuz, 2020 ; Ottenbreit-Leftwich et al., 2021 ; Touretzky et al., 2019a ; Wong et al., 2020 ; Xiao & Song, 2021 ; Zhang et al., 2020 ). Recently, Van Brummelen et al. ( 2021a ) designed a curriculum for a five-day online workshop based on the specific AI competencies proposed by Long and Magerko ( 2020 ). They used several types of questionnaires to assess the quality of the program through the knowledge acquired by the students in these competencies. Therefore, clearly defined competency-based learning experiences can provide a rigorous assessment of student learning outcomes.

The research shows that clear guidelines are needed on what students are expected to learn about AI in K-12 (Chiu, 2021 ; Chiu & Chai, 2020 ; Lee et al., 2020 ). These studies highlight the need for a competency framework to guide the design of didactic proposals for AI literacy in K-12 in educational institutions. This framework would provide a benchmark for describing the areas of competency that K-12 learners should develop and which specific educational projects can be designed. Furthermore, it would support the definition of a curriculum reflecting sequence and academic continuity (Woo et al., 2020 ). Such a curriculum should be modular and personalized (Gong et al., 2019 ) and adjusted to the conditions of the schools (Wang et al., 2020 ). In the teaching of AI, an exploratory education should be adopted, which integrates science, computer science and integral practice (Wang et al., 2020 ). It should also address issues related to the ethical dimension, which is fundamental to the literacy of K-12 students as it enables them to understand the basic principles of AI (Henry et al., 2021 ). This training facilitates the development of students’ critical capacity, and this is necessary to understand that technology is not neutral and to benefit from and make appropriate use of it. Ethics, complementary to legal norms, enhances the democratic quality of society by setting legitimate limits in the shaping of technological life. In this sense, different AI literacy proposals in K-12 already support the addressing of ethical, social and security issues linked to AI technologies (Eguchi et al., 2021 ; Micheuz, 2020 ; Wong et al., 2020 ). Moreover, considering designing for social good could foster or help to motivate learning about AI (Chai et al., 2021 ). Without a doubt, all this will impact on the achievement of a more democratic society. Due to the gender gap in issues related to computer science, it is also necessary to address the gender perspective. In this vein, the research proposes, among other strategies, to focus AI literacy on real-world elements since this approach favors the motivation of girls and greater involvement in learning (Jagannathan & Komives, 2019 ). However, little attention is paid to the undesirable consequences of an indiscriminate and insufficiently thought-out application of AI, both in higher education and especially in K-12. For example, the increase in socio-economic inequality between countries and within countries, resulting from the increasing automation of employment, is of particular concern. This is leading to growing inequality in wages and preservation of human employment, but it is not usually a subject of interest in education.

Currently, the challenges of this AI literacy require an interdisciplinary and critical approach (Henry et al., 2021 ). We believe that AI literacy can be leveraged to enhance the learning of disciplinary core subjects by integrating AI into the teaching process of those subjects. AI literacy should rely on transferring AI knowledge and methods to core subjects, allowing education to cross disciplinary boundaries, but staying within the framework of disciplinary core subjects. To achieve this change, educators need to take a closer look at the current capabilities of AI. This would enable them to identify all options to improve the core of educational practice and thus optimize the educational process. For example, understanding and using word clouds is a powerful educational strategy to enhance education in core subjects such as science (e.g., to facilitate object classification), language (e.g. to enable the matching of different topics or authors’ works), music (e.g., to support the analysis of song lyrics) or social sciences (e.g., to assist in comparing different discourses). Since AI is highly interdisciplinary in nature, it has a broad projection on multiple fields and problems that require a transversal and applied approach. For example, the basic algorithms of ML could be taught in Mathematics and related disciplines, the design of supervised classifiers could be performed for the study of taxonomies in Biology, natural language processing could be used to make the study of a language more attractive, or the ethical issues surrounding AI could be discussed in Philosophy and Social Sciences subjects.

Finally, for this meaningful learning to take place, AI teaching must be addressed through holistic, active, and collaborative pedagogical strategies in which real problem solving is the starting point of the learning process. An important gap regarding the integration of AI in K-12 concerns teachers, as it is unclear how to prepare and involve them in the process (Chiu & Chai, 2020 ). Teachers’ attitudes towards AI have a significant influence on the effectiveness of using AI in education. Teachers can swing between total resistance and overconfidence. The first could arise from inadequate, inappropriate, irrelevant, or outdated professional development. On the one hand, teachers must be digitally-competent enough to integrate AI into the teaching–learning processes of their subjects. Therefore, teacher training is also necessary following a framework of standard competencies. This should include new ways of organizing the professional role of teachers, as well as enhancing students’ attitudes towards these changes. On the other hand, research reveals that it is essential for didactic proposals to be co-designed and implemented by the teachers at those schools involved (Henry et al., 2021 ), to undergo training in the specific AI subjects and for this knowledge to be integrated into non-computer subjects (Lin & Brummelen, 2021 ). To this end, it is crucial to identify the perception and knowledge that teachers have about AI and involve them in the design of curricular proposals (Chiu, 2021 ; Chiu & Chai, 2020 ; Chiu et al., 2021 ).

This study aimed to understand how AI literacy is being integrated into K-12 education. To achieve this, we conducted a search process following the systematic literature review method and using Scopus. Two broad groups of AI literacy approaches were identified, namely learning experiences and theoretical perspective. The study revealed that learning experiences in schools have focused mainly on technical and applied skills limited to a specific domain without rigorously assessing student learning outcomes. In contrast, the US and China are leading the way in AI literacy implementation schemes which are broader in scope and involve a more ambitious approach. However, there is still a need to test these initiatives through comprehensive learning experiences that incorporate an analysis of learning outcomes. This work has allowed us to draw several conclusions that can be considered in the design of AI literacy proposals in K-12. Firstly, AI literacy should be based on an interdisciplinary and competency-based approach and integrated into the school curriculum. There is no need to include a new AI subject in the curriculum, but rather to build on the competencies and content of disciplinary subjects and then integrate AI literacy into those subjects. Given the interdisciplinary nature of AI, AI education can break disciplinary boundaries and adopt a global, practical, and active approach in which project-based and contextualized work plays an important role. Secondly, AI literacy should be leveraged to extend and enhance learning in curricular subjects. As a final point, AI literacy must prioritize the competency of teachers and their active participation in the co-design of didactic proposals, together with pedagogues and AI experts.

Availability of data and materials

Last revision round required update the review. Thus, Additional file 1 contains a.csv file with the listing of papers that are not cited but are part of the reviewed papers. The papers cited in text already appear in the Reference section and, therefore, not in the Additional file.

1 Conference categorization and ranking based on the GII-GRIN-SCIE (GGS) Conference Ratings: https://scie.lcc.uma.es/

Aguar, K., Arabnia, H. R., Gutierrez, J. B., Potter, W. D., & Taha, T. R. (2016). Making cs inclusive: An overview of efforts to expand and diversify cs education. In International Conference on Computational Science and Computational Intelligence (CSCI). (pp. 321–326). https://doi.org/10.1109/CSCI.2016.0067

Alexandre, F., Becker, J., Comte, M. H., Lagarrigue, A., Liblau, R., Romero, M., & Viéville, T. (2021). Why, What and How to help each citizen to understand artificial intelligence? KI Kunstliche Intelligenz, 35 (2), 191–199. https://doi.org/10.1007/s13218-021-00725-7

Article   Google Scholar  

Ali, S., DiPaola, D., Lee, I., Hong, J., & Breazeal, C. (2021a). Exploring generative models with middle school students. In Proceedings of the 2021a CHI Conference on Human Factors in Computing Systems. (pp. 1–13). https://doi.org/10.1145/3411764.3445226

Ali, S., DiPaola, D., Lee, I., Sindato, V., Kim, G., Blumofe, R., & Breazeal, C. (2021b). Children as creators, thinkers and citizens in an AI-driven future. Computers and Education Artificial Intelligence . https://doi.org/10.1016/j.caeai.2021.100040

An, X., Chai, C. S., Li, Y., Zhou, Y., Shen, X., Zheng, C., & Chen, M. (2022). Modeling English teachers’ behavioral intention to use artificial intelligence in middle schools. Education and Information Technologies . https://doi.org/10.1007/s10639-022-11286-z

Aung, Z. H., Sanium, S., Songsaksuppachok, C., Kusakunniran, W., Precharattana, M., Chuechote, S., & Ritthipravat, P. (2022). Designing a novel teaching platform for AI: A case study in a Thai school context. Journal of Computer Assisted Learning, 38 (6), 1714–1729. https://doi.org/10.1111/jcal.12706

Avanzato, R. L. (2009). Autonomous Outdoor Mobile Robot Challenge. Computer in Education Journal (July-September 2009).

Bai, H., & Yang, S. (2019, October). Research on the Sustainable Development Model of Information Technology Literacy of Normal Students Based on Deep Learning Recommendation System. In 2019 4th International Conference on Mechanical, Control and Computer Engineering (ICMCCE). (pp. 837–840). https://doi.org/10.1109/ICMCCE48743.2019.00192

Barlex, D., Steeg, T., & Givens, N. (2020). Teaching about disruption: A key feature of new and emerging technologies. Learning to Teach Design and Technology in the Secondary School, 4 , 137–154. https://doi.org/10.4324/9780429321191-9

Bollin, A., Kesselbacher, M., & Mößlacher, C. (2020). Ready for computing science? A closer look at personality, interests and self-concept of girls and boys at secondary level. In  Informatics in Schools. Engaging Learners in Computational Thinking: 13th International Conference, ISSEP. (pp. 107–118). https://doi.org/10.1007/978-3-030-63212-0_9

Burgsteiner, H., Kandlhofer, M., & Steinbauer, G. (2016). IRobot: teaching the basics of artificial intelligence in high schools. Proceedings of the AAAI Conference on Artificial Intelligence . https://doi.org/10.1609/aaai.v30i1.9864

Chai, C.S., Lin, P.-Y., Jong, M.S.-Y., Dai, Y., Chiu, T.K., & Huang, B. (2020a). Factors influencing students' behavioral intention to continue artificial intelligence learning. In 2020a International Symposium on Educational Technology (ISET). (pp. 147–150).  https://doi.org/10.1109/ISET49818.2020.00040

Chai, C. S., Wang, X., & Xu, C. (2020b). An extended theory of planned behavior for the modelling of chinese secondary school students’ intention to learn artificial intelligence. Mathematics, 8 (11), 1–18. https://doi.org/10.3390/math8112089

Chai, C.S., Lin, P.-Y., Jong, M.S.-Y., Dai, Y., Chiu, T.K., & Qin, J. (2021). Perceptions of and behavioral intentions towards learning artificial intelligence in primary school students. Educational Technology & Society , 24 (3), 89–101. Retrieved from https://www.jstor.org/stable/27032858

Chen, S., Qian, B., & Cheng, H. (2017). Voice recognition for STEM education using robotics. In Volume 9: 13th ASME/IEEE International Conference on Mechatronic and Embedded Systems and Applications. ASME . https://doi.org/10.1115/DETC2017-68368

Chen, M., Zhou, C., & Wu, Y. (2020). Research on the model and development status of information literacy self-improvement ability of primary and secondary school teachers. In Ninth International Conference of Educational Innovation through Technology (EITT). (pp. 87–91). https://doi.org/10.1109/EITT50754.2020.00021

Cheung, S. K., Lam, J., Li, K. C., Au, O., Ma, W. W., & Ho, W. S. (Eds.). (2018).  Technology in Education. Innovative Solutions and Practices: Third International Conference, ICTE 2018. Springer.

Chittora, S., & Baynes, A. (2020, October). Interactive Visualizations to Introduce Data Science for High School Students. In  Proceedings of the 21st Annual Conference on Information Technology Education.  (pp. 236–241). https://doi.org/10.1145/3368308.3415360

Chiu, T. K. F. (2021). A holistic approach to the design of artificial intelligence (AI) education for k-12 schools. TechTrends, 65 (5), 796–807. https://doi.org/10.1007/s11528-021-00637-1

Chiu, T. K., & Chai, C.-S. (2020). Sustainable curriculum planning for artificial intelligence education: A self-determination theory perspective. Sustainability (switzerland) . https://doi.org/10.3390/su12145568

Chiu, T. K., Meng, H., Chai, C. S., King, I., Wong, S., & Yam, Y. (2021). Creation and evaluation of a pretertiary artificial intelligence (AI) curriculum. IEEE Transactions on Education, 65 (1), 30–39. https://doi.org/10.1109/TE.2021.3085878

Chounta, I.-A., Bardone, E., Raudsep, A., & Pedaste, M. (2021). Exploring teachers’ perceptions of artificial intelligence as a tool to support their practice in estonian k-12 education. International Journal of Artificial Intelligence in Education, 32 , 725–755. https://doi.org/10.1007/s40593-021-00243-5

Crompton, H., Jones, M. V., & Burke, D. (2022). Affordances and challenges of artificial intelligence in K-12 education: a systematic review. Journal of Research on Technology in Education . https://doi.org/10.1080/15391523.2022.2121344

Croxell, J., Mead, R., & Weinberg, J. (2007). Designing robot competitions that promote ai solutions: Lessons learned competing and designing. Technical Report of the 2007 American Association of Artificial Intelligence. Spring Symposia , SS-07–09. (pp. 29–34).

Cummings D., Anthony M., Watson C., Watson A., & Boone S. (2021). Combating social injustice and misinformation to engage minority youth in computing sciences. In Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. (pp.1006–1012). https://doi.org/10.1145/3408877.3432452 .

Dai, Y., Liu, A., Qin, J., Guo, Y., Jong, M. S. Y., Chai, C. S., & Lin, Z. (2023). Collaborative construction of artificial intelligence curriculum in primary schools. Journal of Engineering Education, 112 (1), 23–42. https://doi.org/10.1002/jee.20503

Díaz, J., Queiruga, C., Tzancoff, C., Fava, L., & Harari, V. (2015). Educational robotics and videogames in the classroom. In 2015 10th Iberian Conference on Information Systems and Technologies (CISTI) . Aveiro, Portugal. (pp. 1–6). https://doi.org/10.1109/CISTI.2015.7170616

du Boulay, B. (2016). Artificial intelligence as an effective classroom assistant. IEEE Intelligent Systems, 31 (6), 76–81. https://doi.org/10.1109/MIS.2016.93

Eguchi, A. (2021). AI-robotics and ai literacy. Studies in Computational Intelligence, 982 , 75–85. https://doi.org/10.1007/978-3-030-77022-8

Eguchi, A., & Okada, H. (2018). If you give students a social robot? - world robot summit pilot study. In  Companion of the 2018 ACM/IEEE International Conference on Human-Robot Interaction.  (pp. 103–104). https://doi.org/10.1145/3173386.3177038

Eguchi, A., & Okada, H. (2020). Imagine the Future with Social Robots - World Robot Summit’s Approach: Preliminary Investigation. In M. Moro, D. Alimisis, & L. Iocchi, L. (Eds) Educational Robotics in the Context of the Maker Movement. Edurobotics 2018. Advances in Intelligent Systems and Computing, (p. 946). Springer. https://doi.org/10.1007/978-3-030-18141-3_10

Eguchi, A., Okada, H., & Muto, Y. (2021). Contextualizing AI education for k- 12 students to enhance their learning of ai literacy through culturally responsive approaches. KI Kunstliche Intelligenz, 35 (2), 153–161. https://doi.org/10.1007/s13218-021-00737-3

Ellis, G., Ory, E., Bhushan, N. (2005). Organizing a K-12 AI curriculum using philosophy of the mind. Engineering: Faculty Publications, Smith College. Retrieved from https://scholarworks.smith.edu/egr_facpubs/96

Ellis, G., Silva, K., Epstein, T., & Giammaria, N. (2009). Artificial intelligence in pre-college education: Learning within a philosophy of the mind framework. International Journal of Engineering Education, 25 (3), 511–522.

Google Scholar  

Evangelista, I., Blesio, G., & Benatti, E. (2019). Why are we not teaching machine learning at high school? a proposal. In 2018 World Engineering Education Forum - Global Engineering Deans Council (WEEF-GEDC). (pp. 1–6). https://doi.org/10.1109/WEEF-GEDC.2018.8629750

Gao, J., & Wang, L. (2019). Reverse thinking teaching discussion in high school information technology under new curriculum standards. In 14th International Conference on Computer Science & Education (ICCSE). (pp. 222–226). https://doi.org/10.1109/ICCSE.2019.8845429

Giannakos, M., Voulgari, I., Papavlasopoulou, S., Papamitsiou, Z., & Yannakakis, G. (2020). Games for artificial intelligence and machine learning education: Review and perspectives. Lecture Notes in Educational Technology . https://doi.org/10.1007/978-981-15-6747-6_7

Gong, X., Zhao, L., Tang, R., Guo, Y., Liu, X., He, J., … Wang, X. (2019). AI education system for primary and secondary schools. In  2019 ASEE Annual Conference & Exposition .

Gonzalez, A. J., Hollister, J. R., DeMara, R. F., Leigh, J., Lanman, B., Lee, S. Y., & Wilder, B. (2017). AI in informal science education: bringing turing back to life to perform the turing test. International Journal of Artificial Intelligence in Education, 27 (2), 353–384. https://doi.org/10.1007/s40593-017-0144-1

Guan, C., Mou, J., & Jiang, Z. (2020). Artificial intelligence innovation in education: A twenty-year data-driven historical analysis. International Journal of Innovation Studies, 4 (4), 134–147. https://doi.org/10.1016/j.ijis.2020.09.001

Gutiérrez, L. F., & Henriques, A. (2020). Prospective mathematics teachers’ tpack in a context of a teacher education experiment. Revista Latinoamericana De Investigación En Matemática Educativa, 23 (2), 175–202. https://doi.org/10.12802/relime.20.2322

Han, X., Hu, F., Xiong, G., Liu, X., Gong, X., Niu, X., … Wang, X. (2019). Design of AI + curriculum for primary and secondary schools in Qingdao. In Chinese Automation Congress (CAC) . (pp. 4135–4140). https://doi.org/10.1109/CAC.2018.8623310

Harris, E., Lamonica, A., & Weinberg. JB. (2004) Interfacing the public and technology: a web controlled mobile robot. In Accessible hands-on artificial intelligence and robotics education: working papers of the 2004. AAAI spring symposium series. AAAI Press. (pp.106–110)

He, Y.-T., Guo, B.-J., Lu, J., Xu, Y.-P., & Gong, M. (2020). Research of scratch programming recommendation system based on med and knowledge graph. In 2020 5th International Conference on Mechanical, Control and Computer Engineering (ICMCCE) . (pp. 2158–2163). https://doi.org/10.1109/ICMCCE51767.2020.00469

Heintz, F. (2021). Three interviews about k-12 ai education in america, europe, and singapore. KI Kunstliche Intelligenz, 35 (2), 233–237. https://doi.org/10.1007/s13218-021-00730-w

Henry, J., Hernalesteen, A., & Collard, A.-S. (2021). Teaching artificial intelli- gence to k-12 through a role-playing game questioning the intelligence concept. KI Kunstliche Intelligenz, 35 (2), 171–179. https://doi.org/10.1007/s13218-021-00733-7

Herrero, Á., Cambra, C., Urda, D., Sedano, J., Quintián, H., & Corchado, E. (Eds) (2020). 11th International Conference on European Transnational Educational (ICEUTE 2020). ICEUTE 2020. Advances in Intelligent Systems and Computing, 1266 . Springer. https://doi.org/10.1007/978-3-030-57799-5_8

Ho, J. W., Scadding, M., Kong, S. C., Andone, D., Biswas, G., Hoppe, H. U., & Hsu, T. C. (2019). Classroom activities for teaching artificial intelligence to primary school students. In  Proceedings of international conference on computational thinking education. The Education University of Hong Kong. (pp. 157–159).

Holowka, P. (2020). Teaching robotics during COVID-19: Machine learning, simulation, and aws deepracer. In 17th International Conference on Cognition and Exploratory Learning in Digital Age, CELDA .

Holstein, K., McLaren, B. M., & Aleven, V. (2019). Designing for complementarity: Teacher and student needs for orchestration support in ai-enhanced classrooms. In S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, & R. Luckin (Eds.), Artificial intelligence in education. AIED 2019. Lecture notes in computer science (p. 11625). Springer. 10.1007/978-3-030-23204-7_14.

Itmazi, J., & Khlaif, Z. N. (2022). Science education in Palestine: Hope for a better future. Lecture Notes in Educational Technology . https://doi.org/10.1007/978-981-16-6955-2_9

Jagannathan, R. K., & Komives, C. (2019). Teaching by induction: Project- based learning for Silicon Valley. Journal of Engineering Education Transformations, 33 (1), 22–26. https://doi.org/10.16920/jeet/2019/v33i1/149003

Jaskie, K., Larson, J., Johnson, M., Turner, K., O’Donnell, M., Christen, J.B., & Spanias, A. (2021). Research experiences for teachers in machine learning. In IEEE Frontiers in Education Conference (FIE) . Lincoln, NE, USA. (pp. 1–5). https://doi.org/10.1109/FIE49875.2021.9637132

Judd, S. (2020). Activities for Building Understanding: How AI4ALL Teaches AI to Diverse High School Students. In Proceedings of the 51st ACM Technical Symposium on Computer Science Education. (pp. 633–634). https://doi.org/10.1145/3328778.3366990

Kahn, K., Megasari, R., Piantari, E., & Junaeti, E. (2018). AI programming by children using Snap! block programming in a developing country. In Thirteenth European Conference on Technology Enhanced Learning . (p. 11082). https://doi.org/10.1007/978-3-319-98572-5

Kandlhofer, M., Steinbauer, G., Hirschmugl-Gaisch, S., & Huber, P. (2016). Artificial intelligence and computer science in education: From kinder- garten to university. In IEEE Frontiers in Education Conference. (pp. 1–9). https://doi.org/10.1109/FIE.2016.7757570

Kandlhofer, M., Steinbauer, G., Lasnig, J.P., Baumann, W., Plomer, S., Ballagi, A., & Alfoldi, I. (2019). Enabling the creation of intelligent things: Bringing artificial intelligence and robotics to schools. In IEEE Frontiers in Education Conference (FIE) . (pp. 1–5). https://doi.org/10.1109/FIE43999.2019.9028537

Kandlhofer, M., & Steinbauer, G. (2021). AI k-12 education service. KI Kunstliche Intelligenz, 35 (2), 125–126. https://doi.org/10.1007/s13218-021-00715-9

Kandlhofer, M., Steinbauer, G., Lassnig, J., Menzinger, M., Baumann, W., Ehardt-Schmiederer, M., & Szalay, I. (2021). EDLRIS: A European driving license for robots and intelligent systems. KI Kunstliche Intelligenz, 35 (2), 221–232. https://doi.org/10.1007/s13218-021-00716-8

Kaspersen, M. H., Bilstrup, K. E. K., Van Mechelen, M., Hjort, A., Bouvin, N. O., & Petersen, M. G. (2022). High school students exploring machine learning and its societal implications Opportunities and challenges. International Journal of Child-Computer Interaction, 34 , 1–12. https://doi.org/10.1016/j.ijcci.2022.100539

Kelly, J., Binney, J., Pereira, A., Khan, O., & Sukhatme, G. (2008). Just add wheels: Leveraging commodity laptop hardware for robotics and ai education. In  Proceedings of AAAI Education Colloquium , 22.

Kim, K., Kwon, K., Ottenbreit-Leftwich, A., Bae, H., & Glazewski, K. (2023). Exploring middle school students’ common naive conceptions of Artificial Intelligence concepts, and the evolution of these ideas. Education and Information Technologies . https://doi.org/10.1007/s10639-023-11600-3

Kim, S., Jang, Y., Choi, S., Kim, W., Jung, H., Kim, S., & Kim, H. (2021). Analyzing teacher competency with tpack for k-12 ai education. KI Kunstliche Intelligenz, 35 (2), 139–151. https://doi.org/10.1007/s13218-021-00731-9

Kitchenham, B. (2004). Procedures for performing systematic reviews (Vol. 33, pp. 1–26). Keele: Keele University.

Knijnenburg, B., Bannister, N., & Caine, K. (2021). Using mathematically- grounded metaphors to teach ai-related cybersecurity. In IJCAI-21 Workshop on Adverse Impacts and Collateral Effects of Artificial Intelligence Technologies (AIofAI) .

Kong, S. C., ManYinCheung, W., & Zhang, G. (2021). Evaluation of an artificial intelligence literacy course for university students with diverse study backgrounds. Computers and Education: Artificial Intelligence . https://doi.org/10.1016/j.caeai.2021.100026

Korenova, L. (2016). Digital technologies in teaching mathematics on the faculty of education of the Comenius University in Bratislava. In 15 Conference on Applied Mathematics. Slovak University of Technology in Bratislava. (p. 690–699).

Lee, S., Mott, B., Ottenbriet-Leftwich, A., Scribner, A., Taylor, S., Glazewski, K.,…Lester, J. (2020). Designing a collaborative game-based learning environment for ai-infused inquiry learning in elementary school class- rooms. In Proceedings of the 2020 ACM conference on innovation and technology in computer science education . (pp. 566–566). https://doi.org/10.1145/3341525.3393981

Lee, I., Ali, S., Zhang, H., Dipaola, D., & Breazeal, C. (2021). Developing middle school students’ ai literacy. In Association for Computing Machinery, Inc. (pp. 191–197). https://doi.org/10.1145/3408877.3432513

Lenoir, Y., & Hasni, A. (2016). Interdisciplinarity in primary and secondary school: Issues and perspectives. Creative Education, 7 (16), 2433–2458. https://doi.org/10.4236/ce.2016.716233

Lin, P., & Brummelen, J. (2021). Engaging teachers to co-design integrated ai curriculum for k-12 classrooms. In CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . (pp.1–12). https://doi.org/10.1145/3411764.3445377

Lin, X. F., Chen, L., Chan, K. K., Peng, S., Chen, X., Xie, S., & Hu, Q. (2022). Teachers’ perceptions of teaching sustainable artificial intelligence: A design frame perspective. Sustainability, 14 (13), 1–20. https://doi.org/10.3390/su14137811

Lindner, A., & Berges, M. (2020). Can you explain ai to me? teachers’ pre- concepts about artificial intelligence. In IEEE Frontiers in Education Conference (FIE). (pp. 1–9). https://doi.org/10.1109/FIE44824.2020.9274136

Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. In Proceedings of the 2020 chi conference on human factors in computing systems. (pp. 1–16). https://doi.org/10.1145/3313831.3376727

Malach, J., & Vicherková, D. (2020). Background of the Revision of the Secondary School Engineering Curriculum in the Context of the Society 4.0. In M. Auer, H. Hortsch & P. Sethakul (Eds). The Impact of the 4th Industrial Revolution on Engineering Education. ICL Advances in Intelligent Systems and Computing, vol 1135. Springer. https://doi.org/10.1007/978-3-030-40271-6_27

Matarić, M.J., Koenig, N., & Feil-Seifer, D. (2007). Materials for enabling hands- on robotics and stem education. In AAAI Spring Symposium: Semantic Scientific Knowledge Integration. (pp. 99–102). http://www.aaai.org/Papers/Symposia/Spring/2007/SS-07-09/SS07-09-022.pdf

Miao, F., Holmes, W., Huang, R., & Zhang, H. (2021). AI and education: A guidance for policymakers . UNESCO Publishing.

Micheuz, P. (2020). Approaches to Artificial Intelligence as a Subject in School Education. In T. Brinda, D. Passey, & T. Keane (Eds), Empowering Teaching for Digital Equity and Agency. OCCE 2020. IFIP Advances in Information and Communication Technology, 595. Springer. https://doi.org/10.1007/978-3-030-59847-1_1

Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108 (6), 1017–1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x

Mongan, W.M., & Regli, W.C. (2008). A cyber-infrastructure for supporting k-12 engineering education through robotics, WS-08-02, 68–73.

Narahara, T., & Kobayashi, Y. (2018). Personalizing homemade bots with plug & play ai for steam education. In SIGGRAPH Asia 2018 technical briefs . (pp. 1–4). https://doi.org/10.1145/3283254.3283270

Ng, D. T. K., Lee, M., Tan, R. J. Y., Hu, X., Downie, J. S., & Chu, S. K. W. (2022). A review of AI teaching and learning from 2000 to 2020. Education and Information Technologies . https://doi.org/10.1007/s10639-022-11491-w

Nisheva-Pavlova, M.M. (2021). Ai courses for secondary and high school - comparative analysis and conclusions. In CEUR Workshop Proceedings, 3061. (pp. 9–16).

Nurbekova, Z., Mukhamediyeva, K., & Assainova, A. (2018). Educational robotics technologies in Kazakhstan and in the world: Comparative analysis, current state and perspectives. Astra Salvensis, 6 (1), 665–686.

Oh, W. (2020). Physics teachers’ perception of it convergence-based physics education. New Physics: Sae Mulli, 70 (8), 660–666. https://doi.org/10.3938/NPSM.70.660

Olari, V., & Romeike, R. (2021). Addressing ai and data literacy in teacher education: A review of existing educational frameworks. In WiPSCE '21: The 16th Workshop in Primary and Secondary Computing Education, 17. (pp. 1–2) https://doi.org/10.1145/3481312.3481351

Ottenbreit-Leftwich, A., Glazewski, K., Jeon, M., Hmelo-Silver, C., Mott, B., Lee, S., & Lester, J. (2021). How do elementary students conceptualize artificial intelligence? In SIGCSE '21: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education . (pp. 1261). https://doi.org/10.1145/3408877.3439642

Park, K., Mott, B., Lee, S., Glazewski, K., Scribner, J., Ottenbreit-Leftwich, A., & Lester, J. (2021). Designing a visual interface for elementary students to formulate ai planning tasks. In IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). (pp. 1–9). https://doi.org/10.1109/VL/HCC51201.2021.9576163

Perlin, K., Flanagan, M., & Hollingshead, A. (2005). The Rapunsel Project. In Subsol, G. (Eds). Virtual Storytelling. Using Virtual Reality Technologies for Storytelling. ICVS 2005. Lecture Notes in Computer Science , 3805. Springer. https://doi.org/10.1007/11590361_29

Reyes, A., Elkin, C., Niyaz, Q., Yang, X., Paheding, S., & Devabhaktuni, V. (2020). A preliminary work on visualization-based education tool for high school machine learning education. In IEEE Integrated STEM Education Conference (ISEC). (pp. 1–5). https://doi.org/10.1109/ISEC49744.2020.9280629

Rodríguez-García, J., Moreno-León, J., Román-González, M., & Robles, G. (2020). Introducing artificial intelligence fundamentals with learning ML: Artificial intelligence made easy. In TEEM'20: Eighth International Conference on Technological Ecosystems for Enhancing Multiculturality . (pp. 18–20). https://doi.org/10.1145/3434780.3436705

Russell, S. (2021). The history and future of AI. Oxford Review of Economic Policy, 37 (3), 509–520. https://doi.org/10.1093/oxrep/grab013

Russell, S., & Norvig, P. (2021). Artificial Intelligence, global edition a modern approach . Pearson Deutschland.

Sabuncuoglu, A. (2020). Designing one year curriculum to teach artificial intelligence for middle school. In Proceedings of the 2020 ACM conference on innovation and technology in computer science education . (pp. 96–102). https://doi.org/10.1145/3341525.3387364

Sakulkueakulsuk, B., Witoon, S., Ngarmkajornwiwat, P., Pataranutapom, P., Surareungchai, W., Pataranutaporn, P., & Subsoontorn, P. (2019). Kids making ai: Integrating machine learning, gamification, and social context. In 2018 IEEE international conference on teaching, assessment, and learning for engineering (TALE) . (pp. 1005–1010). https://doi.org/10.1109/TALE.2018.8615249

Sanusi, I. T., Oyelere, S. S., Vartiainen, H., Suhonen, J., & Tukiainen, M. (2022). A systematic review of teaching and learning machine learning in K-12 education. Education and Information Technologies . https://doi.org/10.1007/s10639-022-11416-7

Shin, S. (2021). A study on the framework design of artificial intelligence thinking for artificial intelligence education. International Journal of Information and Education Technology, 11 (9), 392–397. https://doi.org/10.18178/ijiet.2021.11.9.1540

Sing, C. C., Teo, T., Huang, F., Chiu, T. K., & Xing Wei, W. (2022). Secondary school students’ intentions to learn AI: Testing moderation effects of readiness, social good and optimism. Educational Technology Research and Development, 70 (3), 765–782. https://doi.org/10.1007/s11423-022-10111-1

Sorensen, L., & Koefoed, N. (2018). The future of teaching—what are students’ expectations. In 2018 11th CMI International Conference: Prospects and Challenges Towards Developing a Digital Economy within the EU . (pp. 62–66). https://doi.org/10.1109/PCTDDE.2018.8624771

Sperling, A., & Lickerman, D. (2012). Integrating AI and machine learning in software engineering course for high school students. In Proceedings of the 17th ACM annual conference on Innovation and technology in computer science education . (pp. 244–249). https://doi.org/10.1145/2325296.2325354

Su, J., Zhong, Y., & Ng, D. T. K. (2022). A meta-review of literature on educational approaches for teaching AI at the K-12 levels in the Asia-Pacific region. Computers and Education: Artificial Intelligence, 3 , 1–18. https://doi.org/10.1016/j.caeai.2022.100065

Suh, W., & Ahn, S. (2022). Development and validation of a scale measuring student attitudes toward artificial intelligence. SAGE Open, 12 (2), 1–12. https://doi.org/10.1177/21582440221100463

Summers, B.G., Hicks, H., & Oliver, C. (1995). Reaching minority, female and disadvantaged students. In Proceedings Frontiers in Education 1995 25th Annual Conference. Engineering Education for the 21st Century , 1 . (992a4–16). https://doi.org/10.1109/FIE.1995.483030

Tedre, M., Toivonen, T., Kahila, J., Vartiainen, H., Valtonen, T., Jormanainen, I., & Pears, A. (2021). Teaching machine learning in k-12 classroom: Pedagogical and technological trajectories for artificial intelligence education. IEEE Access, 9 , 110558–110572. https://doi.org/10.1109/ACCESS.2021.3097962

Tims, H., Turner III, G., Cazes, G., & Marshall, J. (2012). Junior cyber discovery: Creating a vertically integrated middle school cyber camp. In  2012 ASEE Annual Conference & Exposition.  (pp. 25–867). Retrieved from https://peer.asee.org/21624

Toivonen, T., Jormanainen, I., Kahila, J., Tedre, M., Valtonen, T., Vartiainen, H. (2020). Co-designing machine learning apps in k-12 with primary school children. In  2020 IEEE 20th International Conference on Advanced Learning Technologies (ICALT). IEEE. (pp. 308–310). https://doi.org/10.1109/ICALT49669.2020.00099

Touretzky, D., Gardner-McCune, C., Breazeal, C., Martin, F., & Seehorn, D. (2019a). A year in k-12 ai education. AI Magazine, 40 (4), 88–90. https://doi.org/10.1609/aimag.v40i4.5289

Touretzky, D., Gardner-McCune, C., Martin, F., & Seehorn, D. (2019b). Envisioning ai for k-12: What should every child know about ai? Proceedings of the AAAI Conference on Artificial Intelligence, 33 (01), 9795–9799. https://doi.org/10.1609/aaai.v33i01.33019795

Vachovsky, M., Wu, G., Chaturapruek, S., Russakovsky, O., Sommer, R., & Fei-Fei, L. (2016). Towards more gender diversity in cs through an arti- ficial intelligence summer program for high school girls. In  Proceedings of the 47th ACM technical symposium on computing science education.  (pp. 303–308). https://doi.org/10.1145/2839509.2844620

Van Brummelen, J., Heng, T., & Tabunshchyk, V. (2021a). Teaching tech to talk: K-12 conversational artificial intelligence literacy curriculum and development tools. Proceedings of the AAAI Conference on Artificial Intelligence, 35 (17), 15655–15663. https://doi.org/10.1609/aaai.v35i17.17844

Van Brummelen, J., Tabunshchyk, V., & Heng, T. (2021b). Alexa, can i program you? Student perceptions of conversational artificial intelligence before and after programming Alexa. In IDC '21: Interaction Design and ChildrenJune. (pp. 305–313) https://doi.org/10.1145/3459990.3460730

Vartiainen, H., Tedre, M., & Valtonen, T. (2020). Learning machine learning with very young children: Who is teaching whom? International Journal of Child-Computer Interaction, 25 , 1–11. https://doi.org/10.1016/j.ijcci.2020.100182

Vartiainen, H., Toivonen, T., Jormanainen, I., Kahila, J., Tedre, M., & Valtonen, T. (2021). Machine learning for middle schoolers: Learning through data- driven design. International Journal of Child-Computer Interaction, 29 , 1–12. https://doi.org/10.1016/j.ijcci.2021.100281

Verner, I., Cuperman, D., & Reitman, M. (2021). Exploring robot connectivity and collaborative sensing in a high-school enrichment program. Robotics, 10 (1), 1–19. https://doi.org/10.3390/robotics10010013

von Wangenheim, C. G., Hauck, J. C., Pacheco, F. S., & Bueno, M. F. B. (2021). Visual tools for teaching machine learning in K-12: A ten-year systematic mapping. Education and Information Technologies, 26 (5), 5733–5778. https://doi.org/10.1007/s10639-021-10570-8

Wan, X., Zhou, X., Ye, Z., Mortensen, C., & Bai, Z. (2020). Smileyclus- ter: Supporting accessible machine learning in k-12 scientific discovery. In  proceedings of the Interaction Design and Children Conference.  (pp. 23–35). https://doi.org/10.1145/3392063.3394440

Wang, H., Liu, Y., Han, Z., & Wu, J. (2020). Extension of media literacy from the perspective of artificial intelligence and implementation strategies of artificial intelligence courses in junior high schools. In  2020 International Conference on Artificial Intelligence and Education (ICAIE).  (pp. 63–66). https://doi.org/10.1109/ICAIE50891.2020.00022

Wei, Y. (2021). Influence factors of using modern teaching technology in the classroom of junior middle school teachers under the background of artificial intelligence-analysis based on HLM. Advances in Intelligent Systems and Computing, 1282 , 110–118. https://doi.org/10.1007/978-3-030-62743-0_16

Wei, Q., Li, M., Xiang, K., & Qiu, X. (2020). Analysis and strategies of the professional development of information technology teachers under the vision of artificial intelligence. In  2020 15th International Conference on Computer Science & Education (ICCSE).  (pp. 716–721). https://doi.org/10.1109/ICCSE49874.2020.9201652

West, D.M., & Allen, J.R. (2018). How artificial intelligence is transforming the world. Report. Retrieved April 24, 2018, f rom https://www.brookings.edu/research/how-artificial-intelligence-is-transforming-the-world/

Wong, K.-C. (2020). Computational thinking and artificial intelligence education: A balanced approach using both classical AI and modern AI. CoolThink@ JC , 108.

Wong, G. K., Ma, X., Dillenbourg, P., & Huen, J. (2020). Broadening artificial intelligence education in k-12: Where to start? ACM Inroads, 11 (1), 20–29. https://doi.org/10.1145/3381884

Woo, H., Kim, J., Kim, J., & Lee, W. (2020). Exploring the ai topic composition of k-12 using nmf-based topic modeling. International Journal on Advanced Science, Engineering and Information Technology, 10 (4), 1471–1476. https://doi.org/10.18517/ijaseit.10.4.12787

Wu, D., Zhou, C., Meng, C., & Chen, M. (2020). Identifying multilevel factors influencing ICT self-efficacy of k-12 teachers in China. In  Blended Learning. Education in a Smart Learning Environment: 13th International Conference, ICBL 2020. (pp. 303–314). Springer International Publishing. https://doi.org/10.1007/978-3-030-51968-1

Xia, Q., Chiu, T. K., & Chai, C. S. (2022). The moderating effects of gender and need satisfaction on self-regulated learning through Artificial Intelligence (AI). Education and Information Technologies . https://doi.org/10.1007/s10639-022-11547-x

Xia, L., & Zheng, G. (2020). To meet the trend of AI: The ecology of developing ai talents for pre-service teachers in China. International Journal of Learning, 6 (3), 186–190. https://doi.org/10.18178/IJLT.6.3.186-190

Xiao, W., & Song, T. (2021). Current situation of artificial intelligence education in primary and secondary schools in China. In  The Sixth International Conference on Information Management and Technology. (pp. 1–4). https://doi.org/10.1145/3465631.3465980

Yau, K. W., Chai, C. S., Chiu, T. K., Meng, H., King, I., Wong, S. W. H., & Yam, Y. (2022). Co-designing artificial intelligence curriculum for secondary schools: A grounded theory of teachers' experience. In 2022 International Symposium on Educational Technology (ISET). (pp. 58–62). https://doi.org/10.1109/ISET55194.2022.00020

Yue, M., Dai, Y., Siu-Yung, M., & Chai, C.-S. (2021). An analysis of k-12 artificial intelligence curricula in eight countries. In  Proceedings of the 29th International Conference on Computers in Education.  (pp. 22–26).

Yue, M., Jong, M. S. Y., & Dai, Y. (2022). Pedagogical design of K-12 artificial intelligence education: A systematic review. Sustainability, 14 (23), 15620. https://doi.org/10.3390/su142315620

Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., & Li, Y. (2021). A review of artificial intelligence (AI) in education from 2010 to 2020. Complexity . https://doi.org/10.1155/2021/8812542

Zhang, N., Biswas, G., McElhaney, K.W., Basu, S., McBride, E., & Chiu, J.L. (2020). Studying the interactions between science, engineering, and computational thinking in a learning-by-modeling environment. In  International conference on artificial intelligence in education. (pp. 598–609). Springer.

Download references

Acknowledgements

Authors would like to thank the reviewers and editors, whose comments and feedback helped us to improve the original manuscript.

This work has partially been funded by the Spanish Ministry of Science, Innovation and Universities (PID2021-123152OB-C21), and the Consellería de Educación, Universidade e Formación Profesional (accreditation 2019–2022 ED431C2022/19 and reference competitive group, ED431G2019/04) and the European Regional Development Fund (ERDF), which acknowledges the CiTIUS—Centro Singular de Investigación en Tecnoloxías Intelixentes da Universidade de Santiago de Compostela as a Research Center of the Galician University System. This work also received support from the Educational Knowledge Transfer (EKT), the Erasmus + project (reference number 612414-EPP-1-2019-1-ES-EPPKA2-KA) and the Knowledge Alliances call (Call EAC/A03/2018).

Author information

Authors and affiliations.

Pedagogy and Didactics Department, University of Santiago de Compostela, 15782, Santiago de Compostela, Spain

Lorena Casal-Otero, Carmen Fernández-Morante & Beatriz Cebreiro

Departamento de Electrónica e Computación, Universidade de Santiago de Compostela, 15782, Santiago de Compostela, Spain

Alejandro Catala & Maria Taboada

Centro Singular de Investigación en Tecnoloxías Intelixentes (CiTIUS), Universidade de Santiago de Compostela, 15782, Santiago de Compostela, Spain

Alejandro Catala & Senén Barro

You can also search for this author in PubMed   Google Scholar

Contributions

All authors have contributed significantly to the authorship of this work in all stages of conceptualization, discussions, definition of the methodology, carrying out the analysis as well as writing—review & editing. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Alejandro Catala .

Ethics declarations

Ethics approval and consent to participate.

This research is carried out in accordance to ethics recommendations. As it focuses on a systematic literature review as a research method, ethics approval by the University ethics committee does not apply.

Competing interests

The authors declare that they have no competing interests. The authors have no other relevant financial or non-financial interests to disclose and no further competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1..

Additional listing.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Casal-Otero, L., Catala, A., Fernández-Morante, C. et al. AI literacy in K-12: a systematic literature review. IJ STEM Ed 10 , 29 (2023). https://doi.org/10.1186/s40594-023-00418-7

Download citation

Received : 06 September 2022

Accepted : 30 March 2023

Published : 19 April 2023

DOI : https://doi.org/10.1186/s40594-023-00418-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Secondary education
• Teaching/learning strategies
• Twenty-first century skills
• Cultural and social implications

Advertisement

Feedback literacy: a critical review of an emerging concept

• Published: 19 July 2022
• Volume 85 , pages 1381–1400, ( 2023 )

Cite this article

• Juuso Henrik Nieminen   ORCID: orcid.org/0000-0003-3087-8933 1 &
• David Carless   ORCID: orcid.org/0000-0003-1449-5174 1  

8788 Accesses

30 Citations

47 Altmetric

Explore all metrics

Systemic challenges for feedback practice are widely discussed in the research literature. The expanding mass higher education systems, for instance, seem to inhibit regular and sustained teacher-student interactions. The concept of feedback literacy , representing students’ and teachers’ capacities to optimize the benefits of feedback opportunities, has gained widespread attention by offering new ways of tackling these challenges. This study involves a critical review of the first 49 published articles on feedback literacy. Drawing on science and technology studies, and in particular on Popkewitz’s concept of fabrication, we explore how research has invented feedback literacy as a way of reframing feedback processes through the idea of individual skill development. First, we analyze how research has fabricated students and teachers through their feedback literacies that can be tracked, measured, and developed. Here, there exists a conceptual shift from analyzing feedback as external input to feedback literacy as a psychological construct residing within individuals. This interpretation carries positive implications of student and teacher empowerment, whilst downplaying policy-level challenges facing feedback interactions. The second contrasting fabrication positions feedback literate students and teachers as socio-culturally situated, communal agents. We conclude that feedback literacy is a powerful idea that, if used carefully, carries potential for reimagining feedback in higher education. It also, however, risks psychologizing students’ and teachers’ feedback behaviors amidst prevalent assessment and grading policies. We call for further reflexivity in considering whether feedback literacy research aims to challenge or complement the broader socio-political landscapes of higher education.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

• Get 10 units per month
• Download Article/Chapter or Ebook
• 1 Unit = 1 Article or 1 Chapter
• Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Similar content being viewed by others

Swedish students’ everyday school life and teachers’ assessment dilemmas: peer strategies for ameliorating schoolwork for assessment

Mapping research in student engagement and educational technology in higher education: a systematic evidence map

The power of assessment feedback in teaching and learning: a narrative review and synthesis of the literature

The studies in our review are marked with an asterisk (*)..

* Arts, J. G., Jaspers, M., & Joosten-ten Brinke, D. (2021). Enhancing written feedback: The use of a cover sheet influences feedback quality. Cogent Education , 8(1), 1901641.

Bagger, A. (2022). Opportunities to display knowledge during national assessment in mathematics: A matter of access and participation. European Journal of Special Needs Education, 37 (1), 104–117.

Article   Google Scholar  

* Banister, C. (2020). Exploring peer feedback processes and peer feedback meta-dialogues with learners of academic and business English. Language Teaching Research . https://doi.org/10.1177/1362168820952222

Biesta, G. (2009). Good education in an age of measurement: On the need to reconnect with the question of purpose in education. Educational Assessment, Evaluation and Accountability, 21 (1), 33–46.

* Boud, D., & Dawson, P. (2021). What feedback literate teachers do: an empirically-derived competency framework. Assessment & Evaluation in Higher Education , 1-14.  https://doi.org/10.1080/02602938.2021.1910928

Broadfoot, P. (1998). Quality, standards and control in higher education: What price life-long learning? International Studies in Sociology of Education, 8 (2), 155–180.

Buckley, A. (2020). Crisis? What crisis? Interpreting student feedback on assessment. Assessment & Evaluation in Higher Education , 1-12.  https://doi.org/10.1080/02602938.2020.1846015

Bunce, L., Baird, A., & Jones, S. E. (2017). The student-as-consumer approach in higher education and its effects on academic performance. Studies in Higher Education, 42 (11), 1958–1978.

Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65 (3), 245–281.

Carless, D. (2015). Excellence in university assessment: Learning from award-winning practice . Routledge.

Book   Google Scholar  

* Carless, D. (2019). Feedback loops and the longer-term: towards feedback spirals. Assessment & Evaluation in Higher Education , 44 (5), 705–714.

* Carless, D. (2020). Longitudinal perspectives on students’ experiences of feedback: a need for teacher–student partnerships. Higher Education Research & Development , 39 (3), 425–438.

* Carless, D. (2022). From teacher transmission of information to student feedback literacy: Activating the learner role in feedback processes. Active Learning in Higher Education , 23 (2), 143-153.  https://doi.org/10.1177/1469787420945845

* Carless, D., & Winstone, N. (2020). Teacher feedback literacy and its interplay with student feedback literacy. Teaching in Higher Education , 1-14.  https://doi.org/10.1080/13562517.2020.1782372

* Carless, D., & Boud, D. (2018). The development of student feedback literacy: enabling uptake of feedback. Assessment & Evaluation in Higher Education , 43 (8), 1315–1325.

* Chan, C. K. Y., & Luo, J. (2021). Exploring teacher perceptions of different types of ‘feedback practices’ in higher education: implications for teacher feedback literacy. Assessment & Evaluation in Higher Education , 1-16.  https://doi.org/10.1080/02602938.2021.1888074

* Chong, S. W. (2021). Reconsidering student feedback literacy from an ecological perspective. Assessment & Evaluation in Higher Education , 46 (1), 92–104.

* Dawson, P., Carless, D., & Lee, P. P. W. (2021). Authentic feedback: supporting learners to engage in disciplinary feedback practices. Assessment & Evaluation in Higher Education , 46 (2), 286–296.

* de Kleijn, R. A. (2021). Supporting student and teacher feedback literacy: an instructional model for student feedback processes. Assessment & Evaluation in Higher Education , 1-15.  https://doi.org/10.1080/02602938.2021.1967283

* Deneen, C. C., & Hoo, H. T. (2021). Connecting teacher and student assessment literacy with self-evaluation and peer feedback. Assessment & Evaluation in Higher Education , 1-13.  https://doi.org/10.1080/02602938.2021.1967284

Evans, A. M. (2011). Governing student assessment: Administrative rules, silenced academics and performing students. Assessment & Evaluation in Higher Education, 36 (2), 213–223.

Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 83 (1), 70–120.

* Esterhazy, R., de Lange, T., & Damşa, C. (2021). Performing teacher feedback literacy in peer mentoring meetings. Assessment & Evaluation in Higher Education , 1–14.  https://doi.org/10.1080/02602938.2021.1980768

* Fernández-Toro, M. & Duensing, A. (2021) Repositioning peer marking for feedback literacy in higher education. Assessment & Evaluation in Higher Education , 46 (8), 1202–1220

Grant, M. J., & Booth, A. (2009). A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26 (2), 91–108.

* Gravett, K. (2022). Feedback literacies as sociomaterial practice. Critical Studies in Education , 63 (2), 261–274.

* Han, Y., & Xu, Y. (2020). The development of student feedback literacy: the influences of teacher feedback on peer feedback. Assessment & Evaluation in Higher Education , 45 (5), 680–696.

* Han, Y., & Xu, Y. (2021). Student feedback literacy and engagement with feedback: a case study of Chinese undergraduate students. Teaching in Higher Education , 26 (2), 181–196.

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77 (1), 81–112.

* Heron, M., Medland, E., Winstone, N., & Pitt, E. (2021). Developing the relational in teacher feedback literacy: exploring feedback talk. Assessment & Evaluation in Higher Education , 1-14.  https://doi.org/10.1080/02602938.2021.1932735

* Hey-Cunningham, A. J., Ward, M. H., & Miller, E. J. (2021). Making the most of feedback for academic writing development in postgraduate research: Pilot of a combined programme for students and supervisors. Innovations in Education and Teaching International , 58 (2), 182–194.

* Hill, J., & West, H. (2019). Improving the student learning experience through dialogic feed-forward assessment. Assessment & Evaluation in Higher Education , 45 (1), 82–97.

* Hoo, H. T., Deneen, C., & Boud, D. (2021). Developing student feedback literacy through self and peer assessment interventions. Assessment & Evaluation in Higher Education , 1-14.  https://doi.org/10.1080/02602938.2021.1925871

Ibarra-Sáiz, M. S., Rodríguez-Gómez, G., & Boud, D. (2020). Developing student competence through peer assessment: the role of feedback, self-regulation and evaluative judgement. Higher Education , 80 (1), 137–156. https://doi.org/10.1007/s10734-019-00469-2

* Joughin, G., Boud, D., Dawson, P., & Tai, J. (2021). What can higher education learn from feedback seeking behaviour in organisations? Implications for feedback literacy. Assessment & Evaluation in Higher Education , 46 (1), 80–91.

Lea, M. R., & Street, B. V. (1998). Student writing in higher education: An academic literacies approach. Studies in Higher Education, 23 (2), 157–172.

Lea, M. R., & Street, B. V. (2006). The" academic literacies" model: Theory and applications. Theory into Practice, 45 (4), 368–377.

* Li, F., & Han, Y. (2022). Student feedback literacy in L2 disciplinary writing: insights from international graduate students at a UK university. Assessment & Evaluation in Higher Education , 47 (2), 198–212. https://doi.org/10.1080/02602938.2021.1908957

Lipnevich, A., & Panadero, E. (2021). A Review of Feedback Models and Theories: Descriptions, Definitions, and Conclusions. Frontiers in Education, 6 , 720195. https://doi.org/10.3389/feduc.2021.720195

Mahoney, P., Macfarlane, S., & Ajjawi, R. (2019). A qualitative synthesis of video feedback in higher education. Teaching in Higher Education, 24 (2), 157–179.

* Malecka, B., Boud, D., & Carless, D. (2020). Eliciting, processing and enacting feedback: mechanisms for embedding student feedback literacy within the curriculum. Teaching in Higher Education , 1–15. https://doi.org/10.1080/13562517.2020.1754784

* Malecka, B., Boud, D., Tai, J., & Ajjawi, R. (2022). Navigating feedback practices across learning contexts: implications for feedback literacy. Assessment & Evaluation in Higher Education , 1–15.  https://doi.org/10.1080/02602938.2022.2041544

Marginson, S. (2016). The worldwide trend to high participation higher education: Dynamics of social stratification in inclusive systems. Higher Education, 72 (4), 413–434.

* Molloy, E., Boud, D., & Henderson, M. (2020). Developing a learning-centred framework for feedback literacy. Assessment & Evaluation in Higher Education , 45 (4), 527–540.

Nicol, D. (2021). The power of internal feedback: Exploiting natural comparison processes. Assessment & Evaluation in Higher Education, 46 (5), 756–778.

Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher education: A peer review perspective. Assessment & Evaluation in Higher Education, 39 (1), 102–122.

Nieminen, J. H., Bearman, M., & Tai, J. (2022a). How is theory used in assessment and feedback research? A critical review. Assessment & Evaluation in Higher Education , 1-18.  https://doi.org/10.1080/02602938.2022.2047154

* Nieminen, J. H., Tai, J., Boud, D., & Henderson, M. (2022b). Student agency in feedback: beyond the individual. Assessment & Evaluation in Higher Education , 47 (1), 95–108. https://doi.org/10.1080/02602938.2021.1887080

* Noble, C., Billett, S., Armit, L., Collier, L., Hilder, J., Sly, C., & Molloy, E. (2020). “It’s yours to take”: generating learner feedback literacy in the workplace. Advances in Health Sciences Education , 25 (1), 55–74.

* O’Connor, A., & McCurtin, A. (2021). A feedback journey: employing a constructivist approach to the development of feedback literacy among health professional learners. BMC Medical Education , 21 (1), 1–13.

O’Donovan, B. (2017). How student beliefs about knowledge and knowing influence their satisfaction with assessment and feedback. Higher Education, 74 (4), 617–633.

Ostrowicka, H. (2022). Economization of discourse on education and pedagogization of economic problems: media debate on higher education reform in Poland. Discourse: Studies in the Cultural Politics of Education , 43 (1), 86–100. https://doi.org/10.1080/01596306.2020.1811957

Pangrazio, L., & Selwyn, N. (2019). ‘Personal data literacies’: A critical literacies approach to enhancing understandings of personal digital data. New Media & Society, 21 (2), 419–437.

Pitt, E., & Carless, D. (2021). Signature feedback practices in the creative arts: Integrating feedback within the curriculum. Assessment & Evaluation in Higher  Education, 1–13. https://doi.org/10.1080/02602938.2021.1980769

* Pitt, E., Bearman, M., & Esterhazy, R. (2019). The conundrum of low achievement and feedback for learning. Assessment & Evaluation in Higher Education , 45 (2), 239–250.

Popkewitz, T. S. (2008a). The social, psychological, and education sciences: From educationalization to pedagogicalization of the family and the child. In P. Smeyers & M. Depaepe (Eds.), Educational Research: The Educationalization of Social Problems (pp. 171–190). Springer.

Chapter   Google Scholar  

Popkewitz, T. S. (2008b). Cosmopolitanism and the age of school reform: Science, education, and making society by making the Child . Routledge.

Google Scholar  

Popkewitz, T. S. (2013). The sociology of education as the history of the present: Fabrication, difference and abjection. Discourse: Studies in the Cultural Politics of Education, 34 (3), 439–456.

Raaper, R. (2017). Tracing assessment policy discourses in neoliberalised higher education settings. Journal of Education Policy, 32 (3), 322–339.

* Rovagnati, V., & Pitt, E. (2021). Exploring intercultural dialogic interactions between individuals with diverse feedback literacies. Assessment & Evaluation in Higher Education , 1–14.  https://doi.org/10.1080/02602938.2021.2006601

* Rovagnati, V., Pitt, E., & Winstone, N. (2022). Feedback cultures, histories and literacies: international postgraduate students’ experiences. Assessment & Evaluation in Higher Education , 47 (3), 347–359. https://doi.org/10.1080/02602938.2021.1916431

* Song, B. K. (2022). Bifactor modelling of the psychological constructs of learner feedback literacy: conceptions of feedback, feedback trust and self-efficacy. Assessment & Evaluation in Higher Education , 1–14.  https://doi.org/10.1080/02602938.2022.2042187

* Sutton, P. (2012). Conceptualizing feedback literacy: Knowing, being, and acting. Innovations in Education and Teaching International , 49(1), 31–40.

* Tai, J., Bearman, M., Gravett, K., & Molloy, E. (2021). Exploring the notion of teacher feedback literacies through the theory of practice architectures. Assessment & Evaluation in Higher Education , 1–13.  https://doi.org/10.1080/02602938.2021.1948967

Tannock, S. (2017). No grades in higher education now! Revisiting the place of graded assessment in the reimagination of the public university. Studies in Higher Education, 42 (8), 1345–1357.

Torrance, H. (2017). Blaming the victim: assessment, examinations, and the responsibilisation of students and teachers in neo-liberal governance. Discourse: Studies in the Cultural politics of Education, 38 (1), 83–96.

Vincent, D. (2003). Literacy literacy. Interchange, 34 (2), 341–357.

* Wei, W., Sun, Y., & Xu, X. (2020). Investigating the impact of increased student feedback literacy level on their expectations on university teachers’ feedback. Assessment & Evaluation in Higher Education , 46 (7), 1092–1103.

Wheelahan, L., Moodie, G., & Doughney, J. (2022). Challenging the skills fetish. British Journal of Sociology of Education , 43 (3), 475–494. https://doi.org/10.1080/01425692.2022.2045186

Winstone, N. E., & Carless, D. (2019). Designing effective feedback processes in higher education: A learning-focused approach . Routledge.

Winstone, N. E., Bourne, J., Medland, E., Niculescu, I., & Rees, R. (2021a). “Check the grade, log out”: Students’ engagement with feedback in learning management systems. Assessment & Evaluation in Higher Education, 46 (4), 631–643.

Winstone, N. E., Ajjawi, R., Dirkx, K., & Boud, D. (2021b). Measuring what matters: the positioning of students in feedback processes within national student satisfaction surveys. Studies in Higher Education , 1–13.

Winstone, N., Boud, D., Dawson, P., & Heron, M. (2022a). From feedback-as-information to feedback-as-process: a linguistic analysis of the feedback literature. Assessment & Evaluation in Higher Education , 47 (2), 213–230. https://doi.org/10.1080/02602938.2021.1902467

* Winstone, N. E., Balloo, K., & Carless, D. (2022b). Discipline-specific feedback literacies: A framework for curriculum design. Higher Education , 83(1), 57–77.  https://doi.org/10.1007/s10734-020-00632-0

* Winstone, N. E., Mathlin, G., & Nash, R. A. (2019). Building feedback literacy: students’ perceptions of the Developing Engagement with Feedback Toolkit. Frontiers in Education , 4 , 39.

* Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017). Supporting learners' agentic engagement with feedback: a systematic review and a taxonomy of recipience processes. Educational Psychologist , 52 (1), 17–37.

Wisniewski, B., Zierer, K., & Hattie, J. (2020). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10 , 3087.

* Wood, J. (2020). A dialogic technology-mediated model of feedback uptake and literacy. Assessment & Evaluation in Higher Education , 46 (8), 1173–1190.

* Wood, J. (2022). Making peer feedback work: the contribution of technology-mediated dialogic peer feedback to feedback uptake and literacy. Assessment & Evaluation in Higher Education , 47 (3), 327–346. https://doi.org/10.1080/02602938.2021.1914544

* Xiang, X., Yuan, R., & Yu, B. (2021). Implementing assessment as learning in the L2 writing classroom: a Chinese case. Assessment & Evaluation in Higher Education , 1–15.  https://doi.org/10.1080/02602938.2021.1965539

* Xu, Y., & Carless, D. (2017). ‘Only true friends could be cruelly honest’: cognitive scaffolding and social-affective support in teacher feedback literacy. Assessment & Evaluation in Higher Education , 42 (7), 1082–1094.

* Yan, Z., & Carless, D. (2021). Self-assessment is about more than self: the enabling role of feedback literacy. Assessment & Evaluation in Higher Education , 1–13.  https://doi.org/10.1080/02602938.2021.2001431

* Yu, S. (2021). Feedback-giving practice for L2 writing teachers: friend or foe? Journal of Second Language Writing , 52 , 100798.

* Yu, S., & Liu, C. (2021). Improving student feedback literacy in academic writing: an evidence-based framework. Assessing Writing , 48 , 100525.

* Zhan, Y. (2021a). Developing and validating a student feedback literacy scale. Assessment & Evaluation in Higher Education.  https://doi.org/10.1080/02602938.2021.2001430

* Zhan, Y. (2021b). Are they ready? An investigation of university students’ difficulties in peer assessment from dual perspectives. Teaching in Higher Education, 1–18. https://doi.org/10.1080/13562517.2021.2021393 .

* Zhou, J., Dawson, P., Tai, J. H. M., & Bearman, M. (2021). How conceptualising respect can inform feedback pedagogies. Assessment & Evaluation in Higher Education , 46 (1), 68–79.

Download references

Acknowledgements

We would like to express our heartfelt gratitude to Dr. Karen Gravett and Dr. Zi Yan for their thoughtful comments on an earlier version of the manuscript. Their feedback literacies were vital for sharpening our argumentation.

Author information

Authors and affiliations.

Faculty of Education, The University of Hong Kong, Hong Kong, SAR, China

Juuso Henrik Nieminen & David Carless

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Juuso Henrik Nieminen .

Ethics declarations

Conflict of interest.

The authors declare that they have no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Nieminen, J.H., Carless, D. Feedback literacy: a critical review of an emerging concept. High Educ 85 , 1381–1400 (2023). https://doi.org/10.1007/s10734-022-00895-9

Download citation

Accepted : 23 June 2022

Published : 19 July 2022

Issue Date : June 2023

DOI : https://doi.org/10.1007/s10734-022-00895-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Feedback literacy
• Learner-centred feedback
• Critical review
• Science and technology studies
• Find a journal
• Publish with us
• Track your research

• Craft and Criticism
• Fiction and Poetry
• News and Culture
• Lit Hub Radio
• Reading Lists

• Literary Criticism
• Craft and Advice
• In Conversation
• On Translation
• Short Story
• From the Novel
• Bookstores and Libraries
• Film and TV
• Art and Photography
• Freeman’s
• The Virtual Book Channel
• Behind the Mic
• Beyond the Page
• The Cosmic Library
• The Critic and Her Publics
• Emergence Magazine
• Fiction/Non/Fiction
• First Draft: A Dialogue on Writing
• The History of Literature
• I’m a Writer But
• Lit Century
• Tor Presents: Voyage Into Genre
• Windham-Campbell Prizes Podcast
• Write-minded
• The Best of the Decade
• Best Reviewed Books
• BookMarks Daily Giveaway
• The Daily Thrill
• CrimeReads Daily Giveaway

The Best Reviewed Short Story Collections of 2021

Featuring haruki murakami, brandon taylor, elizabeth mccracken, kevin barry, lily king, and more.

Well, friends, another grim and grueling plague year is drawing to a close, and that can mean only one thing: it’s time to put on our Book Marks stats hats and tabulate the best reviewed books of the past twelve months.

Yes, using reviews drawn from more than 150 publications, over the next two weeks we’ll be revealing the most critically-acclaimed books of 2021, in the categories of (deep breath): Memoir and Biography ; Sci-Fi, Fantasy, and Horror ; Short Story Collections; Essay Collections; Poetry; Mystery and Crime; Graphic Literature; Literature in Translation; General Fiction; and General Nonfiction.

Today’s installment: Short Story Collections .

Brought to you by Book Marks , Lit Hub’s “Rotten Tomatoes for books.”

1. Afterparties by Anthony Veasna So (Ecco)

22 Rave • 5 Positive • 1 Mixed

“The presence of the author is so vivid in Afterparties , Anthony Veasna So’s collection of stories, he seems to be at your elbow as you read … The personality that animates Afterparties is unmistakably youthful, and the stories themselves are mainly built around conditions of youth—vexed and tender relationships with parents, awkward romances, nebulous worries about the future. But from his vantage on the evanescent bridge to maturity, So is puzzling out some big questions, ones that might be exigent from different vantages at any age. The stories are great fun to read—brimming over with life and energy and comic insight and deep feeling.”

–Deborah Eisenberg ( New York Review of Books )

2. Filthy Animals by Brandon Taylor (Riverhead)

19 Rave • 7 Positive • 2 Mixed Read an interview with Brandon Taylor here

“Taylor plays the Lionel-Charles-Sophie storyline for all its awkwardness and resentment, but it can feel like a note held too long to suspend commitment, which is the resolution we’re trained to expect … The violence is neither glamorous nor gratuitous; it is senseless without being pointless. In contrast, Taylor presents such earnest moments of vulnerability in Anne of Cleves that my breath hitched … Some writers have the gift of perfect pitch when writing dialogue; Taylor’s gift is perfect tempo. In a band of writers, he’d be the drummer who sticks to a steady moderato. He neither rushes a story to its high notes nor drags the pace so that we can admire his voice. And as a plotter, he doesn’t rely on gasp-inducing reveals … Taylor’s superpower is compressing a lifetime of backstory into a paragraph – sometimes just a sentence … I’ve come to expect, in fiction, the story of the Sad Gay Youth who is rejected by his often religious family and thereafter becomes self-destructive or reckless. And while Taylor refracts versions of this story throughout the collection, he does so without overly romanticising it … He is a writer of enormous subtlety and of composure beyond his years.”

–Ian Williams ( The Guardian )

3. First Person Singular by Haruki Murakami (Knopf)

13 Rave • 17 Positive • 7 Mixed • 5 Pan

“… a blazing and brilliant return to form … a taut and tight, suspenseful and spellbinding, witty and wonderful group of eight stories … there isn’t a weak one in the bunch. The stories echo with Murakami’s preoccupations. Nostalgia and longing for the charged, evocative moments of young adulthood. Memory’s power and fragility; how identity forms from random decisions, ‘minor incidents,’ and chance encounters; the at once intransigent and fragile nature of the ‘self.’ Guilt, shame, and regret for mistakes made and people damaged by foolish or heartless choices. The power and potency of young love and the residual weight of fleeting erotic entanglements. Music’s power to make indelible impressions, elicit buried memories, connect otherwise very different people, and capture what words cannot. The themes become a kind of meter against which all the stories make their particular, chiming rhythms … The reading experience is unsettled by a pervasive blurring of the lines between fantasy and reality, dream and waking … Most of the narrators foreground the act of telling and ruminate on the intention behind and effects of disclosing secrets, putting inchoate impulses, fears, or yearnings into clear, logical prose … This mesmerizing collection would make a superb introduction to Murakami for anyone who hasn’t yet fallen under his spell; his legion of devoted fans will gobble it up and beg for more.”

–Pricilla Gilman ( The Boston Globe )

4. That Old Country Music by Kevin Barry (Doubleday)

13 Rave • 10 Positive •1 Mixed

“There’s not a bad story in the bunch, and it’s as accomplished a book as Barry has ever written … Barry does an excellent job probing the psyche of his diffident protagonist, and ends the story with an unexpected moment of sweetness that’s anything but cloying—realism doesn’t need to be miserablism, he seems to hint; sometimes things actually do work out … Barry has a rare gift for crafting characters the reader cares about despite their flaws; in just 13 pages, he manages to make Hannah and Setanta come to life through sharp dialogue and keen observations … Barry proves to be a master of writing about both love and cruelty … Barry brilliantly evokes both the good and bad sides of love, and does so with stunningly gorgeous writing … There’s not an aspect of writing that Barry doesn’t excel at. His dialogue rings true, and he’s amazingly gifted at scene-setting—he evokes both the landscape of western Ireland and the landscape of the human heart beautifully. His greatest accomplishment, perhaps, is his understanding of the ways our collective psyche works; he seems to have an innate sense of why people behave the way we do, and exactly what we’re capable of, both good and bad.”

–Michael Schaub ( NPR )

5. Milk Blood Heat by Dantiel W. Moniz (Grove)

17 Rave • 1 Positive Listen to an interview with Dantiel W. Moniz here

“Mortality is the undercurrent in Dantiel W. Moniz’s electrifying debut story collection, Milk Blood Heat , but where there’s death there is the whir of life, too. A lot of collections consist of some duds, yet every single page in this book is a shimmering seashell that contains the sound of multiple oceans. Reading one of Moniz’s stories is like holding your breath underwater while letting the salt sting your fresh wounds. It’s exhilarating and shocking and even healing. The power in these stories rests in their veracity, vitality and vulnerability.”

–Michelle Filgate ( The Washington Post )

6. The Dangers of Smoking in Bed by Mariana Enriquez (Hogarth)

15 Rave 2 Positive Read a story from The Dangers of Smoking in Bed here

“There’s something thrilling about other people’s suffering—at least within this collection’s 12 stories of death, sex and the occult. Horrors are relayed in a stylish deadpan … Enriquez’s plots deteriorate with satisfying celerity … Largely it’s insatiable women, raggedy slum dwellers and dead children—those who are ordinarily powerless—who wield unholy power in this collection, and they seem uninterested in being reasonable. And Enriquez is particularly adept at capturing the single-minded intensity of teenage girls … If some of these stories end vaguely, the best ones close on the verge of some transgressive climax … To Enriquez, there’s pleasure in the perverse.”

–Chelsea Leu ( The New York Times Book Review )

7. The Souvenir Museum by Elizabeth McCracken (Ecco)

13 Rave • 2 Positive • 1 Mixed Read Elizabeth McCracken on savoring the mystery of stories here

“Elizabeth McCracken’s The Souvenir Museum begins with one of the funniest short stories I’ve read in a long time … I had to stop reading ‘The Irish Wedding’ several times to explain to my husband why I was laughing so hard. I kept thinking: I wish I were reading a whole book about these people … they’re all beguiling … This tale, like much of McCracken’s work, captures the mixed bag that characterizes most people’s lives … McCracken’s writing is never dull. She ends this fantastic collection with a second English wedding and its aftermath, nearly 20 years after the first, delivering happiness tempered by sobering circumstances—and a satisfying symmetry.”

–Heller McAlpin ( NPR )

8. Wild Swims by Dorthe Nors (Graywolf)

13 Rave • 1 Positive Read an excerpt from Wild Swims here

“How slippery the work of the Danish writer Dorthe Nors is, how it sideswipes and gleams … The stories are vivid the way a flash of immobilizing pain is vivid … Perhaps because they’re so very short and because they mostly sketch slight interior shifts in her characters, Nors’s stories all feel a little bashful, a little tender. Surely this is intentional … Most of her stories are too short to linger deeply in time or consciousness; the characters spin back into their silence almost as soon as they emerge on the page. Nors is a master at portraying female rage, but here there is also no violent explosion outward, instead a sort of inner collapse; her characters assiduously resist confronting their fury until it rises up against them and attacks their bodies … The sense of simultaneous, furious upwelling into text and retraction into shame or reticence gives the stories a powerful undercurrent, as if they were constantly wrestling with themselves. Inherently self-contradicting, they wobble interestingly on their axes, pulled between outraged individualism and the restrictive Janteloven.”

–Lauren Groff ( The New York Review of Books )

9. Walking on Cowrie Shells by Nana Nkweti (Graywolf)

12 Rave • 1 Mixed Read an interview with Nana Nkweti here

“The pure energy of the words strikes first, the thrumming, soaring, frenetic pace of Nana Nkweti’s expression … None of these stories end with a miraculous healing. Even where revelations occur, they never erase scars. Nkweti uses genre tropes to subvert our expectations. She employs the zombie story, the fairy tale, and the confessional in order to invert conventions … The levity of Nkweti’s writing can make even passing descriptions a delight … Occasionally the writing veers into the overwrought … But the sheer speed of Nkweti’s expression allows for correction in midair, and her keen descriptive eye provides more pleasures than missteps … Her inventiveness dazzles.”

–Lee Thomas ( Los Angeles Review of Books )

10. My Monticello by Jocelyn Nicole Johnson (Henry Holt)

9 Rave • 4 Positive 1 Mixed Read Jocelyn Nicole Johnson on how writing “vengeful fiction” can make you a better person, here

“Jocelyn Nicole Johnson uses history to spectacular effect in her debut fiction collection … What makes My Monticello particularly resonant is that it does not stray far from life as we know it today. In the near future conjured by Johnson, there are the heat waves and wildfires that bring climate change into view. There is fallout from a fraught election. There is the vile replacement theory rhetoric of the right wing. But the lives of Johnson’s richly drawn characters—their personal stories—are always in focus. And, because of it, the storytelling is propulsive, as we follow these refugees along a harrowing journey, with danger ever at their heels. My Monticello is, quite simply, an extraordinary debut from a gifted writer with an unflinching view of history and what may come of it.”

–Anissa Gray ( The Washington Post )

Our System:

RAVE = 5 points • POSITIVE = 3 points • MIXED = 1 point • PAN = -5 points

• Share on Facebook (Opens in new window)
• Click to share on Twitter (Opens in new window)
• Click to share on Google+ (Opens in new window)
• Click to share on LinkedIn (Opens in new window)
• Click to share on Reddit (Opens in new window)
• Click to share on Tumblr (Opens in new window)
• Click to share on Pinterest (Opens in new window)
• Click to share on Pocket (Opens in new window)

Previous Article

Next article, support lit hub..

Join our community of readers.

to the Lithub Daily

Popular posts.

Follow us on Twitter

On the Fleeting Wonder of Youth and the Surreal Permanence of Motherhood

• RSS - Posts

Literary Hub

Created by Grove Atlantic and Electric Literature

Sign Up For Our Newsletters

How to Pitch Lit Hub

Advertisers: Contact Us

Privacy Policy

Support Lit Hub - Become A Member

Become a Lit Hub Supporting Member : Because Books Matter

For the past decade, Literary Hub has brought you the best of the book world for free—no paywall. But our future relies on you. In return for a donation, you’ll get an ad-free reading experience , exclusive editors’ picks, book giveaways, and our coveted Joan Didion Lit Hub tote bag . Most importantly, you’ll keep independent book coverage alive and thriving on the internet.

Become a member for as low as $5/month

Map Options

Education Rankings by Country 2024

There is a correlation between a country's educational system quality and its economic status, with developed nations offering higher quality education.

The U.S., despite ranking high in educational system surveys, falls behind in math and science scores compared to many other countries.

Educational system adequacy varies globally, with some countries struggling due to internal conflicts, economic challenges, or underfunded programs.

While education levels vary from country to country, there is a clear correlation between the quality of a country's educational system and its general economic status and overall well-being. In general, developing nations tend to offer their citizens a higher quality of education than the least developed nations do, and fully developed nations offer the best quality of education of all. Education is clearly a vital contributor to any country's overall health.

According to the Global Partnership for Education , education is considered to be a human right and plays a crucial role in human, social, and economic development . Education promotes gender equality, fosters peace, and increases a person's chances of having more and better life and career opportunities.

"Education is the most powerful weapon which you can use to change the world." — Nelson Mandela

The annual Best Countries Report , conducted by US News and World Report, BAV Group, and the Wharton School of the University of Pennsylvania , reserves an entire section for education. The report surveys thousands of people across 78 countries, then ranks those countries based upon the survey's responses. The education portion of the survey compiles scores from three equally-weighted attributes: a well-developed public education system, would consider attending university there, and provides top-quality education. As of 2023, the top ten countries based on education rankings are:

Countries with the Best Educational Systems - 2021 Best Countries Report*

Ironically, despite the United States having the best-surveyed education system on the globe, U.S students consistently score lower in math and science than students from many other countries. According to a Business Insider report in 2018, the U.S. ranked 38th in math scores and 24th in science. Discussions about why the United States' education rankings have fallen by international standards over the past three decades frequently point out that government spending on education has failed to keep up with inflation.

It's also worthwhile to note that while the Best Countries study is certainly respectable, other studies use different methodologies or emphasize different criteria, which often leads to different results. For example, the Global Citizens for Human Rights' annual study measures ten levels of education from early childhood enrollment rates to adult literacy. Its final 2020 rankings look a bit different:

Education Rates of Children Around the World

Most findings and ranking regarding education worldwide involve adult literacy rates and levels of education completed. However, some studies look at current students and their abilities in different subjects.

One of the most-reviewed studies regarding education around the world involved 470,000 fifteen-year-old students. Each student was administered tests in math, science, and reading similar to the SAT or ACT exams (standardized tests used for college admissions in the U.S.) These exam scores were later compiled to determine each country's average score for each of the three subjects. Based on this study, China received the highest scores , followed by Korea, Finland , Hong Kong , Singapore , Canada , New Zealand , Japan , Australia and the Netherlands .

On the down side, there are many nations whose educational systems are considered inadequate. This could be due to internal conflict, economic problems, or underfunded programs. The United Nations Educational, Scientific, and Cultural Organization's Education for All Global Monitoring Report ranks the following countries as having the world's worst educational systems:

Countries with the Lowest Adult Literacy Rates

• Education rankings are sourced from both the annual UN News Best Countries report and the nonprofit organization World Top 20

Download Table Data

Enter your email below, and you'll receive this table's data in your inbox momentarily.

Which country ranks first in education?

Which country ranks last in education, frequently asked questions.

• Best Countries for Education - 2023 - US News
• Literacy rate, adult total (% of people ages 15 and above) - World Bank
• World Best Education Systems - Global Citizens for Human Rights
• UNESCO - Global Education Monitoring Reports
• World’s 10 Worst Countries for Education - Global Citizen
• International Education Database - World Top 20

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Springer Nature - PMC COVID-19 Collection

A systematic review on digital literacy

Hasan tinmaz.

1 AI & Big Data Department, Endicott College of International Studies, Woosong University, Daejeon, South Korea

Yoo-Taek Lee

2 Endicott College of International Studies, Woosong University, Daejeon, South Korea

Mina Fanea-Ivanovici

3 Department of Economics and Economic Policies, Bucharest University of Economic Studies, Bucharest, Romania

Hasnan Baber

4 Abu Dhabi School of Management, Abu Dhabi, United Arab Emirates

Associated Data

The authors present the articles used for the study in “ Appendix A ”.

The purpose of this study is to discover the main themes and categories of the research studies regarding digital literacy. To serve this purpose, the databases of WoS/Clarivate Analytics, Proquest Central, Emerald Management Journals, Jstor Business College Collections and Scopus/Elsevier were searched with four keyword-combinations and final forty-three articles were included in the dataset. The researchers applied a systematic literature review method to the dataset. The preliminary findings demonstrated that there is a growing prevalence of digital literacy articles starting from the year 2013. The dominant research methodology of the reviewed articles is qualitative. The four major themes revealed from the qualitative content analysis are: digital literacy, digital competencies, digital skills and digital thinking. Under each theme, the categories and their frequencies are analysed. Recommendations for further research and for real life implementations are generated.

Introduction

The extant literature on digital literacy, skills and competencies is rich in definitions and classifications, but there is still no consensus on the larger themes and subsumed themes categories. (Heitin, 2016 ). To exemplify, existing inventories of Internet skills suffer from ‘incompleteness and over-simplification, conceptual ambiguity’ (van Deursen et al., 2015 ), and Internet skills are only a part of digital skills. While there is already a plethora of research in this field, this research paper hereby aims to provide a general framework of digital areas and themes that can best describe digital (cap)abilities in the novel context of Industry 4.0 and the accelerated pandemic-triggered digitalisation. The areas and themes can represent the starting point for drafting a contemporary digital literacy framework.

Sousa and Rocha ( 2019 ) explained that there is a stake of digital skills for disruptive digital business, and they connect it to the latest developments, such as the Internet of Things (IoT), cloud technology, big data, artificial intelligence, and robotics. The topic is even more important given the large disparities in digital literacy across regions (Tinmaz et al., 2022 ). More precisely, digital inequalities encompass skills, along with access, usage and self-perceptions. These inequalities need to be addressed, as they are credited with a ‘potential to shape life chances in multiple ways’ (Robinson et al., 2015 ), e.g., academic performance, labour market competitiveness, health, civic and political participation. Steps have been successfully taken to address physical access gaps, but skills gaps are still looming (Van Deursen & Van Dijk, 2010a ). Moreover, digital inequalities have grown larger due to the COVID-19 pandemic, and they influenced the very state of health of the most vulnerable categories of population or their employability in a time when digital skills are required (Baber et al., 2022 ; Beaunoyer, Dupéré & Guitton, 2020 ).

The systematic review the researchers propose is a useful updated instrument of classification and inventory for digital literacy. Considering the latest developments in the economy and in line with current digitalisation needs, digitally literate population may assist policymakers in various fields, e.g., education, administration, healthcare system, and managers of companies and other concerned organisations that need to stay competitive and to employ competitive workforce. Therefore, it is indispensably vital to comprehend the big picture of digital literacy related research.

Literature review

Since the advent of Digital Literacy, scholars have been concerned with identifying and classifying the various (cap)abilities related to its operation. Using the most cited academic papers in this stream of research, several classifications of digital-related literacies, competencies, and skills emerged.

Digital literacies

Digital literacy, which is one of the challenges of integration of technology in academic courses (Blau, Shamir-Inbal & Avdiel, 2020 ), has been defined in the current literature as the competencies and skills required for navigating a fragmented and complex information ecosystem (Eshet, 2004 ). A ‘Digital Literacy Framework’ was designed by Eshet-Alkalai ( 2012 ), comprising six categories: (a) photo-visual thinking (understanding and using visual information); (b) real-time thinking (simultaneously processing a variety of stimuli); (c) information thinking (evaluating and combining information from multiple digital sources); (d) branching thinking (navigating in non-linear hyper-media environments); (e) reproduction thinking (creating outcomes using technological tools by designing new content or remixing existing digital content); (f) social-emotional thinking (understanding and applying cyberspace rules). According to Heitin ( 2016 ), digital literacy groups the following clusters: (a) finding and consuming digital content; (b) creating digital content; (c) communicating or sharing digital content. Hence, the literature describes the digital literacy in many ways by associating a set of various technical and non-technical elements.

Digital competencies

The Digital Competence Framework for Citizens (DigComp 2.1.), the most recent framework proposed by the European Union, which is currently under review and undergoing an updating process, contains five competency areas: (a) information and data literacy, (b) communication and collaboration, (c) digital content creation, (d) safety, and (e) problem solving (Carretero, Vuorikari & Punie, 2017 ). Digital competency had previously been described in a technical fashion by Ferrari ( 2012 ) as a set comprising information skills, communication skills, content creation skills, safety skills, and problem-solving skills, which later outlined the areas of competence in DigComp 2.1, too.

Digital skills

Ng ( 2012 ) pointed out the following three categories of digital skills: (a) technological (using technological tools); (b) cognitive (thinking critically when managing information); (c) social (communicating and socialising). A set of Internet skill was suggested by Van Deursen and Van Dijk ( 2009 , 2010b ), which contains: (a) operational skills (basic skills in using internet technology), (b) formal Internet skills (navigation and orientation skills); (c) information Internet skills (fulfilling information needs), and (d) strategic Internet skills (using the internet to reach goals). In 2014, the same authors added communication and content creation skills to the initial framework (van Dijk & van Deursen). Similarly, Helsper and Eynon ( 2013 ) put forward a set of four digital skills: technical, social, critical, and creative skills. Furthermore, van Deursen et al. ( 2015 ) built a set of items and factors to measure Internet skills: operational, information navigation, social, creative, mobile. More recent literature (vaan Laar et al., 2017 ) divides digital skills into seven core categories: technical, information management, communication, collaboration, creativity, critical thinking, and problem solving.

It is worth mentioning that the various methodologies used to classify digital literacy are overlapping or non-exhaustive, which confirms the conceptual ambiguity mentioned by van Deursen et al. ( 2015 ).

Digital thinking

Thinking skills (along with digital skills) have been acknowledged to be a significant element of digital literacy in the educational process context (Ferrari, 2012 ). In fact, critical thinking, creativity, and innovation are at the very core of DigComp. Information and Communication Technology as a support for thinking is a learning objective in any school curriculum. In the same vein, analytical thinking and interdisciplinary thinking, which help solve problems, are yet other concerns of educators in the Industry 4.0 (Ozkan-Ozen & Kazancoglu, 2021 ).

However, we have recently witnessed a shift of focus from learning how to use information and communication technologies to using it while staying safe in the cyber-environment and being aware of alternative facts. Digital thinking would encompass identifying fake news, misinformation, and echo chambers (Sulzer, 2018 ). Not least important, concern about cybersecurity has grown especially in times of political, social or economic turmoil, such as the elections or the Covid-19 crisis (Sulzer, 2018 ; Puig, Blanco-Anaya & Perez-Maceira, 2021 ).

Ultimately, this systematic review paper focuses on the following major research questions as follows:

• Research question 1: What is the yearly distribution of digital literacy related papers?
• Research question 2: What are the research methods for digital literacy related papers?
• Research question 3: What are the main themes in digital literacy related papers?
• Research question 4: What are the concentrated categories (under revealed main themes) in digital literacy related papers?

This study employed the systematic review method where the authors scrutinized the existing literature around the major research question of digital literacy. As Uman ( 2011 ) pointed, in systematic literature review, the findings of the earlier research are examined for the identification of consistent and repetitive themes. The systematic review method differs from literature review with its well managed and highly organized qualitative scrutiny processes where researchers tend to cover less materials from fewer number of databases to write their literature review (Kowalczyk & Truluck, 2013 ; Robinson & Lowe, 2015 ).

Data collection

To address major research objectives, the following five important databases are selected due to their digital literacy focused research dominance: 1. WoS/Clarivate Analytics, 2. Proquest Central; 3. Emerald Management Journals; 4. Jstor Business College Collections; 5. Scopus/Elsevier.

The search was made in the second half of June 2021, in abstract and key words written in English language. We only kept research articles and book chapters (herein referred to as papers). Our purpose was to identify a set of digital literacy areas, or an inventory of such areas and topics. To serve that purpose, systematic review was utilized with the following synonym key words for the search: ‘digital literacy’, ‘digital skills’, ‘digital competence’ and ‘digital fluency’, to find the mainstream literature dealing with the topic. These key words were unfolded as a result of the consultation with the subject matter experts (two board members from Korean Digital Literacy Association and two professors from technology studies department). Below are the four key word combinations used in the search: “Digital literacy AND systematic review”, “Digital skills AND systematic review”, “Digital competence AND systematic review”, and “Digital fluency AND systematic review”.

A sequential systematic search was made in the five databases mentioned above. Thus, from one database to another, duplicate papers were manually excluded in a cascade manner to extract only unique results and to make the research smoother to conduct. At this stage, we kept 47 papers. Further exclusion criteria were applied. Thus, only full-text items written in English were selected, and in doing so, three papers were excluded (no full text available), and one other paper was excluded because it was not written in English, but in Spanish. Therefore, we investigated a total number of 43 papers, as shown in Table ​ Table1. 1 . “ Appendix A ” shows the list of these papers with full references.

Number of papers identified sequentially after applying all inclusion and exclusion criteria

Data analysis

The 43 papers selected after the application of the inclusion and exclusion criteria, respectively, were reviewed the materials independently by two researchers who were from two different countries. The researchers identified all topics pertaining to digital literacy, as they appeared in the papers. Next, a third researcher independently analysed these findings by excluded duplicates A qualitative content analysis was manually performed by calculating the frequency of major themes in all papers, where the raw data was compared and contrasted (Fraenkel et al., 2012 ). All three reviewers independently list the words and how the context in which they appeared and then the three reviewers collectively decided for how it should be categorized. Lastly, it is vital to remind that literature review of this article was written after the identification of the themes appeared as a result of our qualitative analyses. Therefore, the authors decided to shape the literature review structure based on the themes.

As an answer to the first research question (the yearly distribution of digital literacy related papers), Fig.  1 demonstrates the yearly distribution of digital literacy related papers. It is seen that there is an increasing trend about the digital literacy papers.

Yearly distribution of digital literacy related papers

Research question number two (The research methods for digital literacy related papers) concentrates on what research methods are employed for these digital literacy related papers. As Fig.  2 shows, most of the papers were using the qualitative method. Not stated refers to book chapters.

Research methods used in the reviewed articles

When forty-three articles were analysed for the main themes as in research question number three (The main themes in digital literacy related papers), the overall findings were categorized around four major themes: (i) literacies, (ii) competencies, (iii) skills, and (iv) thinking. Under every major theme, the categories were listed and explained as in research question number four (The concentrated categories (under revealed main themes) in digital literacy related papers).

The authors utilized an overt categorization for the depiction of these major themes. For example, when the ‘creativity’ was labelled as a skill, the authors also categorized it under the ‘skills’ theme. Similarly, when ‘creativity’ was mentioned as a competency, the authors listed it under the ‘competencies’ theme. Therefore, it is possible to recognize the same finding under different major themes.

Major theme 1: literacies

Digital literacy being the major concern of this paper was observed to be blatantly mentioned in five papers out forty-three. One of these articles described digital literacy as the human proficiencies to live, learn and work in the current digital society. In addition to these five articles, two additional papers used the same term as ‘critical digital literacy’ by describing it as a person’s or a society’s accessibility and assessment level interaction with digital technologies to utilize and/or create information. Table ​ Table2 2 summarizes the major categories under ‘Literacies’ major theme.

Categories (more than one occurrence) under 'literacies' major theme

Computer literacy, media literacy and cultural literacy were the second most common literacy (n = 5). One of the article branches computer literacy as tool (detailing with software and hardware uses) and resource (focusing on information processing capacity of a computer) literacies. Cultural literacy was emphasized as a vital element for functioning in an intercultural team on a digital project.

Disciplinary literacy (n = 4) was referring to utilizing different computer programs (n = 2) or technical gadgets (n = 2) with a specific emphasis on required cognitive, affective and psychomotor skills to be able to work in any digital context (n = 3), serving for the using (n = 2), creating and applying (n = 2) digital literacy in real life.

Data literacy, technology literacy and multiliteracy were the third frequent categories (n = 3). The ‘multiliteracy’ was referring to the innate nature of digital technologies, which have been infused into many aspects of human lives.

Last but not least, Internet literacy, mobile literacy, web literacy, new literacy, personal literacy and research literacy were discussed in forty-three article findings. Web literacy was focusing on being able to connect with people on the web (n = 2), discover the web content (especially the navigation on a hyper-textual platform), and learn web related skills through practical web experiences. Personal literacy was highlighting digital identity management. Research literacy was not only concentrating on conducting scientific research ability but also finding available scholarship online.

Twenty-four other categories are unfolded from the results sections of forty-three articles. Table ​ Table3 3 presents the list of these other literacies where the authors sorted the categories in an ascending alphabetical order without any other sorting criterion. Primarily, search, tagging, filtering and attention literacies were mainly underlining their roles in information processing. Furthermore, social-structural literacy was indicated as the recognition of the social circumstances and generation of information. Another information-related literacy was pointed as publishing literacy, which is the ability to disseminate information via different digital channels.

Other mentioned categories (n = 1)

While above listed personal literacy was referring to digital identity management, network literacy was explained as someone’s social networking ability to manage the digital relationship with other people. Additionally, participatory literacy was defined as the necessary abilities to join an online team working on online content production.

Emerging technology literacy was stipulated as an essential ability to recognize and appreciate the most recent and innovative technologies in along with smart choices related to these technologies. Additionally, the critical literacy was added as an ability to make smart judgements on the cost benefit analysis of these recent technologies.

Last of all, basic, intermediate, and advanced digital assessment literacies were specified for educational institutions that are planning to integrate various digital tools to conduct instructional assessments in their bodies.

Major theme 2: competencies

The second major theme was revealed as competencies. The authors directly categorized the findings that are specified with the word of competency. Table ​ Table4 4 summarizes the entire category set for the competencies major theme.

Categories under 'competencies' major theme

The most common category was the ‘digital competence’ (n = 14) where one of the articles points to that category as ‘generic digital competence’ referring to someone’s creativity for multimedia development (video editing was emphasized). Under this broad category, the following sub-categories were associated:

• Problem solving (n = 10)
• Safety (n = 7)
• Information processing (n = 5)
• Content creation (n = 5)
• Communication (n = 2)
• Digital rights (n = 1)
• Digital emotional intelligence (n = 1)
• Digital teamwork (n = 1)
• Big data utilization (n = 1)
• Artificial Intelligence utilization (n = 1)
• Virtual leadership (n = 1)
• Self-disruption (in along with the pace of digitalization) (n = 1)

Like ‘digital competency’, five additional articles especially coined the term as ‘digital competence as a life skill’. Deeper analysis demonstrated the following points: social competences (n = 4), communication in mother tongue (n = 3) and foreign language (n = 2), entrepreneurship (n = 3), civic competence (n = 2), fundamental science (n = 1), technology (n = 1) and mathematics (n = 1) competences, learning to learn (n = 1) and self-initiative (n = 1).

Moreover, competencies were linked to workplace digital competencies in three articles and highlighted as significant for employability (n = 3) and ‘economic engagement’ (n = 3). Digital competencies were also detailed for leisure (n = 2) and communication (n = 2). Furthermore, two articles pointed digital competencies as an inter-cultural competency and one as a cross-cultural competency. Lastly, the ‘digital nativity’ (n = 1) was clarified as someone’s innate competency of being able to feel contented and satisfied with digital technologies.

Major theme 3: skills

The third major observed theme was ‘skills’, which was dominantly gathered around information literacy skills (n = 19) and information and communication technologies skills (n = 18). Table ​ Table5 5 demonstrates the categories with more than one occurrence.

Categories under 'skills' major theme

Table ​ Table6 6 summarizes the sub-categories of the two most frequent categories of ‘skills’ major theme. The information literacy skills noticeably concentrate on the steps of information processing; evaluation (n = 6), utilization (n = 4), finding (n = 3), locating (n = 2) information. Moreover, the importance of trial/error process, being a lifelong learner, feeling a need for information and so forth were evidently listed under this sub-category. On the other hand, ICT skills were grouped around cognitive and affective domains. For instance, while technical skills in general and use of social media, coding, multimedia, chat or emailing in specific were reported in cognitive domain, attitude, intention, and belief towards ICT were mentioned as the elements of affective domain.

Sub-categories under ‘information literacy’ and ‘ICT’ skills

Communication skills (n = 9) were multi-dimensional for different societies, cultures, and globalized contexts, requiring linguistic skills. Collaboration skills (n = 9) are also recurrently cited with an explicit emphasis for virtual platforms.

‘Ethics for digital environment’ encapsulated ethical use of information (n = 4) and different technologies (n = 2), knowing digital laws (n = 2) and responsibilities (n = 2) in along with digital rights and obligations (n = 1), having digital awareness (n = 1), following digital etiquettes (n = 1), treating other people with respect (n = 1) including no cyber-bullying (n = 1) and no stealing or damaging other people (n = 1).

‘Digital fluency’ involved digital access (n = 2) by using different software and hardware (n = 2) in online platforms (n = 1) or communication tools (n = 1) or within programming environments (n = 1). Digital fluency also underlined following recent technological advancements (n = 1) and knowledge (n = 1) including digital health and wellness (n = 1) dimension.

‘Social intelligence’ related to understanding digital culture (n = 1), the concept of digital exclusion (n = 1) and digital divide (n = 3). ‘Research skills’ were detailed with searching academic information (n = 3) on databases such as Web of Science and Scopus (n = 2) and their citation, summarization, and quotation (n = 2).

‘Digital teaching’ was described as a skill (n = 2) in Table ​ Table4 4 whereas it was also labelled as a competence (n = 1) as shown in Table ​ Table3. 3 . Similarly, while learning to learn (n = 1) was coined under competencies in Table ​ Table3, 3 , digital learning (n = 2, Table ​ Table4) 4 ) and life-long learning (n = 1, Table ​ Table5) 5 ) were stated as learning related skills. Moreover, learning was used with the following three terms: learning readiness (n = 1), self-paced learning (n = 1) and learning flexibility (n = 1).

Table ​ Table7 7 shows other categories listed below the ‘skills’ major theme. The list covers not only the software such as GIS, text mining, mapping, or bibliometric analysis programs but also the conceptual skills such as the fourth industrial revolution and information management.

Categories (one-time occurrence) under 'skills' major theme

Major theme 4: thinking

The last identified major theme was the different types of ‘thinking’. As Table ​ Table8 8 shows, ‘critical thinking’ was the most frequent thinking category (n = 4). Except computational thinking, the other categories were not detailed.

Categories under ‘thinking’ major theme

Computational thinking (n = 3) was associated with the general logic of how a computer works and sub-categorized into the following steps; construction of the problem (n = 3), abstraction (n = 1), disintegration of the problem (n = 2), data collection, (n = 2), data analysis (n = 2), algorithmic design (n = 2), parallelization & iteration (n = 1), automation (n = 1), generalization (n = 1), and evaluation (n = 2).

A transversal analysis of digital literacy categories reveals the following fields of digital literacy application:

• Technological advancement (IT, ICT, Industry 4.0, IoT, text mining, GIS, bibliometric analysis, mapping data, technology, AI, big data)
• Networking (Internet, web, connectivity, network, safety)
• Information (media, news, communication)
• Creative-cultural industries (culture, publishing, film, TV, leisure, content creation)
• Academia (research, documentation, library)
• Citizenship (participation, society, social intelligence, awareness, politics, rights, legal use, ethics)
• Education (life skills, problem solving, teaching, learning, education, lifelong learning)
• Professional life (work, teamwork, collaboration, economy, commerce, leadership, decision making)
• Personal level (critical thinking, evaluation, analytical thinking, innovative thinking)

This systematic review on digital literacy concentrated on forty-three articles from the databases of WoS/Clarivate Analytics, Proquest Central, Emerald Management Journals, Jstor Business College Collections and Scopus/Elsevier. The initial results revealed that there is an increasing trend on digital literacy focused academic papers. Research work in digital literacy is critical in a context of disruptive digital business, and more recently, the pandemic-triggered accelerated digitalisation (Beaunoyer, Dupéré & Guitton, 2020 ; Sousa & Rocha 2019 ). Moreover, most of these papers were employing qualitative research methods. The raw data of these articles were analysed qualitatively using systematic literature review to reveal major themes and categories. Four major themes that appeared are: digital literacy, digital competencies, digital skills and thinking.

Whereas the mainstream literature describes digital literacy as a set of photo-visual, real-time, information, branching, reproduction and social-emotional thinking (Eshet-Alkalai, 2012 ) or as a set of precise specific operations, i.e., finding, consuming, creating, communicating and sharing digital content (Heitin, 2016 ), this study reveals that digital literacy revolves around and is in connection with the concepts of computer literacy, media literacy, cultural literacy or disciplinary literacy. In other words, the present systematic review indicates that digital literacy is far broader than specific tasks, englobing the entire sphere of computer operation and media use in a cultural context.

The digital competence yardstick, DigComp (Carretero, Vuorikari & Punie, 2017 ) suggests that the main digital competencies cover information and data literacy, communication and collaboration, digital content creation, safety, and problem solving. Similarly, the findings of this research place digital competencies in relation to problem solving, safety, information processing, content creation and communication. Therefore, the findings of the systematic literature review are, to a large extent, in line with the existing framework used in the European Union.

The investigation of the main keywords associated with digital skills has revealed that information literacy, ICT, communication, collaboration, digital content creation, research and decision-making skill are the most representative. In a structured way, the existing literature groups these skills in technological, cognitive, and social (Ng, 2012 ) or, more extensively, into operational, formal, information Internet, strategic, communication and content creation (van Dijk & van Deursen, 2014 ). In time, the literature has become richer in frameworks, and prolific authors have improved their results. As such, more recent research (vaan Laar et al., 2017 ) use the following categories: technical, information management, communication, collaboration, creativity, critical thinking, and problem solving.

Whereas digital thinking was observed to be mostly related with critical thinking and computational thinking, DigComp connects it with critical thinking, creativity, and innovation, on the one hand, and researchers highlight fake news, misinformation, cybersecurity, and echo chambers as exponents of digital thinking, on the other hand (Sulzer, 2018 ; Puig, Blanco-Anaya & Perez-Maceira, 2021 ).

This systematic review research study looks ahead to offer an initial step and guideline for the development of a more contemporary digital literacy framework including digital literacy major themes and factors. The researchers provide the following recommendations for both researchers and practitioners.

Recommendations for prospective research

By considering the major qualitative research trend, it seems apparent that more quantitative research-oriented studies are needed. Although it requires more effort and time, mixed method studies will help understand digital literacy holistically.

As digital literacy is an umbrella term for many different technologies, specific case studies need be designed, such as digital literacy for artificial intelligence or digital literacy for drones’ usage.

Digital literacy affects different areas of human lives, such as education, business, health, governance, and so forth. Therefore, different case studies could be carried out for each of these unique dimensions of our lives. For instance, it is worth investigating the role of digital literacy on lifelong learning in particular, and on education in general, as well as the digital upskilling effects on the labour market flexibility.

Further experimental studies on digital literacy are necessary to realize how certain variables (for instance, age, gender, socioeconomic status, cognitive abilities, etc.) affect this concept overtly or covertly. Moreover, the digital divide issue needs to be analysed through the lens of its main determinants.

New bibliometric analysis method can be implemented on digital literacy documents to reveal more information on how these works are related or centred on what major topic. This visual approach will assist to realize the big picture within the digital literacy framework.

Recommendations for practitioners

The digital literacy stakeholders, policymakers in education and managers in private organizations, need to be aware that there are many dimensions and variables regarding the implementation of digital literacy. In that case, stakeholders must comprehend their beneficiaries or the participants more deeply to increase the effect of digital literacy related activities. For example, critical thinking and problem-solving skills and abilities are mentioned to affect digital literacy. Hence, stakeholders have to initially understand whether the participants have enough entry level critical thinking and problem solving.

Development of digital literacy for different groups of people requires more energy, since each group might require a different set of skills, abilities, or competencies. Hence, different subject matter experts, such as technologists, instructional designers, content experts, should join the team.

It is indispensably vital to develop different digital frameworks for different technologies (basic or advanced) or different contexts (different levels of schooling or various industries).

These frameworks should be updated regularly as digital fields are evolving rapidly. Every year, committees should gather around to understand new technological trends and decide whether they should address the changes into their frameworks.

Understanding digital literacy in a thorough manner can enable decision makers to correctly implement and apply policies addressing the digital divide that is reflected onto various aspects of life, e.g., health, employment, education, especially in turbulent times such as the COVID-19 pandemic is.

Lastly, it is also essential to state the study limitations. This study is limited to the analysis of a certain number of papers, obtained from using the selected keywords and databases. Therefore, an extension can be made by adding other keywords and searching other databases.

See Table ​ Management9 9 .

List of papers (n = 43) included in the qualitative analysis—ordered alphabetically by title

Author contributions

The authors worked together on the manuscript equally. All authors have read and approved the final manuscript.

This research is funded by Woosong University Academic Research in 2022.

Availability of data and materials

Declarations.

The authors declare that they have no competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Hasan Tinmaz, Email: rk.ca.ttocidne@zamnith .

Yoo-Taek Lee, Email: rk.ca.usw@eelty .

Mina Fanea-Ivanovici, Email: [email protected] .

Hasnan Baber, Email: [email protected] .

• Baber H, Fanea-Ivanovici M, Lee YT, Tinmaz H. A bibliometric analysis of digital literacy research and emerging themes pre-during COVID-19 pandemic. Information and Learning Sciences. 2022 doi: 10.1108/ILS-10-2021-0090. [ CrossRef ] [ Google Scholar ]
• Beaunoyer E, Dupéré S, Guitton MJ. COVID-19 and digital inequalities: Reciprocal impacts and mitigation strategies. Computers in Human Behavior. 2020; 111 :10642. doi: 10.1016/j.chb.2020.106424. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Blau I, Shamir-Inbal T, Avdiel O. How does the pedagogical design of a technology-enhanced collaborative academic course promote digital literacies, self-regulation, and perceived learning of students? The Internet and Higher Education. 2020; 45 :100722. doi: 10.1016/j.iheduc.2019.100722. [ CrossRef ] [ Google Scholar ]
• Carretero, S., Vuorikari, R., & Punie, Y. (2017). DigComp 2.1: The Digital Competence Framework for Citizens with eight proficiency levels and examples of use (No. JRC106281). Joint Research Centre, https://publications.jrc.ec.europa.eu/repository/handle/JRC106281
• Eshet, Y. (2004). Digital literacy: A conceptual framework for survival skills in the digital era. Journal of Educational Multimedia and Hypermedia , 13 (1), 93–106, https://www.learntechlib.org/primary/p/4793/
• Eshet-Alkalai Y. Thinking in the digital era: A revised model for digital literacy. Issues in Informing Science and Information Technology. 2012; 9 (2):267–276. doi: 10.28945/1621. [ CrossRef ] [ Google Scholar ]
• Ferrari, A. (2012). Digital competence in practice: An analysis of frameworks. JCR IPTS, Sevilla. https://ifap.ru/library/book522.pdf
• Fraenkel JR, Wallen NE, Hyun HH. How to design and evaluate research in education. 8. Mc Graw Hill; 2012. [ Google Scholar ]
• Heitin, L. (2016). What is digital literacy? Education Week, https://www.edweek.org/teaching-learning/what-is-digital-literacy/2016/11
• Helsper EJ, Eynon R. Distinct skill pathways to digital engagement. European Journal of Communication. 2013; 28 (6):696–713. doi: 10.1177/0267323113499113. [ CrossRef ] [ Google Scholar ]
• Kowalczyk N, Truluck C. Literature reviews and systematic reviews: What is the difference ? Radiologic Technology. 2013; 85 (2):219–222. [ PubMed ] [ Google Scholar ]
• Ng W. Can we teach digital natives digital literacy? Computers & Education. 2012; 59 (3):1065–1078. doi: 10.1016/j.compedu.2012.04.016. [ CrossRef ] [ Google Scholar ]
• Ozkan-Ozen YD, Kazancoglu Y. Analysing workforce development challenges in the Industry 4.0. International Journal of Manpower. 2021 doi: 10.1108/IJM-03-2021-0167. [ CrossRef ] [ Google Scholar ]
• Puig B, Blanco-Anaya P, Perez-Maceira JJ. “Fake News” or Real Science? Critical thinking to assess information on COVID-19. Frontiers in Education. 2021; 6 :646909. doi: 10.3389/feduc.2021.646909. [ CrossRef ] [ Google Scholar ]
• Robinson L, Cotten SR, Ono H, Quan-Haase A, Mesch G, Chen W, Schulz J, Hale TM, Stern MJ. Digital inequalities and why they matter. Information, Communication & Society. 2015; 18 (5):569–582. doi: 10.1080/1369118X.2015.1012532. [ CrossRef ] [ Google Scholar ]
• Robinson P, Lowe J. Literature reviews vs systematic reviews. Australian and New Zealand Journal of Public Health. 2015; 39 (2):103. doi: 10.1111/1753-6405.12393. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Sousa MJ, Rocha A. Skills for disruptive digital business. Journal of Business Research. 2019; 94 :257–263. doi: 10.1016/j.jbusres.2017.12.051. [ CrossRef ] [ Google Scholar ]
• Sulzer A. (Re)conceptualizing digital literacies before and after the election of Trump. English Teaching: Practice & Critique. 2018; 17 (2):58–71. doi: 10.1108/ETPC-06-2017-0098. [ CrossRef ] [ Google Scholar ]
• Tinmaz, H., Fanea-Ivanovici, M., & Baber, H. (2022). A snapshot of digital literacy. Library Hi Tech News , (ahead-of-print).
• Uman LS. Systematic reviews and meta-analyses. Journal of the Canadian Academy of Child and Adolescent Psychiatry. 2011; 20 (1):57–59. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Van Deursen AJAM, Helsper EJ, Eynon R. Development and validation of the Internet Skills Scale (ISS) Information, Communication & Society. 2015; 19 (6):804–823. doi: 10.1080/1369118X.2015.1078834. [ CrossRef ] [ Google Scholar ]
• Van Deursen AJAM, van Dijk JAGM. Using the internet: Skills related problems in users’ online behaviour. Interacting with Computers. 2009; 21 :393–402. doi: 10.1016/j.intcom.2009.06.005. [ CrossRef ] [ Google Scholar ]
• Van Deursen AJAM, van Dijk JAGM. Measuring internet skills. International Journal of Human-Computer Interaction. 2010; 26 (10):891–916. doi: 10.1080/10447318.2010.496338. [ CrossRef ] [ Google Scholar ]
• Van Deursen AJAM, van Dijk JAGM. Internet skills and the digital divide. New Media & Society. 2010; 13 (6):893–911. doi: 10.1177/1461444810386774. [ CrossRef ] [ Google Scholar ]
• van Dijk JAGM, Van Deursen AJAM. Digital skills, unlocking the information society. Palgrave MacMillan; 2014. [ Google Scholar ]
• van Laar E, van Deursen AJAM, van Dijk JAGM, de Haan J. The relation between 21st-century skills and digital skills: A systematic literature review. Computer in Human Behavior. 2017; 72 :577–588. doi: 10.1016/j.chb.2017.03.010. [ CrossRef ] [ Google Scholar ]

Now Published: Systematic literature review on religious leader well-being, burnout, and trauma

Our second publication from the Helping the Helpers project is a systematic literature review of 82 empirical studies that look at burnout, trauma impacts, and/or well-being among religious leaders. We were able to highlight relational, systemic/organizational, and diversity issues that are crucial for gaining a more holistic understanding of these issues. The citation and abstract are below.

Hydinger, K. R., Wu, X., Captari, L. E., & Sandage, S. (2024). Burnout, Trauma Impacts, and Well-Being Among Clergy and Chaplains: A Systematic Review and Recommendations to Guide Best Practice. Pastoral Psychology . Advanced online publication. https://doi.org/10.1007/s11089-024-01150-x

Religious leaders (i.e., clergy and chaplains) face unique, ongoing stressors that can increase risks for psychosocial and vocational vulnerabilities. Emerging evidence indicates concerning prevalence rates of distress and attrition among these professionals, particularly since the COVID-19 pandemic. To date, most empirical work has focused on compromised functioning among religious leaders. Utilizing a more holistic approach, this systematic review explores individual, relational, and organizational factors associated with diverse outcomes. Following the PRISMA methodology, we identified 82 empirical articles investigating (a) risk and protective factors related to burnout, trauma impacts, spiritual distress, and other occupational hazards and/or (b) factors associated with well-being and flourishing, over and above distress reduction. We summarize the state of the available evidence, distinguishing between  risk increasers ,  protective factors , and  well-being enhancers . Attention is given to three domains:  individual  (e.g., demographics, personality factors, virtue development, coping and formation practices),  relational  (e.g., peer, family, and collegial supports; navigation of conflicts and polarized issues in one’s community of care), and  institutional  (e.g., role ambiguity or clarity, resource availability, systemic expectations and demands). We identify notable gaps to be addressed in future research; for example, most studies are cross-sectional, lack diversity in religion, gender, and geography, and operationalize well-being as the absence of symptoms rather than the presence of positive states and functioning. Considering the available evidence, we present best practices to guide psychological practitioners, denominational bodies, and others involved in religious leaders’ formation.

• Introduction
• Conclusions
• Article Information

The reports to the Vaccine Adverse Event Reporting System met the case definition of myocarditis (reported cases). Among individuals older than 40 years of age, there were no more than 8 reports of myocarditis for any individual age after receiving either vaccine. For the BNT162b2 vaccine, there were 114 246 837 first vaccination doses and 95 532 396 second vaccination doses; and for the mRNA-1273 vaccine, there were 78 158 611 and 66 163 001, respectively. The y-axis range differs between panels A and B.

The reports to the Vaccine Adverse Event Reporting System met the case definition of myocarditis (reported cases). Among recipients of either vaccine, there were only 13 reports or less of myocarditis beyond 10 days for any individual time from vaccination to symptom onset. The y-axis range differs between panels A and B.

A, For the BNT162b2 vaccine, there were 138 reported cases of myocarditis with known date for symptom onset and dose after 114 246 837 first vaccination doses and 888 reported cases after 95 532 396 second vaccination doses.

B, For the mRNA-1273 vaccine, there were 116 reported cases of myocarditis with known date for symptom onset and dose after 78 158 611 first vaccination doses and 311 reported cases after 66 163 001 second vaccination doses.

eMethods. Medical Dictionary for Regulatory Activities Preferred Terms, Definitions of Myocarditis and Pericarditis, Myocarditis medical review form

eFigure. Flow diagram of cases of myocarditis and pericarditis reported to Vaccine Adverse Event Reporting System (VAERS) after receiving mRNA-based COVID-19 vaccine, United States, December 14, 2020-August 31, 2021.

eTable 1. Characteristics of all myocarditis cases reported to Vaccine Adverse Event Reporting System (VAERS) after mRNA-based COVID-19 vaccination, United States, December 14, 2020–August 31, 2021.

eTable 2. Characteristics of all pericarditis cases reported to Vaccine Adverse Event Reporting System (VAERS) after mRNA-based COVID-19 vaccination, United States, December 14, 2020–August 31, 2021.

eTable 3. Characteristics of myocarditis cases reported to Vaccine Adverse Event Reporting System after mRNA-based COVID-19 vaccination by case definition status.

• Myocarditis and Pericarditis After Vaccination for COVID-19 JAMA Research Letter September 28, 2021 This study investigates the incidence of myocarditis and pericarditis emergency department or inpatient hospital encounters before COVID-19 vaccine availability (January 2019–January 2021) and during a COVID-19 vaccination period (February-May 2021) in a large US health care system. George A. Diaz, MD; Guilford T. Parsons, MD, MS; Sara K. Gering, BS, BSN; Audrey R. Meier, MPH; Ian V. Hutchinson, PhD, DSc; Ari Robicsek, MD
• Myocarditis Following a Third BNT162b2 Vaccination Dose in Military Recruits in Israel JAMA Research Letter April 26, 2022 This study assessed whether a third vaccine dose was associated with the risk of myocarditis among military personnel in Israel. Limor Friedensohn, MD; Dan Levin, MD; Maggie Fadlon-Derai, MHA; Liron Gershovitz, MD; Noam Fink, MD; Elon Glassberg, MD; Barak Gordon, MD
• Myocarditis Cases After mRNA-Based COVID-19 Vaccination in the US—Reply JAMA Comment & Response May 24, 2022 Matthew E. Oster, MD, MPH; David K. Shay, MD, MPH; Tom T. Shimabukuro, MD, MPH, MBA
• Myocarditis Cases After mRNA-Based COVID-19 Vaccination in the US JAMA Comment & Response May 24, 2022 Sheila R. Weiss, PhD
• JAMA Network Articles of the Year 2022 JAMA Medical News & Perspectives December 27, 2022 This Medical News article is our annual roundup of the top-viewed articles from all JAMA Network journals. Melissa Suran, PhD, MSJ
• Diagnosis and Treatment of Acute Myocarditis—A Review JAMA Review April 4, 2023 This Review summarizes current evidence regarding the diagnosis and treatment of acute myocarditis. Enrico Ammirati, MD, PhD; Javid J. Moslehi, MD
• Patient Information: Acute Myocarditis JAMA JAMA Patient Page August 8, 2023 This JAMA Patient Page describes acute myocarditis and its symptoms, causes, diagnosis, and treatment. Kristin Walter, MD, MS
• Myocarditis Following Immunization With mRNA COVID-19 Vaccines in Members of the US Military JAMA Cardiology Brief Report October 1, 2021 This case series describes myocarditis presenting after COVID-19 vaccination within the Military Health System. Jay Montgomery, MD; Margaret Ryan, MD, MPH; Renata Engler, MD; Donna Hoffman, MSN; Bruce McClenathan, MD; Limone Collins, MD; David Loran, DNP; David Hrncir, MD; Kelsie Herring, MD; Michael Platzer, MD; Nehkonti Adams, MD; Aliye Sanou, MD; Leslie T. Cooper Jr, MD
• Patients With Acute Myocarditis Following mRNA COVID-19 Vaccination JAMA Cardiology Brief Report October 1, 2021 This study describes 4 patients who presented with acute myocarditis after mRNA COVID-19 vaccination. Han W. Kim, MD; Elizabeth R. Jenista, PhD; David C. Wendell, PhD; Clerio F. Azevedo, MD; Michael J. Campbell, MD; Stephen N. Darty, BS; Michele A. Parker, MS; Raymond J. Kim, MD
• Association of Myocarditis With BNT162b2 Vaccination in Children JAMA Cardiology Brief Report December 1, 2021 This case series reviews comprehensive cardiac imaging in children with myocarditis after COVID-19 vaccine. Audrey Dionne, MD; Francesca Sperotto, MD; Stephanie Chamberlain; Annette L. Baker, MSN, CPNP; Andrew J. Powell, MD; Ashwin Prakash, MD; Daniel A. Castellanos, MD; Susan F. Saleeb, MD; Sarah D. de Ferranti, MD, MPH; Jane W. Newburger, MD, MPH; Kevin G. Friedman, MD

See More About

Select your interests.

Customize your JAMA Network experience by selecting one or more topics from the list below.

• Academic Medicine
• Acid Base, Electrolytes, Fluids
• Allergy and Clinical Immunology
• American Indian or Alaska Natives
• Anesthesiology
• Anticoagulation
• Art and Images in Psychiatry
• Artificial Intelligence
• Assisted Reproduction
• Bleeding and Transfusion
• Caring for the Critically Ill Patient
• Challenges in Clinical Electrocardiography
• Climate and Health
• Climate Change
• Clinical Challenge
• Clinical Decision Support
• Clinical Implications of Basic Neuroscience
• Clinical Pharmacy and Pharmacology
• Complementary and Alternative Medicine
• Consensus Statements
• Coronavirus (COVID-19)
• Critical Care Medicine
• Cultural Competency
• Dental Medicine
• Dermatology
• Diabetes and Endocrinology
• Diagnostic Test Interpretation
• Drug Development
• Electronic Health Records
• Emergency Medicine
• End of Life, Hospice, Palliative Care
• Environmental Health
• Equity, Diversity, and Inclusion
• Facial Plastic Surgery
• Gastroenterology and Hepatology
• Genetics and Genomics
• Genomics and Precision Health
• Global Health
• Guide to Statistics and Methods
• Hair Disorders
• Health Care Delivery Models
• Health Care Economics, Insurance, Payment
• Health Care Quality
• Health Care Reform
• Health Care Safety
• Health Care Workforce
• Health Disparities
• Health Inequities
• Health Policy
• Health Systems Science
• History of Medicine
• Hypertension
• Images in Neurology
• Implementation Science
• Infectious Diseases
• Innovations in Health Care Delivery
• JAMA Infographic
• Law and Medicine
• Leading Change
• Less is More
• LGBTQIA Medicine
• Lifestyle Behaviors
• Medical Coding
• Medical Devices and Equipment
• Medical Education
• Medical Education and Training
• Medical Journals and Publishing
• Mobile Health and Telemedicine
• Narrative Medicine
• Neuroscience and Psychiatry
• Notable Notes
• Nutrition, Obesity, Exercise
• Obstetrics and Gynecology
• Occupational Health
• Ophthalmology
• Orthopedics
• Otolaryngology
• Pain Medicine
• Palliative Care
• Pathology and Laboratory Medicine
• Patient Care
• Patient Information
• Performance Improvement
• Performance Measures
• Perioperative Care and Consultation
• Pharmacoeconomics
• Pharmacoepidemiology
• Pharmacogenetics
• Pharmacy and Clinical Pharmacology
• Physical Medicine and Rehabilitation
• Physical Therapy
• Physician Leadership
• Population Health
• Primary Care
• Professional Well-being
• Professionalism
• Psychiatry and Behavioral Health
• Public Health
• Pulmonary Medicine
• Regulatory Agencies
• Reproductive Health
• Research, Methods, Statistics
• Resuscitation
• Rheumatology
• Risk Management
• Scientific Discovery and the Future of Medicine
• Shared Decision Making and Communication
• Sleep Medicine
• Sports Medicine
• Stem Cell Transplantation
• Substance Use and Addiction Medicine
• Surgical Innovation
• Surgical Pearls
• Teachable Moment
• Technology and Finance
• The Art of JAMA
• The Arts and Medicine
• The Rational Clinical Examination
• Tobacco and e-Cigarettes
• Translational Medicine
• Trauma and Injury
• Treatment Adherence
• Ultrasonography
• Users' Guide to the Medical Literature
• Vaccination
• Venous Thromboembolism
• Veterans Health
• Women's Health
• Workflow and Process
• Wound Care, Infection, Healing

Others Also Liked

• Download PDF
• X Facebook More LinkedIn
• CME & MOC

Oster ME , Shay DK , Su JR, et al. Myocarditis Cases Reported After mRNA-Based COVID-19 Vaccination in the US From December 2020 to August 2021. JAMA. 2022;327(4):331–340. doi:10.1001/jama.2021.24110

Manage citations:

© 2024

• Permissions

Myocarditis Cases Reported After mRNA-Based COVID-19 Vaccination in the US From December 2020 to August 2021

• 1 US Centers for Disease Control and Prevention, Atlanta, Georgia
• 2 School of Medicine, Emory University, Atlanta, Georgia
• 3 Children’s Healthcare of Atlanta, Atlanta, Georgia
• 4 Vanderbilt University Medical Center, Nashville, Tennessee
• 5 Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio
• 6 Boston Medical Center, Boston, Massachusetts
• 7 Duke University, Durham, North Carolina
• 8 US Food and Drug Administration, Silver Spring, Maryland
• Research Letter Myocarditis and Pericarditis After Vaccination for COVID-19 George A. Diaz, MD; Guilford T. Parsons, MD, MS; Sara K. Gering, BS, BSN; Audrey R. Meier, MPH; Ian V. Hutchinson, PhD, DSc; Ari Robicsek, MD JAMA
• Research Letter Myocarditis Following a Third BNT162b2 Vaccination Dose in Military Recruits in Israel Limor Friedensohn, MD; Dan Levin, MD; Maggie Fadlon-Derai, MHA; Liron Gershovitz, MD; Noam Fink, MD; Elon Glassberg, MD; Barak Gordon, MD JAMA
• Comment & Response Myocarditis Cases After mRNA-Based COVID-19 Vaccination in the US—Reply Matthew E. Oster, MD, MPH; David K. Shay, MD, MPH; Tom T. Shimabukuro, MD, MPH, MBA JAMA
• Comment & Response Myocarditis Cases After mRNA-Based COVID-19 Vaccination in the US Sheila R. Weiss, PhD JAMA
• Medical News & Perspectives JAMA Network Articles of the Year 2022 Melissa Suran, PhD, MSJ JAMA
• Review Diagnosis and Treatment of Acute Myocarditis—A Review Enrico Ammirati, MD, PhD; Javid J. Moslehi, MD JAMA
• JAMA Patient Page Patient Information: Acute Myocarditis Kristin Walter, MD, MS JAMA
• Brief Report Myocarditis Following Immunization With mRNA COVID-19 Vaccines in Members of the US Military Jay Montgomery, MD; Margaret Ryan, MD, MPH; Renata Engler, MD; Donna Hoffman, MSN; Bruce McClenathan, MD; Limone Collins, MD; David Loran, DNP; David Hrncir, MD; Kelsie Herring, MD; Michael Platzer, MD; Nehkonti Adams, MD; Aliye Sanou, MD; Leslie T. Cooper Jr, MD JAMA Cardiology
• Brief Report Patients With Acute Myocarditis Following mRNA COVID-19 Vaccination Han W. Kim, MD; Elizabeth R. Jenista, PhD; David C. Wendell, PhD; Clerio F. Azevedo, MD; Michael J. Campbell, MD; Stephen N. Darty, BS; Michele A. Parker, MS; Raymond J. Kim, MD JAMA Cardiology
• Brief Report Association of Myocarditis With BNT162b2 Vaccination in Children Audrey Dionne, MD; Francesca Sperotto, MD; Stephanie Chamberlain; Annette L. Baker, MSN, CPNP; Andrew J. Powell, MD; Ashwin Prakash, MD; Daniel A. Castellanos, MD; Susan F. Saleeb, MD; Sarah D. de Ferranti, MD, MPH; Jane W. Newburger, MD, MPH; Kevin G. Friedman, MD JAMA Cardiology

Question   What is the risk of myocarditis after mRNA-based COVID-19 vaccination in the US?

Findings   In this descriptive study of 1626 cases of myocarditis in a national passive reporting system, the crude reporting rates within 7 days after vaccination exceeded the expected rates across multiple age and sex strata. The rates of myocarditis cases were highest after the second vaccination dose in adolescent males aged 12 to 15 years (70.7 per million doses of the BNT162b2 vaccine), in adolescent males aged 16 to 17 years (105.9 per million doses of the BNT162b2 vaccine), and in young men aged 18 to 24 years (52.4 and 56.3 per million doses of the BNT162b2 vaccine and the mRNA-1273 vaccine, respectively).

Meaning   Based on passive surveillance reporting in the US, the risk of myocarditis after receiving mRNA-based COVID-19 vaccines was increased across multiple age and sex strata and was highest after the second vaccination dose in adolescent males and young men.

Importance   Vaccination against COVID-19 provides clear public health benefits, but vaccination also carries potential risks. The risks and outcomes of myocarditis after COVID-19 vaccination are unclear.

Objective   To describe reports of myocarditis and the reporting rates after mRNA-based COVID-19 vaccination in the US.

Design, Setting, and Participants   Descriptive study of reports of myocarditis to the Vaccine Adverse Event Reporting System (VAERS) that occurred after mRNA-based COVID-19 vaccine administration between December 2020 and August 2021 in 192 405 448 individuals older than 12 years of age in the US; data were processed by VAERS as of September 30, 2021.

Exposures   Vaccination with BNT162b2 (Pfizer-BioNTech) or mRNA-1273 (Moderna).

Main Outcomes and Measures   Reports of myocarditis to VAERS were adjudicated and summarized for all age groups. Crude reporting rates were calculated across age and sex strata. Expected rates of myocarditis by age and sex were calculated using 2017-2019 claims data. For persons younger than 30 years of age, medical record reviews and clinician interviews were conducted to describe clinical presentation, diagnostic test results, treatment, and early outcomes.

Results   Among 192 405 448 persons receiving a total of 354 100 845 mRNA-based COVID-19 vaccines during the study period, there were 1991 reports of myocarditis to VAERS and 1626 of these reports met the case definition of myocarditis. Of those with myocarditis, the median age was 21 years (IQR, 16-31 years) and the median time to symptom onset was 2 days (IQR, 1-3 days). Males comprised 82% of the myocarditis cases for whom sex was reported. The crude reporting rates for cases of myocarditis within 7 days after COVID-19 vaccination exceeded the expected rates of myocarditis across multiple age and sex strata. The rates of myocarditis were highest after the second vaccination dose in adolescent males aged 12 to 15 years (70.7 per million doses of the BNT162b2 vaccine), in adolescent males aged 16 to 17 years (105.9 per million doses of the BNT162b2 vaccine), and in young men aged 18 to 24 years (52.4 and 56.3 per million doses of the BNT162b2 vaccine and the mRNA-1273 vaccine, respectively). There were 826 cases of myocarditis among those younger than 30 years of age who had detailed clinical information available; of these cases, 792 of 809 (98%) had elevated troponin levels, 569 of 794 (72%) had abnormal electrocardiogram results, and 223 of 312 (72%) had abnormal cardiac magnetic resonance imaging results. Approximately 96% of persons (784/813) were hospitalized and 87% (577/661) of these had resolution of presenting symptoms by hospital discharge. The most common treatment was nonsteroidal anti-inflammatory drugs (589/676; 87%).

Conclusions and Relevance   Based on passive surveillance reporting in the US, the risk of myocarditis after receiving mRNA-based COVID-19 vaccines was increased across multiple age and sex strata and was highest after the second vaccination dose in adolescent males and young men. This risk should be considered in the context of the benefits of COVID-19 vaccination.

Myocarditis is an inflammatory condition of the heart muscle that has a bimodal peak incidence during infancy and adolescence or young adulthood. 1 - 4 The clinical presentation and course of myocarditis is variable, with some patients not requiring treatment and others experiencing severe heart failure that requires subsequent heart transplantation or leads to death. 5 Onset of myocarditis typically follows an inciting process, often a viral illness; however, no antecedent cause is identified in many cases. 6 It has been hypothesized that vaccination can serve as a trigger for myocarditis; however, only the smallpox vaccine has previously been causally associated with myocarditis based on reports among US military personnel, with cases typically occurring 7 to 12 days after vaccination. 7

With the implementation of a large-scale, national COVID-19 vaccination program starting in December 2020, the US Centers for Disease Control and Prevention (CDC) and the US Food and Drug Administration began monitoring for a number of adverse events of special interest, including myocarditis and pericarditis, in the Vaccine Adverse Event Reporting System (VAERS), a long-standing national spontaneous reporting (passive surveillance) system. 8 As the reports of myocarditis after COVID-19 vaccination were reported to VAERS, the Clinical Immunization Safety Assessment Project, 9 a collaboration between the CDC and medical research centers, which includes physicians treating infectious diseases and other specialists (eg, cardiologists), consulted on several of the cases. In addition, reports from several countries raised concerns that mRNA-based COVID-19 vaccines may be associated with acute myocarditis. 10 - 15

Given this concern, the aims were to describe reports and confirmed cases of myocarditis initially reported to VAERS after mRNA-based COVID-19 vaccination and to provide estimates of the risk of myocarditis after mRNA-based COVID-19 vaccination based on age, sex, and vaccine type.

VAERS is a US spontaneous reporting (passive surveillance) system that functions as an early warning system for potential vaccine adverse events. 8 Co-administered by the CDC and the US Food and Drug Administration, VAERS accepts reports of all adverse events after vaccination from patients, parents, clinicians, vaccine manufacturers, and others regardless of whether the events could plausibly be associated with receipt of the vaccine. Reports to VAERS include information about the vaccinated person, the vaccine or vaccines administered, and the adverse events experienced by the vaccinated person. The reports to VAERS are then reviewed by third-party professional coders who have been trained in the assignment of Medical Dictionary for Regulatory Activities preferred terms. 16 The coders then assign appropriate terms based on the information available in the reports.

This activity was reviewed by the CDC and was conducted to be consistent with applicable federal law and CDC policy. The activities herein were confirmed to be nonresearch under the Common Rule in accordance with institutional procedures and therefore were not subject to institutional review board requirements. Informed consent was not obtained for this secondary use of existing information; see 45 CFR part 46.102(l)(2), 21 CFR part 56, 42 USC §241(d), 5 USC §552a, and 44 USC §3501 et seq.

The exposure of concern was vaccination with one of the mRNA-based COVID-19 vaccines: the BNT162b2 vaccine (Pfizer-BioNTech) or the mRNA-1273 vaccine (Moderna). During the analytic period, persons aged 12 years or older were eligible for the BNT162b2 vaccine and persons aged 18 years or older were eligible for the mRNA-1273 vaccine. The number of COVID-19 vaccine doses administered during the analytic period was obtained through the CDC’s COVID-19 Data Tracker. 17

The primary outcome was the occurrence of myocarditis and the secondary outcome was pericarditis. Reports to VAERS with these outcomes were initially characterized using the Medical Dictionary for Regulatory Activities preferred terms of myocarditis or pericarditis (specific terms are listed in the eMethods in the Supplement ). After initial review of reports of myocarditis to VAERS and review of the patient’s medical records (when available), the reports were further reviewed by CDC physicians and public health professionals to verify that they met the CDC’s case definition for probable or confirmed myocarditis (descriptions previously published and included in the eMethods in the Supplement ). 18 The CDC’s case definition of probable myocarditis requires the presence of new concerning symptoms, abnormal cardiac test results, and no other identifiable cause of the symptoms and findings. Confirmed cases of myocarditis further require histopathological confirmation of myocarditis or cardiac magnetic resonance imaging (MRI) findings consistent with myocarditis.

Deaths were included only if the individual had met the case definition for confirmed myocarditis and there was no other identifiable cause of death. Individual cases not involving death were included only if the person had met the case definition for probable myocarditis or confirmed myocarditis.

We characterized reports of myocarditis or pericarditis after COVID-19 vaccination that met the CDC’s case definition and were received by VAERS between December 14, 2020 (when COVID-19 vaccines were first publicly available in the US), and August 31, 2021, by age, sex, race, ethnicity, and vaccine type; data were processed by VAERS as of September 30, 2021. Race and ethnicity were optional fixed categories available by self-identification at the time of vaccination or by the individual filing a VAERS report. Race and ethnicity were included to provide the most complete baseline description possible for individual reports; however, further analyses were not stratified by race and ethnicity due to the high percentage of missing data. Reports of pericarditis with evidence of potential myocardial involvement were included in the review of reports of myocarditis. The eFigure in the Supplement outlines the categorization of the reports of myocarditis and pericarditis reviewed.

Further analyses were conducted only for myocarditis because of the preponderance of those reports to VAERS, in Clinical Immunization Safety Assessment Project consultations, and in published articles. 10 - 12 , 19 - 21 Crude reporting rates for myocarditis during a 7-day risk interval were calculated using the number of reports of myocarditis to VAERS per million doses of COVID-19 vaccine administered during the analytic period and stratified by age, sex, vaccination dose (first, second, or unknown), and vaccine type. Expected rates of myocarditis by age and sex were calculated using 2017-2019 data from the IBM MarketScan Commercial Research Database. This database contains individual-level, deidentified, inpatient and outpatient medical and prescription drug claims, and enrollment information submitted to IBM Watson Health by large employers and health plans. The data were accessed using version 4.0 of the IBM MarketScan Treatment Pathways analytic platform. Age- and sex-specific rates were calculated by determining the number of individuals with myocarditis ( International Statistical Classification of Diseases and Related Health Problems, Tenth Revision [ICD-10] codes B33.20, B33.22, B33.24, I40.0, I40.1, I40.8, I40.9, or I51.4) 22 identified during an inpatient encounter in 2017-2019 relative to the number of individuals of similar age and sex who were continually enrolled during the year in which the myocarditis-related hospitalization occurred; individuals with any diagnosis of myocarditis prior to that year were excluded. Given the limitations of the IBM MarketScan Commercial Research Database to capture enrollees aged 65 years or older, an expected rate for myocarditis was not calculated for this population. A 95% CI was calculated using Poisson distribution in SAS version 9.4 (SAS Institute Inc) for each expected rate of myocarditis and for each observed rate in a strata with at least 1 case.

In cases of probable or confirmed myocarditis among those younger than 30 years of age, their clinical course was then summarized to the extent possible based on medical review and clinician interviews. This clinical course included presenting symptoms, diagnostic test results, treatment, and early outcomes (abstraction form appears in the eMethods in the Supplement ). 23

When applicable, missing data were delineated in the results or the numbers with complete data were listed. No assumptions or imputations were made regarding missing data. Any percentages that were calculated included only those cases of myocarditis with adequate data to calculate the percentages.

Between December 14, 2020, and August 31, 2021, 192 405 448 individuals older than 12 years of age received a total of 354 100 845 mRNA-based COVID-19 vaccines. VAERS received 1991 reports of myocarditis (391 of which also included pericarditis) after receipt of at least 1 dose of mRNA-based COVID-19 vaccine (eTable 1 in the Supplement ) and 684 reports of pericarditis without the presence of myocarditis (eTable 2 in the Supplement ).

Of the 1991 reports of myocarditis, 1626 met the CDC’s case definition for probable or confirmed myocarditis ( Table 1 ). There were 208 reports that did not meet the CDC’s case definition for myocarditis and 157 reports that required more information to perform adjudication (eTable 3 in the Supplement ). Of the 1626 reports that met the CDC’s case definition for myocarditis, 1195 (73%) were younger than 30 years of age, 543 (33%) were younger than 18 years of age, and the median age was 21 years (IQR, 16-31 years) ( Figure 1 ). Of the reports of myocarditis with dose information, 82% (1265/1538) occurred after the second vaccination dose. Of those with a reported dose and time to symptom onset, the median time from vaccination to symptom onset was 3 days (IQR, 1-8 days) after the first vaccination dose and 74% (187/254) of myocarditis events occurred within 7 days. After the second vaccination dose, the median time to symptom onset was 2 days (IQR, 1-3 days) and 90% (1081/1199) of myocarditis events occurred within 7 days ( Figure 2 ).

Males comprised 82% (1334/1625) of the cases of myocarditis for whom sex was reported. The largest proportions of cases of myocarditis were among White persons (non-Hispanic or ethnicity not reported; 69% [914/1330]) and Hispanic persons (of all races; 17% [228/1330]). Among persons younger than 30 years of age, there were no confirmed cases of myocarditis in those who died after mRNA-based COVID-19 vaccination without another identifiable cause and there was 1 probable case of myocarditis but there was insufficient information available for a thorough investigation. At the time of data review, there were 2 reports of death in persons younger than 30 years of age with potential myocarditis that remain under investigation and are not included in the case counts.

Symptom onset of myocarditis was within 7 days after vaccination for 947 reports of individuals who received the BNT162b2 vaccine and for 382 reports of individuals who received the mRNA-1273 vaccine. The rates of myocarditis varied by vaccine type, sex, age, and first or second vaccination dose ( Table 2 ). The reporting rates of myocarditis were highest after the second vaccination dose in adolescent males aged 12 to 15 years (70.73 [95% CI, 61.68-81.11] per million doses of the BNT162b2 vaccine), in adolescent males aged 16 to 17 years (105.86 [95% CI, 91.65-122.27] per million doses of the BNT162b2 vaccine), and in young men aged 18 to 24 years (52.43 [95% CI, 45.56-60.33] per million doses of the BNT162b2 vaccine and 56.31 [95% CI, 47.08-67.34] per million doses of the mRNA-1273 vaccine). The lower estimate of the 95% CI for reporting rates of myocarditis in adolescent males and young men exceeded the upper bound of the expected rates after the first vaccination dose with the BNT162b2 vaccine in those aged 12 to 24 years, after the second vaccination dose with the BNT162b2 vaccine in those aged 12 to 49 years, after the first vaccination dose with the mRNA-1273 vaccine in those aged 18 to 39 years, and after the second vaccination dose with the mRNA-1273 vaccine in those aged 18 to 49 years.

The reporting rates of myocarditis in females were lower than those in males across all age strata younger than 50 years of age. The reporting rates of myocarditis were highest after the second vaccination dose in adolescent females aged 12 to 15 years (6.35 [95% CI, 4.05-9.96] per million doses of the BNT162b2 vaccine), in adolescent females aged 16 to 17 years (10.98 [95% CI, 7.16-16.84] per million doses of the BNT162b2 vaccine), in young women aged 18 to 24 years (6.87 [95% CI, 4.27-11.05] per million doses of the mRNA-1273 vaccine), and in women aged 25 to 29 years (8.22 [95% CI, 5.03-13.41] per million doses of the mRNA-1273 vaccine). The lower estimate of the 95% CI for reporting rates of myocarditis in females exceeded the upper bound of the expected rates after the second vaccination dose with the BNT162b2 vaccine in those aged 12 to 29 years and after the second vaccination dose with the mRNA-1273 vaccine in those aged 18 to 29 years.

Among the 1372 reports of myocarditis in persons younger than 30 years of age, 1305 were able to be adjudicated, with 92% (1195/1305) meeting the CDC’s case definition. Of these, chart abstractions or medical interviews were completed for 69% (826/1195) ( Table 3 ). The symptoms commonly reported in the verified cases of myocarditis in persons younger than 30 years of age included chest pain, pressure, or discomfort (727/817; 89%) and dyspnea or shortness of breath (242/817; 30%). Troponin levels were elevated in 98% (792/809) of the cases of myocarditis. The electrocardiogram result was abnormal in 72% (569/794) of cases of myocarditis. Of the patients who had received a cardiac MRI, 72% (223/312) had abnormal findings consistent with myocarditis. The echocardiogram results were available for 721 cases of myocarditis; of these, 84 (12%) demonstrated a notable decreased left ventricular ejection fraction (<50%). Among the 676 cases for whom treatment data were available, 589 (87%) received nonsteroidal anti-inflammatory drugs. Intravenous immunoglobulin and glucocorticoids were each used in 12% of the cases of myocarditis (78/676 and 81/676, respectively). Intensive therapies such as vasoactive medications (12 cases of myocarditis) and intubation or mechanical ventilation (2 cases) were rare. There were no verified cases of myocarditis requiring a heart transplant, extracorporeal membrane oxygenation, or a ventricular assist device. Of the 96% (784/813) of cases of myocarditis who were hospitalized, 98% (747/762) were discharged from the hospital at time of review. In 87% (577/661) of discharged cases of myocarditis, there was resolution of the presenting symptoms by hospital discharge.

In this review of reports to VAERS between December 2020 and August 2021, myocarditis was identified as a rare but serious adverse event that can occur after mRNA-based COVID-19 vaccination, particularly in adolescent males and young men. However, this increased risk must be weighed against the benefits of COVID-19 vaccination. 18

Compared with cases of non–vaccine-associated myocarditis, the reports of myocarditis to VAERS after mRNA-based COVID-19 vaccination were similar in demographic characteristics but different in their acute clinical course. First, the greater frequency noted among vaccine recipients aged 12 to 29 years vs those aged 30 years or older was similar to the age distribution seen in typical cases of myocarditis. 2 , 4 This pattern may explain why cases of myocarditis were not discovered until months after initial Emergency Use Authorization of the vaccines in the US (ie, until the vaccines were widely available to younger persons). Second, the sex distribution in cases of myocarditis after COVID-19 vaccination was similar to that seen in typical cases of myocarditis; there is a strong male predominance for both conditions. 2 , 4

However, the onset of myocarditis symptoms after exposure to a potential immunological trigger was shorter for COVID-19 vaccine–associated cases of myocarditis than is typical for myocarditis cases diagnosed after a viral illness. 24 - 26 Cases of myocarditis reported after COVID-19 vaccination were typically diagnosed within days of vaccination, whereas cases of typical viral myocarditis can often have indolent courses with symptoms sometimes present for weeks to months after a trigger if the cause is ever identified. 1 The major presenting symptoms appeared to resolve faster in cases of myocarditis after COVID-19 vaccination than in typical viral cases of myocarditis. Even though almost all individuals with cases of myocarditis were hospitalized and clinically monitored, they typically experienced symptomatic recovery after receiving only pain management. In contrast, typical viral cases of myocarditis can have a more variable clinical course. For example, up to 6% of typical viral myocarditis cases in adolescents require a heart transplant or result in mortality. 27

In the current study, the initial evaluation and treatment of COVID-19 vaccine–associated myocarditis cases was similar to that of typical myocarditis cases. 28 - 31 Initial evaluation usually included measurement of troponin level, electrocardiography, and echocardiography. 1 Cardiac MRI was often used for diagnostic purposes and also for possible prognostic purposes. 32 , 33 Supportive care was a mainstay of treatment, with specific cardiac or intensive care therapies as indicated by the patient’s clinical status.

Long-term outcome data are not yet available for COVID-19 vaccine–associated myocarditis cases. The CDC has started active follow-up surveillance in adolescents and young adults to assess the health and functional status and cardiac outcomes at 3 to 6 months in probable and confirmed cases of myocarditis reported to VAERS after COVID-19 vaccination. 34 For patients with myocarditis, the American Heart Association and the American College of Cardiology guidelines advise that patients should be instructed to refrain from competitive sports for 3 to 6 months, and that documentation of a normal electrocardiogram result, ambulatory rhythm monitoring, and an exercise test should be obtained prior to resumption of sports. 35 The use of cardiac MRI is unclear, but it may be useful in evaluating the progression or resolution of myocarditis in those with abnormalities on the baseline cardiac MRI. 36 Further doses of mRNA-based COVID-19 vaccines should be deferred, but may be considered in select circumstances. 37

This study has several limitations. First, although clinicians are required to report serious adverse events after COVID-19 vaccination, including all events leading to hospitalization, VAERS is a passive reporting system. As such, the reports of myocarditis to VAERS may be incomplete, and the quality of the information reported is variable. Missing data for sex, vaccination dose number, and race and ethnicity were not uncommon in the reports received; history of prior SARS-CoV-2 infection also was not known. Furthermore, as a passive system, VAERS data are subject to reporting biases in that both underreporting and overreporting are possible. 38 Given the high verification rate of reports of myocarditis to VAERS after mRNA-based COVID-19 vaccination, underreporting is more likely. Therefore, the actual rates of myocarditis per million doses of vaccine are likely higher than estimated.

Second, efforts by CDC investigators to obtain medical records or interview physicians were not always successful despite the special allowance for sharing information with the CDC under the Health Insurance Portability and Accountability Act of 1996. 39 This challenge limited the ability to perform case adjudication and complete investigations for some reports of myocarditis, although efforts are still ongoing when feasible.

Third, the data from vaccination administration were limited to what is reported to the CDC and thus may be incomplete, particularly with regard to demographics.

Fourth, calculation of expected rates from the IBM MarketScan Commercial Research Database relied on administrative data via the use of ICD-10 codes and there was no opportunity for clinical review. Furthermore, these data had limited information regarding the Medicare population; thus expected rates for those older than 65 years of age were not calculated. However, it is expected that the rates in those older than 65 years of age would not be higher than the rates in those aged 50 to 64 years. 4

Based on passive surveillance reporting in the US, the risk of myocarditis after receiving mRNA-based COVID-19 vaccines was increased across multiple age and sex strata and was highest after the second vaccination dose in adolescent males and young men. This risk should be considered in the context of the benefits of COVID-19 vaccination.

Corresponding Author: Matthew E. Oster, MD, MPH, US Centers for Disease Control and Prevention, 1600 Clifton Rd, Atlanta, GA 30333 ( [email protected] ).

Correction: This article was corrected March 21, 2022, to change “pericarditis” to “myocarditis” in the first row, first column of eTable 1 in the Supplement.

Accepted for Publication: December 16, 2021.

Author Contributions: Drs Oster and Su had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.

Concept and design: Oster, Shay, Su, Creech, Edwards, Dendy, Schlaudecker, Woo, Shimabukuro.

Acquisition, analysis, or interpretation of data: Oster, Shay, Su, Gee, Creech, Broder, Edwards, Soslow, Schlaudecker, Lang, Barnett, Ruberg, Smith, Campbell, Lopes, Sperling, Baumblatt, Thompson, Marquez, Strid, Woo, Pugsley, Reagan-Steiner, DeStefano, Shimabukuro.

Drafting of the manuscript: Oster, Shay, Su, Gee, Creech, Marquez, Strid, Woo, Shimabukuro.

Critical revision of the manuscript for important intellectual content: Oster, Shay, Su, Creech, Broder, Edwards, Soslow, Dendy, Schlaudecker, Lang, Barnett, Ruberg, Smith, Campbell, Lopes, Sperling, Baumblatt, Thompson, Pugsley, Reagan-Steiner, DeStefano, Shimabukuro.

Statistical analysis: Oster, Su, Marquez, Strid, Woo, Shimabukuro.

Obtained funding: Edwards, DeStefano.

Administrative, technical, or material support: Oster, Gee, Creech, Broder, Edwards, Soslow, Schlaudecker, Smith, Baumblatt, Thompson, Reagan-Steiner, DeStefano.

Supervision: Su, Edwards, Soslow, Dendy, Schlaudecker, Campbell, Sperling, DeStefano, Shimabukuro.

Conflict of Interest Disclosures: Dr Creech reported receiving grants from the National Institutes of Health for the Moderna and Janssen clinical trials and receiving personal fees from Astellas and Horizon. Dr Edwards reported receiving grants from the National Institutes of Health; receiving personal fees from BioNet, IBM, X-4 Pharma, Seqirus, Roche, Pfizer, Merck, Moderna, and Sanofi; and receiving compensation for being the associate editor of Clinical Infectious Diseases . Dr Soslow reported receiving personal fees from Esperare. Dr Schlaudecker reported receiving grants from Pfizer and receiving personal fees from Sanofi Pasteur. Drs Barnett, Ruberg, and Smith reported receiving grants from Pfizer. Dr Lopes reported receiving personal fees from Bayer, Boehringer Ingleheim, Bristol Myers Squibb, Daiichi Sankyo, GlaxoSmithKline, Medtronic, Merck, Pfizer, Portola, and Sanofi and receiving grants from Bristol Myers Squibb, GlaxoSmithKline, Medtronic, Pfizer, and Sanofi. No other disclosures were reported.

Funding/Support: This work was supported by contracts 200-2012-53709 (Boston Medical Center), 200-2012-53661 (Cincinnati Children’s Hospital Medical Center), 200-2012-53663 (Duke University), and 200-2012-50430 (Vanderbilt University Medical Center) with the US Centers for Disease Control and Prevention (CDC) Clinical Immunization Safety Assessment Project.

Role of the Funder/Sponsor: The CDC provided funding via the Clinical Immunization Safety Assessment Project to Drs Creech, Edwards, Soslow, Dendy, Schlaudecker, Lang, Barnett, Ruberg, Smith, Campbell, and Lopes. The authors affiliated with the CDC along with the other coauthors conducted the investigations; performed collection, management, analysis, and interpretation of the data; were involved in the preparation, review, and approval of the manuscript; and made the decision to submit the manuscript for publication.

Disclaimer: The findings and conclusions in this article are those of the authors and do not necessarily represent the official position of the CDC or the US Food and Drug Administration. Mention of a product or company name is for identification purposes only and does not constitute endorsement by the CDC or the US Food and Drug Administration.

Additional Contributions: We thank the following CDC staff who contributed to this article without compensation outside their normal salaries (in alphabetical order and contribution specified in parenthesis at end of each list of names): Nickolas Agathis, MD, MPH, Stephen R. Benoit, MD, MPH, Beau B. Bruce, MD, PhD, Abigail L. Carlson, MD, MPH, Meredith G. Dixon, MD, Jonathan Duffy, MD, MPH, Charles Duke, MD, MPH, Charles Edge, MSN, MS, Robyn Neblett Fanfair, MD, MPH, Nathan W. Furukawa, MD, MPH, Gavin Grant, MD, MPH, Grace Marx, MD, MPH, Maureen J. Miller, MD, MPH, Pedro Moro, MD, MPH, Meredith Oakley, DVM, MPH, Kia Padgett, MPH, BSN, RN, Janice Perez-Padilla, MPH, BSN, RN, Robert Perry, MD, MPH, Nimia Reyes, MD, MPH, Ernest E. Smith, MD, MPH&TM, David Sniadack, MD, MPH, Pamela Tucker, MD, Edward C. Weiss, MD, MPH, Erin Whitehouse, PhD, MPH, RN, Pascale M. Wortley, MD, MPH, and Rachael Zacks, MD (for clinical investigations and interviews); Amelia Jazwa, MSPH, Tara Johnson, MPH, MS, and Jamila Shields, MPH (for project coordination); Charles Licata, PhD, and Bicheng Zhang, MS (for data acquisition and organization); Charles E. Rose, PhD (for statistical consultation); and Scott D. Grosse, PhD (for calculation of expected rates of myocarditis). We also thank the clinical staff who cared for these patients and reported the adverse events to the Vaccine Adverse Event Reporting System.

• Register for email alerts with links to free full-text articles
• Access PDFs of free articles
• Manage your interests
• Save searches and receive search alerts

More From Forbes

Confront The Agency AI Fear Factor With Workforce Literacy

• Share to Facebook
• Share to Twitter
• Share to Linkedin

Marketing and advertising agencies’ ability to tap into the seemingly boundless opportunities of generative AI (genAI) is running into a stubborn human obstacle: the lack of AI expertise and employee concerns of obsolescence. A new Forrester report, The State Of Generative AI Inside US Agencies, 2024, shows that among the top challenges to adopting AI is concerns for employee expertise and readiness, which we defined as lack of necessary skills or training and employee cultural resistance. This human obstacle is a part of an agency AI fear factor that the industry must take immediate steps to overcome.

A Secure Economic Future Is At Stake

The failure to rise to the AI moment puts the future of the agency industry and a decade of agencies’ business change at risk. GenAI not only promises to be the next biggest thing since the smartphone but also the likely ingredient to unlock the expansion of the agency economic model away from commoditized services and toward value-add creativity algorithms. GenAI infused with the creative process is the final element to a new system of value for brands: data intelligence and technology scale combined with human creative intuition. This intelligent creativity will energize the creative process (i.e., change how marketing is created and by whom), expand the value produced for brands, and modernize the economic model governing marketing services. To say that agency leaders understand the significance of the moment is an understatement, because 77% believe that genAI is a disruptor and nearly a third call genAI a major disruption that will change their business forever .

Emotional Safety Runs Unchecked

Today’s employees are still recovering from multiple workplace disruptions over the last four years: pandemic-forced remote work, return to the office, the great resignation, quiet quitting, and, now, white-collar automation. Agencies are no exception; arguably, agency workers are even more at risk given their industry’s reputation for long hours and low wages . The fear of AI job displacement is not new, even in creative industries: Look no further than the tension between the Hollywood business model and writers’ creativity . But agency employees find themselves in a particularly precarious position: The exponential rise of genAI inside agencies acts as a superconductor for their fear of obsolescence, challenging their artistic identities and personal livelihoods.

Invest In Essential AI Literacy

Agency employees need the skills and confidence to master AI and not let the AI fear factor master them. A compulsory system of AI literacy and training will solve this dilemma. It’s true that there’s no shortage of advertising and marketing education, training, and continuing education opportunities with university programs such as the Savannah College of Art and Design or Utrecht University , technology partner certification programs , or even Cannes’ Creative Academy . While these programs and hundreds of others like them are significant, they’re not nearly enough. Additional investments are needed to provide agency employees the skills and practice to integrate AI into their daily jobs and the emotional safety to adapt to the AI era in advertising and marketing creativity.

Agencies, technology partners, universities, trade groups, and even award shows need to rise to the occasion to reskill, invest in retraining, and support the creative workforce as it learns to incorporate AI as part of the creative team. This can be accomplished at the university level, in formal training, via technology/software certification, and through informal on-the-job training. No doubt, some agencies are providing some training, but the data indicates that AI literacy must be a top priority among agency and industry leaders.

Best High-Yield Savings Accounts Of 2024

Best 5% interest savings accounts of 2024.

To embrace the first meaningful innovation of its creative process and business model in decades, the industry should commit to invest in every employee inside an agency or marketing organization and every student pursuing an advertising or marketing degree. This will help agencies — which are currently leading in AI adoption and innovation — maintain their edge and bring elevated value to their clients in the form of agency algorithms powered by creativity.

This post was written by VP, Principal Analyst Jay Pattisall, and it originally appeared here.

• Editorial Standards
• Reprints & Permissions

Join The Conversation

One Community. Many Voices. Create a free account to share your thoughts. 

Forbes Community Guidelines

Our community is about connecting people through open and thoughtful conversations. We want our readers to share their views and exchange ideas and facts in a safe space.

In order to do so, please follow the posting rules in our site's  Terms of Service.   We've summarized some of those key rules below. Simply put, keep it civil.

Your post will be rejected if we notice that it seems to contain:

• False or intentionally out-of-context or misleading information
• Insults, profanity, incoherent, obscene or inflammatory language or threats of any kind
• Attacks on the identity of other commenters or the article's author
• Content that otherwise violates our site's  terms.

User accounts will be blocked if we notice or believe that users are engaged in:

• Continuous attempts to re-post comments that have been previously moderated/rejected
• Racist, sexist, homophobic or other discriminatory comments
• Attempts or tactics that put the site security at risk
• Actions that otherwise violate our site's  terms.

So, how can you be a power user?

• Stay on topic and share your insights
• Feel free to be clear and thoughtful to get your point across
• ‘Like’ or ‘Dislike’ to show your point of view.
• Protect your community.
• Use the report tool to alert us when someone breaks the rules.

Thanks for reading our community guidelines. Please read the full list of posting rules found in our site's  Terms of Service.

• Advanced Search

Key Success Factors for Integration of Blockchain and ERP Systems: : A Systematic Literature Review

New citation alert added.

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options, recommendations, evaluation of key success factors influencing erp implementation success.

Enterprise Resource Planning (ERP) application is often viewed as a strategic investment that can provide significant competitive advantage with positive return thus contributing to the firms' revenue and growth. Despite such strategic importance given ...

Factors for Adopting ERP as SaaS amongst SMEs: The Customers vs. Vendor Point of View

This paper identifies the factors for adopting Enterprise Resource Planning ERP delivered as Software-as-a-Service SaaS among small and medium enterprises SMEs. The authors conducted a two phases' qualitative-methodology: interviews with 20 experts from ...

Critical success factors in enterprise resource planning systems: Review of the last decade

Organizations perceive ERP as a vital tool for organizational competition as it integrates dispersed organizational systems and enables flawless transactions and production. This review examines studies investigating Critical Success Factors (CSFs) in ...

Information

Published in.

Elsevier Science Publishers B. V.

Netherlands

Publication History

Author tags.

• Enterprise Resource Planning
• Research-article

Contributors

Other metrics, bibliometrics, article metrics.

• 0 Total Citations
• 0 Total Downloads
• Downloads (Last 12 months) 0
• Downloads (Last 6 weeks) 0

View options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

California’s new high school requirement: Balance a checkbook, manage credit, avoid scams

• Copy Link URL Copied!

California students will have to complete a course in pocketbook economics — balancing a checkbook, managing credit cards, avoiding scams — to graduate from high school under a bill that will become law, state lawmakers announced Thursday.

“We need to help Californians prepare for their financial futures as early as possible,” Gov. Gavin Newsom said in a statement. “Saving for the future, making investments and spending wisely are lifelong skills that young adults need to learn before they start their careers, not after.”

This bill — which has drawn criticism from those concerned about another requirement on crammed academic schedules — orders school districts and charter schools to offer a stand-alone, one-semester course in personal finance. To meet the requirement, the class cannot be combined with any other course beginning in the 2027-28 school year.

Students graduating in 2031 will have to pass the class.

The agreement among state lawmakers avoids a ballot-box verdict by voters. Backers of the new requirement had gathered enough signatures to place the proposal, dubbed the California Personal Finance Initiative , on the November ballot. They will now shut down that effort.

The new requirement and the bill that will make it law “will benefit countless future generations of Californians,” said Tim Ranzetta, a wealthy Silicon Valley businessman who bankrolled the signature gathering for the ballot initiative and also supported the legislation.

Thursday was the legal deadline for Ranzetta to withdraw the ballot initiative, which he said he would do if an adequate version of the requirement was guaranteed to become law.

Ranzetta heads a nonprofit, Next Gen Personal Finance, that provides free curriculum and teacher training. He said the materials have reached nearly 100,000 teachers across the country, including more than 6,000 in California.

Although there is broad agreement on the importance of financial literacy, not everyone supports the requirement or the process that brought it about.

LAUSD approves $18.4-billion budget amid concerns over police, the arts and the future

Late additions to the Los Angeles Unified School District budget protect jobs and benefits and add arts instruction. Police funding stays about the same.

June 28, 2024

“There is a philosophical opposition to governance by ballot measure — where billionaires by virtue of their wealth — are exerting a disproportionate impact on determining curriculum in our schools,” said Troy Flint, chief information officer for the California School Boards Assn. “We don’t believe that’s the best system.”

He said financial literacy could have been incorporated into the existing one-semester economics requirement.

“Financial literacy instruction could be included within that larger preexisting economics course without further cluttering the class schedule for high school students — and reducing their ability to take an elective course or a course of interest to them, which this new bill will do.”

The final version of the bill attempts to speak to this concern, according to a legislative analysis , by allowing students to substitute personal finance in place of the one-semester course in economics.

Former L.A. schools Supt. Austin Beutner also expressed concern: “What is it that one is going to subtract to create time for financial literacy?”

“It’s more important for kids to build a foundation in literacy and math before they get to high school,” he said. “ If they have that, then there’s little mystery in personal finance.”

Several students liked the subject matter, but Angelica Gonzalez, who just graduated from Rancho Dominguez Preparatory School, said, “It should just be based on what the student wants, not what the student has to do in order to graduate.”

As an elective, “a course in financial literacy is more of a necessity than other electives, such as leadership.” Often students value “how easy an elective is rather than what the elective actually has in its course,” she said.

Chidubem Okigbo, a student at Narbonne High School, was less concerned about the requirement “crowding out electives ... because the course has potential to be practical and creative. For example, if the course included a lesson on how one could monetize their passion, students would most likely be interested and engaged.”

Odalis Lopez, who just graduated from Angelou Community High School, said personal finance is “hardly ever talked about in other required classes/courses, not even in business classes. ... I personally think it should be a one-year course to better prepare students.”

The legislation was introduced by Assemblymember Kevin McCarty (D-Sacramento). Not everyone was fully on board from the outset. McCarty introduced a similar bill last year that was amended to make financial literacy an optional component of economic classes, which could be done already. Ranzetta dropped his support of that bill, and even the watered-down version failed to pass.

The fate of the bill on this round changed with the backing of the governor and leaders of each house.

“Financial literacy is a critical tool that pays dividends for a lifetime,” said Senate President Pro Tempore Mike McGuire (D-North Coast). “There’s a wealth of data about the benefits of learning these valuable lessons in high school, from improving credit scores and reducing default rates to increasing the likelihood that our future generations will maintain three months of savings for emergencies and have at least one kind of retirement account.”

“Ensuring our students have the skills and knowledge to thrive is paramount to California’s continued success,” said Assembly Speaker Robert Rivas (D-Salinas).

Separately, California lawmakers recently added an ethnic studies course to the list of mandated classes.

More parents are delaying their kids’ vaccines, and it’s alarming pediatricians

More parents are choosing to delay childhood vaccinations, such as the MMR vaccine. Doctors worry toddlers remain vulnerable as measles spreads.

March 11, 2024

Minimum graduation requirements include three years of English and two of mathematics , including one year of algebra. There also are two years of science , including biological and physical sciences and three of social studies , as well as two years of physical education, one year of visual or performing arts , world language or career technical education.

There are additional requirements if a student wishes to apply to a four-year state college, and selective universities carefully evaluate the rigor of a student’s advanced coursework. Individual school districts often have their own added requirements as well.

L.A. high school teacher Colleen Ancrile said her school builds financial literacy into its advisory program, a class similar to the homeroom of old. “Adding a course to all of the other requirements will be a scheduling difficulty. Financial Literacy should be embedded starting in elementary school. Outreach to accounting firms to come in [is] actually a better idea.”

“Great idea but difficult to implement,” said L.A. parent Beth Owen. “The requirements to graduate are already quite cumbersome and often at the end a student discovers they are missing something and have to scramble. ... Electives are often courses that happen yearly, like band. It doesn’t work to have to drop something like that for a semester. Or it’s leadership or yearbook— yearlong commitments that are valuable.”

Los Angeles-area parent Irene Luczynski was surprised by how few opportunities there are for her ninth-grade son to take electives: “There’s really no room for him to branch out and try something new, and isn’t that what electives are supposed to do? ... Perhaps this is trivial, but where’s the fun in school?”

LAUSD approves cellphone ban as Newsom calls for statewide action

Student cellphone use at L.A. public schools will be banned starting in January to improve learning, limit distractions and decrease cyberbullying.

June 18, 2024

However, momentum appears to be building for financial literacy. The number of states that guarantee personal finance education for high school students has grown from eight in 2021 to 26, according to Ranzetta’s organization, which tracks the issue .

In an earlier analysis, the Center for Financial Literacy at Champlain College gave California an F in the topic: “Personal finance is not included in the graduation requirements, either as a stand-alone course or embedded in another course, and schools are not required to offer financial literacy courses.”

Researchers gave California some credit because the state education department offers “a robust list of financial literacy resources.”

In addition, the state’s CalMoneySmart program provides annual grants of up to $200,000 to nonprofit organizations to “provide financial education and financial empowerment programs and services for unbanked and underbanked Californians.”

A report by the consulting firm Tyton Partners concluded that the lifetime benefit for California students of taking a one-semester high school personal finance course is $127,000 — although such figures are hard to prove and ultimately abstract to the real-world experience of young adults.

More to Read

Can California control its boom-and-bust budget?

June 11, 2024

Overwhelmed by student loans? A free new California program can help you

June 4, 2024

California Assembly passes bill to make kindergarten mandatory

May 21, 2024

Start your day right

Sign up for Essential California for news, features and recommendations from the L.A. Times and beyond in your inbox six days a week.

You may occasionally receive promotional content from the Los Angeles Times.

Howard Blume covers education for the Los Angeles Times. He’s won the top investigative reporting prize from the L.A. Press Club and print Journalist of the Year from the L.A. Society of Professional Journalists chapter. He recently retired “Deadline L.A.,” a past honoree for best public-affairs radio program, which he produced and co-hosted on KPFK-FM (90.7) for 15 years. He teaches tap dancing and has two superior daughters.

More From the Los Angeles Times

World & Nation

Appeals court allows part of Biden student loan repayment plan to go forward

July 1, 2024

How will Louisiana’s new Ten Commandments classroom requirement be funded and enforced?

California voters could see schools bond and historic climate initiative on November ballot

Newsom and Democratic lawmakers announce ballot measure to rival more conservative crime reform