text annotation protocol

How to Annotate Texts

Use the links below to jump directly to any section of this guide:

Annotation Fundamentals

How to start annotating , how to annotate digital texts, how to annotate a textbook, how to annotate a scholarly article or book, how to annotate literature, how to annotate images, videos, and performances, additional resources for teachers.

Writing in your books can make you smarter. Or, at least (according to education experts), annotation–an umbrella term for underlining, highlighting, circling, and, most importantly, leaving comments in the margins–helps students to remember and comprehend what they read. Annotation is like a conversation between reader and text. Proper annotation allows students to record their own opinions and reactions, which can serve as the inspiration for research questions and theses. So, whether you're reading a novel, poem, news article, or science textbook, taking notes along the way can give you an advantage in preparing for tests or writing essays. This guide contains resources that explain the benefits of annotating texts, provide annotation tools, and suggest approaches for diverse kinds of texts; the last section includes lesson plans and exercises for teachers.

Why annotate? As the resources below explain, annotation allows students to emphasize connections to material covered elsewhere in the text (or in other texts), material covered previously in the course, or material covered in lectures and discussion. In other words, proper annotation is an organizing tool and a time saver. The links in this section will introduce you to the theory, practice, and purpose of annotation. 

How to Mark a Book, by Mortimer Adler

This famous, charming essay lays out the case for marking up books, and provides practical suggestions at the end including underlining, highlighting, circling key words, using vertical lines to mark shifts in tone/subject, numbering points in an argument, and keeping track of questions that occur to you as you read. 

How Annotation Reshapes Student Thinking (TeacherHUB)

In this article, a high school teacher discusses the importance of annotation and how annotation encourages more effective critical thinking.

The Future of Annotation (Journal of Business and Technical Communication)

This scholarly article summarizes research on the benefits of annotation in the classroom and in business. It also discusses how technology and digital texts might affect the future of annotation. 

Annotating to Deepen Understanding (Texas Education Agency)

This website provides another introduction to annotation (designed for 11th graders). It includes a helpful section that teaches students how to annotate reading comprehension passages on tests.

Once you understand what annotation is, you're ready to begin. But what tools do you need? How do you prepare? The resources linked in this section list strategies and techniques you can use to start annotating. 

What is Annotating? (Charleston County School District)

This resource gives an overview of annotation styles, including useful shorthands and symbols. This is a good place for a student who has never annotated before to begin.

How to Annotate Text While Reading (YouTube)

This video tutorial (appropriate for grades 6–10) explains the basic ins and outs of annotation and gives examples of the type of information students should be looking for.

Annotation Practices: Reading a Play-text vs. Watching Film (U Calgary)

This blog post, written by a student, talks about how the goals and approaches of annotation might change depending on the type of text or performance being observed. 

Annotating Texts with Sticky Notes (Lyndhurst Schools)

Sometimes students are asked to annotate books they don't own or can't write in for other reasons. This resource provides some strategies for using sticky notes instead.

Teaching Students to Close Read...When You Can't Mark the Text (Performing in Education)

Here, a sixth grade teacher demonstrates the strategies she uses for getting her students to annotate with sticky notes. This resource includes a link to the teacher's free Annotation Bookmark (via Teachers Pay Teachers).

Digital texts can present a special challenge when it comes to annotation; emerging research suggests that many students struggle to critically read and retain information from digital texts. However, proper annotation can solve the problem. This section contains links to the most highly-utilized platforms for electronic annotation.

Evernote is one of the two big players in the "digital annotation apps" game. In addition to allowing users to annotate digital documents, the service (for a fee) allows users to group multiple formats (PDF, webpages, scanned hand-written notes) into separate notebooks, create voice recordings, and sync across all sorts of devices. 

OneNote is Evernote's main competitor. Reviews suggest that OneNote allows for more freedom for digital note-taking than Evernote, but that it is slightly more awkward to import and annotate a PDF, especially on certain platforms. However, OneNote's free version is slightly more feature-filled, and OneNote allows you to link your notes to time stamps on an audio recording.

Diigo is a basic browser extension that allows a user to annotate webpages. Diigo also offers a Screenshot app that allows for direct saving to Google Drive.

While the creators of Hypothesis like to focus on their app's social dimension, students are more likely to be interested in the private highlighting and annotating functions of this program.

Foxit PDF Reader

Foxit is one of the leading PDF readers. Though the full suite must be purchased, Foxit offers a number of annotation and highlighting tools for free.

Nitro PDF Reader

This is another well-reviewed, free PDF reader that includes annotation and highlighting. Annotation, text editing, and other tools are included in the free version.

Goodreader is a very popular Mac-only app that includes annotation and editing tools for PDFs, Word documents, Powerpoint, and other formats.

Although textbooks have vocabulary lists, summaries, and other features to emphasize important material, annotation can allow students to process information and discover their own connections. This section links to guides and video tutorials that introduce you to textbook annotation. 

Annotating Textbooks (Niagara University)

This PDF provides a basic introduction as well as strategies including focusing on main ideas, working by section or chapter, annotating in your own words, and turning section headings into questions.

A Simple Guide to Text Annotation (Catawba College)

The simple, practical strategies laid out in this step-by-step guide will help students learn how to break down chapters in their textbooks using main ideas, definitions, lists, summaries, and potential test questions.

Annotating (Mercer Community College)

This packet, an excerpt from a literature textbook, provides a short exercise and some examples of how to do textbook annotation, including using shorthand and symbols.

Reading Your Healthcare Textbook: Annotation (Saddleback College)

This powerpoint contains a number of helpful suggestions, especially for students who are new to annotation. It emphasizes limited highlighting, lots of student writing, and using key words to find the most important information in a textbook. Despite the title, it is useful to a student in any discipline.

Annotating a Textbook (Excelsior College OWL)

This video (with included transcript) discusses how to use textbook features like boxes and sidebars to help guide annotation. It's an extremely helpful, detailed discussion of how textbooks are organized.

Because scholarly articles and books have complex arguments and often depend on technical vocabulary, they present particular challenges for an annotating student. The resources in this section help students get to the heart of scholarly texts in order to annotate and, by extension, understand the reading.

Annotating a Text (Hunter College)

This resource is designed for college students and shows how to annotate a scholarly article using highlighting, paraphrase, a descriptive outline, and a two-margin approach. It ends with a sample passage marked up using the strategies provided. 

Guide to Annotating the Scholarly Article (ReadWriteThink.org)

This is an effective introduction to annotating scholarly articles across all disciplines. This resource encourages students to break down how the article uses primary and secondary sources and to annotate the types of arguments and persuasive strategies (synthesis, analysis, compare/contrast).

How to Highlight and Annotate Your Research Articles (CHHS Media Center)

This video, developed by a high school media specialist, provides an effective beginner-level introduction to annotating research articles. 

How to Read a Scholarly Book (AndrewJacobs.org)

In this essay, a college professor lets readers in on the secrets of scholarly monographs. Though he does not discuss annotation, he explains how to find a scholarly book's thesis, methodology, and often even a brief literature review in the introduction. This is a key place for students to focus when creating annotations. 

A 5-step Approach to Reading Scholarly Literature and Taking Notes (Heather Young Leslie)

This resource, written by a professor of anthropology, is an even more comprehensive and detailed guide to reading scholarly literature. Combining the annotation techniques above with the reading strategy here allows students to process scholarly book efficiently. 

Annotation is also an important part of close reading works of literature. Annotating helps students recognize symbolism, double meanings, and other literary devices. These resources provide additional guidelines on annotating literature.

AP English Language Annotation Guide (YouTube)

In this ~10 minute video, an AP Language teacher provides tips and suggestions for using annotations to point out rhetorical strategies and other important information.

Annotating Text Lesson (YouTube)

In this video tutorial, an English teacher shows how she uses the white board to guide students through annotation and close reading. This resource uses an in-depth example to model annotation step-by-step.

Close Reading a Text and Avoiding Pitfalls (Purdue OWL)

This resources demonstrates how annotation is a central part of a solid close reading strategy; it also lists common mistakes to avoid in the annotation process.

AP Literature Assignment: Annotating Literature (Mount Notre Dame H.S.)

This brief assignment sheet contains suggestions for what to annotate in a novel, including building connections between parts of the book, among multiple books you are reading/have read, and between the book and your own experience. It also includes samples of quality annotations.

AP Handout: Annotation Guide (Covington Catholic H.S.)

This annotation guide shows how to keep track of symbolism, figurative language, and other devices in a novel using a highlighter, a pencil, and every part of a book (including the front and back covers).

In addition to written resources, it's possible to annotate visual "texts" like theatrical performances, movies, sculptures, and paintings. Taking notes on visual texts allows students to recall details after viewing a resource which, unlike a book, can't be re-read or re-visited ( for example, a play that has finished its run, or an art exhibition that is far away). These resources draw attention to the special questions and techniques that students should use when dealing with visual texts.

How to Take Notes on Videos (U of Southern California)

This resource is a good place to start for a student who has never had to take notes on film before. It briefly outlines three general approaches to note-taking on a film. 

How to Analyze a Movie, Step-by-Step (San Diego Film Festival)

This detailed guide provides lots of tips for film criticism and analysis. It contains a list of specific questions to ask with respect to plot, character development, direction, musical score, cinematography, special effects, and more. 

How to "Read" a Film (UPenn)

This resource provides an academic perspective on the art of annotating and analyzing a film. Like other resources, it provides students a checklist of things to watch out for as they watch the film.

Art Annotation Guide (Gosford Hill School)

This resource focuses on how to annotate a piece of art with respect to its formal elements like line, tone, mood, and composition. It contains a number of helpful questions and relevant examples. 

Photography Annotation (Arts at Trinity)

This resource is designed specifically for photography students. Like some of the other resources on this list, it primarily focuses on formal elements, but also shows students how to integrate the specific technical vocabulary of modern photography. This resource also contains a number of helpful sample annotations.

How to Review a Play (U of Wisconsin)

This resource from the University of Wisconsin Writing Center is designed to help students write a review of a play. It contains suggested questions for students to keep in mind as they watch a given production. This resource helps students think about staging, props, script alterations, and many other key elements of a performance.

This section contains links to lessons plans and exercises suitable for high school and college instructors.

Beyond the Yellow Highlighter: Teaching Annotation Skills to Improve Reading Comprehension (English Journal)

In this journal article, a high school teacher talks about her approach to teaching annotation. This article makes a clear distinction between annotation and mere highlighting.

Lesson Plan for Teaching Annotation, Grades 9–12 (readwritethink.org)

This lesson plan, published by the National Council of Teachers of English, contains four complete lessons that help introduce high school students to annotation.

Teaching Theme Using Close Reading (Performing in Education)

This lesson plan was developed by a middle school teacher, and is aligned to Common Core. The teacher presents her strategies and resources in comprehensive fashion.

Analyzing a Speech Using Annotation (UNC-TV/PBS Learning Media)

This complete lesson plan, which includes a guide for the teacher and relevant handouts for students, will prepare students to analyze both the written and presentation components of a speech. This lesson plan is best for students in 6th–10th grade.

Writing to Learn History: Annotation and Mini-Writes (teachinghistory.org)

This teaching guide, developed for high school History classes, provides handouts and suggested exercises that can help students become more comfortable with annotating historical sources.

Writing About Art (The College Board)

This Prezi presentation is useful to any teacher introducing students to the basics of annotating art. The presentation covers annotating for both formal elements and historical/cultural significance.

Film Study Worksheets (TeachWithMovies.org)

This resource contains links to a general film study worksheet, as well as specific worksheets for novel adaptations, historical films, documentaries, and more. These resources are appropriate for advanced middle school students and some high school students. 

Annotation Practice Worksheet (La Guardia Community College)

This worksheet has a sample text and instructions for students to annotate it. It is a useful resource for teachers who want to give their students a chance to practice, but don't have the time to select an appropriate piece of text. 

• PDFs for all 136 Lit Terms we cover
• Downloads of 1924 LitCharts Lit Guides
• Teacher Editions for every Lit Guide
• Explanations and citation info for 40,556 quotes across 1924 books
• Downloadable (PDF) line-by-line translations of every Shakespeare play

Need something? Request a new guide .

How can we improve? Share feedback .

LitCharts is hiring!

The Ultimate Guide to Text Annotation: Techniques, Tools, and Best Practices

Puneet Jindal

Introduction.

Welcome to the realm where language meets machine intelligence : text annotation - the catalyst propelling artificial intelligence to understand, interpret, and communicate in human language. Evolving from editorial footnotes to a cornerstone in data science, text annotation now drives Natural Language Processing (NLP) and Computer Vision , reshaping industries across the globe.

Imagine AI models decoding sentiments, recognizing entities, and grasping human nuances in a text. Text annotation is the magical key to making this possible. Join us on this journey through text annotation - exploring its techniques, challenges, and the transformative potential it holds for healthcare, finance, government, logistics, and beyond.

In this exploration, witness text annotation's evolution and its pivotal role in fueling AI's understanding of language. Explore how tools such as Labellerr help in text annotation and work.  Let's unravel the artistry behind text annotation, shaping a future where AI comprehends, adapts, and innovates alongside human communication.

1. What is Text Annotation?

Text annotation is a crucial process that involves adding labels, comments, or metadata to textual data to facilitate machine learning algorithms' understanding and analysis.

This practice, known for its traditional role in editorial reviews by adding comments or footnotes to text drafts, has evolved significantly within the realm of data science, particularly in Natural Language Processing (NLP) and Computer Vision applications .

In the context of machine learning, text annotation takes on a more specific role. It involves systematically labeling pieces of text to create a reference dataset, enabling supervised machine learning algorithms to recognize patterns, learn from labeled data, and make accurate predictions or classifications when faced with new, unseen text.

To elaborate on what it means to annotate text: In data science and NLP, annotating text demands a comprehensive understanding of the problem domain and the dataset. It involves identifying and marking relevant features within the text. This can be akin to labeling images in image classification tasks, but in text, it includes categorizing sentences or segments into predefined classes or topics.

For instance, labeling sentiments in online reviews, distinguishing fake and real news articles, or marking parts of speech and named entities in text.

1.1 Text Annotation Tasks: A Multifaceted Approach to Data Labeling

(i) Text Classification : Assigning predefined categories or labels to text segments based on their content, such as sentiment analysis or topic classification.

(ii) Named Entity Recognition (NER) : Identifying and labeling specific entities within the text, like names of people, organizations, locations, dates, etc.

(iii) Parts of Speech Tagging : Labeling words in a sentence with their respective grammatical categories, like nouns, verbs, adjectives, etc.

(iv) Summarization : Condensing a lengthy text into a shorter, coherent version while retaining its key information.

1.2 Significant Benefits of Text Annotation

(i) Improved Machine Learning Models : Annotated data provides labeled examples for algorithms to learn from, enhancing their ability to make accurate predictions or classifications when faced with new, unlabeled text.

(ii) Enhanced Performance and Efficiency : Annotations expedite the learning process by offering clear indicators to algorithms, leading to improved performance and faster model convergence.

(iii) Nuance Recognition : Text annotations help algorithms understand contextual nuances, sarcasm, or subtle linguistic cues that might not be immediately apparent, enhancing their ability to interpret text accurately.

(iv) Applications in Various Industries : Text annotation is vital across industries, aiding in tasks like content moderation, sentiment analysis for customer feedback , information extraction for search engines , and much more.

Text annotation is a critical process in modern machine learning, empowering algorithms to comprehend, interpret, and extract valuable insights from textual data, thereby enabling various applications across different sectors.

2. Types of Text Annotation

Text annotation, in the realm of data labeling and Natural Language Processing (NLP), encompasses a diverse range of techniques used to label, categorize, and extract meaningful information from textual data. This multifaceted process involves several types of annotations, each serving a distinct purpose in enhancing machine understanding and analysis of text.

These annotation types include sentiment annotation, intent annotation, entity annotation, text classification, linguistic annotation, named entity recognition (NER), part-of-speech tagging, keyphrase tagging, entity linking, document classification, language identification, and toxicity classification.

1. Sentiment Annotation

Sentiment annotation is a technique crucial for understanding emotions conveyed in text. Assigning sentiments like positive, negative, or neutral to sentences aids in sentiment analysis .

This process involves deciphering emotions in customer reviews on e-commerce platforms (e.g., Amazon, Flipkart), enabling businesses to gauge customer satisfaction.

Precise sentiment annotation is vital for training machine learning models that categorize texts into various emotions, facilitating a deeper understanding of user sentiments towards products or services.

Let's consider various instances where sentiment annotation encounters complexities:

(i) Clear Emotions: In the initial examples, emotions are distinctly evident. The first instance exudes happiness and positivity, while the second reflects disappointment and negative feelings. However, in the third case, emotions become intricate. Phrases like "nostalgic" or "bittersweet" evoke mixed sentiments, making it challenging to classify into a single emotion.

(ii) Success versus Failure: Analyzing phrases such as "Yay! Argentina beat France in the World Cup Finale" presents a paradox. Initially appearing positive, this sentence also implies negative emotions for the opposing side, complicating straightforward sentiment classification.

(iii) Sarcasm and Ridicule: Capturing sarcasm involves comprehending nuanced human communication styles, relying on context, tone, and social cues—characteristics often intricate for machines to interpret.

(iv) Rhetorical Questions: Phrases like "Why do we have to quibble every time?" may seem neutral initially. However, the speaker's tone and delivery convey a sense of frustration and negativity, posing challenges in categorizing the sentiment accurately.

(v) Quoting or Re-tweeting: Sentiment annotation confronts difficulties when dealing with quoted or retweeted content. The sentiment expressed might not align with the opinions of the one sharing the quote, creating discrepancies in sentiment classification.

In essence, sentiment annotation encounters challenges due to the complexity of human emotions, contextual nuances, and the subtleties of language expression, making accurate classification a demanding task for automated systems.

Intent Annotation

Intent annotation is a crucial aspect in the development of chatbots and virtual assistants , forming the backbone of their functionality. It involves labeling or categorizing user messages or sentences to identify the underlying purpose or intention behind the communication.

This annotation process aims to understand and extract the user's intent, enabling these AI systems to provide contextually relevant and accurate responses. Intent annotation involves labeling sentences to discern the user's intention behind a message. By annotating intents like greetings, complaints, or inquiries, systems can generate appropriate responses.

Key points regarding intent text annotation include:

Purpose Identification: Intent annotation involves categorizing user messages into specific intents such as greetings, inquiries, complaints, feedback, orders, or any other actionable user intents. Each category represents a different user goal or purpose within the conversation.

Training Data Creation: Creating labeled datasets is crucial for training machine learning models to recognize and classify intents accurately. Annotated datasets consist of labeled sentences or phrases paired with their corresponding intended purposes, forming the foundation for model training.

Contextual Understanding: Intent annotation often requires a deep understanding of contextual nuances within language. It's not solely about identifying keywords but comprehending the broader meaning and context of user queries or statements.

Natural Language Understanding (NLU) : It falls under the realm of natural language processing (NLP) and requires sophisticated algorithms capable of interpreting and categorizing user intents accurately. Machine learning models, such as classifiers or neural networks, are commonly used for this purpose.

Iterative Process: Annotation of intents often involves an iterative process. Initially, a set of intent categories is defined based on common user interactions. As the system encounters new user intents, the annotation process may expand or refine these categories to ensure comprehensive coverage.

Quality Assurance and Validation: It's essential to validate and ensure the quality of labeled data. This may involve multiple annotators labeling the same data independently to assess inter-annotator agreement and enhance annotation consistency.

Adaptation and Evolution: Intent annotation isn't a one-time task. As user behaviors, language use, and interaction patterns evolve, the annotated intents also need periodic review and adaptation to maintain accuracy and relevance.

Enhancing User Experience: Accurate intent annotation is pivotal in enhancing user experience. It enables chatbots and virtual assistants to understand user needs promptly and respond with relevant and helpful information or actions, improving overall user satisfaction.

Industry-Specific Customization: Intent annotation can be industry-specific. For instance, in healthcare, intents may include appointment scheduling, medication queries, or symptom descriptions, while in finance, intents may revolve around account inquiries, transaction history, or support requests.

Continuous Improvement: Feedback loops and analytics derived from user interactions help refine intent annotation. Analyzing user feedback on system responses can drive improvements in intent categorization and response generation.

For instance, Siri or Alexa, trained on annotated data for specific intents, responds accurately to user queries, enhancing user experience. Below are given examples:

• Greeting Intent: Hello there, how are you?
• Complaint Intent:  I am very disappointed with the service I received.
• Inquiry Intent: What are your business hours?
• Confirmation Intent:  Yes, I'd like to confirm my appointment for tomorrow at 10 AM.
• Request Intent: Could you please provide me with the menu?
• Gratitude Intent: Thank you so much for your help!
• Feedback Intent:  I wanted to give feedback about the recent product purchase.
• Apology Intent:  I'm sorry for the inconvenience caused.
• Assistance Intent:  Can you assist me with setting up my account?
• Goodbye Intent:  Goodbye, have a great day!

These annotations serve as training data for AI models to learn and understand different user intentions, enabling chatbots or virtual assistants to respond accurately and effectively.

Entity Annotation:

Entity annotation focuses on labeling key phrases, named entities, or parts of speech in text. This technique emphasizes crucial details in lengthy texts and aids in training models for entity extraction. Named entity recognition (NER) is a subset of entity annotation, labeling entities like people's names, locations, dates, etc., enabling machines to comprehend text more comprehensively by distinguishing semantic meanings.

Text Classification

Text classification assigns categories or labels to text segments. This annotation technique is essential for organizing text data into specific classes or topics, such as document classification or sentiment analysis. Categorizing tweets into education, politics, etc., helps organize content and enables better understanding.

Let's look at each of these forms separately.

Document Classification: This involves assigning a single label to a document, aiding in the efficient sorting of vast textual data based on its primary theme or content.

Product Categorization: It's the process of organizing products or services into specific classes or categories. This helps enhance search results in eCommerce platforms, improving SEO strategies and boosting visibility in product ranking pages.

Email Classification: This task involves categorizing emails into either spam or non-spam (ham) categories, typically based on their content, aiding in email filtering and prioritization.

News Article Classification: Categorizing news articles based on their content or topics such as politics, entertainment, sports, technology, etc. This categorization assists in better organizing and presenting news content to readers.

Language Identification: This task involves determining the language used in a given text, is useful in multilingual contexts or language-specific applications.

Toxicity Classification: Identifying whether a social media comment or post contains toxic content, hate speech, or is non-toxic. This classification helps in content moderation and creating safer online environments.

Each form of text annotation serves a specific purpose, enabling better organization, classification, and understanding of textual data, and contributing to various applications across industries and domains.

Linguistic Annotation

Linguistic annotation focuses on language-related details in text or speech, including semantics, phonetics, and discourse. It encompasses intonation, stress, pauses, and discourse relations. It helps systems understand linguistic nuances, like coreference resolution linking pronouns to their antecedents, semantic labeling, and annotating stress or tone in speech.

Named Entity Recognition (NER)

NER identifies and labels named entities like people's names, locations, dates, etc., in text. It plays a pivotal role in NLP applications, allowing systems like Google Translate or Siri to understand and process textual data accurately.

Part-of-Speech Tagging

Part-of-speech tagging labels words in a sentence with their grammatical categories (nouns, verbs, adjectives). It assists in parsing sentences and understanding their structure.

Keyphrase Tagging

Keyphrase tagging locates and labels keywords or keyphrases in text, aiding in tasks like summarization or extracting key concepts from large text documents.

Entity Linking

Entity linking maps words in text to entities in a knowledge base, aiding in disambiguating entities' meanings and connecting them to larger datasets for contextual understanding.

3. Text Annotation use cases

(i) healthcare.

Text annotation significantly transforms healthcare operations by leveraging AI and machine learning techniques to enhance patient care, streamline processes, and improve overall efficiency:

Automatic Data Extraction: Text annotation aids in extracting critical information from clinical trial records, facilitating better access and analysis of medical documents. It expedites research efforts and supports comprehensive data-driven insights.

Patient Record Analysis: Annotated data enables thorough analysis of patient records, leading to improved outcomes and more accurate medical condition detection. It aids healthcare professionals in making informed decisions and providing tailored treatments.

Insurance Claims Processing: Within healthcare insurance, text annotation helps recognize medically insured patients, identify loss amounts, and extract policyholder information. This speeds up claims processing, ensuring faster service delivery to policyholders.

(II) Insurance

Text annotation in the insurance industry revolutionizes various facets of operations, making tasks more efficient and accurate:

Risk Evaluation: By annotating and extracting contextual data from contracts and forms, text annotation supports risk evaluation, enabling insurance companies to make more informed decisions while minimizing potential risks.

Claims Processing: Annotated data assists in recognizing entities like involved parties and loss amounts, significantly expediting the claims processing workflow. It aids in detecting dubious claims, contributing to fraud detection efforts.

Fraud Detection: Through text annotation, insurance firms can monitor and analyze documents and forms more effectively, enhancing their capabilities to detect fraudulent claims and irregularities.

(III) Banking

The banking sector utilizes text annotation to revolutionize operations and ensure better accuracy and customer satisfaction:

Fraud Identification: Text annotation techniques aid in identifying potential fraud and money laundering patterns, allowing banks to take proactive measures and ensure security.

Custom Data Extraction: Annotated text facilitates the extraction of critical information from contracts, improving workflows and ensuring compliance. It enables efficient data extraction for various attributes like loan rates and credit scores, supporting compliance monitoring.

(IV) Government

In government operations, text annotation facilitates various tasks, ensuring better efficiency and compliance:

Regulatory Compliance: Text annotation streamlines financial operations by ensuring regulatory compliance through advanced analytics . It helps maintain compliance standards more effectively.

Document Classification: Through text classification and annotation, different types of legal cases can be categorized, ensuring efficient document management and access to digital documents.

Fraud Detection & Analytics: Text annotation assists in the early detection of fraudulent activities by utilizing linguistic annotation, semantic annotation, tone detection , and entity recognition. It enables analytics on vast amounts of data for insights.

(V) Logistics

Text annotation in logistics plays a pivotal role in handling massive volumes of data and improving customer experiences:

Invoice Annotation: Annotated text assists in extracting crucial details such as amounts, order numbers, and names from invoices. It streamlines billing and invoicing processes.

Customer Feedback Analysis: By utilizing sentiment and entity annotation, logistics companies can analyze customer feedback, ensuring better service improvements and customer satisfaction.

(VI) Media and News

Text annotation's role in the media industry is indispensable for content categorization and credibility:

Content Categorization: Annotation is crucial for categorizing news content into various segments such as sports, education, government, etc., enabling efficient content management and retrieval.

Entity Recognition: Annotating entities like names, locations, and key phrases in news articles aids in information retrieval and fact-checking. It contributes to credibility and accurate reporting.

Fake News Detection: Utilizing text annotation techniques such as NLP annotation and sentiment analysis enables the identification of fake news by analyzing the credibility and sentiment of the content.

These comprehensive applications across sectors showcase how text annotation significantly impacts various industries, making operations more efficient, accurate, and streamlined.

4. Text Annotation Guidelines

Annotation guidelines serve as a comprehensive set of instructions and rules for annotators when labeling or annotating text data for machine learning tasks. These guidelines are crucial as they define the objectives of the modeling task and the purpose behind the labels assigned to the data. They are crafted by a team familiar with the data and the intended use of the annotations.

Starting with defining the modeling problem and the desired outcomes, annotation guidelines cover various aspects:

(i) Annotation Techniques: Guidelines may start by choosing appropriate annotation methods tailored to the specific problem being addressed.

(ii) Case Definitions: They define common and potentially ambiguous cases that annotators might encounter in the data, along with instructions on how to handle each scenario.

(iii) Handling Ambiguity: Guidelines include examples from the data and strategies to deal with outliers, ambiguous instances, or unusual cases that might arise during annotation.

Text Annotation Workflow

An annotation workflow typically consists of several stages:

(i) Curating Annotation Guidelines: Define the problem, set the expected outcomes, and create comprehensive guidelines that are easy to follow and revisit.

(ii) Selecting a Labeling Tool: Choose appropriate text annotation tools, considering options like Labellerr or other available tools that suit the task's requirements.

(iii) Defining Annotation Process: Create a reproducible workflow that encompasses organizing data sources, utilizing guidelines, employing annotation tools effectively, documenting step-by-step annotation processes, defining formats for saving and exporting annotations, and reviewing each labeled sample.

(iv) Review and Quality Control: Regularly review labeled data to prevent generic label errors, biases, or inconsistencies. Multiple annotators may label the same samples to ensure consistency and reduce interpretational bias. Statistical measures like Cohen's kappa statistic can assess annotator agreement to identify and address discrepancies or biases in annotations.

Ensuring a streamlined flow of incoming data samples, rigorous review processes, and consistent adherence to annotation guidelines are crucial for generating high-quality labeled datasets for machine learning models. Regular monitoring and quality checks help maintain the reliability and integrity of the annotated data.

5. Text Annotation Tools and Technologies

Text annotation tools play a vital role in preparing data for AI and machine learning, particularly in natural language processing (NLP) applications. These tools fall into two main categories: open-source and commercial offerings. Open-source tools, available at no cost, are customizable and widely used in startups and academic projects for their affordability. Conversely, commercial tools offer advanced functionalities and support, making them suitable for large-scale and enterprise-level projects.

Commercial Text Annotation Tools

(i) labellerr.

Labellerr is a text annotation tool that provides high-quality and accurate text annotations for training AI models at scale. The tool, Labellerr, offers various features and services tailored to text annotation needs.

Labellerr boasts the following functionalities and services:

Text Annotation Features:

(i) Sentiment Analysis: Identifies sentiments and emotions in text, categorizing statements as positive, negative, or neutral.

(ii) Summarization: Highlights key sentences or phrases within text to create a summarized version.

(iii) Translation: Translates selected text segments into different languages, such as English to French or German to Italian.

(iv) Named-Entity Recognition: Tags named entities (e.g., ID, Name, Place, Price) in text based on predefined categories.

(v) Text Classification: Classifies text by assigning appropriate classes based on their content.

(vi) Question Answering: Matches questions with their respective answers to train models for generating accurate responses.

Automated Workflows:

(i) Customization: Allows users to create custom automated data workflows, collaborate in real-time, perform QA reviews, and gain complete visibility into AI operations.

(ii) Pipeline Management: Enables the creation and automation of text labeling workflows, multiple user roles, review cycles, inter-annotator agreements, and various annotation stages.

Text Labeling Services:

(i) Provides professional text annotators and linguists focused on ensuring quality and accuracy in annotations.

(ii) Offers fully managed services, allowing users to concentrate on other important aspects while delegating text annotation tasks.

Labellerr emerges as a comprehensive and versatile commercial text annotation tool that streamlines the process of annotating large text datasets for AI model training purposes. It provides a wide array of annotation capabilities and customizable workflows, catering to diverse text annotation requirements.

(II) SuperAnnotate

SuperAnnotate is an advanced text annotation tool designed to facilitate the creation of high-quality and accurate annotations essential for training top-performing AI models. This tool offers a wide array of features and functionalities aimed at streamlining text annotation processes for various industries and use cases.

Key Features of SuperAnnotate's Text Annotation Tool:

Cloud Integrations: Supports integration with various cloud storage systems, allowing users to easily add items from their cloud repositories to the SuperAnnotate platform.

Versatile Use Cases: Encompasses all use cases, ensuring its applicability across different industries and scenarios.

Advanced Annotation Tools: Equipped with an array of advanced tools tailored for efficient text annotation.

Functionalities Offered by SuperAnnotate:

Sentiment Analysis: Capable of identifying sentiments expressed in text, determining whether statements are positive, negative, or neutral, and even detecting emotions like happiness or anger.

Summarization: Annotations can focus on key sentences or phrases within text, aiding in the creation of summarized versions.

Translation Assistance: Annotations assist in identifying elements for translation, such as sentences, terms, and specific entities.

Named-Entity Recognition: Detects and classifies named entities within text, sorting them into predefined categories like dates, locations, names of individuals, and more.

Text Classification: Assigns classes to texts based on their content and characteristics.

Question Answering: Enables the pairing of questions with corresponding answers to train models for generating accurate responses.

Efficiency-Boosting Features:

Token Annotation: Splits texts into units using linguistic knowledge, ensuring seamless and accurate annotation.

Classify All: Instantly assigns the same class to every occurrence of a word or phrase in a text, enhancing efficiency.

Quality-Focused Elements:

Collaboration System: Involves stakeholders in the quality review process through comments, fostering seamless collaboration and task distribution.

Status Tracking: Provides visibility into the status of items and projects, allowing users to track progress effectively.

Detailed Instructions: Sets a solid foundation for project execution by offering comprehensive project instructions to the team.

(III) V7 Labs

The V7 Text Annotation Tool is a feature within the V7 platform that facilitates the annotation of text data within images and documents. This tool automates the process of detecting and reading text from various types of visual content, including images, photos, documents, and videos.

Key features and steps associated with the V7 Text Annotation Tool include:

Text Scanner Model : V7 has incorporated a public Text Scanner model within its Neural Networks page. This model is designed to automatically detect and read text within images and documents.

Integration into Workflow : Add a model stage to the workflow under the Settings page of your dataset. Select the Text Scanner model from the dropdown list and map the newly created text class. If desired, enable the Auto-Start option to automatically process new images through the model at the beginning of the workflow.

Automatic Text Detection and Reading : Once set up, the V7 Text Annotation Tool will automatically scan and read text from different types of images, including documents, photos, and videos. The tool is extensively pre-trained, enabling it to interpret characters that might be challenging for humans to decipher accurately.

Overall, the V7 Text Annotation Tool streamlines the process of text annotation by leveraging a pre-trained model to automatically detect and read text within visual content, providing an efficient and accurate solution for handling text data in images and documents.

Open Source Text Annotation Tools

(i) piaf platform.

• Led by Etalab, this tool aims to create a public Q&A dataset in French.
• Initially designed for question/answer annotation, it allows users to write questions and highlight text segments that answer them.
• Offers an easy installation process and collaborative annotation capabilities.
• Export annotations in the format of the Stanford SQuAD dataset.
• Limited to question/answer annotation but has potential for adaptation to other use cases like sentiment analysis or named entity recognition.

(II) Label Studio

• Free and open-source tool suitable for various tasks like natural language processing, computer vision, and more.
• Highly scalable and configurable labeling interface.
• Provides templates for common tasks (sentiment analysis, named entities, object detection) for easy setup.
• Allows exporting labeled data in multiple formats, compatible with learning algorithms.
• Supports collaborative annotation and can be deployed on servers for simultaneous annotation by multiple collaborators.

(III) Doccano

• Originally designed for text annotation tasks and recently extended to image classification, object detection, and speech-to-text annotations.
• Offers local installation via pip, supporting SQLite3 or PostgreSQL databases for saving annotations and datasets.
• Docker image available for deployment on various cloud providers.
• Simple user interface, collaborative features, and customizable labeling templates.
• Allows importing datasets in various formats (CSV, JSON, fastText) and exporting annotations accordingly.

These open-source tools provide valuable solutions for annotating text data, with each tool having its unique features and suitability for specific annotation tasks. While PIAF is focused on Q&A datasets in French, Label Studio offers extensive customization, and Doccano supports diverse annotation tasks, expanding beyond text to cover image and speech annotations.

Open-source NLP Service Toolkits

• spaCy : A Python library designed for production-level NLP tasks. While not a standalone annotation tool, it's often used with tools like Prodigy or Doccano for text annotation.
• NLTK (Natural Language Toolkit) : A popular Python platform that provides numerous text-processing libraries for various language-related tasks. It can be combined with other tools for text annotation purposes.
• Stanford CoreNLP : A Java-based toolkit capable of performing diverse NLP tasks like part-of-speech tagging, named entity recognition, parsing, and coreference resolution. It's typically used as a backend for annotation tools.
• GATE (General Architecture for Text Engineering) : An extensive open-source toolkit equipped with components for text processing, information extraction, and semantic annotation.
• Apache OpenNLP : A machine learning-based toolkit supporting tasks such as tokenization, part-of-speech tagging, entity extraction, and more. It's used alongside other tools for text annotation.
• UIMA (Unstructured Information Management Architecture) : An open-source framework facilitating the development of applications for analyzing unstructured information like text, audio, and video. It's used in conjunction with other tools for text annotation.

Commercial NLP Service Platforms

• Amazon Comprehend : A machine learning-powered NLP service offering entity recognition, sentiment analysis, language detection, and other text insights. APIs facilitate easy integration into applications.
• Google Cloud Natural Language API : Provides sentiment analysis, entity analysis, content classification, and other NLP features. Part of Google Cloud's Machine Learning APIs.
• Microsoft Azure Text Analytics : Offers sentiment analysis, key phrase extraction, language detection, and named entity recognition among its text processing capabilities.
• IBM Watson Natural Language Understanding : Utilizes deep learning to extract meaning, sentiment, entities, relations, and more from unstructured text. Available through IBM Cloud with REST APIs and SDKs for integration.
• MeaningCloud : A text analytics platform supporting sentiment analysis, topic extraction, entity recognition, and classification across multiple languages through APIs and SDKs.
• Rosette Text Analytics : Provides entity extraction, sentiment analysis, relationship extraction, and language identification functionalities across various languages. Can be integrated into applications using APIs and SDKs.

6. Challenges in Text Annotation

AI and ML companies face numerous hurdles in text annotation processes. These encompass ensuring data quality, efficiently handling large datasets, mitigating annotator biases, safeguarding sensitive information, and scaling operations as data volumes expand. Tackling these issues is crucial to achieving precise model training and robust AI outcomes.

(i) Ambiguity

This occurs when a word, phrase, or sentence holds multiple meanings, leading to inconsistencies in annotations. Resolving such ambiguities is vital for accurate machine learning model training. For instance, the phrase "I saw the man with the telescope" can be interpreted in different ways, impacting annotation accuracy.

(ii) Subjectivity

Annotating subjective language, containing personal opinions or emotions, poses challenges due to differing interpretations among annotators. Labeling sentiment in customer reviews can vary based on annotators' perceptions, resulting in inconsistencies in annotations.

(iii) Contextual Understanding

Accurate annotation relies on understanding the context in which words or phrases are used. Failing to consider context, such as the dual meaning of "bank" referring to a financial institution or a river side, can lead to incorrect annotations and hinder model performance.

(iv) Language Diversity

The need for proficiency in multiple languages poses challenges in annotating diverse datasets. Finding annotators proficient in less common languages or dialects is difficult, leading to inconsistencies in annotations and proficiency levels among annotators.

(v) Scalability

Annotating large volumes of data is time-consuming and resource-intensive. Handling increasing data volumes demands more annotators, posing challenges in efficiently scaling annotation efforts.

Hiring and training annotators and investing in annotation tools can be expensive. The significant investment required in the data labeling market emphasizes the challenge of balancing accurate annotations with the associated costs for AI and machine learning implementation.

7. The Future of Text Annotation

Text annotation, an integral part of data annotation, is experiencing several future trends that align with the broader advancements in data annotation processes. These trends are likely to shape the landscape of text annotation in the coming years:

(i) Natural Language Processing (NLP) Advancements

With the rapid progress in NLP technologies, text annotation is expected to witness the development of more sophisticated tools that can understand and interpret textual data more accurately. This includes improvements in sentiment analysis, entity recognition, named entity recognition, and other text categorization tasks.

(ii) Contextual Understanding

Future trends in text annotation will likely focus on capturing contextual understanding within language models. This involves annotating text with a deeper understanding of nuances, tone, and context, leading to the creation of more context-aware and accurate language models.

(iii) Multilingual Annotation

As the demand for multilingual AI models grows, text annotation will follow suit. Future trends involve annotating and curating datasets in multiple languages, enabling the training of AI models that can understand and generate content in various languages.

(iv) Fine-grained Annotation for Specific Applications

Industries such as healthcare, legal, finance, and customer service are increasingly utilizing AI-driven solutions. Future trends will involve more fine-grained and specialized text annotation tailored to these specific domains, ensuring accurate and domain-specific language models.

(v) Emphasis on Bias Mitigation

Recognizing and mitigating biases within text data is crucial for fair and ethical AI. Future trends in text annotation will focus on identifying and mitigating biases in textual datasets to ensure AI models are fair and unbiased across various demographics and social contexts.

(vi) Semi-supervised and Active Learning Approaches

To optimize annotation efforts, future trends in text annotation might include the integration of semi-supervised and active learning techniques. These methods intelligently select the most informative samples for annotation, reducing the annotation workload while maintaining model performance.

(vii) Privacy-Centric Annotation Techniques

In alignment with broader data privacy concerns, text annotation will likely adopt techniques that ensure the anonymization and protection of sensitive information within text data, balancing the need for annotation with privacy preservation.

(viii) Enhanced Collaboration and Crowdsourcing Platforms

Similar to other data annotation domains, text annotation will benefit from collaborative and crowdsourced platforms that allow distributed teams to annotate text data efficiently. These platforms will offer improved coordination, quality control mechanisms, and scalability.

(ix) Continual Learning and Adaptation

As language evolves and new linguistic patterns emerge, text annotation will evolve towards continual learning paradigms. This will enable AI models to adapt and learn from ongoing annotations, ensuring they remain relevant and up-to-date.

(x) Explainable AI through Annotation

Text annotation may involve creating datasets that facilitate the development of explainable AI models. Annotations focused on explaining decisions made by AI systems can aid in building transparent and interpretable language models.

These future trends in text annotation are driven by the evolving nature of AI technology, the increasing demands for more accurate and specialized AI models, ethical considerations, and the need for scalable and efficient annotation processes.

The exploration of text annotation highlights its crucial role in AI's language understanding. This journey revealed:

(i) Text annotation is vital for AI to interpret human language nuances across industries like healthcare, finance, and more.

(ii) Challenges in annotation, like dealing with ambiguity and subjectivity, stress the need for ongoing innovation.

(iii) The best practices and guidelines for text annotation and various available text annotation tools.

(iv) The future promises advancements in language processing, bias mitigation, and contextual understanding.

Overall, text annotation is a cornerstone in AI's language comprehension, fostering innovation and laying the groundwork for seamless human-machine communication in the future.

Frequently Asked Questions

1. what is text annotation & why is it important.

Text annotation enriches raw text by labeling entities, sentiments, parts of speech , etc. This labeled data trains AI models for better language understanding. It's crucial for improving accuracy in tasks like sentiment analysis, named entity recognition, and more. Annotation aids in creating domain-specific AI models and standardizing data, facilitating precise human-AI interactions.

2. What are the different types of annotation techniques?

Annotation techniques involve labeling different aspects of text data for training AI models. Types include Entity Annotation (identifying entities), Sentiment Annotation (labeling emotions), Intent Annotation (categorizing purposes), Linguistic Annotation (marking grammar), Relation Extraction, Coreference Resolution, Temporal Annotation , and Speech Recognition Annotation .

These techniques are vital for training models in various natural language processing tasks, aiding accurate comprehension and response generation by AI systems.

3. What is in-text annotation?

In-text annotation involves adding labels directly within the text to highlight attributes like phrases, keywords, or sentences. These labels guide machine learning models. Quality in-text annotations are essential for building accurate models as they provide reliable training data for AI systems to understand and process language more effectively.

Book our demo with one of our product specialist

Sign up for more like this.

• Productivity
• Thoughtful learning

Annotating text: The complete guide to close reading

As students, researchers, and self-learners, we understand the power of reading and taking smart notes . But what happens when we combine those together? This is where annotating text comes in.

Annotated text is a written piece that includes additional notes and commentary from the reader. These notes can be about anything from the author's style and tone to the main themes of the work. By providing context and personal reactions, annotations can turn a dry text into a lively conversation.

Creating text annotations during close readings can help you follow the author's argument or thesis and make it easier to find critical points and supporting evidence. Plus, annotating your own texts in your own words helps you to better understand and remember what you read.

This guide will take a closer look at annotating text, discuss why it's useful, and how you can apply a few helpful strategies to develop your annotating system.

What does annotating text mean?

Text annotation refers to adding notes, highlights, or comments to a text. This can be done using a physical copy in textbooks or printable texts. Or you can annotate digitally through an online document or e-reader.

Generally speaking, annotating text allows readers to interact with the content on a deeper level, engaging with the material in a way that goes beyond simply reading it. There are different levels of annotation, but all annotations should aim to do one or more of the following:

• Summarize the key points of the text
• Identify evidence or important examples
• Make connections to other texts or ideas
• Think critically about the author's argument
• Make predictions about what might happen next

When done effectively, annotation can significantly improve your understanding of a text and your ability to remember what you have read.

What are the benefits of annotation?

There are many reasons why someone might wish to annotate a document. It's commonly used as a study strategy and is often taught in English Language Arts (ELA) classes. Students are taught how to annotate texts during close readings to identify key points, evidence, and main ideas.

In addition, this reading strategy is also used by those who are researching for self-learning or professional growth. Annotating texts can help you keep track of what you’ve read and identify the parts most relevant to your needs. Even reading for pleasure can benefit from annotation, as it allows you to keep track of things you might want to remember or add to your personal knowledge management system .

Annotating has many benefits, regardless of your level of expertise. When you annotate, you're actively engaging with the text, which can help you better understand and learn new things . Additionally, annotating can save you time by allowing you to identify the most essential points of a text before starting a close reading or in-depth analysis.

There are few studies directly on annotation, but the body of research is growing. In one 2022 study, specific annotation strategies increased student comprehension , engagement, and academic achievement. Students who annotated read slower, which helped them break down texts and visualize key points. This helped students focus, think critically , and discuss complex content.

Annotation can also be helpful because it:

• Allows you to quickly refer back to important points in the text without rereading the entire thing
• Helps you to make connections between different texts and ideas
• Serves as a study aid when preparing for exams or writing essays
• Identifies gaps in your understanding so that you can go back and fill them in

The process of annotating text can make your reading experience more fruitful. Adding comments, questions, and associations directly to the text makes the reading process more active and enjoyable.

Be the first to try it out!

We're developing ABLE, a powerful tool for building your personal knowledge, capturing information from the web, conducting research, taking notes, and writing content.

How do you annotate text?

There are many different ways to annotate while reading. The traditional method of annotating uses highlighters, markers, and pens to underline, highlight, and write notes in paper books. Modern methods have now gone digital with apps and software. You can annotate on many note-taking apps, as well as online documents like Google Docs.

While there are documented benefits of handwritten notes, recent research shows that digital methods are effective as well. Among college students in an introductory college writing course, those with more highlighting on digital texts correlated with better reading comprehension than those with more highlighted sections on paper.

No matter what method you choose, the goal is always to make your reading experience more active, engaging, and productive. To do so, the process can be broken down into three simple steps:

• Do the first read-through without annotating to get a general understanding of the material.
• Reread the text and annotate key points, evidence, and main ideas.
• Review your annotations to deepen your understanding of the text.

Of course, there are different levels of annotation, and you may only need to do some of the three steps. For example, if you're reading for pleasure, you might only annotate key points and passages that strike you as interesting or important. Alternatively, if you're trying to simplify complex information in a detailed text, you might annotate more extensively.

The type of annotation you choose depends on your goals and preferences. The key is to create a plan that works for you and stick with it.

Annotation strategies to try

When annotating text, you can use a variety of strategies. The best method for you will depend on the text itself, your reason for reading, and your personal preferences. Start with one of these common strategies if you don't know where to begin.

• Questioning: As you read, note any questions that come to mind as you engage in critical thinking . These could be questions about the author's argument, the evidence they use, or the implications of their ideas.
• Summarizing: Write a brief summary of the main points after each section or chapter. This is a great way to check your understanding, help you process information , and identify essential information to reference later.
• Paraphrasing: In addition to (or instead of) summaries, try paraphrasing key points in your own words. This will help you better understand the material and make it easier to reference later.
• Connecting: Look for connections between different parts of the text or other ideas as you read. These could be things like similarities, contrasts, or implications. Make a note of these connections so that you can easily reference them later.
• Visualizing: Sometimes, it can be helpful to annotate text visually by drawing pictures or taking visual notes . This can be especially helpful when trying to make connections between different ideas.
• Responding: Another way to annotate is to jot down your thoughts and reactions as you read. This can be a great way to personally engage with the material and identify any areas you need clarification on.

Combining the three-step annotation process with one or more strategies can create a customized, powerful reading experience tailored to your specific needs.

ABLE: Zero clutter, pure flow

Carry out your entire learning, reflecting and writing process from one single, minimal interface. Focus modes for reading and writing make concentrating on what matters at any point easy.

7 tips for effective annotations

Once you've gotten the hang of the annotating process and know which strategies you'd like to use, there are a few general tips you can follow to make the annotation process even more effective.

1. Read with a purpose. Before you start annotating, take a moment to consider what you're hoping to get out of the text. Do you want to gain a general overview? Are you looking for specific information? Once you know what you're looking for, you can tailor your annotations accordingly.

2. Be concise. When annotating text, keep it brief and focus on the most important points. Otherwise, you risk annotating too much, which can feel a bit overwhelming, like having too many tabs open . Limit yourself to just a few annotations per page until you get a feel for what works for you.

3. Use abbreviations and symbols. You can use abbreviations and symbols to save time and space when annotating digitally. If annotating on paper, you can use similar abbreviations or symbols or write in the margins. For example, you might use ampersands, plus signs, or question marks.

4. Highlight or underline key points. Use highlighting or underlining to draw attention to significant passages in the text. This can be especially helpful when reviewing a text for an exam or essay. Try using different colors for each read-through or to signify different meanings.

5. Be specific. Vague annotations aren't very helpful. Make sure your note-taking is clear and straightforward so you can easily refer to them later. This may mean including specific inferences, key points, or questions in your annotations.

6. Connect ideas. When reading, you'll likely encounter ideas that connect to things you already know. When these connections occur, make a note of them. Use symbols or even sticky notes to connect ideas across pages. Annotating this way can help you see the text in a new light and make connections that you might not have otherwise considered.

7. Write in your own words. When annotating, copying what the author says verbatim can be tempting. However, it's more helpful to write, summarize or paraphrase in your own words. This will force you to engage your information processing system and gain a deeper understanding.

These tips can help you annotate more effectively and get the most out of your reading. However, it’s important to remember that, just like self-learning , there is no one "right" way to annotate. The process is meant to enrich your reading comprehension and deepen your understanding, which is highly individual. Most importantly, your annotating system should be helpful and meaningful for you.

Engage your learning like never before by learning how to annotate text

Learning to effectively annotate text is a powerful tool that can improve your reading, self-learning , and study strategies. Using an annotating system that includes text annotations and note-taking during close reading helps you actively engage with the text, leading to a deeper understanding of the material.

Try out different annotation strategies and find what works best for you. With practice, annotating will become second nature and you'll reap all the benefits this powerful tool offers.

I hope you have enjoyed reading this article. Feel free to share, recommend and connect 🙏

Connect with me on Twitter 👉   https://twitter.com/iamborisv

And follow Able's journey on Twitter: https://twitter.com/meet_able

And subscribe to our newsletter to read more valuable articles before it gets published on our blog.

Now we're building a Discord community of like-minded people, and we would be honoured and delighted to see you there.

Straight from the ABLE team: how we work and what we build. Thoughts, learnings, notes, experiences and what really matters.

Read more posts by this author

follow me :

Learning with a cognitive approach: 5 proven strategies to try

What is knowledge management the answer, plus 9 tips to get started.

Managing multiple tabs: how ABLE helps you tackle tab clutter

What is abstract thinking? 10 activities to improve your abstract thinking skills

0 results found.

• Aegis Alpha SA
• We build in public

Building with passion in

Text Annotation for Natural Language Processing – A Comprehensive Guide

Explore the pivotal role of text annotation in shaping NLP algorithms as we walk you through diverse types of text annotation, annotation tools, case studies, trends, and industry applications. The comprehensive guide throws insights into the Human-in-the-loop approach in text annotation.

Text annotation is a crucial part of natural language processing (NLP), through which textual data is labeled to identify and classify its components. Essential for training NLP models, text annotation involves tasks like named entity recognition, sentiment analysis, and part-of-speech tagging. By providing context and meaning to raw text, it plays a central role in enhancing the performance and accuracy of NLP applications.

Text annotation is not just a technical requirement, but a foundation for the growing NLP market, which witnessed a turnover of over $12 billion in 2020. According to Statista, the market for NLP is projected to grow at a compound annual growth rate (CAGR) of about 25% from 2021 to 2025.

Recent studies have shown that around two-thirds of NLP systems fail after they are put to use. The primary reason for this failure is their inability to deal with the complex data encountered outside of testing environments, highlighting the importance of high-quality text annotation .

Challenges in annotating text for NLP projects

Popular text annotation techniques, process of annotating text for nlp, how does hitl (human-in-the-loop) approach help, how ai companies benefit from text annotation for domain-based ai apps, types of text annotation in nlp and their effective use cases, text annotation tools, the future of text annotation.

Text annotation is a critical step in preparing data for Natural Language Processing (NLP) systems, which rely heavily on accurately labeled datasets. However, it faces many challenges ranging from data volumes and speed to consistency and data security.

• Volume of Data: NLP projects often require large datasets to be effective. Annotators face the daunting task of labeling vast amounts of text, which can be time consuming and mentally taxing. For instance, a project aimed at understanding customer sentiment might need to process millions of product reviews. This sheer volume can lead to fatigue, affecting the quality of the annotation.
• Speed of Production: In our fast-paced digital world, the speed at which text data is produced and needs to be processed is staggering. Social media platforms generate enormous amounts of data daily. Annotators are under pressure to work quickly, which can sometimes compromise the accuracy and depth of annotation. This need for speed can also lead to burnout among annotators.
• Resource Intensiveness: Text annotation is often a time-consuming and labor-intensive process. It requires a significant amount of human effort, which can be costly and inefficient, especially for large datasets.
• Scalability: As the amount of data increases, scaling the annotation process efficiently while maintaining quality is a major challenge. Automated tools can help, but they often require human validation to ensure accuracy.
• Ambiguity in Language: Natural language is inherently ambiguous and context-dependent. Capturing the correct meaning, especially in cases of idiomatic expressions, sarcasm, or context-specific usage, can be difficult. This ambiguity can lead to challenges in ensuring that the annotations accurately reflect the intended meaning.
• Language and Cultural Diversity: Dealing with multiple languages and cultural contexts increases the complexity of annotation. It’s challenging to ensure that annotators understand the nuances of different languages and cultural references.
• Domain-Specific Knowledge: Certain NLP applications require domain-specific knowledge (such as legal, medical, or technical fields). Finding annotators with the right expertise can be difficult and expensive.
• Annotation Guidelines and Standards: Developing clear, comprehensive annotation guidelines is crucial for consistency. These guidelines must be regularly updated and annotators adequately trained, which adds to the complexity and costs.
• Subjectivity in Interpretation: Different annotators may interpret the same text differently. Achieving consensus or a standardized interpretation can be challenging.
• Adaptation to Evolving Language: Language is dynamic and constantly evolving. Keeping the annotation process and guidelines up to date with new slang, terminologies, and language usage patterns is an ongoing challenge.
• Human Bias: Annotators, being human, bring their own perspectives and biases to the task. This can affect how the text is interpreted and labeled. For example, in sentiment analysis, what one annotator might label as a negative sentiment, another might view as neutral. This subjectivity can lead to inconsistencies in the dataset, which in turn can skew the NLP model’s learning and outputs.
• Consistency: Maintaining consistency in annotation across different annotators and over time is a significant challenge. Different interpretations of guidelines, varying levels of understanding, and even changes in annotators’ perceptions over time can lead to inconsistent labeling. Inconsistent annotations can confuse the NLP models, leading to poor performance.
• Data Security: Annotators often work with sensitive data, which might include personal information. Ensuring the security and privacy of this data is paramount. Data breaches can have serious consequences, not just for the individuals whose data is compromised, but also for the organizations handling the data. Annotators and their employers must adhere to strict data protection protocols, adding another layer of complexity to their work.

Get your solutions to text annotation challenges.

In Natural Language Processing (NLP), the method of text annotation plays a pivotal role in shaping the effectiveness of the technology. Understanding the different text annotation techniques is crucial for selecting the most appropriate method for a given project and address the regular challenges generally involved in them. Here are three primary annotation techniques: Manual, Automated, and Semi-Automated Annotation, each with its unique attributes and applications.

By leveraging these different annotation techniques, organizations and researchers can tailor their approach to suit the specific needs and constraints of their NLP projects, balancing factors like accuracy, speed, and cost-effectiveness.

Confused about what type of text annotation meets your project needs?

Text annotation in NLP is a systematic process in which raw text data is methodically labeled to identify specific linguistic elements, such as entities, sentiments, and syntactic structures. This process not only aids in the training of NLP models, but also significantly improves their ability to understand and process natural language. The stages in this process, from data collection to building an effective annotation team, are crucial for ensuring high-quality data annotation and, consequently, superior model performance in NLP applications.

This comprehensive table encapsulates the entire process of text annotation for NLP, providing a clear roadmap from the initial stages of data collection to the integration of annotated data with machine learning models.

The Human-in-the-Loop (HITL) approach significantly enhances AI-driven data annotation by integrating human expertise into the AI workflow, thereby ensuring greater accuracy and quality. This collaborative technique addresses the limitations of AI, enabling it to navigate complex data more effectively. Key benefits of the HITL approach in text annotation for NLP include:

• Improved Accuracy and Quality: Human experts are better at understanding ambiguous and complex data, allowing them to identify and correct errors that automated systems might overlook. This is particularly beneficial in scenarios involving rare data or languages with limited examples, where machine learning algorithms alone may struggle.
• Enhanced Contextual Understanding: Humans bring nuanced judgment and contextual knowledge to the annotation process, crucial for tasks requiring subjective interpretations, such as sentiment analysis. This human involvement ensures more precise and meaningful labeling of data.
• Edge Case Resolution: HITL is valuable in addressing challenging edge cases that require human judgment and reasoning, which are often difficult for AI to handle accurately. Human annotators can ensure that these rare or complex instances are correctly labeled, enhancing the reliability and performance of the AI models trained on this data.
• Continuous Improvement: The HITL approach facilitates an iterative feedback loop, where human annotators provide insights and feedback to improve automated systems. This collaboration leads to ongoing refinements in the accuracy and quality of annotations over time.
• Active Learning and Querying: HITL systems can use active learning techniques, where the model queries humans for annotations on uncertain or challenging examples, thereby focusing human effort on the most informative instances. This optimizes the annotation process and improves annotation accuracy while reducing overall effort.
• Quality Control: Human annotators adhere to specific quality control measures and guidelines, ensuring that annotations meet the desired standards. Techniques like involving a third-party annotator for consensus or employing consensus-building strategies among multiple annotators enhance the reliability and reduce the impact of individual biases .

HabileData leverages the HITL approach in text annotation and combines the strengths of human intelligence and AI capabilities, resulting in more reliable, accurate, and contextually nuanced NLP models. This synergy is pivotal in advancing the effectiveness of AI-driven data annotation, particularly in complex, ambiguous, or highly subjective annotation tasks.

Text annotation in NLP is essential for training AI to understand and process language in various industries, enhancing domain-specific applications:

Text annotation involves categorizing and labeling text data, which is crucial for training NLP models. Each type of annotation serves a specific purpose and finds unique applications in various industries.

Entity Annotation: This involves identifying and labeling specific entities in the text, such as names of people, organizations, locations, and more.

Use cases in NLP

• In healthcare, it’s used to extract key patient information from clinical documents, aiding in patient care and research.
• In legal contexts, it helps in identifying and organizing pertinent details like names, dates, and legal terms from vast documents.
• Useful for extracting company names and financial terms from business reports for market analysis.

Entity Linking: This process connects entities in the text to a larger knowledge base or other entities.

• In journalism, it enriches articles by linking people, places, and events to related information or historical databases.
• In financial analysis, it can link company names to their stock profiles or corporate histories.

Text Classification: This involves categorizing text into predefined groups or classes.

• In customer support, it’s used to sort customer inquiries into categories like complaints, queries, or requests, streamlining the response process.
• In content management, it helps in organizing and classifying articles, blogs, and other written content by topics or themes.

Sentiment Annotation: This type of annotation identifies and categorizes the sentiment expressed in a text segment as positive, negative, or neutral.

• In market research, it’s widely used to analyze customer feedback on products or services.
• In social media monitoring, it helps in gauging public sentiment towards events, brands, or personalities.
• In ecommerce, it is used to evaluate customer feedback to assess product satisfaction levels.

Linguistic Annotation: This adds information about the linguistic properties of the text, such as syntax (sentence structure) and semantics (meaning).

• In language learning applications, it provides detailed grammatical analysis to aid language comprehension.
• For text-to-speech systems, it helps in understanding the context for accurate pronunciation and intonation.

Part-of-Speech (POS) Tagging: This involves labeling each word with its corresponding part of speech, such as noun, verb, adjective, etc.

• In search engines, it assists in parsing queries to deliver more relevant results.
• In content creation, it aids in keyword optimization for SEO purposes.
• In transcription, it is used to enhance voice recognition systems by tagging words in speech transcripts for more accurate context understanding.

Document Classification: Similar to text classification, but on a broader scale, it categorizes entire documents.

• In legal tech, it assists in sorting various legal documents into categories such as ‘contracts’, ‘briefs’, or ‘judgments’ for easier retrieval and analysis.
• In academic research, it aids in organizing scholarly articles and papers by fields and topics.

Coreference Resolution: This identifies when different words or phrases refer to the same entity in a text.

• In news aggregation, it’s crucial for linking different mentions of the same person, place, or event across multiple articles.
• In literature analysis, it helps in tracking characters and themes throughout a narrative.

These examples showcase how text annotation empowers various NLP applications, enhancing their functionality and utility across different domains.

How HabileData nailed text annotation for a German construction company

A Germany-based construction technology company sought to enhance its in-house construction leads data platform for sharing comprehensive construction project data across USA and Europe. Their clientele ranged from small businesses to Fortune 500 companies in the real estate and construction sectors. The company used automated crawlers to gather real-time data on construction projects, which was auto-classified into segments like property type, project dates, location, size, cost, and phases.

However, for accuracy and to append missing information, they partnered with HabileData to verify, validate, and manually annotate 20% of the data that couldn’t be auto-classified.

The project involved comprehending and extracting relevant information from articles, tagging this information based on categories like project size and location, and managing large volumes of articles within a tight 24-hour timeline.

The HabileData team conducted an in-depth assessment of the client’s needs, received domain-specific training, and carried out a rigorous two-step quality check on the classified data. Over 10,000 construction-related articles were processed with effective text annotation techniques , significantly improving the accuracy of the AI algorithms used by the company. This collaboration led to enhanced AI model performance, a 50% cost reduction on the project, and a superior customer experience.

Other than understanding the HITL approach, it is crucial to also understand the tools and software that facilitate this process. Text annotation tools are specialized software designed to streamline the labeling of textual data for NLP applications.

Text annotation tools provide an interface for annotators to label data efficiently. These tools often support various annotation types, such as entity recognition, sentiment analysis, and part-of-speech tagging. They range from simple, user-friendly platforms to more advanced systems that offer automation and integration capabilities.

Popular text annotation tools

• Prodigy: A highly interactive and user-friendly tool, Prodigy allows for efficient manual annotation. It supports active learning and is particularly useful in iterative annotation processes.
• Labelbox: This tool is known for its ability to handle large datasets. Labelbox offers a combination of manual and semi-automated annotation features, making it suitable for projects of varying complexity.
• spaCy: spaCy is not just a text annotation tool, but a full-fledged NLP library. It provides functionalities for both annotation and building NLP models, suitable for projects requiring the integration of annotation and model training.

Choosing the right text annotation tool

Selecting an appropriate text annotation tool depends on several factors:

• Project Size and Complexity: For large-scale projects, tools like Labelbox that handle high volumes efficiently are preferable. For more complex annotation tasks, Prodigy with its active learning capabilities, may be more suitable.
• Annotation Type: Different tools excel in different types of annotations. It’s important to choose a tool that aligns well with the specific annotation needs of your project.
• Integration Needs: If integration with other NLP tools or model training is a requirement, spaCy could be an ideal choice.
• Budget and Resource Availability: Some tools are more cost-effective than others and require varying levels of expertise to operate effectively.

The choice of text annotation tools plays a critical role in the efficiency and effectiveness of the text annotation process in NLP projects. The selection should be tailored to the specific needs of the project, considering factors like project scope, annotation requirements, and available resources.

Recent advancements in NLP have introduced important trends, such as transfer learning, where a model trained for one task is repurposed for a related task, thus requiring less labeled data. The introduction of machine learning models like GPT and advancements in BERT and ELMo models have revolutionized the understanding of word context in NLP. Additionally, the emergence of low-code/no-code tools has democratized NLP, enabling non-technical users to perform tasks previously limited to data scientists.

As we look toward the future of text annotation in NLP, several key developments are poised to shape this evolving field:

• Advancements in AI-Powered Annotation Tools: Future annotation tools are expected to be more sophisticated, leveraging AI to a greater extent. This could include enhanced automation capabilities, better context understanding, and more efficient handling of large datasets.
• Enhanced Guidelines and Standards: There will probably be a push toward more standardized and universally accepted annotation guidelines, which will help in improving the consistency and quality of annotated data across different projects and domains.
• The Role of Synthetic Data in Annotation: Synthetic data generation is an emerging area that could revolutionize text annotation. By creating artificial yet realistic text data, it offers the potential to train NLP models in more diverse scenarios, reducing reliance on labor-intensive manual annotation.

These developments indicate a future in which text annotation becomes more efficient, accurate, and adaptable, significantly impacting the capabilities and applications of NLP technologies.

Text annotation plays a vital role in the field of Natural Language Processing (NLP), acting as the backbone for training and improving NLP models. From the initial stages of data collection and preparation to the detailed processes of annotation workflow, quality control, and integration with machine learning models, each step is crucial for ensuring the effectiveness and accuracy of NLP applications.

The future of text annotation, marked by advancements in AI-powered tools, enhanced guidelines, and the utilization of synthetic data, points toward a more efficient and sophisticated landscape. The key takeaway is that, as NLP continues to evolve, the importance of meticulous and advanced text annotation processes will become increasingly important, shaping the future capabilities of AI in understanding and processing human language.

Experience the power of precision in your text annotation projects.

About Author

Snehal Joshi heads the business process management vertical at HabileData , the company offering quality data processing services to companies worldwide. He has successfully built, deployed and managed more than 40 data processing management, research and analysis and image intelligence solutions in the last 20 years. Snehal leverages innovation, smart tooling and digitalization across functions and domains to empower organizations to unlock the potential of their business data.

Related Articles

10 Best Data Cleansing Companies in 2024

5 Images Annotation Types Explained with Common Use Cases

• Privacy Policy

Copyright © 2024 HabileData. All Rights Reserved.

Email:  [email protected] Phone:  +91-794-000-3251

Disclaimer:   

HitechDigital Solutions LLP and HabileData will never ask for money or commission to offer jobs or projects. In the event you are contacted by any person with job offer in our companies, please reach out to us at [email protected]

Writers' Center

Eastern Washington University

Reading and Study Strategies

What is annotating and why do it, annotation explained, steps to annotating a source, annotating strategies.

• Using a Dictionary
• Study Skills

[ Back to resource home ]

[email protected] 509.359.2779

Cheney Campus   JFK Library Learning Commons

Stay Connected!

inside.ewu.edu/writerscenter Instagram  Facebook

Helpful Links

Software for Annotating

ProQuest Flow (sign up with your EWU email)

FoxIt PDF Reader

Adobe Reader Pro  - available on all campus computers

Track Changes in Microsoft Word

What is Annotating?

Annotating is any action that deliberately interacts with a text to enhance the reader's understanding of, recall of, and reaction to the text. Sometimes called "close reading," annotating usually involves highlighting or underlining key pieces of text and making notes in the margins of the text. This page will introduce you to several effective strategies for annotating a text that will help you get the most out of your reading.

Why Annotate?

By annotating a text, you will ensure that you understand what is happening in a text after you've read it. As you annotate, you should note the author's main points, shifts in the message or perspective of the text, key areas of focus, and your own thoughts as you read. However, annotating isn't just for people who feel challenged when reading academic texts. Even if you regularly understand and remember what you read, annotating will help you summarize a text, highlight important pieces of information, and ultimately prepare yourself for discussion and writing prompts that your instructor may give you. Annotating means you are doing the hard work while you read, allowing you to reference your previous work and have a clear jumping-off point for future work.

1. Survey : This is your first time through the reading

You can annotate by hand or by using document software. You can also annotate on post-its if you have a text you do not want to mark up. As you annotate, use these strategies to make the most of your efforts:

• Include a key or legend on your paper that indicates what each marking is for, and use a different marking for each type of information. Example: Underline for key points, highlight for vocabulary, and circle for transition points.
• If you use highlighters, consider using different colors for different types of reactions to the text. Example: Yellow for definitions, orange for questions, and blue for disagreement/confusion.
• Dedicate different tasks to each margin: Use one margin to make an outline of the text (thesis statement, description, definition #1, counter argument, etc.) and summarize main ideas, and use the other margin to note your thoughts, questions, and reactions to the text.

Lastly, as you annotate, make sure you are including descriptions of the text as well as your own reactions to the text. This will allow you to skim your notations at a later date to locate key information and quotations, and to recall your thought processes more easily and quickly.

• Next: Using a Dictionary >>
• Last Updated: Apr 25, 2024 2:50 PM
• URL: https://research.ewu.edu/writers_c_read_study_strategies

Help | Advanced Search

Computer Science > Computation and Language

Title: text annotation handbook: a practical guide for machine learning projects.

Abstract: This handbook is a hands-on guide on how to approach text annotation tasks. It provides a gentle introduction to the topic, an overview of theoretical concepts as well as practical advice. The topics covered are mostly technical, but business, ethical and regulatory issues are also touched upon. The focus lies on readability and conciseness rather than completeness and scientific rigor. Experience with annotation and knowledge of machine learning are useful but not required. The document may serve as a primer or reference book for a wide range of professions such as team leaders, project managers, IT architects, software developers and machine learning engineers.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

BibTeX formatted citation

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

• Our Purpose
• Our Routines
• Video Library
• Professional Development
• Planning Guide: Micro-Planning, Responsiveness & Goals
• Micro-Routines: Printable Micro-Routines Index
• ·: Launch the School Year
• ·: Develop Ideas with Peers
• ·: Make Connections
• ·: Focus on Evidence
• ·: Vocabulary Retention
• ·: Reflect on Learning
• ·: Having Fun with Micro-Routines
• ·: Independent Habits
• ·: Peer Review

Reading: Annotation

• Reading: React Aloud
• Reading: OPTIQ
• Reading: Recall to Relate
• Reading: Backstory
• Discourse: Think Pair Start
• Discourse: Silent Graffiti
• Reading: Zoom In & Out
• Writing: Sort Connect Reflect
• Discourse: Silent Carousel
• Discourse: Reporter
• Writing: Extract
• Writing: Growing Statements
• Discourse: Paraphrase
• Writing: Thought Bubbles
• Discourse: Socratic Seminar
• Remote Learning Guide: Instructional Routines
• Group Routine: Word Hunt
• Group Routine: In Here, Out There
• Planning: Vocabulary Retention Guide
• Lesson Planning: Accessible Lesson Planning
• Planning Guide: Student Centered Learning
• Engagement: Weird Ice Breakers
• Planning Guide: CRT and the Brain
• Planning Guide: Facilitation Moves
• Cross-Content: Brain Breaks
• Planning Guide: Curiosity and Cognition
• Planning Guide: Leveraging Reading and Mistakes
• Planning Guide: Formative Assessment
• Planning Guide: Exit Slips through Bloom's Taxonomy
• Digital Tool: 2021-2022 Teacher Planning Notebook
• Remote Learning: Hybrid Resources
• Find Resources

Annotation Protocol

Read closely to determine what the text says explicitly and to make logical inferences from it; cite specific textual evidence when writing or speaking to support conclusions drawn from the text.

Determine central ideas or themes of a text and analyze their development; summarize the key supporting details and ideas.

Analyze how and why individuals, events, or ideas develop and interact over the course of a text.

Delineate and evaluate the argument and specific claims in a text, including the validity of the reasoning as well as the relevance and sufficiency of the evidence.

Interpret words and phrases as they are used in a text, including determining technical, connotative, and figurative meanings, and analyze how specific word choices shape meaning or tone.

Analyze the structure of texts, including how specific sentences, paragraphs, and larger portions of the text (e.g., a section, chapter, scene, or stanza) relate to each other and the whole.

Assess how point of view or purpose shapes the content and style of a text.

Instructional Routines

Teacher Guide: Annotation Protocol

Teacher Feedback

Annotation Protocol for Textbook Enrichment with Prerequisite Knowledge Graph

• Original research
• Open access
• Published: 21 September 2023
• Volume 29 , pages 197–228, ( 2024 )

Cite this article

You have full access to this open access article

• Chiara Alzetta   ORCID: orcid.org/0000-0002-7850-9611 1 ,
• Ilaria Torre   ORCID: orcid.org/0000-0003-1159-2833 2 &
• Frosina Koceva 3  

896 Accesses

Explore all metrics

Extracting and formally representing the knowledge embedded in textbooks, such as the concepts explained and the relations between them, can support the provision of advanced knowledge-based services for learning environments and digital libraries. In this paper, we consider a specific type of relation in textbooks referred to as prerequisite relations (PR). PRs represent precedence relations between concepts aimed to provide the reader with the knowledge needed to understand a further concept(s). Their annotation in educational texts produces datasets that can be represented as a graph of concepts connected by PRs. However, building good-quality and reliable datasets of PRs from a textbook is still an open issue, not just for automated annotation methods but even for manual annotation. In turn, the lack of good-quality datasets and well-defined criteria to identify PRs affect the development and validation of automated methods for prerequisite identification. As a contribution to this issue, in this paper, we propose PREAP, a protocol for the annotation of prerequisite relations in textbooks aimed at obtaining reliable annotated data that can be shared, compared, and reused in the research community. PREAP defines a novel textbook-driven annotation method aimed to capture the structure of prerequisites underlying the text. The protocol has been evaluated against baseline methods for manual and automatic annotation. The findings show that PREAP enables the creation of prerequisite knowledge graphs that have higher inter-annotator agreement, accuracy, and alignment with text than the baseline methods. This suggests that the protocol is able to accurately capture the PRs expressed in the text. Furthermore, the findings show that the time required to complete the annotation using PREAP are significantly shorter than with the other manual baseline methods. The paper includes also guidelines for using PREAP in three annotation scenarios, experimentally tested. We also provide example datasets and a user interface that we developed to support prerequisite annotation.

Similar content being viewed by others

Knowledge Annotation for Intelligent Textbooks

Constructing an Educational Knowledge Graph with Concepts Linked to Wikipedia

Constructing Low-Redundant and High-Accuracy Knowledge Graphs for Education

Avoid common mistakes on your manuscript.

1 Introduction

Textbooks play a central role in the learning process despite the recent worldwide growth of distant learning, possibly because they provide deep knowledge about a subject and help consolidate learning outcomes (Carvalho et al., 2018 ). Their availability in academic digital libraries and repositories provides learners with the opportunity to access them at lower cost (Eighmy-Brown et al., 2017 ) and exploit further services (Atkinson, 2020 ). Regarding the latter, recent advances in artificial intelligence and natural language processing have opened up possibilities for automating the extraction of knowledge embedded in educational textbooks. Specifically, our focus is on extracting concepts and prerequisite relations (PRs) between them. Together, these components form a graph that represents the content structure (Wang et al., 2016 ; Lu et al., 2019 ). We refer to it as the prerequisite knowledge graph (PR graph, for short), i.e., a graph composed of concepts as nodes and PRs as edges.

The availability of datasets annotated with PRs can support the development of supervised methods for prerequisite learning and can also support semi-supervised approaches and the evaluation of non-machine learning methods. However, the existing literature on prerequisite relations lacks high-quality resources and well-defined annotation criteria. As a result, the datasets generated are difficult to compare and reuse, and often show low inter-annotator agreement scores (Chaplot et al., 2016 ; Gordon et al., 2016 ; Fabbri et al., 2018 ). This issue could be addressed by adopting annotation protocols for PRs since they would provide specifications on the annotation criteria and rationale, along with guidelines for their application (Fort et al., 2011 ; Pustejovsky and Stubbs, 2012 ). Furthermore, we intend to address the lack of approaches that rely solely on the text for the annotation of RPs, without relying on the prior knowledge of the annotators. Such an approach would allow for annotations that faithfully reflect the content of the textbook, as current methods rely heavily on the annotators’ knowledge of the subject matter.

Our research aims to tackle these challenges by designing an annotation protocol that addresses the following goals :

Designing a knowledge engineering procedure for the annotation of prerequisites and the creation of PR datasets, with the aim of reducing the ambiguity of the annotation task and thus achieving more reliable and consistent datasets;

Implementing a textbook-driven annotation procedure aimed to annotate concepts and prerequisites based solely on the content of the text, rather than relying on the annotator’s domain knowledge. By adopting an in-context annotation approach, we seek to explicitly identify the instructional design principles that underlie the organization of content in the textbook, specifically identifying which concepts serve as prerequisites for others.

To achieve these goals, we designed PREAP (PRErequisite Annotation Protocol) using an iterative design methodology. We evaluated the final version of the protocol in a mixed quantitative-qualitative study involving education experts. The study aimed to answer the following Research Questions (RQs):

- RQ1: to what extent PREAP succeeds in obtaining PR-annotated datasets that are reliable in terms of completeness and accuracy (the former intended as the extent to which the annotations cover the relevant information and the latter as the correctness and precision of PRs);

- RQ2: to what extent PREAP succeeds in obtaining textbook-driven annotations.

The contribution we make in this paper lies in the following points:

A knowledge engineering procedure for prerequisite annotations that led to an increased agreement between annotators and higher accuracy compared to existing methods in the literature;

A novel methodology that binds PR annotation to the textbook in order to make explicit and annotate not only the content of the textbook but also the underlying structure of prerequisites.

In addition to these methodological contributions, we also provide resources that are publicly available on GitHub: the dataset resulted from a case study annotation project described in this paper, a tool for the annotation and analysis of PRs, and the recommendations for applying the protocol in different annotation scenarios.

The remainder of the paper is organised as follows. In Sect.  2 we review related works on prerequisite annotation. Section  3 introduces PREAP protocol, focusing on its design process and annotation principles, while Sect.  4 presents PREAP evaluation. Section  5 describes the application of the protocol in an annotation project case study. Section  6 extends the case study, comparing the datasets produced with three options of the protocol and using them to train a machine learning (ML) system for automatic prerequisite extraction. Section  7 concludes the paper and Sect.  8 describes the datasets and the other shared resources.

2 Related Work and Background

The content of educational texts such as textbooks is typically structured and presented according to instructional design principles that authors intuitively or deliberately apply (Gagne, 1962 ; Ruiz-Primo, 2000 ; Council, 2000 ). For example, arithmetic and algebra textbooks typically introduce the concept of “addition” before explaining “multiplication” as it is useful to refer to the former when introducing the latter. Thus, “addition” can be said a prerequisite of “multiplication” from a teaching point of view.

As in (Liang et al., 2017 ), we define a prerequisite relation (PR) as a binary dependency relation connecting a prerequisite and a target concept where the former is a concept that has to be known in order to understand the latter . In other words, the prerequisite concept provides the prior knowledge required to understand the target concept. The set of PRs in a textbook can be represented as a knowledge graph, resembling Educational Concept Maps (Novak et al., 2008 ) as sketched in Fig.  1 , where concepts are nodes and edges represent prerequisite relations between them. The edge in the graph, for instance, between concept B (e.g., “addition”) and C (e.g., “multiplication”), is read as B is prerequisite of C (“addition is prerequisite of multiplication”, \({B \prec C}\) ).

Example of PR knowledge graph representing arithmetic concepts as nodes and their PRs as edges. Dashed edges represent transitive PRs. The label of edges is “prerequisite", e.g., A is prerequisite of B, A is prerequisite of D

2.1 Concepts and Prerequisite Relations

The term “ concept ” refers in general terms to an abstract and general idea conceived in the mind (Carey, 2009 ). Given such a broad definition, the nature of concepts is a matter of debate in many fields. We refer to “ concepts ” similarly to other works in the literature on prerequisite annotation (Talukdar and Cohen, 2012 ; Wang et al., 2016 ; Liang et al., 2017 ; Pan et al., 2017a ; Zhou & Xiao, 2019 ; Adorni et al., 2019 ; Alzetta et al., 2019 ; Limongelli et al., 2015 ; Xiao et al., 2022 ) that basically associate concepts to terms, intended as lexical units composed of single or multiple words with unambiguous sense in the given context. Similarly to (Chau et al., 2020b ; Wang et al., 2021 ), we identify terms representing concepts from an educational text as a subset of words therein (more precisely, noun phrases) that convey a domain-specific meaning Cabré ( 1999 ). This perspective borrows from the approaches in terminology research, according to which the terminology of a domain provides as many lexical units as there are concepts in its subspace (Sager, 1990 ), and also from information extraction, which addresses automatic keyword extraction (Augenstein et al., 2017 ; Shen et al., 2014 ; Martinez-Rodriguez et al., 2020 ). Computational linguistics and natural language processing specifically tackle keyword extraction from unstructured resources, that is, text, defining either (i) pattern-based linguistic approaches, which employ syntactic parsing to identify domain terms among short noun phrases in the text Faure and Nedellec ( 1999 ); Hippisley et al. ( 2005 ); Golik et al. ( 2013 ), or (ii) statistical approaches, that assign a termhood degree to words by relying on distributional properties Suresu and Elamparithi ( 2016 ); Rani et al. ( 2017 ); Zhao and Zhang ( 2018 ) or on sentence-level contextual information Cimiano and Völker ( 2005 ); Velardi et al. ( 2013 ); Dell’Orletta et al. ( 2014 ). Only a few works have considered textbooks as a source for extracting concepts Wang et al. ( 2015 ); Labutov et al. ( 2017 ).

In a prerequisite relation , the concepts involved are referred to as the prerequisite concept and target concepts respectively, meaning that the prerequisite concept must be understood before the target concept (Liang et al., 2017 , 2019 ). According to Hübscher ( 2001 ), the term “prerequisite” has at least two meanings. First, it signifies a pedagogical relationship between two elements that a student should learn. Secondly, it indicates a formal mechanism that can be used to partially order two instructional units (such as concepts, pages, exercises, or similar) into a sequence. Early studies in instructional design (Gagne, 1962 ; Ausubel et al., 1968 ; Carey, 1999 ; Merrill, 2002 ) emphasized the significance of prior knowledge in the process of learning new concepts. These studies proposed that learning occurs in a sequential manner, building upon existing knowledge.

This pedagogical perspective paved the way for representing educational content in the form of graph and concept map structures. Graph structures inherently represent interlinked concepts and are easily exploited for computer-based applications (Novak, 1990 ; Gruber, 1993 ). For example, in automatic lesson plan generation, graph structures enable the inclusion of multiple paths between components to accommodate students’ needs and interests (Brusilovsky & Vassileva, 2003 ; Yu et al., 2021 ).

2.2 Prerequisite Annotated Datasets

A prerequisite annotated dataset is a collection of concept pairs where the information concerning the presence or absence of a PR is explicitly indicated by assigning a ‘prerequisite’ or ‘non-prerequisite’ label to each pair (Wang et al., 2016 ; Chaplot et al., 2016 ; Gordon et al., 2016 ). PR graphs usually display only “prerequisite” edges, as in Fig.  1 . These PR-annotated datasets serve two main purposes: training and testing ML algorithms (Gasparetti et al., 2018 ; Liang et al., 2018 ; Li et al., 2019 ) and evaluating PR extraction methods against a gold dataset (Liang et al., 2015 ; Adorni et al., 2019 ). Ultimately, the aim of PR datasets is to serve as knowledge bases for developing advanced services (Talukdar and Cohen, 2012 ; Liang et al., 2019 ; Changuel et al., 2015 ). This demands reliable and quality PR-annotated datasets. However, the availability of high-quality datasets annotated with PRs between educational concepts is limited, due to the insufficient accuracy of automatically created ones, the high effort required for their manual construction, and the shortage of reliable and systematic annotation procedures. Even more critical is the fact that existing datasets vary with respect to the annotated items and the annotation principles. In fact, PRs can concern prerequisite relations between university courses (Yang et al., 2015 ; Liang et al., 2017 ; Li et al., 2019 ), MOOCs (Chaplot et al., 2016 ; Pan et al., 2017a ; Roy et al., 2019 ; Zhao et al., 2020 ), MOOC videos (Pan et al., 2017c ; Huang et al., 2021 ; Wen et al., 2021 ; Xiao et al., 2021 ), learning objects (Gasparetti, 2022 ), scientific databases (Gordon et al., 2017 ) or Wikipedia pages (Talukdar and Cohen, 2012 ; Gasparetti et al., 2018 ; Miaschi et al., 2019 ; Zhou & Xiao, 2019 ; Sayyadiharikandeh et al., 2019 ; Bai et al., 2021 ; Hu et al., 2021 ), all represented as PR relations between concept pairs. Alternatively, concepts can be relevant domain terms acquired from a text, as in (Wang et al., 2016 ; Lu et al., 2019 ; Adorni et al., 2019 ; Alzetta et al., 2019 ; Chau et al., 2020b ; Wang et al., 2021 ), and in our approach.

2.3 PR Annotation

Automated methods. The most used methods for the automatic identification of PRs are based on relational metrics (Liang et al., 2015 ; Adorni et al., 2019 ) and machine learning approaches. (Talukdar and Cohen, 2012 ; Liang et al., 2019 ; Manrique et al., 2018 ; Gasparetti, 2022 ; Xiao et al., 2021 ). Among ML approaches, we distinguish between approaches exploiting link-based features (Gasparetti et al., 2018 ; Wen et al., 2021 ), text-based features (Miaschi et al., 2019 ; Alzetta et al., 2019 ), or a combination of the two (Liang et al., 2018 ; Hu et al., 2021 ). The former refers to ML approaches that exploit the structure of the source text provided by links, in the sense of connections, between concepts and portions of contents (e.g., Wikipedia graph of categories, DBpedia links, organization in sections and paragraphs, etc.), while the approaches exploiting text-based features use only features from the raw text (e.g., bag-of-words and word embeddings).

The most widely used and effective methods, such as RefD (Liang et al., 2015 ), rely on external knowledge. Recently, the task has been addressed employing neural language models (Angel et al., 2020 ; Li et al., 2021 ; Bai et al., 2021 ). However, automatic methods for concepts and PR extraction are generally still not good enough to be used in knowledge-based services for learning support (Chau et al., 2020b ) and still need gold datasets for evaluation, thus manual annotation is still a crucial task in this field.

Manual methods. Manual PR annotation is commonly carried out by recruiting domain experts (Liang et al., 2015 , 2018 ; Fabbri et al., 2018 ) or graduate students (Wang et al., 2015 ; Pan et al., 2017b ; Zhou & Xiao, 2019 ) to annotate all pairwise combinations of predefined concepts (Chaplot et al., 2016 ; Wang et al., 2016 ; Li et al., 2019 ; Zhou & Xiao, 2019 ) or a random sample of that set (Pan et al., 2017c ; Gordon et al., 2017 ; Gasparetti et al., 2018 ). Asking annotators to autonomously create concept pairs based on their domain knowledge, as in Lu et al. ( 2019 ), is less common. These strategies aim to identify PRs between concepts in the given domain without accounting for concept organization in the text. In fact, annotators generally rely on their prior domain knowledge (Talukdar and Cohen, 2012 ; Chaplot et al., 2016 ; Wang et al., 2016 ; Li et al., 2019 ), at most checking dubious cases on a given collection of documents (Gordon et al., 2016 ), and not on a specific text. In PREAP we use an approach called textbook-driven in-context annotation of PR relations between pairs of concepts, that takes into account how concepts are organized in the annotated text. Differently from the approaches that annotate PR relations in a given domain unbounded from a specific text, this approach does not fit the goal of developing intelligent tutoring systems (ITS) that can be used regardless of the textbook chosen. Conversely, textbook-driven annotation is thought to produce training and testing datasets for NLP tools that are mostly used to extract information from corpora, since it is essential to feed these models with training examples that can be associated with a text passage written in natural language. The two approaches can be said to be complementary. In Sect.  4 we will compare datasets produced by using the PREAP approach against datasets produced through pairwise combinations of predefined concepts, showing that the former approach not only better expresses the content explained in the text, as expected, but also improves the coherence and consistency between annotations produced by different annotators.

Indeed, PR-annotated datasets frequently report low annotation agreement and performance variability of systems trained on such data (Chaplot et al., 2016 ; Gordon et al., 2016 ; Fabbri et al., 2018 ; Alzetta et al., 2020 ), possibly due to the lack of reproducible procedures for creating them. In fact, although properly defining an annotation task is vital to reduce annotation inconsistencies (Ide & Pustejovsky, 2017 ), the available PR-annotated datasets are mostly poorly documented, and annotation guidelines tend to be absent or fairly basic, mostly relying on a naive definition of prerequisite relation. PREAP tries to fill this gap in PR literature as it defines a systematic procedure for annotating educational texts: we could not find any other knowledge engineering procedure for prerequisite annotation and PR dataset creation, while methods exist for the mere task of concept annotation, including a recent one from (Wang et al., 2021 ).

Additionally, to improve the documentation of the released PR datasets, PREAP recommends that they are published and described following the principles of the Linked Data paradigm, a W3C standard for sharing machine-readable interlinked data on the Web using standard vocabularies and unique identification of resources (URI/IRI). The linked data approach has not been used much in PR annotation, while it is very common in other types of annotation. We freely distribute our datasets described accordingly.

3 PREAP Protocol: Design and Description

3.1 design of preap protocol.

The design of the PREAP protocol for manual annotation of prerequisite relations has been guided by the MATTER framework (Pustejovsky, 2006 ; Ide & Pustejovsky, 2017 ), which defines an iterative methodology to obtain annotated corpora for machine learning tasks. We took into account in particular the recommendations for model and annotation definition.

Figure  2 shows the process that led to the definition of the PREAP protocol. The Goals definition block in the figure represents the initial input for the overall iterative design of the protocol. The goals have been defined through the experience and groundwork (Adorni & Koceva, 2016 ; Alzetta et al., 2018 ; Alzetta et al., 2019 ; Adorni et al., 2019 ; Alzetta et al., 2020a , 2020b ) that guided toward the identification of the goals stated in the Introduction in Sect.  1 .

Iterative design of PREAP annotation protocol

The central part of the figure shows the four-step cycle for the definition of the protocol: definition-testing-evaluation-revision . The first step, protocol definition , includes input decisions for that cycle (i.e., annotation and revision methods). It is followed by testing , which involves the annotation task performed by annotators according to the annotation protocol and the resulting annotated dataset . The third step is evaluation , where both the annotation process and the datasets are evaluated using quantitative (inter-agreement and dataset analysis) and qualitative (focus group with annotators) methods in order to identify unclear instructions. The outcome of the evaluation drove the revision of the protocol and the start of a new cycle. For consistency, the annotation tasks for each cycle were performed on the same introductory computer science textbook (Brookshear & Brylow, 2015 ). The annotators involved in the annotation tasks were four master’s students in Computer Science, different in each cycle.

The current version of the PREAP protocol, presented in Sect.  3.2 , is the result of three iterative cycles that lasted about two years. As a final step, PREAP underwent an Evaluation with experts in Education reported in Sect.  4 .

To support the testing and evaluation phases, we built a tool, PRAT, described in Sect.  5 . PRAT provides an interface for manual annotation of prerequisites and facilities for quantitative and visual analysis.

3.2 Description of PREAP Protocol

The main principle addressed in PREAP is the textbook-driven annotation approach: annotations are anchored to the text portion where a PR relation occurs between concepts. As a result, the application of PREAP results in the creation of a gold-PR dataset (or gold dataset , for short). This dataset is annotated with PR relations following the systematic annotation procedure defined by PREAP. The dataset can be represented as a PR graph whose nodes are the concepts explained in the textbook and the edges are the prerequisite relations between the concepts expressed in the text. The dataset can be directly employed in services for augmented textbooks that demand high-quality manual annotation, or as ground truth data for the development and evaluation of automated methods for prerequisite extraction.

To attain a gold dataset, the person or the team managing the annotation project must set up and coordinate the set of tasks shown in Fig.  3 . PREAP provides specifications for dealing with all the tasks: (1) Project Management, (2) Annotation Task, (3) Annotation Revision, and (4) Post–annotation procedures for gold dataset creation.

PREAP tasks to carry out a PR Annotation Project

The Project Management task concerns supervising the whole project and making decisions especially regarding task (4). Tasks (2) and (3) are performed by the annotators recruited for the specific annotation project. Below we describe the main points of each task.

3.2.1 Project Management

The design and setup of an annotation project are handled by the manager(s) of the annotation project (Fort et al., 2011 ). As shown in Fig.  3 , the decisions concern: (i) annotation goals and features (what the annotation project is intended for, domain and language); (ii) corpus (textbook) selection and tool to be used for the annotation, if any; (iii) annotators recruitment and training (selecting annotators with adequate expertise to properly understand the textbook content; setting up a trial task to train the annotators and then assess their understanding of the guidelines); (iv) coordination of all the other tasks described below.

3.2.2 Annotation Task

The text annotation phase, performed by annotators and supervised by the annotation manager, is the core part of a PR annotation project. The annotation recommendations are systematised within the Annotation Manual which comprises two complementary resources: the Annotation Guidelines (AG), describing how the annotation process should be carried on, and a list of Knowledge Elicitation Questions (KEQ), aimed at clarifying dubious cases through questions and examples. Prior to the actual annotation, both AG and KEQ should be given to annotators in a trial annotation task where the manager(s) of the project can check whether the annotators interpreted the instructions correctly. Training annotators is recommended in annotation projects to reduce the biases caused by annotators’ background knowledge and subjective interpretation of the task instructions (Hovy & Lavid, 2010 ). The manual remains accessible to the annotators throughout the entire annotation process.

As shown in Fig. 3 , the Annotation task encompasses the concept and prerequisite annotation subtasks.

(i) Concept annotation. The Annotation Manual provides a definition of what should be regarded as concept in the annotation task and it also provides examples in order to increase the reliability of the identification: therein, concepts are described as the building blocks of learning, namely what a student should understand in order to comprehend a subject matter. Depending on the topic and detail level of the given textbook, concepts can be general (e.g., algebra, geometry, mathematics etc.) or very specific (e.g., radius, integer multiplication, fraction denominator). Either way, they are domain terms represented in texts as lexical entities (more precisely, noun phrases) constituted by a single or multi-word term.

In PREAP, the identification of domain concepts in the text (see Fig. 3 ) can be carried out in two ways: autonomously or simultaneously with the prerequisite annotation task, based on the project management decisions. In the former approach, the list of concepts, i.e., the terminology , can be obtained through manual extraction or (semi)automated extraction approaches. In these cases, the work of Chau et al. ( 2020a ) proved the benefit of including the evaluation of a domain expert to refine the list of concepts. Footnote 1 Alternatively, in the simultaneous approach, the identification of domain concepts is performed by the annotators alongside the task of prerequisite annotation. This option seems appealing for saving time. However, it is likely to result in less consistent annotations and lower agreement, as shown in our experimental tests reported in Sect.  6 , which thus demand heavy revision and time consumption. Hence, our recommendation is to adopt this option carefully, e.g., when obtaining a rich although less consistent annotation complies with the project goal.

(ii) Prerequisite annotation. The in-context annotation approach of PREAP requires annotators to perform the annotation of prerequisite relations while reading the educational text. This implies identifying PRs based on the explanations provided by the textbook rather than relying on the annotator’s background knowledge about the topic. Differently from existing PR datasets (ref. Sect.  2.2 ), PREAP aims to capture the view of the textbook’s author on which concepts should be presented, and how they should be presented, to allow students to understand the target concepts. This approach is referred to as textbook-driven annotation, as it aligns with the content and organization of the textbook itself.

The specific properties of PRs, as intended in PREAP, should be preserved in the annotation to avoid invalid relations from a structural and semantic point of view. Specifically, PRs are binary relations characterised by the following properties: (i) irreflexive: if A and B show a PR relation, A must be different from B ; (ii) asymmetry: if \({A \prec B}\) , the opposite cannot be true (e.g., if \({\textit{network} \prec \textit{internet}}\) , \({\textit{internet} \prec \textit{network}}\) can’t be true); iii) transitivity: for every A, B, and C, if \({A \prec B}\) and \({B \prec C}\) , then \({A \prec C}\) (e.g., if \({\textit{computer} \prec \textit{network}}\) and \({\textit{network} \prec \textit{internet}}\) , then \({\textit{computer} \prec \textit{internet}}\) ).

Note that, differently from (Chaplot et al., 2016 ; Wang et al., 2016 ; Li et al., 2019 ; Zhou & Xiao, 2019 ), annotators are not required to explicitly annotate non-PR pairs. In the proposed textbook-driven annotation approach, non-annotated transitive relations (dashed edges in Fig.  1 ) remain implicit, but they can be inferred using PR properties. Specifically, transitivity allows retrieving PRs that derive from paths involving intermediate concepts; in addition, asymmetry can be used to infer those non-PRs represented by inverse relations.

Considering the semantic properties of the relation, an extension of PREAP would be accounting for different strengths of PR as a weight assigned by the annotator to each detected relation. Consistently with the PR annotation approach, a strong weight should be assigned if the prerequisite is described in the textbook as absolutely necessary to understand the target concept, while a weak weight could be used to indicate that the prerequisite is useful for a deeper comprehension of the concept but not strictly necessary.

To guide the annotation of PRs, KEQs offer examples of lexical taxonomic relations that can easily subtend PR, such as hyponyms, hypernyms and meronyms, or semantic relations like causal or temporal relations. In fact, the goal of KEQ is to provide examples in order to build a shared understanding of the PR interpretation. The instructions in KEQ for assigning PR weights, are a first draft whose results are still under evaluation and possibly subject to future refinements.

3.2.3 Annotation Revision

Manual annotation is known to be error-prone, as well-recognised in the literature (Fort et al., 2011 ; Dickinson, 2015 ; Wang et al., 2021 ) and also studied in our own work (Alzetta et al. 2020a ). Therefore, PREAP recommends a revision phase ( Annotation revision task in Fig.  3 ) after the annotation task: searching for errors and inconsistencies is aimed at improving the reliability and consistency of the annotations (Plank et al., 2014 ).

In line with the Annotation task, the Revision phase of PREAP consists of two subtasks: Concept annotation revision and Prerequisite annotation revision . For both subtasks, PREAP recommends “in-context revision” in order to comply with the textbook-driven annotation approach.

(i) Concept annotation revision . When concept annotation is conducted autonomously using semi-automatic or automatic extraction tools, it is recommended the support of experts to review the set of concepts. According to Chau et al. ( 2020a ), domain experts are best suited for this task as they provide high-quality annotations, are less burdened by difficult annotation instances and are more capable to spot erroneous automatic annotations than non-experts Lee et al. ( 2022 ). The manager of the annotation project provides both AG and KEQ to the experts so that they can revise the semi(automatically) extracted concepts based on the examples and definitions of the PREAP manual. The validated set can be then provided to the annotators for the annotation of prerequisite relations.

When concept annotation is simultaneous to PR annotation, annotators who earlier identified and annotated the concepts should revise the set using the approach for PR revision that will be explained below.

(ii) Prerequisite annotation revision . To comply with the in-context annotation approach, annotators are required to read again the portion of text where they found a PR relation before making the final decision of approving, excluding or modifying the relation. While reading the textual context, each annotator reconsiders her/his own annotations and checks if the inserted pairs comply with the formal and semantic requirements of prerequisite relations described in the annotation manual. Note that, like PR annotation, PR revision is carried out by each annotator individually.

Since revision is a time-consuming process, a convenience approach to balance the benefit of revision and its cost might be revising only a subset of annotations, specifically PR pairs that are more likely to contain annotation errors, i.e., those with lower agreement. This is because the highest chance of finding errors lies in phenomena that are rarely annotated (Eskin, 2000 ). In this case, the criteria for selecting the PR sample to be checked should not be shared with annotators to avoid biased revisions. The same approach can be used also in the case of simultaneous annotation of concepts and PRs. However, if incorrect concepts are identified, it is necessary to revise all the direct and indirect PRs related to those concepts.

3.2.4 Post-annotation Procedures for Gold Dataset Creation

Once the revision task is completed, the manager(s) of the annotation project has to undertake actions toward the creation of the gold dataset as a result of the combination of the revised annotations. The main actions are shown in Fig.  3 and explained in the following.

Agreement evaluation , using agreement metrics to assess the homogeneity and consistency of annotations produced by different annotators;

Annotations combination , using appropriate combination criteria;

Gold dataset revision after annotations combination (e.g., looking for loops in the resulted PR graph);

Gold dataset release : meta-annotation and documentation, to enable sharing and reuse of the resulted PR graph.

The first three actions are unnecessary if only one annotator has been recruited, although this is generally not recommended to minimize errors.

The use of agreement metrics is recommended to quantify the consistency and homogeneity of annotations produced by different annotators: Footnote 2 while disagreement can be due to multiple factors, as long studied in the literature (Bayerl & Paul, 2011 ), high agreement is generally assumed as an indicator of common understanding of the annotation specifications as well as of the specific phenomenon to annotate (Artstein & Poesio, 2008 ). Thus, in case of low agreement, the annotation manager should check the annotators’ understanding of the annotation specifications and investigate any possible issues with the annotation instructions (Di Eugenio & Glass, 2004 ). Among agreement metrics, pairwise Cohen’s Kappa coefficient ( k ) (Cohen, 1960 ) is a de facto standard for manual annotation evaluation. However, it presents some weaknesses, particularly when dealing with skewed distributions of the phenomena within the annotated set (Di Eugenio & Glass, 2004 ; Byrt et al., 1993 ). Moreover, as traditionally employed, k only accounts for the match between the labels assigned by two annotators to the same item. This means that it does not account for ‘implicit agreement’, i.e. agreement given by the transitive property, specifically relevant to PR annotation. Hence, it is necessary to process the dataset in a way that allows applying k properly. To this aim, we assume that two annotators agree on the PR \({A \prec C}\) in both the following cases: (i) both annotators manually created the pair \({A \prec C}\) ; (ii) one annotator created the pair \({A \prec C}\) and the other created the pairs \({A \prec B}\) and \({B \prec C}\) . Footnote 3 Then, the k metric can be computed as follows: given the terminology T of concepts used during annotation, consider as total items of the annotation task the list P of each pairs-wise combination p of concepts in T , including both \({A \prec B}\) and \({B \prec A}\) in P . For each annotator, consider as positive PR each p that is either manually created by the annotator or that can be derived for the transitive property. Consider p as non-PR otherwise. Then, compute k for each pair of annotators using equation 1 .

probability that a concept pair is annotated as PR or non-PR by both annotators, i.e. the number of concept pairs annotated in the same way in both annotations over all possible concept pairs

probability of agreement occurring by chance, i.e. the probability that a pair is annotated and not annotated as PR.

For the whole group of annotators, use Fleiss’ variant of Cohen k (Fleiss, 1971 ).

Depending on the obtained agreement and the project goals, more or less inclusive annotations combination methods can be chosen. At the two ends, taking the Union \(\cup\) of PRs means including all the PRs identified by the annotators, while taking their Intersection \(\cap\) means including only shared PRs (i.e., PRs detected by all the annotators). In general, when the goal of the PR project is to analyze every case where annotators claim to encounter a relation, it is advisable to use more inclusive combination approaches such as the union. This is particularly relevant when the goal is to discover linguistic patterns in the textual realizations of PRs or when the annotators’ judgments are highly reliable due to their strong domain expertise, assuming that annotation revisions have been carried out. On the other hand, this approach is not recommended with low-experienced annotators and when the annotations revision has not been performed. Less inclusive combination approaches offer higher certainty and guarantee higher consensus about the relations included in the gold dataset. However, they result in more limited datasets, particularly when there is low agreement among annotators.

It is worth noting that, when possible, a good practice consists in discussing among annotators about disagreement cases in order to converge toward an agreed PR graph, as suggested in Wang et al. ( 2021 ) for concept annotation.

The final phases of Revision and Release of the gold dataset will be detailed in Sect.  5 through the description of an annotation project and its meta-annotation using a standard vocabulary, following Linked Data principles.

4 Evaluation of the PREAP Protocol

In this section, we present the final evaluation (lower block of Fig.  2 , Evaluation with education experts ) that we carried out on different domains by comparing five datasets produced using PREAP against datasets obtained through alternative PR annotation methods.

To evaluate if PREAP succeeds in reaching the goals stated in the introduction, we formulated the following Research Questions (RQ).

RQ1: to what extent does PREAP succeed in obtaining PR-annotated datasets that are reliable? Specifically:

RQ1.1: to what extent are PR relations consistently annotated by the annotators?

RQ1.2: to what extent is the gold-PR dataset resulting from the combination of individual PR datasets complete and accurate ?

RQ2: to what extent does PREAP succeed in obtaining textbook-driven annotations, i.e., PR-annotated datasets that match the text in terms of prerequisite concepts used by the textbook’s author to make the reader understand the target concepts?

4.1 Methods

We conducted a mixed-method study based on quantitative and qualitative dimensions for data quality assessment (Zaveri et al., 2013 ), detailed below. These were used to compare the datasets produced using PREAP against datasets annotated by employing alternative approaches, referred to as baseline methods .

To answer RQ1.1 concerning consistency , i.e., the extent to which the dataset does not report conflicting annotations for similar phenomena (Mendes et al., 2012 ), we exploited agreement metrics between manually produced annotations as usual in such cases (Artstein & Poesio, 2008 ; Artstein, 2017 ; Hripcsak & Wilcox, 2002 ).

To answer RQ1.2 concerning completeness (the extent to which the annotations cover the relevant information of the data (Mendes et al., 2012 ; Zaveri et al., 2013 )) and accuracy (the degree of correctness and precision with which the annotation represents information (Zaveri et al., 2013 )) we performed an evaluation where education experts, i.e. teachers in the respective domains and a pedagogist, were asked to evaluate the annotated PR datasets represented as graphs and face-to-face interviewed to discuss the answers.

To answer RQ2 teachers were asked to assess the match between text and PR annotations in their respective domain, by evaluating the adherence between the annotation and the content of the source text, focusing on the way concepts are presented, and relevancy , i.e. the extent to which the annotated data are applicable and helpful for a task at hand (Zaveri et al., 2013 ), in our case learning support. The assessment was followed by a face-to-face interview.

Additionally, in order to obtain a comprehensive comparison of PREAP against the baseline methods, we computed the average completion time required to perform each annotation.

4.1.1 Baseline Annotation Methods

Four PR-annotation methods were used as baselines.

Manual Methods (MMs):

MMP, a Manual Method for concept Pairs annotation of PRs (Li et al., 2019 ). In this method, annotators annotate if a PR exists between all possible pairwise combinations of pre-defined concepts using their background knowledge.

MMT is an adaptation from MMP since we could not find Textbook-driven approaches in the literature. Instead of relying on their background knowledge, annotators are given a text to check if a PR exists between pairs of concepts therein.

Automated Methods (AMs):

RefD (Liang et al., 2015 ), a widely adopted method for PR identification (cf Sect.  2 ), which exploits knowledge external to the text: basically, a PR is found between concepts that result associated from the analysis of links between their corresponding Wikipedia pages;

Burst-based method (Adorni et al., 2019 ) annotates PRs based on the text content. Specifically, it uses Burst Analysis to identify portions of texts where each concept is estimated as relevant and then exploits temporal patterns between them to find concept pairs showing a PR.

4.1.2 Source Texts and Participants

For the annotation task, we used five source texts from three domain areas: two texts in mathematics (algebra, statistics), two texts in natural science (biology, biochemistry) and one text in archaeology. Each text was acquired from a textbook targeting undergraduate students not majoring in the field of the book. Footnote 4

We recruited six annotators, two for each domain area (post-graduate level expertise, age range between 25 and 49, AVG=29.8, SD=9.8). For the evaluation, we recruited 12 university teachers, grouped for domain area (age range between 32 and 65, AVG=45.4, SD=12.9), and one pedagogist (senior researcher in Education, age 47).

4.1.3 Study Setup and Procedures

1. Creation of the PR annotated datasets. To ensure a consistent experimental setting, human annotators and automatic methods were provided with the same set of concepts extracted from the source texts as in the semi-automatic autonomous option of PREAP.

PR datasets creation through MMs: the annotators were asked to perform the annotation task using MMP, MMT and PREAP, following the respective annotation procedures, but varying their order to avoid biases. This resulted in 30 individual PR datasets (annotators*methods*domains), then combined using the union option to obtain 15 gold datasets (i.e., one for each method for each domain).

PR datasets creation through Automated Methods (AMs): we implemented the RefD and Burst-based methods as described in the cited references, then we generated the PR datasets using them. This resulted in 10 PR datasets (methods*domains: one dataset for each method for each domain).

Table 1 provides details about the resulting gold datasets. Figure  4 shows a portion of one of the datasets visualized as a PR graph on the PRAT tool.

2. Evaluation with education experts. We organized individual face-to-face meetings with the teachers and the pedagogist. After general instructions, teachers were provided with the set of concepts and the PR graphs obtained using MMs and AMs in their area of expertise. They were given about 1 h, or more when required, to analyze and evaluate the graphs. Later, they were asked to read the source texts and evaluate the graphs according to the dimensions introduced in Sect.  4.1 . Finally, we discussed the answers in an open-ended interview. The average time of each meeting with teachers was 130 min. In a final meeting, all the results were discussed with a pedagogist, commenting on the use of the PR graphs for educational purposes.

Example of a PR graph from PRAT user interface and corresponding textbook portion (Archaeology domain)

4.2 Results

Annotations consistency. To investigate the effect of PREAP on annotations’ consistency (RQ1.1), we measured the inter-annotator agreement using the approach described in Sect.  3.2.4 between the individual PR datasets produced with PREAP and the manual baseline methods. Results show better performance of PREAP compared to the manual methods MMP (AVG +0.98) and MMT (AVG +2.05) (Table 2 , left side).

Completeness and accuracy of the combined PR datasets. To investigate RQ1.2, we relied on the evaluations of teachers. Specifically, completeness is evaluated by detecting the number of PR pairs in common between the datasets produced by the annotation methods and the PRs identified by the teachers. To this aim, teachers were asked to identify the PRs for each concept as as if they had to explain them to a student, drawing a concept map of prerequisites. In this process, they were free to look at the graphs under evaluation and to modify their identified PRs in order to produce their optimal map as in a process of ground truth creation. Then, for each prerequisite in their map, teachers were asked to confirm its presence in the graph being evaluated. In detail: a ‘good’ score is given if a direct or indirect PR exists in the evaluated graph, while an ‘acceptable’ score is given if the two concepts are not linked but their PRs are consistent with the graph. No scores otherwise. Labels are then converted to numbers and combined. The result is normalized by the total number of PRs identified by the teacher and mapped to a five-point scale. Summary results are reported in Fig.  5 a, and detailed data are reported in Appendix. We used the Kruskal–Wallis non-parametric test to check if any significant difference exists among the completeness scores of the six methods, finding that there is a significant difference among the groups (X \(^2(4)\) = 53.98, \(p<.001\) ). Then we used the Post-Hoc Mann Whitney U test for pairwise comparison. We did not find any significant difference between the Manual Methods pairs, while we found that the difference between the MMs and the AMs is significant with \(p<.001\) for each MM-AM pair. The Bonferroni correction ( \(\alpha =0.005\) ) did not change the statistical significance of any of the outcomes above, since all of these have p values \(<.001\) .

Accuracy , as defined in Sect.  4 , is measured by asking teachers to evaluate the correctness of a set of randomly extracted paths of three nodes form each graph. Evaluating paths instead of single pairs is coherent with the definition of PRs characterized by transitivity, and thus relevant to assess accuracy. In detail, if both the PRs in the path are correct then, a ‘very good’ score is given; if one of the PRs is correct and the other weakly wrong, but consistent with the graph, then a ‘good’ score is given; ‘bad’ score is given otherwise. As with the completeness score, labels are converted to numbers, combined, normalized, and mapped to a five-point scale. Results are reported in Fig.  5 b, details on each evaluation test are in Appendix.

Results for a completeness, b accuracy, and c match with the text (PR correspondence)

By performing the same statistical analysis, we found a significant difference among the six methods, according to the Kruskal-Wallis test (X \(^2(4)\) = 64.43, \(p<.001\) ). The Mann Whitney test showed a significant difference in the mean ranks of each Manual Method compared to each Automated Method. Moreover, it revealed a significant difference in PREAP accuracy compared against both MMP and MMT ( \(p<.001\) in both cases) and a difference between MMP and MMT ( \(P=0.0140\) ), while there was no significant difference between the two AMs. Bonferroni correction ( \(\alpha =0.005\) ) did not modify the results except for the difference between MMP and MMT that became not significant.

Match with text. To investigate the effect of PREAP on the correspondence of annotations to the prerequisites expressed in the text (RQ2), teachers were asked to read the source texts of their domain area and write down, for each concept, the prerequisite concepts used to explain it. Then, they were asked to repeat the procedure for completeness assessment, but checking only the first condition, i.e. the existence of the same PRs in the evaluated graphs. Results are reported in Fig.  5 c and in Appendix. We performed the same statistical analysis as above, finding statistical differences among the groups using the Kruskal–Wallis test (X \(^2(4)\) = 65.63, \(p<.001\) ). As well as for accuracy, the pairwise comparison of the methods in terms of ‘match with the text’ showed a significant difference in the mean ranks of each Manual Method compared to each Automated Method. Moreover, it revealed a significant difference in PREAP accuracy compared against both MMP and MMT ( \(p<.001\) in both cases), and also a significant difference between MMP and MMT but with a higher p-value ( \(p=0.0027\) ). No significant difference was found between the two AM methods. Bonferroni correction ( \(\alpha =0.005\) ) did not modify the results.

Concerning the relevancy dimension, based on the interviews conducted, it can be concluded that most of the teachers (9) consider PR graphs tied to the text as a potentially very useful feature for educational purposes. Some teachers (3) argue that usefulness depends on several factors. Additionally, 77% claim that it can support learners, 85% believes that it can be useful for teachers to organize the contents of lectures.

Task completion time. We computed the average time used to annotate each PR dataset generated using PREAP, MMP and MMT. Results show that the average time is lower for PREAP than for the other methods, indicating 43% and 29% less time for PREAP than for MMP and MMT, respectively (Table 2 , right side). We used one-way analysis of variance (ANOVA) to check if the difference between the averages of the three groups was significant. Results revealed statistically significant differences among the three groups, each of equal size (annotators*domains) for each method ( F (2,27)=12.57, n \(^{2}\) =0.48, \(p<.001\) ). After the one-way ANOVA Test, Tukey Test was used as a complementary Post-Hoc analysis for pairwise comparisons. The difference resulted to be significant ( \(p < 0.05\) ) for the pairs PREAP-MMP ( \(p<.001\) ) and PREAP-MMT ( \(p=0.027\) ), while it was not found to be significant for the MMP-MMT pair ( \(p=0.08003\) ).

4.3 Discussion

The evaluation provided a rich source of quantitative and qualitative data. Limiting the analysis to what concerns the research questions of this study, we highlight the following results.

RQ1 was aimed at evaluating the reliability of the datasets annotated using PREAP (Goal1) by considering: the consistency of annotations in terms of inter-annotator agreement (RQ1.1) and the completeness and accuracy of the resulting datasets (RQ1.2). As for RQ1.1, results in the previous section show that the inter-annotator agreement scores obtained on the PREAP datasets are much higher than those obtained relying on the other manual methods. Apart from one case (biology annotated using MMP), the agreement scores obtained using MMP and MMT are generally slight, while they raise to moderate and substantial with PREAP (Landis and Koch, 1977 ). Considering RQ1.2, we can observe that all the manual MM methods perform considerably better than the automatic AM methods, both for completeness and accuracy. Focusing on PREAP against MMs, it appears that the three methods are mostly comparable in terms of completeness of the datasets, while in terms of accuracy PREAP turns out to yield better results. The main reason for the lower accuracy of MMP and MMT is the incoherence of some resulting PR paths. This can be attributed to the requirement of these methods of annotating set of concept pairs, identifying if a PR relation exists or not among the two. This seems to induce annotators to find more relations than necessary. For instance, teachers evaluated as wrong or borderline acceptable, but not good, PR relations between ‘product’ and ‘enzyme’, ‘product’ and ‘activation energy’, that were included in the biochemistry dataset annotated using the MMP method.

RQ2 was aimed to evaluate the textbook-driven annotation approach (Goal2). The results reported in Fig.  5 c. The statistical analysis shows that PREAP-annotated datasets perform better in terms of correspondence between the annotation and the content of the source text. The interview clarified also the errors attributed to each method. In the case of PREAP, the main error reported was false prerequisite concepts mentioned in the explanation of another concept, whereas they were rather supplementary explanations or primary notions. For example, three teachers noted that the sentence ‘Elementary algebra differs from arithmetic in the use of abstractions, such as using letters to stand for numbers’ means that letters and numbers are prerequisites for abstraction but elementary algebra is not a prerequisite for them, as it resulted in the PREAP-annotated dataset. Another example is the sentence ‘A horizontal layer interface will be recorded on a plan which shows the boundary contours of the deposit and, therefore, the limits of the interface’, which made teachers raise concerns about the correctness of boundary contour as the prerequisite of deposit. Relevancy of the annotated datasets was discussed with the teachers and the pedagogist. The aim was to get hints about the value of such PR graphs for educational uses. As seen, almost all considered them a useful support for teachers and most of them for learners. The pedagogist pointed out concerns about its practical direct use with large graphs, suggesting splitting into sub-graphs. It was also observed that graph accuracy is essential for its usefulness ( relevancy ), and that PREAP is the method that most accurately helps to highlight the lesson structure underlying the educational text ( exact correspondence ), also thanks to its higher readability.

Finally, if we look at such results in light of the average completion time required for completing the annotations, we observe that not only PREAP improves annotation consistency, accuracy, and match with the text, but it is also faster than the baseline methods. No specific and recurrent differences have been found across domains for any of the metrics.

5 Annotation Project Case Study

This section presents the annotation project we carried out in the last cycle of development, following PREAP procedures described in Sect.  3.2 , Fig.  3 .

5.1 Project Management

(i) Annotation goals and features: obtaining a gold dataset to be used for linguistic analysis of PR instances and for testing an automatic PR learning system based on linguistic features. While the latter use is presented in Sect.  6 of this paper, the linguistic analysis is left out for space limits.

(ii) Corpus selection and annotation tool: the annotation project relies on the fourth chapter of the computer science textbook (Brookshear & Brylow, 2015 ), ‘Networking and the Internet’ (20,964 tokens distributed along 780 sentences). The chosen tool for supporting PR annotation is the PRAT tool that we developed for PR annotation and analysis.

(iii) Annotators recruitment and training: the project manager recruited four master’s students in Computer Science. Although they were domain experts with regard to the book content, none of them was familiar with annotation procedures or the annotation protocol. Hence, a preliminary training phase was conducted before starting the annotation task. The guidelines in the annotation manual were first explained by the project manager and then tried individually by each annotator in a trial annotation task. Then, annotators compared and commented on their individual annotations in a group discussion to address doubts.

(iv) Coordination of the other tasks: described below.

5.2 Annotation Task

PRAT tool annotation interface

After the training phase, each annotator performed text annotation individually without consulting the other annotators.

(i) Concept annotation. Concept annotation, supervised by the project manager, was performed as an autonomous step with respect to PR annotation, adopting a semi-automatic approach. Specifically, the text underwent linguistic analysis Footnote 5 and semi-automatic terminology extraction through the Text-To-Knowledge 2 platform (Dell’Orletta et al., 2014 ). The platform returned a list of 185 candidate terms, then manually revised according to PREAP guidelines in order to remove non-concepts (e.g., busy channel, own network, term gateway, same machine) and add missing ones (e.g., router). The ultimate result was a terminology T of 140 concepts. The lists of automatically extracted and revised concepts are available among the shared resources. Note that Sect.  6 discusses different concept annotation options.

(ii) Prerequisite annotation. The PR annotation was carried out on PRAT tool. As shown in Fig.  6 , the “Text” area displays the text and highlights the concepts of T (also listed in the upper part of the “Concepts” area). To create a PR pair, the expert selected the occurrence of the target concept in the text and entered its prerequisite concept, along with the weight of the relation (weak or strong) as specified in the annotation manual. The newly created PR is shown in the “Relations” area as a tuple encoding the following information: the pair ID, i.e. the id of the sentence where the target concept occurs and where the relation was entered, the prerequisite and target concepts, and the relation weight. The statistics about the annotations of each annotator [A1-A4] are reported in the ‘Annotation’ block of Table 3 . As can be noted, although each expert produced different amounts of pairs, the distribution of weight labels is consistent. This is encouraging with regard to the effectiveness of KEQ in making annotators understand how weights should be assigned. Future analyses will investigate this in more depth.

5.3 Annotation Revision

After completing the annotation, experts performed the in-context revision of the PR annotations, checking the correctness of their own created pairs. As recommended by PREAP, each expert checked only the subset of PRs identified solely by her/himself and decided on confirming, deleting or modifying the weight of the pair.

Table 3 , ‘Revision’ block, summarizes the statistics of the revision task. With respect to the overall number of PR pairs (‘PRs’ column), the revision involved a comparable number of pairs among annotators (between 25% and 33%). Considering the modified and deleted pairs, we obtain the following distributions: 38,46%, 54,12%, 50,00%, 63,04% for A1–A4 respectively. This means that an average of more than 50% of the checked PRs have been corrected in the revision phase, which shows the importance of this process in order to have reliable datasets.

5.4 Agreement and gold dataset

(i) Agreement evaluation. Annotations’ consistency was computed pre- and post-revision using the inter-annotator agreement metrics adapted for PR introduced in Sect.  3.2.4 . We computed both pairwise Cohen’s (Cohen, 1960 ) and Fleiss’ (Fleiss, 1971 ) k for all annotators. According to the common interpretation of k (Landis and Koch, 1977 ), we observe an average moderate agreement (0.60) among the original annotations when considering pairwise agreement (Cohen’s k ), which improves to 0.62 on the revised annotations. In fact, a small but consistent improvement is reported for all pairs of experts, confirming that the revision allowed obtaining more coherent and consistent annotations. Fleiss’ k value rises from 0.43 to 0.45 when considering revised annotations. Confirming the results of the protocol evaluation (Sect.  4.2 ), PREAP seems to mitigate the disagreement attested when adopting different PR annotation strategies: Chaplot et al. ( 2016 ) and Fabbri et al. ( 2018 ), e.g., report an average pairwise agreement of around 0.30.

(ii) Annotations combination. The gold dataset was built by merging the four revised annotations ( Union option): the 385 PR pairs annotated as PR by at least one expert appear in the gold dataset as positive PRs , i.e. showing a prerequisite relation Footnote 6 . The Union option aligns well with the project goal of creating a gold-PR dataset suitable for linguistic analysis of PR relations and for training a PR learning system using linguistic features. The conditions for the applicability of the Union option are also satisfied (ref. Sect.  3.2.4 ). These include the expertise level of the annotators, which ensures the understanding of the textbook content. Additionally, the average agreement among annotators provides assurance regarding the comprehension of the annotation guidelines. Moreover, the process of annotation revision resulted in not only a slight improvement in agreement (thus consistency) but above all augmented in correctness and, subsequently, reliability.

(iii) Gold dataset revision. To address potential inconsistencies and loops that may arise from the combination of annotations, we relied on the visualisation aids included in the PRAT tool. These allow to navigate the PR graph resulting from the combination of annotations and identify issues such as loops and lengthy paths, that stemmed from the annotations combination. Such issues were addressed through discussion among annotators, led by the annotation manager, similarly to Wang et al. ( 2021 ).

(iv) Gold dataset release. The gold-PR dataset is made available with the related documentation. It was also annotated with metadata according to schema.org vocabulary Dataset class (schema.org/Dataset) based on the W3C Data Catalog Vocabulary, encoded in JSON-LD format.

6 PREAP Options and Machine Learning Tests

This section discusses the use of PREAP options for concept annotation proposing three application scenarios that complete the case study presented in Sect.  5 . In that case, concept annotation was performed using a semi-automatic approach as an autonomous step of the annotation. Here we present the results of two further annotation projects that differ only in the way concepts are annotated according to the other PREAP options: autonomous automatic and simultaneous manual annotation. The autonomous manual option can be assimilated to the case of autonomous semi-automatic annotation since candidate concepts were manually revised, as described. The example is intended to provide suggestions about the use of PREAP options for different purposes.

First, we describe the annotation projects using the three options, the resulting gold datasets, and the effect on the inter-annotator agreement. Then, we present the use of the datasets to train a machine learning algorithm for PR learning and discuss the effects on algorithm performance.

6.1 Annotation Projects Employing Different Options for Concept Annotation

Table 4 provides information on the three projects, including the details of the resulting datasets. The term ‘Autonomous semi-automatic’ corresponds to the case study discussed in Sect.  5 . All projects rely on the same corpus and combination method for gold dataset creation described in Sect.  5 . T2K 2 was employed for concept extraction in both projects relying on the autonomous option. Each project involved four different annotators, each with comparable levels of expertise.

As reported in Table 4 , the number of concepts in each dataset version reflects the option of the protocol employed: v1 includes only the automatically extracted terms, in v2 the project manager post-processed the automatically extracted terms, as explained in Sect.  5.2 , mostly removing non–concepts. Dataset v3 included also concepts manually added by annotators (agreement on concept annotations=0.71). This explains its larger size compared to v1 and v2, and also the huge increase in PR relations identified by annotators. The inter-annotator agreement shows lower average Cohen agreement on v3 ( k =0.25) compared to v2 ( k =0.62), and also to v1 ( k =0.40). This suggests that, while adding new concepts during annotation produced a richer set of concepts, it also created a less coherent dataset.

6.2 Training a Machine Learning Model and Performance Comparison

To show the use of the PR datasets to train a ML model for PR learning and to investigate the effect of the three options on the performance of the algorithm, we employed the deep learning classification model and the experimental setting of (Alzetta et al., 2019 ). This model acquires lexical (i.e. word embeddings) and global features (i.e. number of occurrences and measures of lexical similarity) for each pair of PRs from the raw textual corpus without relying on external knowledge bases, as in (Liang et al., 2015 ; Gasparetti et al., 2018 ; Talukdar and Cohen, 2012 ), which reflects PREAP annotation principles. The performance of the classifier trained with the three datasets is evaluated using precision, recall, F1, and accuracy computed in a 5-fold cross-validation setting, and compared against a Zero Rule baseline (accuracy=50%, F1=66.66%). As reported in Table 5 , the results obtained by the three gold datasets exceed the baseline. The best performance is observed when the model is trained with v2 dataset and the worst with v3.

6.3 Discussion and Annotation Suggestions

Space constraints do not allow us to report the result of the analysis and to discuss them in detail. We just note that the results in terms of agreement and automatic extraction suggest a positive effect of annotating concepts as an autonomous step, as in v1 and v2. This is coherent with our recommendation to avoid simultaneous annotation of concepts and prerequisites (as in v3) unless specific requirements are given, e.g., in terms of dataset richness.

If we now focus the analysis on comparing v1 and v2 datasets, we observe that v2 results in higher agreement and better PR extraction performance. However, the recommendation for semi-automatic vs automatic concept annotation is not straightforward, and annotation managers should consider at least two factors. The first one is the time required for post-processing the automatically extracted concepts. Even though we found that it is, on average, lower than performing manual annotation, post-processing takes time to read the text and revise the list, as explained in the case study. The percentage increase in performance of v2 compared to v1 is too low to warrant the effort. However, the choice depends on the annotation project goals and the expected quality of annotated data. In particular, if the project aims to produce a dataset for ML training, using an automatic approach for concept extraction can be reasonable. In such cases, the subsequent manual PR annotation step can help mitigate the errors in automatic extraction since annotators are expected not to add PR relations between terms that do not represent domain concepts. This likely accounts for the slight performance decrease observed in v1. Conversely, if the knowledge graph has to be used per se, e.g. for intelligent textbook applications or as ground truth for evaluation tasks, higher correctness and coherence should guide the choice. The second factor to take into account in the decision is that PR extraction results are much affected by the algorithms employed for concept and PR learning, thus better performance might be achieved by other models than those used in this case study.

7 Conclusion and Limits

In this paper, we presented the PREAP protocol for textbook annotation, a systematic procedure for the creation of gold datasets annotated with prerequisite relations.

As a first goal, the protocol is intended to cover a gap in the current literature on prerequisite annotation which lacks systematic procedures for the manual annotation of prerequisites. The aim is to produce reliable datasets built using reproducible methods, adequate for reuse and comparison in learning tasks. The mixed quantitative-qualitative evaluation of the protocol against baseline methods for manual annotation in five domains shows that PREAP succeeds in obtaining datasets that present higher consistency of annotations and accuracy. While dataset completeness is generally comparable across methods, the annotation process using PREAP significantly reduces the required time compared to the other methods. Additionally, a comparison between PREAP and automated methods for PR annotation reveals that automated approaches are not yet able to match the annotation quality achieved through manual methods.

The second goal of PREAP was to design an annotation method aimed at capturing the prerequisite relations as expressed in the text: we refer to it as textbook-driven annotation approach, a method that is very common in concept annotation but still not widely addressed for prerequisite annotation. The annotation approach defined by PREAP also proposes to weight PRs differently based on the concepts’ description in the textbook. This use of PR weights is still a proposal and will be further investigated in future studies. However, we did discuss this PR feature with annotators during the protocol design phases and they expressed a preference for being able to indicate the degree of importance of a prerequisite for a specific target concept. For the evaluation, we used the metrics of annotation correspondence with the source text content and relevancy, defined as the extent to which the annotated data are applicable and helpful for learning support. Also for this evaluation, we compared PREAP against manual and automated methods. Results confirm the validity of PREAP for the two metrics and highlight the expected value of such datasets for applications in education, including learning support for students, support to teachers for instructional design and for textbook comparison.

The paper reports also an annotation project case study that provides a detailed example of protocol application and discusses some of its options and uses for prerequisite extraction in a ML task. The datasets and all the text sources are publicly available with documentation and semantic meta-annotation based on the W3C Web Annotation Vocabulary (see Sect.  8 ).

The protocol has been applied to several texts belonging to different domains. Although we did not find specific and recurrent differences across domains, we cannot claim that the protocol fits all domains and needs, and further evaluations are necessary in this respect. However, we believe that PREAP contributes to the literature by introducing a method that addresses the aforementioned gaps and achieves the goals it was designed for, recalled above. We hope the results presented in this contribution can represent the starting point for the creation of novel resources for analysing PRs in new domains and scenarios, given the relevance of prerequisite relations for enhanced educational systems.

As a limit of the approach and a direction for future work, we observe that having annotations produced based on the content of multiple textbooks would be highly useful for comparing the content reported in different educational resources dealing with the same topic. Also, producing a unique dataset starting from multiple resources could be useful for educational purposes and for ITSs (for improving the dataset coverage, for instance). However, this would require careful and accurate combination strategies to avoid inconsistencies and conflicts in the annotation. We are currently experimenting whether PR weights could be effectively exploited in this scenario, but this research goes beyond the goals of this manuscript. Moreover, future research could investigate other approaches for automatic concept extraction and reconciling annotators’ revisions of concepts extracted from corpora through automatic methods. The high inter-annotator agreement reported in Sect.  6 and in previous analyses (Alzetta et al., 2020a ) suggests a shared understanding of PREAP guidelines about the notion of concept used in the protocol. However, further experiments could be carried out in order to confirm this result, since the proper identification of concepts is a requirement for the reliability of PR annotation. Tests could also be conducted to investigate the balance between reducing the costs associated with concept annotation by involving non-experts and maintaining annotation quality. In this regard, the work of Lee et al. ( 2022 ) can provide inspiration for future research towards this direction.

8 Datasets and Resources

The materials and data presented in this paper are publicly available and have been archived in a public online repository, which can be accessed via the following link: https://github.com/IntAIEdu/PRAT/ . Below is a list of the available datasets, documents and software that can be found in the repository. Researchers and interested parties are encouraged to visit the repository to access and utilize these materials for further exploration and analysis. Footnote 7

PREAP annotation protocol:

PREAP Annotation Manual for Annotators

PREAP Specifications for Project Management

Datasets used for PREAP evaluation and case study

Evaluation: Datasets and Source Texts used in PREAP Evaluation with education experts

Case Study: Annotation project example

List of concepts, annotated PR-dataset, row text

JSON-LD and visual RDF graph encoding metadata information about the dataset and the related annotation process.

Case Study: Datasets and related data used in the ML Experimental tests

PRAT tool for PR annotation and analysis

Sect.  5 presents an Annotation Project case study where more details on this scenario are provided.

This is a consolidated practice for evaluating the reliability of manually produced annotations. Refer to Artstein ( 2017 ) for an overview of inter-annotator agreement measures and their use.

Note that, in this case, annotators are regarded as agreeing on the annotation of the pair \({A \prec C}\) , but not on that of \({A \prec B}\) and \({B \prec C}\) .

Jarboui A, et al. (2016) Fundamentals of Algebra, Magnum Publishing. Tabak J (2009) Probability and statistics: The science of uncertainty, W.H. Freeman & Co. Bartee L, et al. (2019) General Biology I, Open Oregon Educational Resources. Molnar C, et al. (2013) Concepts of Biology, OpenStax College. Harris E (2014) Principles of archaeological stratigraphy, Elsevier.

Performed at the morpho-syntactic level by UDPipe pipeline (Straka et al., 2016 ).

Given that the annotation of PR weights remains a proposal, we did not take into account relation weights in this project.

https://github.com/IntAIEdu/PRAT/ .

Adorni, G., & Koceva, F. (2016). Educational concept maps for personalized learning path generation. In AI* IA 2016 Advances in Artificial Intelligence: XVth International Conference of the Italian Association for Artificial Intelligence, Genova, Italy, November 29–December 1, 2016, Proceedings XV (pp. 135–148). Springer International Publishing.

Adorni, G., Alzetta, C., Koceva, F., Passalacqua, S., & Torre, I. (2019). Towards the identification of propaedeutic relations in textbooks. International conference on Artificial Intelligence in Education : Springer.

Alzetta, C., Koceva, F., Passalacqua, S., Torre, I., & Adorni, G. (2018). PRET: Prerequisite-Enriched Terminology. A Case Study on Educational Texts. In Proceedings of the Italian Conference on Computational Linguistics (CLiC-it 2018).

Alzetta, C., Miaschi, A., Adorni, G., Dell’Orletta, F., Koceva, F., Passalacqua, S., & Torre, I. (2019). Prerequisite or not prerequisite? That’s the problem! an NLP-based approach for concept prerequisites learning. In: 6th Italian Conference on Computational Linguistics, CLiC-it 2019, CEUR-WS, vol 2481.

Alzetta, C., Galluccio, I., Koceva, F., Passalacqua, S., & Torre, I. (2020a). Digging Into Prerequisite Annotation. In iTextbooks@ AIED (pp. 29–34).

Alzetta, C., Miaschi, A., Dell’Orletta, F., Koceva, F., & Torre, I. (2020b). PRELEARN@EVALITA 2020: Overview of the prerequisite relation learning task for Italian. In Proceedings of 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian (EVALITA 2020) .

Angel, J., Aroyehun, S.T., Gelbukh, A. (2020). NLP-CIC@ PRELEARN: Mastering prerequisites relations, from handcrafted features to embeddings. In: Proceedings of 7th Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2020).

Artstein, R. (2017). Inter-annotator agreement. Handbook of linguistic annotation (pp. 297–313). Springer.

Google Scholar  

Artstein, R., & Poesio, M. (2008). Inter-coder agreement for computational linguistics. Computational Linguistics, 34 (4), 555–596.

Atkinson, J. (2020). Technology, change and the academic library: Case studies. Trends and reflections . Chandos Publishing.

Augenstein, I., Das, M., Riedel, S., Vikraman, L., McCallum, A. (2017). Semeval 2017 task 10: Scienceie-extracting keyphrases and relations from scientific publications. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), (pp. 546–555).

Ausubel, D. P., Novak, J. D., Hanesian, H., et al. (1968). Educational psychology: A cognitive view . Rinehart and Winston.

Bai, Y., Zhang, Y., Xiao, K., Lou, Y., & Sun, K. (2021). A BERT-based approach for extracting prerequisite relations among Wikipedia concepts. Mathematical Problems in Engineering . https://doi.org/10.1155/2021/3510402

Article   Google Scholar  

Bayerl, P. S., & Paul, K. I. (2011). What determines inter-coder agreement in manual annotations? A meta-analytic investigation. Computational Linguistics, 37 (4), 699–725. https://doi.org/10.1162/COLI_a_00074

Brookshear, G., & Brylow, D. (2015). Computer science: An overview (Global Edition). Pearson Education Limited.

Brusilovsky, P., & Vassileva, J. (2003). Course sequencing techniques for large-scale web-based education. International Journal of Continuing Engineering Education and Life Long Learning, 13 (1–2), 75–94.

Byrt, T., Bishop, J., & Carlin, J. B. (1993). Bias, prevalence and kappa. Journal of Clinical Epidemiology, 46 (5), 423–429.

CAS   PubMed   Google Scholar  

Cabré, M. T. (1999). Terminology: Theory, methods, and applications (Vol. 1). John Benjamins Publishing.

Carey, S. (1999). Knowledge acquisition: Enrichment or conceptual change. Concepts: Core readings (pp. 459–487). MIT Press.

Carey, S. (2009). The origin of concepts . Oxford University Press.

Carvalho, P.F., Gao, M., Motz, B.A., Koedinger, K.R. (2018). Analyzing the relative learning benefits of completing required activities and optional readings in online courses. International Educational Data Mining Society.

Changuel, S., Labroche, N., & Bouchon-Meunier, B. (2015). Resources sequencing using automatic prerequisite-outcome annotation. ACM Transactions on Intelligent Systems and Technology (TIST), 6 (1), 1–30.

Chaplot, D.S., Yang, Y., Carbonell, J.G., Koedinger, K.R. (2016). Data-driven automated induction of prerequisite structure graphs. In: Proceedings of the 9th International Conference on Educational Data Mining, EDM 2016, (pp. 318–323).

Chau, H., Balaneshin, S., Liu, K., Linda, O. (2020a). Understanding the tradeoff between cost and quality of expert annotations for keyphrase extraction. In: Proceedings of of the 14th Linguistic Annotation Workshop, (pp. 74–86).

Chau, H., Labutov, I., Thaker, K., He, D., & Brusilovsky, P. (2020). Automatic concept extraction for domain and student modeling in adaptive textbooks. International Journal of Artificial Intelligence in Education, 31 , 820–846.

Cimiano, P., Völker, J. (2005). Text2Onto. Natural language processing and information systems. In: International Conference on Applications of Natural Language to Information Systems (NLDB), (pp 15–17).

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20 (1), 37–46.

Council, N. R., et al. (2000). How people learn: Brain, mind, experience, and school: Expanded edition . National Academies Press.

Dell’Orletta, F., Venturi, G., Cimino, A., Montemagni, S. (2014). T2K 2 : A system for automatically extracting and organizing knowledge from texts. In: Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC-2014).

Di Eugenio, B., & Glass, M. (2004). The kappa statistic: A second look. Computational Linguistics, 30 (1), 95–101.

Dickinson, M. (2015). Detection of annotation errors in corpora. Language and Linguistics Compass, 9 (3), 119–138.

Eighmy-Brown, M., McCready, K., & Riha, E. (2017). Textbook access and affordability through academic library services: A department develops strategies to meet the needs of students. Journal of Access Services, 14 (3), 93–113.

Eskin, E. (2000). Detecting errors within a corpus using anomaly detection. In: 1st Meeting of the North American Chapter of the Association for Computational Linguistics.

Fabbri, A.R., Li, I., Trairatvorakul, P., He Y., Ting, W., Tung, R., Westerfield, C., Radev, D. (2018). Tutorialbank: A manually-collected corpus for prerequisite chains, survey extraction and resource recommendation. In: Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, (pp. 611–620).

Faure, D., & Nedellec, C. (1999). Knowledge acquisition of predicate argument structures from technical texts using machine learning: The system asium. International Confernce on Knowledge Engineering and Knowledge Management, (pp. 329–334) Springer.

Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76 (5), 378.

Fort, K., Nazarenko, A., Claire, R. (2011). Corpus linguistics for the annotation manager. In: Corpus Linguistics.

Gagne, R. M. (1962). The acquisition of knowledge. Psychological Review, 69 (4), 355.

Gasparetti, F. (2022). Discovering prerequisite relations from educational documents through word embeddings. Future Generation Computer Systems, 127 , 31–41.

Gasparetti, F., De Medio, C., Limongelli, C., Sciarrone, F., & Temperini, M. (2018). Prerequisites between learning objects: Automatic extraction based on a machine learning approach. Telematics and Informatics, 35 (3), 595–610.

Golik, W., Bossy, R., Ratkovic, Z., & Nédellec, C. (2013). Improving term extraction with linguistic analysis in the biomedical domain. Research in Computing Science, 70 , 157–172.

Gordon, J., Zhu, L., Galstyan, A., Natarajan, P., Burns, G. (2016). Modeling concept dependencies in a scientific corpus. In: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, vol 1, (pp. 866–875).

Gordon, J., Aguilar, S., Sheng, E., Burns, G. (2017). Structured generation of technical reading lists. In: Proceedings of the 12th Workshop on Innovative Use of NLP for Building Educational Applications, (pp. 261–270).

Gruber, T. R. (1993). A translation approach to portable ontology specifications. Knowledge Acquisition, 5 (2), 199–220.

Hippisley, A., Cheng, D., & Ahmad, K. (2005). The head-modifier principle and multilingual term extraction. Natural Language Engineering, 11 (2), 129–157.

Hovy, E., & Lavid, J. (2010). Towards a ‘science’ of corpus annotation: A new methodological challenge for corpus linguistics. International Journal of Translation, 22 (1), 13–36.

Hripcsak, G., & Wilcox, A. (2002). Reference standards, judges, and comparison subjects: roles for experts in evaluating system performance. Journal of the American Medical Informatics Association, 9 (1), 1–15.

PubMed   PubMed Central   Google Scholar  

Hu, X., He, Y., Sun, G. (2021). Active learning for concept prerequisite learning in Wikipedia. In: 13th International Conference on Machine Learning and Computing, (pp. 582–587).

Huang, C., Li, Q., Chen, Y., Zhan, D. (2021). An effective method for constructing knowledge graph of online course. In: 4th International Conference on Big Data and Education, (pp. 12–18).

Hübscher, R. (2001) What’s in a prerequisite. In: International Confernace on Advanced Learning Technology (ICALT), Citeseer.

Ide, N., & Pustejovsky, J. (2017). Handbook of linguistic annotation . Springer.

Labutov, I., Huang, Y., Brusilovsky P, He, D. (2017). Semi-supervised techniques for mining learning outcomes and prerequisites. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM, (pp. 907–915).

Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33 , 159–174.

Lee, J. U., Klie, J. C., & Gurevych, I. (2022). Annotation curricula to implicitly train non-expert annotators. Computational Linguistics, 48 (2), 343–373.

Li, B., et al. (2021). Prerequisite learning with pre-trained language and graph embedding. In: International Conference on NLP and Chinese Computing, (pp. 98–108)

Li, I., Fabbri, A.R., Tung, R.R., Radev, D.R. (2019). What should i learn first: Introducing lecturebank for NLP education and prerequisite chain learning. In: Proceedings of AAAI 2019.

Liang, C., Wu, Z., Huang, W., Giles, C.L. (2015). Measuring prerequisite relations among concepts. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, (pp. 1668–1674).

Liang, C., Ye, J., Wu, Z., Pursel, B., Giles, C.L. (2017). Recovering concept prerequisite relations from university course dependencies. In: AAAI, (pp 4786–4791).

Liang, C., Ye, J., Wang, S., Pursel, B., Giles, C.L. (2018). Investigating active learning for concept prerequisite learning. In: Proceedings of EAAI.

Liang, C., Ye, J., Zhao, H., Pursel, B., Giles, C.L. (2019). Active learning of strict partial orders: A case study on concept prerequisite relations. In: 12th International Conference on Educational Data Mining, EDM 2019, (pp. 348–353).

Limongelli, C., Gasparetti, F., Sciarrone, F. (2015). Wiki course builder: A system for retrieving and sequencing didactic materials from Wikipedia. In: 2015 International Conference on Information Technology Based Higher Education and Training (ITHET), IEEE, (pp. 1–6).

Lu, W., Zhou, Y., Yu, J., Jia, C. (2019). Concept extraction and prerequisite relation learning from educational data. In: Proceedings of the Conference on Artificial Intelligence, vol 33, (pp. 9678–9685).

Manrique, R., Sosa, J., Marino, O., Nunes, B. P., Cardozo, N. (2018). Investigating learning resources precedence relations via concept prerequisite learning. 2018 IEEE/WIC/ACM Int IEEE: Conference on Web Intelligence (pp. 198–205).

Martinez-Rodriguez, J. L., Hogan, A., & Lopez-Arevalo, I. (2020). Information extraction meets the semantic web: A survey. Semantic Web, 11 (2), 255–335.

Mendes, P.N., Mühleisen, H., Bizer, C. (2012). Sieve: Linked data quality assessment and fusion. In: Proceedings of the 2012 joint EDBT/ICDT workshops, (pp. 116–123).

Merrill, M. D. (2002). First principles of instruction. Educational Technology Research and Development, 50 (3), 43–59.

ADS   Google Scholar  

Miaschi, A., Alzetta, C., Cardillo, F.A., Dell’Orletta, F. (2019). Linguistically-driven strategy for concept prerequisites learning on Italian. In: Proceedings of 14th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2019).

Novak, J. D. (1990). Concept mapping: A useful tool for science education. Journal of Research in Science Teaching, 27 (10), 937–949.

Novak, J. D., & Cañas, A. J. (2008). The theory underlying concept maps and how to construct and use them . Institute for Human and Machine Cognition.

Pan, L., Li, C., Li, J., Tang, J. (2017a). Prerequisite relation learning for concepts in MOOCS. In: Proceedings of the 55th Meeting of the Association for Computational Linguistics, (pp. 1447–1456).

Pan, L., Li, C., Li, J., Tang, J. (2017b). Prerequisite relation learning for concepts in MOOCS. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, vol 1, (pp. 1447–1456).

Pan, L., Wang, X., Li, C., Li, J., Tang, J. (2017c). Course concept extraction in moocs via embedding-based graph propagation. In: Proceedings of the Eighth International Joint Conference on Natural Language Processing, (pp. 875–884)

Plank, B., Hovy, D., Søgaard, A. (2014). Linguistically debatable or just plain wrong? In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, (pp. 507–511).

Pustejovsky, J. (2006). Unifying linguistic annotations: A time ML case study. In: Proceedings of Text, Speech, and Dialogue Conference.

Pustejovsky, J., & Stubbs, A. (2012). Natural language annotation for machine learning: A guide to corpus-building for applications . O Reilly Media.

Rani, M., Dhar, A. K., & Vyas, O. (2017). Semi-automatic terminology ontology learning based on topic modeling. Engineering Applications of Artificial Intelligence, 63 , 108–125.

Roy, S., Madhyastha, M., Lawrence, S., Rajan, V. (2019). Inferring concept prerequisite relations from online educational resources. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol 33, (pp. 9589–9594).

Ruiz-Primo, M. A. (2000). On the use of concept maps as an assessment tool in science: What we have learned so far. REDIE Revista Electrónica de Investigación Educativa, 2 (1), 29–53.

Sager, J. C. (1990). Practical course in terminology processing . John Benjamins Publishing.

Sayyadiharikandeh, M., Gordon, J., Ambite, J.L., Lerman, K. (2019), Finding prerequisite relations using the Wikipedia clickstream. In: Companion Proceedings of the WWW Conference, (pp. 1240–1247).

Shen, W., Wang, J., & Han, J. (2014). Entity linking with a knowledge base: Issues, techniques, and solutions. IEEE Transactions on Knowledge and Data Engineering, 27 (2), 443–460.

Straka, M., Hajic, J., Straková, J. (2016), UDPipe: Trainable pipeline for processing CoNLL-U files performing tokenization, morphological analysis, pos tagging and parsing. In: Proceedings of the International Conference on Language Resources and Evaluation, (pp. 4290–4297).

Suresu, S., Elamparithi, M. (2016). Probabilistic relational concept extraction in ontology learning. International Journal of Information Technology, 2(6)

Talukdar, P.P., Cohen, WW. (2012). Crowdsourced comprehension: predicting prerequisite structure in Wikipedia. In: Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, Association for Computational Linguistics, (pp 307–315).

Velardi, P., Faralli, S., & Navigli, R. (2013). Ontolearn reloaded: A graph-based algorithm for taxonomy induction. Computational Linguistics, 39 (3), 665–707.

Wang, M., Chau, H., Thaker, K., Brusilovsky, P., & He, D. (2021). Knowledge annotation for intelligent textbooks. Technology, Knowledge and Learning, 28 , 1–22.

Wang, S., Liang, C., Wu, Z., Williams, K., Pursel, B., Brautigam, B., Saul, S., Williams, H., Bowen, K., Giles, C.L. (2015), Concept hierarchy extraction from textbooks. In: Proceedings of the 2015 ACM Symposium on Document Engineering, (pp. 147–156).

Wang, S., Ororbia, A., Wu, Z., Williams, K., Liang, C., Pursel, B., Giles, C.L. (2016), Using prerequisites to extract concept maps from textbooks. In: Proceedings of the 25th ACM International on Confernce on information and knowledge management, ACM, (pp. 317–326).

Wen, H., Zhu, X., Zhang, M., Zhang, C., & Yin, C. (2021). Combining Wikipedia to identify prerequisite relations of concepts in MOOCS. In: International Conference on Neural Information Processing, (pp. 739–747) Springer.

Xiao, K., Bai, Y., & Wang, S. (2021). Mining precedence relations among lecture videos in MOOCS via concept prerequisite learning. Mathematical Problems in Engineering . https://doi.org/10.1155/2021/7655462

Xiao, K., Bai, Y., & Wang, Z. (2022). Extracting prerequisite relations among concepts from the course descriptions. International Journal of Software Engineering and Knowledge Engineering, 32 (04), 503–523.

Yang, Y., Liu, H., Carbonell, J., Ma, W. (2015). Concept graph learning from educational data. In: Proceedings of the Eighth ACM International Confernce on Web Search and Data Mining, ACM, (pp. 159–168).

Yu, X., Stahr, M., Chen, H., Yan, R. (2021). Design and implementation of curriculum system based on knowledge graph. In: IEEE International Conference on Consumer Electronics and Computer Engineering (ICCECE), IEEE, (pp 767–770).

Zaveri, A., Rula, A., Maurino, A., Pietrobon, R., Lehmann, J., Auer, S., & Hitzler, P. (2013). Quality assessment methodologies for linked open data. Submitted to Semantic Web Journal, 1 (1), 1–5.

Zhao, G., Zhang, X. (2018), Domain-specific ontology concept extraction and hierarchy extension. In: Proceedings of the 2nd International Confernce on Natural Language Processing and Information Retrieval, ACM, (pp. 60–64).

Zhao, Z., Yang, Y., Li, C., & Nie, L. (2020). Guessuneed: Recommending courses via neural attention network and course prerequisite relation embeddings. ACM Transactions on Multimedia Computing, Communications, and Applications, 16 (4), 1–17.

Zhou, Y., Xiao, K. (2019). Extracting prerequisite relations among concepts in Wikipedia. 2019 International IEEE: Joint Conference on Neural Networks. (pp. 1–8).

Download references

Acknowledgements

We thank the colleagues from the Institute for Computational Linguistics ``A. Zampolli'' (CNR-ILC, IT), from the PhD Committee of Digital Humanities (University of Genoa), and the PAWS Lab (School of Computing and Information, Pittsburgh University), who respectively took part in the definition and evaluation phases, and provided fruitful insights and discussion.

Open access funding provided by Università degli Studi di Genova within the CRUI-CARE Agreement. No funding was received for conducting this study.

Author information

Authors and affiliations.

Institute of Computational Linguistics “A. Zampolli”, ItaliaNLP Lab, CNR-ILC, Pisa, Italy

Chiara Alzetta

Department of Informatics, Bioengineering, Robotics and Systems Engineering, University of Genoa, Genoa, Italy

Ilaria Torre

E-Learning and Knowledge Management Lab, DIBRIS, University of Genoa, Genoa, Italy

Frosina Koceva

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ilaria Torre .

Ethics declarations

Conflict of interest.

The authors have no competing interests to declare that are relevant to the content of this article.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

See Table 6 .

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Alzetta, C., Torre, I. & Koceva, F. Annotation Protocol for Textbook Enrichment with Prerequisite Knowledge Graph. Tech Know Learn 29 , 197–228 (2024). https://doi.org/10.1007/s10758-023-09682-6

Download citation

Accepted : 01 August 2023

Published : 21 September 2023

Issue Date : March 2024

DOI : https://doi.org/10.1007/s10758-023-09682-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Text annotation
• Annotation protocol
• Knowledge engineering
• Educational textbooks
• Find a journal
• Publish with us
• Track your research

What we do best

AI Data Services

Data Collection Create global audio, images, text & video.

Data Annotation & Labeling Accurately annotate to make AI/ML think faster

Data De-identification Protecting sensitive information, preserving privacy

Generative AI Fuel your Gen AI with our premium training data.

• Question & Answering Pairs
• Text Summarization
• LLM Data Evaluation & Comparision
• Synthetic Dialogue Creation
• Image Summarization, Rating & Validation

Off-the-shelf Data Catalog & Licensing

Medical Datasets Gold standard, de-identified data

Physician Dictation Datasets

Transcribed Medical Records

Electronic Health Records (EHR)

CT Scan Images Datasets

X-Ray Images Datasets

Computer Vision Datasets Image & Video data for ML development

Bank Statement Dataset

Damaged Car Image Dataset

Facial Recognition Datasets

Landmark Image Dataset

Pay Slips Dataset

Speech/Audio Datasets Transcribed & annotated data in 50+ languages.

New York English | TTS

Chinese Traditional | Utterance/Wake Word

Spanish (Mexico) | Call-Center

Canadian French | Scripted Monologue

Arabic | General Conversation

Banking & Finance Improve ML models for secure user experience

Automotive Accurate training data for Autonomous Vehicles

eCommerce Improve Conversion, Order Value, & Revenue

Named Entity Recognition Extract critical info in unstructured data

Facial Recognition Auto-detect faces via facial landmarks

Search Queries Optimization Improving online store search results

Text-To-Speech (TTS) Enhance interactions with global TTS datasets

Content Moderation Services Boost AI trust & brand reputation

Optical Character Recognition (OCR) Optimize data digitally

• Events & Webinar
• Security & Compliance
• Buyer’s Guide
• Infographics
• In The Media
• Sample Datasets

• July 4, 2023

Text Annotation in Machine Learning: A Comprehensive Guide

What is text annotation in machine learning.

Text annotation in machine learning refers to adding metadata or labels to raw textual data to create structured datasets for training, evaluating, and improving machine learning models. It is a crucial step in natural language processing (NLP) tasks, as it helps algorithms understand, interpret, and make predictions based on textual inputs.

Text annotation is important because it helps bridge the gap between unstructured textual data and structured, machine-readable data. This enables machine learning models to learn and generalize patterns from the annotated examples.

High-quality annotations are vital for building accurate and robust models. This is why careful attention to detail, consistency, and domain expertise is essential in text annotation.

Types of Text Annotation

Text Classification – Importance, Use Cases, and Process

AI-Based Document Classification – Benefits, Process, and Use-cases

• Data Annotation
• Data Collection
• Data De-Identification
• Generative AI
• Conversational AI
• Computer Vision
• Automotive AI
• Banking & Finance
• ShaipCloud™ Platform

(US): (866) 473-5655

[email protected] [email protected] [email protected]

Vendor Enrollment Form

© 2018 – 2024 Shaip | All Rights Reserved

Web Annotation Protocol

W3c recommendation 23 february 2017.

Please check the errata for any errors or issues reported since publication.

This document is also available in this non-normative format: ePub

The English version of this specification is the only normative version. Non-normative translations may also be available.

Copyright © 2017 W3C ® ( MIT , ERCIM , Keio , Beihang ). W3C liability , trademark and document use rules apply.

Annotations are typically used to convey information about a resource or associations between resources. Simple examples include a comment or tag on a single web page or image, or a blog post about a news article.

The Web Annotation Protocol describes the transport mechanisms for creating and managing annotations in a method that is consistent with the Web Architecture and REST best practices.

Status of This Document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at https://www.w3.org/TR/.

By publishing this Recommendation, W3C expects that the functionality specified in this Recommendation will not be affected by changes to the Activity Streams 2.0 [ activitystreams-core ] and Activity Vocabulary [ activitystreams-vocabulary ] as those specifications proceed to Recommendation.

This document was published by the Web Annotation Working Group as a Recommendation. If you wish to make comments regarding this document, please send them to [email protected] ( subscribe , archives ). All comments are welcome.

Please see the Working Group's implementation report .

This document has been reviewed by W3C Members, by software developers, and by other W3C groups and interested parties, and is endorsed by the Director as a W3C Recommendation. It is a stable document and may be used as reference material or cited from another document. W3C 's role in making the Recommendation is to draw attention to the specification and to promote its widespread deployment. This enhances the functionality and interoperability of the Web.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy . W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy .

This document is governed by the 1 September 2015 W3C Process Document .

1. Introduction

This section is non-normative.

Interoperability between systems has two basic aspects: the syntax and semantics of the data that is moved between the systems, and the transport mechanism for that movement. The HTTP protocol and the Web architecture provides us with a great starting point for a standardized transport layer, and can be used to move content between systems easily and effectively. Building upon these foundations allows us to make use of existing technology and patterns to ensure consistency and ease of development.

The Web Annotation Protocol describes a transport mechanism for creating, managing, and retrieving Annotations. Annotations in this specification are assumed to follow the requirements of the Web Annotation Data Model [ annotation-model ] and Web Annotation Vocabulary [ annotation-vocab ]. This specification builds upon REST principles and the Linked Data Platform [ ldp ] recommendation, and familiarity with it is recommended.

1.1 Aims of the Protocol

The primary aim of the Web Annotation Protocol is to provide a standard set of interactions that allow annotation clients and servers to interoperate seamlessly. By being able to discover annotation protocol end-points and how to interact with them, clients can be configured either automatically or by the user to store annotations in any compatible remote system, rather than being locked in to a single client and server pair.

1.2 Summary

For those familiar with the Web Annotation model, LDP, and REST, much of the Annotation Protocol will be very obvious. The following aspects are the most important new requirements.

• The media type to use for Annotations is: application/ld+json;profile="http://www.w3.org/ns/anno.jsonld"
• Annotation Containers are constrained by the set of constraints described in this specification, and thus the ldp:constrainedBy URL is http://www.w3.org/TR/annotation-protocol/
• The link header can refer from any resource to an Annotation Container using a rel type of: http://www.w3.org/ns/oa#annotationService
• The response from a Container after creating an Annotation SHOULD include a representation of the Annotation, after any changes have been made to it, in the JSON-LD serialization.
• Annotation Containers SHOULD only contain Annotations, and not other resources.
• Activity Streams Collection [ activitystreams-core ] model is used for paging, as in-page ordering is an important requirement.

1.3 Conformance

As well as sections marked as non-normative, all authoring guidelines, diagrams, examples, and notes in this specification are non-normative. Everything else in this specification is normative.

The key words MAY , MUST , MUST NOT , RECOMMENDED , SHOULD , and SHOULD NOT are to be interpreted as described in [ RFC2119 ].

1.4 Terminology

2. web annotation protocol principles.

The Web Annotation Protocol is defined using the following basic principles:

• The protocol is developed within the framework laid out by the Web Architecture.
• Interactions will follow the REST best practice guidelines when there is a resource being acted upon.
• Interactions are designed to take place over HTTP.
• Existing specifications and systems will be re-used whenever possible, constrained further when necessary, with invention of new specifications only as a last resort.
• Simplicity and ease of implementation are important design criteria, but ultimately subjective and less important than the above principles.

3. Annotation Retrieval

The Annotation Server MUST support the following HTTP methods on the Annotation's IRI:

• GET (retrieve the description of the Annotation),
• HEAD (retrieve the headers of the Annotation without an entity-body),
• OPTIONS (enable CORS pre-flight requests [ cors ]).

Servers SHOULD use HTTPS rather than HTTP for all interactions, including retrieval of Annotations.

Servers MUST support the JSON-LD representation using the Web Annotation profile. These responses MUST have a Content-Type header with the application/ld+json media type, and it SHOULD have the Web Annotation profile IRI of http://www.w3.org/ns/anno.jsonld in the profile parameter.

Servers SHOULD support a Turtle representation, and MAY support other formats. If more than one representation of the Annotation is available, then the server SHOULD support content negotiation. Content negotiation for different serializations is performed by including the desired media type in the HTTP Accept header of the request, however clients cannot assume that the server will honor their preferences [ rfc7231 ].

Servers MAY support different JSON-LD profiles. Content negotiation for different JSON-LD profiles is performed by adding a profile parameter to the JSON-LD media type in a space separated, quoted list as part of the Accept header.

Servers SHOULD use the 200 HTTP status code when no errors occurred while processing the request to retrieve an Annotation, and MAY use 3XX HTTP status codes to redirect to a new location.

The response from the Annotation Server MUST have a Link header entry where the target IRI is http://www.w3.org/ns/ldp#Resource and the rel parameter value is type . The Annotation type of http://www.w3.org/ns/oa#Annotation MAY also be added with the same rel type. This is to let client systems know that the retrieved representation is a Resource and an Annotation, even if the client cannot process the representation's format.

For HEAD and GET requests, the response MUST have an ETag header with an entity reference value that implements the notion of entity tags from HTTP [ rfc7232 ]. This value will be used by the client when sending update or delete requests.

The response MUST have an Allow header that lists the HTTP methods available for interacting with the Annotation [ rfc7231 ].

For HEAD and GET requests, if the server supports content negotiation by format or JSON-LD profile, the response MUST have a Vary header with Accept in the value [ rfc7231 ]. This is to ensure that caches understand that the representation changes based on the value of that request header.

4. Annotation Containers

If the Annotation Server supports the management of Annotations, including one or more of creating, updating, and deleting them, then the following section's requirements apply. The Annotation Protocol is a use of the Linked Data Platform [ ldp ] specification, with some additional constraints derived from the Web Annotation Data Model [ annotation-model ].

An Annotation Server MUST provide one or more Containers within which Annotations can be managed: an Annotation Container. An Annotation Container is at the same time both a Container [ ldp ] (a service for managing Annotations) and an OrderedCollection [ activitystreams-core ] (an ordered list of Annotations). It can have descriptive and technical information associated with it to allow clients to present it to a user in order to allow her to decide if it should be used or not. The classes, properties and representations for the Collection model are described in the Web Annotation Data Model, and the mappings to Activity Streams provided in the Web Annotation Vocabulary [ annotation-vocab ].

Annotation Containers SHOULD implement the LDP Basic Container specification, but MAY instead implement another type of Container, such as a Direct or Indirect Container, to fulfill business needs. The URI of an Annotation Container MUST NOT have a query or fragment component, and the path component MUST end in a "/" character.

Implementations SHOULD use HTTPS rather than HTTP for all interactions with Annotation Containers.

The creation, management, and structure of Annotation Containers are beyond the scope of this specification. Please see the Linked Data Platform specification [ ldp ] for additional information.

4.1 Container Retrieval

The Annotation Server MUST support the following HTTP methods on the Annotation Container's IRI:

• GET (retrieve the description of the Container and the list of its contents, described below),
• HEAD (retrieve the headers of the Container without an entity-body),

When an HTTP GET request is issued against the Annotation Container, the server MUST return a description of the container. That description MUST be available in JSON-LD, SHOULD be available in Turtle, and MAY be available in other formats. The JSON-LD serialization of the Container's description SHOULD use both the LDP context ( http://www.w3c.org/ns/ldp.jsonld ), and the Web Annotation's profile and context [ annotation-model ], unless the request would determine otherwise.

Servers SHOULD use the 200 HTTP status code if the request is successfully completed without errors and does not require redirection based on the client's preferences.

All supported methods for interacting with the Annotation Container SHOULD be advertised in the Allow header of the GET , HEAD and OPTIONS responses from the container's IRI . The Allow header MAY also be included on any other responses.

Annotation Containers MUST return a Link header [ rfc5988 ] on all responses with the following components:

• It MUST advertise its type by including a link where the rel parameter value is type and the target IRI is the appropriate Container Type, such as http://www.w3.org/ns/ldp#BasicContainer for Basic Containers.
• It MUST advertise that it imposes Annotation protocol specific constraints by including a link where the target IRI is http://www.w3.org/TR/annotation-protocol/ , and the rel parameter value is the IRI http://www.w3.org/ns/ldp#constrainedBy .

For HEAD and GET requests, responses from Annotation Containers MUST include an ETag header that implements the notion of entity tags from HTTP [ rfc7232 ]. This value SHOULD be used by administrative clients when updating the container by including it in an If-Match request header in the same way as clients wanting to update an Annotation.

If the Accept header is absent from a GET request, then Annotation Servers MUST respond with a JSON-LD representation of the Annotation Container, however clients with a preference for JSON-LD SHOULD explicitly request it using an Accept request header.

If the server supports content negotiation by format or JSON-LD profile, the response to a HEAD or GET request from the Annotation Container MUST have a Vary header that includes Accept in the value to ensure that caches can determine that the representation will change based on the value of this header in requests.

Responses from Annotation Containers that support the use of the POST method to create Annotations SHOULD include an Accept-Post header on responses to GET, HEAD and OPTIONS requests. The value is a comma separated list of media-types that are acceptable for the client to send via POST [ ldp ].

4.2 Container Representations

As there are likely to be many Annotations in a single Container, the Annotation Protocol adopts the ActivityStreams collection paging mechanism for returning the contents of the Container. Each Collection Page contains an ordered list with a subset of the managed Annotations, such that if every page is traversed, a client can reconstruct the complete, ordered contents of the container/collection. The number of IRIs or Annotation descriptions included on each page is at the server's discretion, and may be inconsistent between pages. The feature or features by which the Annotations are sorted are not explicit in the response.

The requirements for JSON-LD representation of Annotation Collections are defined in the Web Annotation Data Model, and are summarized here.

The Collection MUST have an IRI that identifies it, and MUST have at least the AnnotationCollection class (the name associated with OrderedCollection in the JSON-LD context) but MAY have other types as well, including the type of LDP Container used. It SHOULD have a human readable label , and MAY have other properties such as creator and created .

If there are greater than zero Annotations in the Container, the representation MUST either include a link to the first page of Annotations as the value of the first property, or include the representation of the first page embedded within the response. If there is more than one page of Annotations, then the representation SHOULD have a link to the last page using the last property.

The representation of the Container SHOULD include the total property with the total number of annotations in the Container. The Container SHOULD include the modified property with the most recent timestamp of when any of the annotations in the Container. This timestamp allows clients to detect when to re-cache data, even if there are the same number of annotations as the same number may have been added and deleted.

The IRI of the Container provided in the response SHOULD differentiate between whether the pages contain just the IRIs, or the full descriptions of the Annotations. It is RECOMMENDED that this be done with a query parameter. The server MAY redirect the client to this IRI and deliver the response there, otherwise it MUST include a Content-Location header with the IRI as its value.

4.2.1 Container Representation Preferences

There are three preferences for Container requests that will govern the representation of the server's responses:

• If the client prefers to only receive the Container description and no Annotations (either URI or full descriptions) embedded in the Container response, then it MUST include a Prefer request header with the value return=representation;include="http://www.w3.org/ns/ldp#PreferMinimalContainer" .
• If the client prefers to receive the Annotations only as IRI references, either embedded in the current Container response or future paged responses, then it MUST include a Prefer request header with the value return=representation;include="http://www.w3.org/ns/oa#PreferContainedIRIs" .
• If the client prefers to receive complete Annotation descriptions, either in the current Container response or future paged responses, then it MUST include a Prefer request header with the value return=representation;include="http://www.w3.org/ns/oa#PreferContainedDescriptions" .

The client MAY send multiple preferences as the value of the include parameter as defined by the Linked Data Platform [ ldp ]. However, the client MUST NOT include both the PreferContainedIRIs and PreferContainedDescriptions preferences on the same request, as the server cannot honor both at the same time. If the PreferMinimalContainer preference is given, then the server SHOULD NOT embed the Annotations or references to them, but SHOULD include a reference to the first and last Annotation Pages. Whether the pages are of IRI references or complete descriptions is governed by the use of PreferContainedIRIs and PreferContainedDescriptions respectively. If no preference is given by the client, the server SHOULD default to the PreferContainedDescriptions behavior. The server MAY ignore the client's preferences.

4.2.2 Representations without Annotations

If the client requests the minimal representation of an Annotation Container, the response MUST NOT include either the ldp:contains predicate nor embed the first page of Annotations within the response.

The linked pages SHOULD follow any PreferContainedDescriptions or PreferContainedIRIs preferences.

The server MAY return a representation without embedded Annotations, even if the PreferMinimalContainer preference is not supplied.

4.2.3 Representations with Annotation IRIs

If the Server supports Container preferences, it MUST respond to PreferContainedIRIs with a response containing an AnnotationPage as the value of first with its items containing only the IRIs of the contained Annotations.

The linked pages SHOULD follow the PreferContainedIRIs preference.

The PreferContainedIRIs and the PreferContainedDescriptions preferences are mutually exclusive.

4.2.4 Representations with Annotation Descriptions

If the Server supports Container preferences, it MUST respond to PreferContainedDescriptions with a response containing an AnnotationPage as the value of first with its items containing complete, inline Annotations.

The linked pages SHOULD follow the PreferContainedDescriptions preference.

4.3 Annotation Pages

Individual pages are instances of the Activity Streams OrderedCollectionPage class, which is refered to as AnnotationPage in the Web Annotation JSON-LD context. The page contains the Annotations, either via their IRIs or full descriptions, in the items property.

The requirements for JSON-LD representation of Annotation Collections Pages are defined in the Web Annotation Data Model, and are summarized here.

The Annotation Page MUST have an IRI that identifies it, and MUST have at least the AnnotationPage class but MAY have other types as well. If the Page is not the last Page in the Collection, then it MUST have a reference to the Page which follows it using the next property. If the Page is not the first Page in the Collection, it SHOULD have a reference ot the previous Page using the prev property. Pages SHOULD give the position of the first Annotation in the items list relative to the order of the Collection using the zero-based startIndex property.

Each page MUST have a link to the Collection that it is part of, using the partOf property. The description of the Collection SHOULD include both the total and modified properties. The response MAY include other properties of the Collection in the response, such as the label or first and last links.

The client SHOULD NOT send the Prefer header when requesting the Page, as it has already been taken into account when requesting the Collection.

This specification does not require any particular functionality when a client makes requests other than GET, HEAD or OPTIONS to a page.

As the Page is not an LDP Container, it does not have the requirement to include a Link header with a type. That the URLs could be constructed with query parameters added to the Container's IRI is an implementation convenience, and does not imply the type of the resource.

Embedded IRIs Interaction Example

Embedded Descriptions Interaction Example

4.4 Discovery of Annotation Containers

As the IRI for Annotation Containers MAY be any IRI, and it is unlikely that every Web Server will support the functionality, it is important to be able to discover the availability of these services.

Any resource MAY link to an Annotation Container when Annotations on the resource SHOULD be created within the referenced Container. This link is carried in an HTTP Link header and the value of the rel parameter MUST be http://www.w3.org/ns/oa#annotationService .

For HTML representations of resources, the equivalent link tag in the header of the document MAY also be used.

For an example image resource, a GET request and response with a link to the above Annotation Container might look like:

5. Creation, Updating and Deletion of Annotations

5.1 create a new annotation.

New Annotations are created via a POST request to an Annotation Container. The Annotation, serialized as JSON-LD, is sent in the body of the request. All of the known information about the Annotation SHOULD be sent, and if there are already IRIs associated with the resources, they SHOULD be included. The serialization SHOULD use the Web Annotation JSON-LD profile, and servers MAY reject other contexts even if they would otherwise produce the same model. The server MAY reject content that is not considered an Annotation according to the Web Annotation specification [ annotation-model ].

Upon receipt of an Annotation, the server MAY assign IRIs to any resource or blank node in the Annotation, and MUST assign an IRI to the Annotation resource in the id property, even if it already has one provided. The server SHOULD use HTTPS IRIs when those resources are able to be retrieved individually. The IRI for the Annotation MUST be the IRI of the Container with an additional component added to the end.

The server MAY add information to the Annotation. Possible additional information includes the agent that created it, the time of the Annotation's creation, or additional types and formats of the constituent resources.

If the Annotation contains a canonical property, then that reference MUST be maintained without change. If the Annotation has an IRI in the id property, then it SHOULD be copied to the via property, and the IRI assigned by the server at which the Annotation will be available MUST be put in the id field to replace it.

The server MUST respond with a 201 Created response if the creation is successful, and an appropriate error code otherwise. The response MUST have a Location header with the Annotation's new IRI.

5.2 Suggesting an IRI for an Annotation

The IRI path segment that is appended to the Container IRI for a resource MAY be suggested by the Annotation Client by using the Slug HTTP header on the request when the resource is created. The server SHOULD use this name, so long as it does not already identify an existing resource, but MAY ignore it and use an automatically assigned name.

5.3 Update an Existing Annotation

Annotations can be updated by using a PUT request to replace the entire state of the Annotation. Annotation Servers SHOULD support this method. Servers MAY also support using a PATCH request to update only the aspects of the Annotation that have changed, but that functionality is not specified in this document.

Replacing the Annotation with a new state MUST be done with the PUT method, where the body of the request is the intended new state of the Annotation. The client SHOULD use the If-Match header with a value of the ETag it received from the server before the editing process began, to avoid collisions of multiple users modifying the same Annotation at the same time. This feature is not mandatory to support, as not every system will have multiple users with the potential to change a single Annotation, or use cases might dictate situations in which overwriting is the desired behavior.

Servers SHOULD reject update requests that modify the values of the canonical or via properties, if they have been already set, unless business logic allows the request to be trusted as authoritatively correctly a previous error.

If successful, the server MUST return a 200 OK status with the Annotation as the body according to the content-type requested. As with creation, the server MUST return the new state of the Annotation in the response.

5.4 Delete an Existing Annotation

Clients MUST use the DELETE HTTP method to request that an Annotation be deleted by the server. Annotation Servers SHOULD support this method. Clients SHOULD send the ETag of the Annotation in the If-Match header to ensure that it is operating against the most recent version of the Annotation.

If the DELETE request is successfully processed, then the server MUST return a 204 status response. The IRIs of deleted Annotations SHOULD NOT be re-used for subsequent Annotations. The IRI of the deleted Annotation MUST be removed from the Annotation Container it was created in. There are no requirements made on the body of the response, and it MAY be empty.

6. Error Conditions

There are inevitably situations where errors occur when retrieving or managing Annotations. The use of the HTTP status codes below provides a method for clients to understand the reason why a request has failed. Some of the situations that might occur, and the preferred HTTP status code are given below. This list is intended to be informative and explanatory, rather than imposing additional requirements beyond those already established by HTTP.

7. Containers for Related Resources

Annotations may have related resources that are required for their correct interpretation and rendering, such as content resources used in or as the Body, CSS stylesheets that determine the rendering of the annotation, SVG documents describing a non-rectangular region of a resource, and so forth. If these resources do not already have IRIs, then they need to be made available somewhere so that they can be referred to.

Annotation Servers MAY support the management of related resources independently from the Annotations. If a server supports the management of these resources, it SHOULD do this with one or more separate Containers. Resources that are not Annotations SHOULD NOT be included in an Annotation Container, as Annotation Clients would not expect to find arbitrary content when dereferencing the IRIs. Containers for related resources MAY contain both RDF Sources and Non-RDF Sources. No restrictions are placed on the type or configuration of the Container beyond those of the Linked Data Platform [ ldp ].

Containers for related resources MUST support the same HTTP methods as described above for the Annotation Container, and MUST support identifying their type with a Link header. The constrainedBy link header on the response when dereferencing the Container SHOULD refer to a server specific set of constraints listing the types of content that are acceptable.

A. Candidate Recommendation Exit Criteria

For this specification to be advanced to Proposed Recommendation, there must be at least two independent implementations of each feature described below. Each feature may be implemented by a different set of products, and there is no requirement that any single product implement every feature.

For the purposes of evaluating exit criteria, the following operations are considered as features:

• HTTP GET of an Annotation
• HTTP GET of an Annotation Collection
• HTTP GET of an Annotation Collection Page, with embedded Annotations
• HTTP GET of an Annotation Collection Page, without embedded Annotations
• POST of an Annotation to an Annotation Collection
• POST of an Annotation to an Annotation Collection, with a Slug to suggest the IRI
• PUT of an Annotation to update an existing Annotation in an Annotation Collection
• DELETE of an Annotation

Each feature must be implemented according to the requirements given in the specification, regarding the HTTP headers, status codes, and entity body. Software that does not alter its behavior in the presence or lack of a given feature is not deemed to implement that feature for the purposes of exiting the Candidate recommendation phase.

B. Changes from Previous Versions

B.1 changes from the proposed recommendation of 2017-01-17.

No significant changes.

B.2 Changes from the Candidate Recommendation of 2016-09-06

Editorial changes in this specification from the Candidate Recommendation of 2016-09-06 are:

• Clarified which header requirements are appropriate for HEAD/GET, rather than OPTIONS.
• Clarified interaction of multiple preferences in a single request.
• Removed incorrect Prefer from the Vary header from the pages example.
• Summarized requirements from the Web Annotation Data Model for Collections and Pages
• Clarified text regarding URI requirements for Containers.
• Clarified use of modified with respect to Containers and Pages.
• Clarified use of canonical with update operations.
• Fixed RFC 7230 / RFC 7232 typo.
• Clarified use of ETag and If-Match.

B.3 Changes from the Candidate Recommendation of 2016-07-12

Editorial changes in this specification from the Candidate Recommendation of 2016-07-12 are:

• Added CR Exit Criteria
• Editorial restructuring of the Pagination content.
• Clarified requirements for Annotation Server's representation preference handling.
• Explained the proper use of the Prefer header when the client has multiple preferences.

B.4 Changes from the Working Draft of 2016-03-31

Significant technical changes in this specification from the Working Draft Published of 2016-03-31 are:

• Use ActivityStreams based Paging mechanism to replace LDP Paging, to allow for in-page order.
• Recommend the use of HTTPS over HTTP.
• Rename PreferContainedURIs to PreferContainedIRIs.
• Add recommendation for Accept-Post, with example.
• Clarify expected status codes for successful interactions.
• Restructure Container Retrieval section and promote Container Representations and Annotation Pages sections.

C. Acknowledgments

The Web Annotation Working Group gratefully acknowledges the contributions of the Open Annotation Community Group . The output of the Community Group was fundamental to the current data model and protocol.

The following people have been instrumental in providing thoughts, feedback, reviews, content, criticism and input in the creation of this specification:

D. References

D.1 normative references.

Shared Tasks in the Digital Humanities

Systematic Analysis of Narrative Texts through Annotation

• Narrative Levels
• Important Dates
• Phase 1: SANTA
• Special Issue

How to Develop Annotation Guidelines

This article describes where to start and how to proceed when developing annotation guidelines. It focuses on the scenario that you are creating new guidelines for a phenomenon or concept that has been described theoretically.

In a single sentence, the goal of annotation guidelines can be formulated as follows: given a theoretically described phenomenon or concept, describe it as generic as possible but as precise as necessary so that human annotators can annotate the concept or phenomenon in any text without running into problems or ambiguity issues.

Developing annotation guidelines is an iterative process: Once a first version has been established, its shortcomings need to be identified and fixed, leading to a second version, which has shortcomings that need to be identified and fixed, etc. This process is displayed schematically in Figure 1. We will describe how to create a first version, and how to come from one version to the next. The most important idea is that in each round, the same text is annotated by multiple annotators independently . This is the main device that allows identifying these shortcomings.

Figure 1: General annotation workflow

Please note that in principle the entire workflow can be performed on paper or digitally. Digital annotation tools make it easier to compare annotations and force deciding on exact annotation boundaries (which words/characters are to be included). Paper-based annotations are more accessible and easier to set up, but make it (too) easy to skip over details. If you decide to make paper-based annotations, please pay attention to exact annotation boundaries.

Making Pilot Annotations

The first round of annotations is best done by annotators who are familiar with the theory that is to be annotated. As with the following annotation rounds, please annotate in parallel and discuss afterwards. It is not necessary to spend a lot of time on preparation. Specifying a list of references or theoretical works, or agreeing on a single text should be sufficient as a starting point.

Your time is spent best on discussing annotation disagreements. In particular in the very first round, many parameters are still undecided and likely to cause disagreement. At the beginning, you need to focus on the big questions:

• What is to be annotated? Every paragraph/sentence/word? Only paragraphs that fulfill a set of conditions?
• What exactly are the annotation categories? Are they related somehow? It sometimes helps to organize them in a hierarchy, as some categories subsume others (e.g., every finite verb is a verb ).
• If you’re using a digital annotation tool: Make sure annotators have the possibility to attach comments to annotations. It helps a lot in the discussions later.

Annotation guidelines typically contain a lot of examples. So you best start collecting interesting/difficult/explanatory examples right away. Examples you find in real texts (possibly with some context) are usually advantageous over made-up ones.

Improving Guidelines

To improve guidelines in this manner, we first need to analyze annotations of the previous “round”, before we reformulate/refine the guidelines. This can be done by inspecting the annotation disagreements : These are cases in which different annotators made different decisions. These can be counted, of course, but it is more informative to talk about the disagreements with the annotators, and to let them explain their decisions.

Such an in-depth discussion with the annotators is fruitful in particular in the first rounds of the process. Once the annotators are trained and annotation guidelines are maturing, a quantitative view might be sufficient. For the latter, a number of metrics have been established (see Wikipedia: Inter-rater reliability for an overview; or Artstein, 2017). Analyzing the inter-annotator-agreement quantitatively gives you a number and allows measuring whether you are actually improving your annotation guidelines, but it does not distinguish different kinds of disagreement.

Some of the disagreements will be caused by annotators not paying attention, or by overlooking something – annotators are human beings after all. These can be fixed easily, without the need to refine the guidelines. It is good practice to let the annotators fix these mistakes by themselves.

Other kinds of disagreement can be expected to have impact on the guidelines: If two annotators made different decisions which are both covered by the annotation guidelines, it is likely that the annotation guidelines are contradictory in this aspect. The source of the contradiction should be identified and resolved.

Many disagreements will be caused when the annotators encounter cases that are not mentioned in the guidelines. In this case, either an existing annotation definition can be applied (perhaps with minor changes), or a new one needs to be defined. If a new definition is added, you need to think about the impact this definition has on the other definitions and annotations . Sometimes, this requires you to re-annotate what you have annotated before.

The actual discussions are likely to be lively and intensive, and tend to jump around between different aspects. It is not always easy, but it makes for better guidelines if this process is well structured and documented. Do not try to fix everything at once, but focus on one aspect at a time.

While going through this iterative process, two processes are likely to be intertwined: The annotation guidelines get better and the annotators get trained. Both are expected and, in principle, welcomed. But: In the end, the annotation guidelines are supposed to be self-contained and also applicable by untrained (or less trained) annotators. It is therefore important to pay attention not to develop rules within a project that are never written down. It will be much harder to integrate new annotators (even if someone drops out and has to be replaced) unwritten rules exist.

List of Annotation Guidelines

The following is a (not exhaustive) list of established annotation guidelines for various, mostly linguistic, phenomena. We provide this list as example for different kinds of tasks.

• Part of speech tagging in the Penn Treebank : The guidelines describe the tag set and its application, and have been developed in the Penn Treebank Project.
• TimeML : The TimeML guidelines describe the annotation of time expressions and related events in news texts.
• Coreference Resolution : Coreference resolution guidelines have been developed in the OntoNotes project. The goal here is to identify mentions in the text that refer to the same real-world entities.

Artstein, Ron. Inter-annotator Agreement. In: Ide Nancy & Pustejovsky James (eds.) Handbook of Linguistic Annotation . Springer, Dordrecht, 2017. DOI  10.1007/978-94-024-0881-2 .

• Architecture and Design
• Asian and Pacific Studies
• Business and Economics
• Classical and Ancient Near Eastern Studies
• Computer Sciences
• Cultural Studies
• Engineering
• General Interest
• Geosciences
• Industrial Chemistry
• Islamic and Middle Eastern Studies
• Jewish Studies
• Library and Information Science, Book Studies
• Life Sciences
• Linguistics and Semiotics
• Literary Studies
• Materials Sciences
• Mathematics
• Social Sciences
• Sports and Recreation
• Theology and Religion
• Publish your article
• The role of authors
• Promoting your article
• Abstracting & indexing
• Publishing Ethics
• Why publish with De Gruyter
• How to publish with De Gruyter
• Our book series
• Our subject areas
• Your digital product at De Gruyter
• Contribute to our reference works
• Product information
• Tools & resources
• Product Information
• Promotional Materials
• Orders and Inquiries
• FAQ for Library Suppliers and Book Sellers
• Repository Policy
• Free access policy
• Open Access agreements
• Database portals
• For Authors
• Customer service
• People + Culture
• Journal Management
• How to join us
• Working at De Gruyter
• Mission & Vision
• De Gruyter Foundation
• De Gruyter Ebound
• Our Responsibility
• Partner publishers

Your purchase has been completed. Your documents are now available to view.

Developing an annotation protocol for evaluative stance and metaphor in discourse: theoretical and methodological considerations

Laura Hidalgo-Downing is Full Professor at Universidad Autónoma, Madrid. Her research interests include discourse analysis at the cross-roads between functional linguistics and cognitive linguistics, stance and evaluation, multimodality and metaphor. Her recent book-length publication is Performing metaphoric creativity across modes and contexts (2020, John Benjamins).

Paula Pérez-Sobrino is a lecturer in Modern Languages at the University of La Rioja. Her work deals with the ways in which metaphors and other types of figurative language help or hinder cross-cultural communication. She has published two research monographs titled Multimodal Metaphor and Metonymy in Advertising (2017, John Benjamins) and Unpacking Creativity: The Power of Figurative Communication in Advertising (2021, Cambridge University Press) and has a third monograph entitled “Emotion-Colour Associations across Languages and Genders” under contract for publication in Cambridge University Press.

The process of identification and annotation of evaluation has received a lot of attention in recent years. However, given the complexity of the topic, the discussion of some of the central issues is still ongoing. The present article contributes to this debate by presenting an annotation scheme that is designed for the identification and annotation of evaluative stance in a corpus of four English genres, namely, newspaper discourse, political discourse, newspaper scientific popularization and fora. A 4,862-word corpus was sampled from a larger 400,000-word corpus compiled within the research project STANCEDISC on the study of stance in discourse varieties. The scheme posits a series of ad hoc categories designed to optimise the transparency, reliability and replicability of the identification, annotation and analysis of evaluative stance. The categories are as follows: parts of speech (Noun Phrase, NP; Adjectival Phrase, AP; Adverbial Phrase, ADVP; Verbal Phrase, VP), function (classifying, predicational and attitude), metaphoricity (metaphoric and non-metaphoric), and value (positive and negative). The aim of this paper is to explain the scheme together with the theoretical justification of the categories and the methodological procedure adopted, and to illustrate the implementation of the scheme by discussing examples taken from different genres.

Funding source: Ministerio de Ciencia, Innovación y Universidades

Award Identifier / Grant number: FFI2017-82730-P

Award Identifier / Grant number: PGC2018-095798

Award Identifier / Grant number: PID2020-118349GB-I00

Award Identifier / Grant number: PID2021-123302NB-I00

About the authors

Research funding: This study was funded by the Ministerio de Ciencia, Innovación y Universidades (FFI2017-82730-P, PGC2018-095798, PID2020-118349GB-I00 and PID2021-123302NB-I00).

Alba-Juez, Laura & Geoff Thompson (eds.). 2014. Evaluation in context . Amsterdam: John Benjamins. Search in Google Scholar

Alba-Juez, Laura & J. Lachlan Mackenzie. 2016. Pragmatics: Cognition, context and culture . Madrid: McGraw Hill. Search in Google Scholar

Bednarek, Monika. 2009. Language patterns and attitude. Functions of Language 16(2). 165–192. https://doi.org/10.1075/fol.16.2.01bed . Search in Google Scholar

Bednarek, Monika. 2014. “An astonishing season of destiny!” Evaluation in blurbs used for advertising TV series. In Laura Alba-Juez & Geoff Thompson (eds.), Evaluation in context , 197–220. Amsterdam: John Benjamins. 10.1075/pbns.242.10bed Search in Google Scholar

Bednarek, Monika & Helen Caple. 2014. Why do news values matter? Towards a new methodological framework for analysing news discourse in critical discourse analysis and beyond. Discourse & Society 25(2). 135–158. https://doi.org/10.1177/0957926513516041 . Search in Google Scholar

Biber, Douglas & Edward Finegan. 1989. Styles of stance in English: Lexical and grammatical marking of evidentiality and affect. Text 9. 93–124. https://doi.org/10.1515/text.1.1989.9.1.93 . Search in Google Scholar

Biber, Douglas, Stig Johansson, Geoffrey Leech, Susan Conrad & Edward Finegan. 1999. The Longman grammar of spoken and written English . London: Longman. Search in Google Scholar

Deignan, Alice. 2010. The evaluative properties of metaphors. In Graham Low, Zazie Todd, Alice Deignan & Lynne Cameron (eds.), Researching and applying metaphor in the real world , 357–374. Amsterdam: John Benjamins. 10.1075/hcp.26.21dei Search in Google Scholar

Du Bois, John W. 2007. The stance triangle. In Robert Engelbreston (ed.), Stancetaking in discourse , 139–182. Amsterdam: John Benjamins. 10.1075/pbns.164.07du Search in Google Scholar

Englebretson, Robert (ed.). 2007 Stancetaking in discourse . Amsterdam: John Benjamins. 10.1075/pbns.164 Search in Google Scholar

Fuoli, Matteo. 2018. A stepwise method for annotating Appraisal. Functions of Language 25(2). 229–258. https://doi.org/10.1075/fol.15016.fuo . Search in Google Scholar

Fuoli, Matteo & Charlotte Hommerberg. 2015. Optimising transparency, reliability and replicability: Annotation principles and inter-coder agreement in the quantification of evaluative expressions. Corpora 10(3). 315–349. https://doi.org/10.3366/cor.2015.0080 . Search in Google Scholar

Fuoli, Matteo, Jeannette Littlemore & Sarah Turner. 2021. Sunken ships and screaming banshees: Metaphor and evaluation in film reviews. English Language and Linguistics . 1–29. https://doi.org/10.1017/S1360674321000046 . Search in Google Scholar

Hidalgo-Downing, Laura & Paula Pérez-Sobrino. In press. Pushing Britain off the precipice: A CDA approach to negative evaluative stance in opinion articles on Brexit. In Juana Marín-Arrese, Laura Hidalgo-Downing & Juan Rafael Zamorano-Mansilla (eds.), Stance, inter/subjectivity and identity in discourse . Frankfurt: Peter Lang. Search in Google Scholar

Hidalgo-Downing, Laura, Paula Pérez-Sobrino, Laura Filardo-Llamas, Carmen Maíz-Arévalo, Begoña Núñez-Perucha, Alfonso Sánchez-Moya & Julia T. Williams Camus. Under review. A protocol for the annotation of evaluative stance and metaphor across four discourse genres. Revista Espanola de Linguistica Aplicada . Search in Google Scholar

Hunston, Susan. 2011. Corpus approaches to evaluation: Phraseology and evaluative language . London: Routledge. 10.4324/9780203841686 Search in Google Scholar

Hunston, Susan & Geoff Thompson. 2000. Evaluation in text . Oxford: Oxford University Press. 10.1093/oso/9780198238546.001.0001 Search in Google Scholar

Hunston, Susan & Hang Su. 2019. Patterns, constructions, and local grammar: A case study of evaluation. Applied Linguistics 40(4). 567–593. https://doi.org/10.1093/applin/amx046 . Search in Google Scholar

Martin, James R. & Peter R. R. White. 2005. The language of evaluation: Appraisal in English . London: Palgrave Macmillan. Search in Google Scholar

Müller, Cornelia. 2009. Metaphors dead and alive, sleeping and walking: A dynamic view . Chicago: University of Chicago Press. 10.7208/chicago/9780226548265.001.0001 Search in Google Scholar

Núñez-Perucha, Begoña & Laura Filardo-Llamas. In press. From “roaring lion” to “chlorinated chicken”: Evaluative stance and ideological positioning in a corpus of British political discourse. In Juana Marín-Arrese, Laura Hidalgo-Downing & Juan Rafael Zamorano-Mansilla (eds.), Stance, inter/subjectivity and identity in discourse . Frankfurt: Peter Lang. Search in Google Scholar

Pragglejazz . 2007. A method for identifying metaphorically used words in discourse. Metaphor and Symbol 22(1). 1–39. https://doi.org/10.1207/s15327868ms2201_1 . Search in Google Scholar

Read, Jonathon & John Carroll. 2012. Annotating expressions of appraisal in English. Language Resources and Evaluation 46(3). 421–447. https://doi.org/10.1007/s10579-010-9135-7 . Search in Google Scholar

Reisigl, Martin & Ruth Wodak. 2009. The discourse historical approach. In Ruth Wodak & Michael Meyer (eds.), Methods for critical discourse analysis , 87–121. London: Sage. Search in Google Scholar

Sánchez-Moya, Alfonso & Carmen Maíz-Arévalo. In press. “Histrionic, appalling, a major Turkey”: The expression of evaluative stance in the discourse of online forums. In Juana Marín-Arrese, Laura Hidalgo-Downing & Juan Rafael Zamorano-Mansilla (eds.), Stance, inter/subjectivity and identity in discourse . Frankfurt: Peter Lang. Search in Google Scholar

Semino, Elena, Zsofia Demjén, Sheila A. Payne & Rayson Paul. 2018. Metaphor, cancer and the end of life: A corpus-based study . London: Routledge. 10.4324/9781315629834 Search in Google Scholar

Simaki, Vasiliki, Carita Paradis & Andreas Kerren. 2018. Evaluating stance-annotated sentences from the Brexit blog corpus: A quantitative linguistic analysis. ICAME Journal 42(1). 133–166. https://doi.org/10.1515/icame-2018-0007 . Search in Google Scholar

Simaki, Vasiliki, Carita Paradis, Maria Skeppstedt, Magnus Sahlgren, Kostiantyn Kucher & Andreas Kerren. 2020. Annotating speaker stance in discourse: The Brexit blog corpus. Corpus Linguistics and Linguistic Theory 16(2). 215–248. Search in Google Scholar

Steen, Gerard, Aletta G. Dorst, J. Berenike Herrmann, Anna A. Kaal & Tina Krennmayr. 2010. Metaphor in usage. Cognitive Linguistics 21(4). 765–796. https://doi.org/10.1515/cogl.2010.024 . Search in Google Scholar

Taboada, Maite & Marta Carretero. 2010. Contrastive analyses of evaluation in text: Key issues in the design of an annotation system for attitude applicable to consumer reviews in English and Spanish. Linguistics and the Human Sciences 6(1–3). 275–295. https://doi.org/10.1558/lhs.v6i1-3.275 . Search in Google Scholar

Thompson, Geoff. 2014. affect and emotion, target-value mismatches, and Russian dolls: Refining the appraisal model. In Laura Alba-Juez & Geoff Thompson (eds.), Evaluation in context , 47–66. Amsterdam: John Benjamins. 10.1075/pbns.242.03tho Search in Google Scholar

Van Dijk, Teun A. 1995. Power and the news media. In David L. Paletz (ed.), Political communication and action , 9–36. Cresskill, NJ: Hampton Press. Search in Google Scholar

van Leeuwen, Theo. 2008. Discourse and practice: New tools for critical discourse analysis . Oxford: Oxford University Press. 10.1093/acprof:oso/9780195323306.001.0001 Search in Google Scholar

White, Peter R. R. 2016. Evaluative contents in verbal communication. In Andrea Rocci & Louis de Saussure (eds.), Verbal communication: Handbook of communication science , 77–96. Berlin: Walter de Gruyter. 10.1515/9783110255478-006 Search in Google Scholar

Williams Camus, Julia T. In press. Evaluative stance in science popularizations in the English press. In Juana Marín-Arrese, Laura Hidalgo-Downing & Juan Rafael Zamorano-Mansilla (eds.), Stance, inter/subjectivity and identity in discourse . Frankfurt: Peter Lang. Search in Google Scholar

© 2022 Walter de Gruyter GmbH, Berlin/Boston

• X / Twitter

Supplementary Materials

Please login or register with De Gruyter to order this product.

Journal and Issue

Articles in the same issue.

Text Annotations Protocol

Description

This FREE text annotation protocol will equip students to develop their reading comprehension skills. This one-pager guides students in annotating and interacting with the text, to strengthen their reading comprehension skills.

The Text Annotation Protocol is 8.5 x 11 and can be printed from your standard printer. Students can keep the printed versions in their notebooks and/or you can enlarge it (with your school’s poster maker) to create an anchor chart for back-to-school AND the entire year. 

Head over to my website for more information on literacy in science.

Related blog posts:

Science is Lit(erary): The Power and Importance of Literacy in science classrooms

Activities for the Beginning of the School Year

3 Engaging and Empowering Examples of Student-Centered Learning Activities

Need more? Want to spend less? Check out this money-saving bundle with a week's + worth of Back to School activities.

***Please note: This resource is not to be used or distributed for use by teachers working as independent contractors for commercial and/or online teaching platforms. I hope your students enjoy and benefit from the activity for classroom teachers. Thank you.***

Questions & Answers

Lit science by jessica payano.

• We're hiring
• Help & FAQ
• Privacy policy
• Student privacy
• Terms of service
• Tell us what you think

• Technology News
• Tech Tips News

How to use annotations in Google Meet

About the Author

The TOI Tech Desk is a dedicated team of journalists committed to delivering the latest and most relevant news from the world of technology to readers of The Times of India. TOI Tech Desk’s news coverage spans a wide spectrum across gadget launches, gadget reviews, trends, in-depth analysis, exclusive reports and breaking stories that impact technology and the digital universe. Be it how-tos or the latest happenings in AI, cybersecurity, personal gadgets, platforms like WhatsApp, Instagram, Facebook and more; TOI Tech Desk brings the news with accuracy and authenticity. Read More

TOP TRENDING

Trending stories.

• This startup wants to turn expensive Ola, Uber airport rides into budget-friendly carpools, here's how
• Sennheiser Momentum True Wireless 4 review: These don’t miss a beat
• 10 apps to capture photos of violators on your Android phone
• ChatGPT-maker OpenAI may have another product to compete with Apple, Google
• Roblox All Star Tower Defense codes May 2024: Steps to redeem All Star Tower Defense codes, how to summon using an ASTD banner, and other information
• Asus announces free 'Like a Dragon: Infinite Wealth' game with select GPU purchases
• 6 tips to turn your tablet into a smart home display hub
• Intel may get $11 billion ‘boost’ for chip facility in Ireland
• OpenAI Spring Update event today: ChatGPT, GPT-4 updates and more Sam Altman may announce
• Typing games on Android: What are they and 5 popular titles to play for free
• Sushil Modi, former Bihar deputy CM, passes away at 72
• Mumbai billboard collapse: 8 dead, 30 still trapped
• You are in your 30s, do you know what money type you are?
• Kashmir sees highest turnout since 1996: EC
• IPL: Gujarat Titans knocked out after washout
• India inks 10-year deal to operate Iran's Chabahar port
• 'How did the Indian student become so stupid?'
• 'Don’t need lectures… ': Ola CEO on ending ties with Azure
• In pics: Several injured as dust storm, heavy rains hit Mumbai
• Explained: Why is Chabahar Port important for India?

Biden’s speech at the Holocaust remembrance ceremony, annotated

By Zachary B. Wolf and Annette Choi , CNN

Published May 7, 2024

President Joe Biden talked about the documented increase of antisemitism in the United States during the annual US Holocaust Memorial Museum’s Days of Remembrance ceremony at the US Capitol building. Every recent president has made remarks at least once at the event, but Biden’s remarks came as pro-Palestinian protests have disrupted classes and commencements at multiple US universities . At times, rhetoric at those protests has veered into antisemitism, offended Jewish students and sparked a fierce debate about free speech.

Biden talked in-depth about the Hamas terror attack against Israel on October 7, 2023, and the Israeli hostages that remain in captivity . He did not mention Israel’s heavy-handed response, which has not only destroyed much of Gaza and cost tens of thousands of lives but has also driven a wedge between Biden and many progressives, particularly on college campuses. See below for what he said , along with context from CNN.

Thank you. Thank you, thank you, thank you. Thank you, Stu Eizenstat, for that introduction, for your leadership of the United States Holocaust Memorial Museum . You are a true scholar and statesman and a dear friend.

Speaker Johnson , Leader Jeffries, members of Congress and especially the survivors of the Holocaust. If my mother were here, she’d look at you and say, “God love you all. God love you all.”

Abe Foxman and all other survivors who embody absolute courage and dignity and grace are here as well.

During these sacred days of remembrance we grieve, we give voice to the 6 million Jews who were systematically targeted and murdered by the Nazis and their collaborators during World War II. We honor the memory of victims, the pain of survivors, the bravery of heroes who stood up to Hitler's unspeakable evil. And we recommit to heading and heeding the lessons that one of the darkest chapters in human history to revitalize and realize the responsibility of never again.

The Days of Remembrance commemoration has been an annual event since 1982. Every US president since Bill Clinton has spoken at least once at a remembrance event.

House Speaker Mike Johnson spoke shortly before Biden and tried to compare the situation on college campuses today with that on college campuses in Germany in the 1930s.

Never again, simply translated for me, means never forget, never forget. Never forgetting means we must must keep telling the story, we must keep teaching the truth, we must keep teaching our children and our grandchildren. And the truth is we are at risk of people not knowing the truth.

That's why, growing up, my dad taught me and my siblings about the horrors of the Shoah at our family dinner table.

Shoah is the Hebrew term for the Holocaust.

That's why I visited Yad Vashem with my family as a senator, as vice president and as president. And that's why I took my grandchildren to Dachau , so they could see and bear witness to the perils of indifference, the complicity of silence in the face of evil that they knew was happening.

Biden visited Yad Vashem , Israel’s Holocaust remembrance site, in 2022 as president.

As vice president, he toured the Nazi concentration camp outside Munich in 2015 with his granddaughter during a trip for an annual security conference.

Germany, 1933, Hitler and his Nazi party rise to power by rekindling one of the world's oldest forms of prejudice and hate — antisemitism.

His rule didn't begin with mass murder. It started slowly across economic, political, social and cultural life — propaganda demonizing Jews, boycotts of Jewish businesses, synagogues defaced with swastikas, harassment of Jews in the street and in the schools, antisemitic demonstrations, pogroms, organized riots.

With the indifference of the world, Hitler knew he could expand his reign of terror by eliminating Jews from Germany, to annihilate Jews across Europe through genocide the Nazis called the final solution. Concentration camps, gas chambers, mass shootings. By the time the war ended, 6 million Jews, one out of every three Jews in the entire world, were murdered.

This ancient hatred of Jews didn't begin with the Holocaust. It didn't end with the Holocaust either, or after, even after our victory in World War II. This hatred continues to lie deep in the hearts of too many people in the world and requires our continued vigilance and outspokenness.

The Holocaust survivor Irene Butter wrote for CNN Opinion in 2021 about Adolf Hitler’s rise and echoes of Nazism in the January 6, 2021, Capitol attack.

That hatred was brought to life on October 7th in 2023. On the sacred Jewish holiday, the terrorist group Hamas unleashed the deadliest day of the Jewish people since the Holocaust.

Read mo re about Hamas .

Driven by ancient desire to wipe out the Jewish people off the face of the Earth, over 1,200 innocent people — babies, parents, grandparents — slaughtered in their kibbutz, massacred at a music festival, brutally raped, mutilated and sexually assaulted .

Evidence of sexual violence has been documented. Here’s the account of one Israeli woman who has spoken publicly about her experience.

Thousands more carrying wounds, bullets and shrapnel from the memory of that terrible day they endured. Hundreds taken hostage, including survivors of the Shoah.

Now here we are, not 75 years later but just seven-and-a-half months later and people are already forgetting, are already forgetting that Hamas unleashed this terror. That it was Hamas that brutalized Israelis. It was Hamas who took and continues to hold hostages. I have not forgotten, nor have you, and we will not forget.

On May 7, 1945, the German High Command agreed to an unconditional surrender in World War II, 79 years ago.

And as Jews around the world still cope with the atrocities and trauma of that day and its aftermath, we've seen a ferocious surge of anti s emitism in America and around the world.

In late October, FBI Director Christopher Wray said reports of antisemitism in the US were reaching “ historic ” levels.

Vicious propaganda on social media, Jews forced to keep their — hide their kippahs under baseball hats, tuck their Jewish stars into their shirts.

On college campuses, Jewish students blocked, harassed, attacked while walking to class . Antisemitism, antisemitic posters , slogans calling for the annihilation of Israel, the world's only Jewish state.

Many Jewish students have described feeling intimidated and attacked on campuses. Others have said they support the protests , citing the situation in Gaza.

Last month, the dean of the University of California Berkeley Law School described antisemitic posters that targeted him.

Too many people denying, downplaying, rationalizing, ignoring the horrors of the Holocaust and October 7th, including Hamas' appalling use of sexual violence to torture and terrorize Jews. It's absolutely despicable and it must stop.

Silence. Silence and denial can hide much but it can erase nothing.

Some injustices are so heinous, so horrific, so grievous they cannot be married – buried, no matter how hard people try.

In my view, a major lesson of the Holocaust is, as mentioned earlier, is it not, was not inevitable.

We know hate never goes away. It only hides. And given a little oxygen, it comes out from under the rocks.

We also know what stops hate. One thing: All of us. The late Rabbi Jonathan Sacks described antisemitism as a virus that has survived and mutated over time.

Together, we cannot continue to let that happen. We have to remember our basic principle as a nation. We have an obligation. We have an obligation to learn the lessons of history so we don't surrender our future to the horrors of the past. We must give hate no safe harbor against anyone. Anyone.

From the very founding, our very founding, Jewish Americans , who represented only about 2% of the US population , have helped lead the cause of freedom for everyone in our nation. From that experience we know scapegoating and demonizing any minority is a threat to every minority and the very foundation of our democracy.

As of 2020, Jewish Americans made up about 2.4% of the US population, according to the Pew Research Center , or about 5.8 million people.

So moments like this we have to put these principles that we're talking about into action.

I understand people have strong beliefs and deep convictions about the world .

In America we respect and protect the fundamental right to free speech, to debate and disagree, to protest peacefully and make our voices heard . I understand. That's America.

The complaint of many protesters is that Israel’s response to the terror attack has claimed more than 30,000 lives and destroyed much of Gaza .

But there is no place on any campus in America, any place in America, for antisemitism or hate speech or threats of violence of any kind.

Whether against Jews or anyone else, violent attacks, destroying property is not peaceful protest. It's against the law and we are not a lawless country. We're a civil society. We uphold the rule of law and no one should have to hide or be brave just to be themselves.

To the Jewish community, I want you to know I see your fear, your hurt and your pain.

Let me reassure you as your president, you're not alone. You belong. You always have and you always will.

And my commitment to the safety of the Jewish people, the security of Israel and its right to exist as an independent Jewish state is ironclad, even when we disagree.

My administration is working around the clock to free remaining hostages, just as we have freed hostages already, and will not rest until we bring them all home.

My administration, with our second gentleman's leadership, has launched our nation's first national strategy to counter antisemitism. That's mobilizing the full force of the federal government to protect Jewish communities.

But we know this is not the work of government alone or Jews alone. That's why I’m calling on all Americans to stand united against antisemitism and hate in all its forms.

My dear friend — and he became a friend — the late Elie Wiesel said, quote, “One person of integrity can make a difference.”

Elie Wiesel , the Holocaust survivor, writer and activist, died in 2016.

We have to remember that, now more than ever.

Here in Emancipation Hall in the US Capitol, among the towering statues of history is a bronze bust of Raoul Wallenberg . Born in Sweden as a Lutheran, he was a businessman and a diplomat. While stationed in Hungary during World War II, he used diplomatic cover to hide and rescue about 100,000 Jews over a six-month period.

Read more about Wallenberg , the Holocaust hero and Swedish diplomat who was formally declared dead in 2016, 71 years after he vanished.

Among them was a 16-year-old Jewish boy who escaped a Nazi labor camp. After the war ended, that boy received a scholarship from the Hillel Foundation to study in America. He came to New York City penniless but determined to turn his pain into purpose. Along with his wife, also a Holocaust survivor, he became a renowned economist and foreign policy thinker, eventually making his way to this very Capitol on the staff of a first-term senator.

That Jewish refugee was Tom Lantos and that senator was me. Tom and his wife and Annette and their family became dear friends to me and my family. Tom would go on to become the only Holocaust survivor ever elected to Congress, where he became a leading voice on civil rights and human rights around the world. Tom never met Raoul, who was taken prisoner by the Soviets, never to be heard from again.

Read more about Lantos , the longtime congressman and Holocaust survivor who died in 2008. Lantos worked for Biden early in his career.

But through Tom's efforts, Raoul’s bust is here in the Capitol. He was also given honorary US citizenship, only the second person ever after Winston Churchill. The Holocaust Museum here in Washington is located in a road in Raoul’s name.

The story of the power of a single person to put aside our differences, to see our common humanity, to stand up to hate and its ancient story of resilience from immense pain, persecution, to find hope, purpose and meaning in life, we try to live and share with one another. That story endures.

Let me close with this. I know these days of remembrance fall on difficult times. We all do well to remember these days also fall during the month we celebrate Jewish American heritage, a heritage that stretches from our earliest days to enrich every single part of American life today.

There are important topics Biden did not address. He referenced the October 7 attacks on Israel but not Israel’s controversial response, which has drawn furious protests. He failed to mention Gaza, where Israel’s military campaign has killed so many, and which has led the World Food Programme to warn of a “full-blown famine .”

A great American — a great Jewish American named Tom Lantos — used the phrase “the veneer of civilization is paper thin.” We are its guardians, and we can never rest.

My fellow Americans, we must, we must be those guardians. We must never rest. We must rise Against hate, meet across the divide, see our common humanity. And God bless the victims and survivors of the Shoah.

May the resilient hearts, the courageous spirit and the eternal flame of faith of the Jewish people forever shine their light on America and around the world, pray God.

Thank you all.