Build powerful annotation layouts with tagtog blocks

Published in

Chatbots Life

7 min readAug 24, 2020

At tagtog, we are constantly innovating. Lately, we have been playing with the potential of Markdown to make the annotation process more enjoyable and clear.

Now we go one step further. For each Markdown code block, you can now select a predefined layout. Say hello to tagtog blocks! 👋

Let's see some examples!

Chatbot conversations 🤖

blocks

These are the block types we are going to use:

bot for the chatbot interaction
human for the human interaction

This is an example of a conversation between a human and Facebook BlenderBot:

```human
Hi, how are you today?
```
```bot
I’m good. How are you? Do you have a favorite food? Mine is lobster!
```
```human
Doing well, my favorite food is cake. I just bought one because I got promoted at work!
```
```bot
Congratulations on the promotion! What kind of work do you do? I work in software.
```
```human
Thanks so much! I just want to make my parents proud! I’m an engineer.
```
```bot
I’m sure they will be very proud of you. What type of engineering do you work in?
```
```human
I’m an environmental engineer. What is it like to work in software?
```
```bot
It can be a lot of work, but it is very rewarding. Do you enjoy your job?
```
```human
I love it! it’s very important to protect the environment.
```
```bot
That’s great! Do you have any hobbies? I like to play tennis in my spare time.
```
```human
I’m not very coordinated so I don’t play sports… I do like to play the guitar, though!
```
```bot
That’s awesome! I’ve always wanted to learn how to play an instrument. What kind of music do you play?
```
```human
Mostly alternative rock!
```
```bot
That’s cool! I like alternative rock as well. What are some of your favorite bands?
```
```human
I like Modest Mouse and the Smashing Pumpkins.
```

… translates into this layout:

Example of a conversation between a human and Facebook BlenderBot chatbot

If you combine this layout with the tagtog annotation capabilities, you could perform chatbot annotation tasks such as:

Entity annotation
Intent classification
Create relations between entities
Attach attributes to entities
Context classification
Overlapping annotations (e.g. entity within intent)

In this case, we are adding an entity label (attribute) to an intent to define its tone.

You can add these blocks in two ways:

a. Import to your tagtog project a .md (Markdown) file with a piece of text similar to the example provided (you can combine human or bot blocks as required). You can use the API or the GUI.

Question Answering datasets ❓

Say you want to build a Question Answering dataset, or you want to visualize your model predictions.

blocks

These are the block types we are going to use:

textbox for a text block
question for questions
answer for answers

This Markdown content:

```textbox
In the U.S. federal health care system (including the VA, the Indian Health Service, and NIH) ambulatory care pharmacists are given full independent prescribing authority. In some states such North Carolina and New Mexico these pharmacist clinicians are given collaborative prescriptive and diagnostic authority. In 2011 the board of Pharmaceutical Specialties approved ambulatory care pharmacy practice as a separate board certification. 

The official designation for pharmacists who pass the ambulatory care pharmacy specialty certification exam will be Board Certified Ambulatory Care Pharmacist and these pharmacists will carry the initials BCACP.
```
```question
Q1: What type of authority are ambulatory care pharmacists given in the U.S. federal health care system?
```
```answer
A1: full independent prescribing authority
```
```question
Q2: In what states are pharmacist clinicians given prescriptive and diagnostic authority?
```
```answer
A2: North Carolina and New Mexico
```
```question
Q3: When was ambulatory care pharmacy approved as its own certification?
```
```answer
A3: 2011
```

…turns to:

SQuAD like dataset sample. On the document, we see at the top the paragraphs supporting the questions. Below, question/answer pairs (Q1-A1, Q2-A2, Q3-A3). On the right, the ground truth answers (A1, A2, A3) provided by Marianne (see at the top bar, this is her annotation’s version for this document). By selecting the annotation’s versions of other users, you can access the ground truth answers from other users.

Tweets 🐦

Useful to annotate tweets (please notice that we are not using iFrames to load original tweets from Twitter).

blocks

These are the block types we are going to use:

tweet

This Markdown content:

```tweet
The NHS has published its phase 3 implementation plan in response to the Covid-19 pandemic, including patient-initiated follow ups, the addressing of NHS inequalities and mental health planning.
```
```tweet
The economic impact of COVID19 on the hotel industry is the worst we have ever faced. The hotel industry is expected to lose more than fifty percent of its total revenue in 2020.
```
```tweet
The World Bank Group's FY20 financial statements are out, highlighting the strength of the World Bank Group's financial position, the strong demand for financing including due to COVID19, and the continued backing from shareholders and capital markets.
```

… translates into:

Example about how to annotate and to classify Tweets with tagtog. A more engaging layout than raw plain text.

Guidelines 📙

Especially at the beginning of an annotation project, it is essential to guide the annotator through the whole annotation flow. Usually, the team puts together a set of guidelines, annotators read through them, and they try to label some documents following these rules. Why not combine theory and practice?

blocks

These are the block types we are going to use:

task to define a task for the annotator
textbox for a text block

This Markdown text:

## Interactive guidelines

Welcome to the interactive guidelines for our medical annotation project.

#### Basic rules:

* The annotator must select the concept codes that better cover the meaning of the clinical concept.
* The annotator must select the fewest number of codes that, together, better covers the meaning of the clinical concept.
* If there is a doubt, don't annotate. 

#### Example

```textbox
Complicated fracture of third rib
```

This text could be annotated with the following codes:
1. |255302009| Complicated.
2. |706922007| Complicated fracture of bone.
3. |125605004| Fracture of bone. If annotators are more literal this code could be full coverage. However, other could see the next code as more appropriate and also use the score "inferred coverage".
4. |20274005|   Fracture of one rib.
5. |25888004|   Bone structure of third rib (body structure).

```task
Explore the different combinations below, click on each annotated entity and see which codes were assigned to it.
```

1,3,5
```textbox
Complicated fracture of third rib
```
2,5
```textbox
Complicated fracture of third rib
```
This would be the best annotation group, because it has a fully coverage and it also has a fewer number of codes#### Your task```task
Now it is your turn! Annotate this sentence to fully cover its content. 
```
```textbox
other complications of unspecified head injuries
```

…will render:

A sample annotation guidelines with an example and a task for the annotators to validate their learning. Afterward, their annotations can be reviewed by a third person to track annotator's progress.

Add-ons

Use them to modify blocks. The add-on name is added after the block name. For example: tweet attention , tweet being the name of the block, and attention the name of the add-on.

Attention

This add-on (attention) marks a block with a warning to indicate the annotator to pay special attention. For example, to illustrate that this block contains critical information or it is a source of annotation conflicts.

This Markdown text:

```bot
Hello, how can I help you?
```
```human attention
I want to report a theft of my account not foreseen 2 days ago, I doubt very much to recover it. However, if it is possible to suspend it, it would be definitely better.
```
```bot
Please let me know your username
```

…translates into: