First of all, define the type of project:<\/p>\n\n\n\n
- \n
- Text classification (what is text classification?<\/a>)<\/li>\n\n\n\n
- Entity detection (what is entity detection<\/a>?)<\/li>\n\n\n\n
- Both text classification and entity detection<\/li>\n<\/ul>\n\n\n\n
In the latter case, create two separate projects and combine the outcomes in a pipeline: one project can use a model from another project.<\/p>\n\n\n\n
Although it is possible to create multilingual projects<\/strong>, it is recommended to create one project for each language<\/strong> to obtain the best possible results. In this case, the language of the documents needs to be defined.<\/p>\n\n\n\n
<\/div>\n\n\n\n
\n\n\n\nText classification<\/h1>\n\n\n\n
Step 1: Initiate the project<\/h2>\n\n\n\n
- \n
- Create<\/a> the project (type=Text classification)<\/li>\n\n\n\n
- Upload<\/a> documents<\/li>\n\n\n\n
- Go to the Documents<\/em> view<\/li>\n\n\n\n
- Inspect<\/a> documents to see what they look like, and explore differences<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-14-1024x495.png\")
<\/figure>\n<\/div>\n\n\nStep 2: Define labels<\/h2>\n\n\n\n
- \n
- Go to the Labels<\/em> view<\/li>\n\n\n\n
- Create<\/a> labels\n
- \n
- One label <\/strong>= one category<\/strong><\/li>\n\n\n\n
- Think whether a document could belong to one or several categories<\/strong><\/li>\n\n\n\n
- It is possible to create a label to obtain better results even if the label is not used to create a classification model<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Write<\/a> annotation guidelines for each label (recommended)<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-15-1024x498.png\")
<\/figure>\n<\/div>\n\n\nStep 3: Pre-annotate documents (optional)<\/h2>\n\n\n\n
- \n
- Why using automatic pre annotation?\n
- \n
- Pre-annotation with an off-the-shelf model<\/strong> or NLP pipeline<\/strong> saves time because a first version of the training dataset is created automatically<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Pre-annotate<\/a> documents\n
- \n
- At least some of the labels <\/strong>(categories) of the existing model\/NLP pipeline should perfectly match the labels you want to create<\/li>\n\n\n\n
- Pre-annotate a small number of documents<\/strong> (say 50) to start with, because you all annotations need to be reviewed individually in order to create a high quality dataset<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Select \u201cLabelled\u201d<\/strong> in the filter \u201cStatus<\/strong>\u201c to access the dataset<\/strong><\/li>\n\n\n\n
- Please note:\n
- \n
- The dataset <\/strong>consists of all labelled documents<\/strong><\/li>\n\n\n\n
- Useless labels (categories) can be deleted<\/a> together with their annotations in the Labels <\/em>view<\/li>\n\n\n\n
- When creating new labels, review all the annotated documents to complete any missing annotations (only for multi-category projects)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-16-1024x498.png\")
<\/figure>\n<\/div>\n\n\nStep 4: Annotate documents<\/h2>\n\n\n\n
- \n
- Annotate<\/a> at the document level\n
- \n
- Single or multiple categories<\/li>\n\n\n\n
- At least 10 to 15 annotations per label (category)<\/strong>, following the annotation guidelines<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Continue even after the first appearance of the blue \u201cpop up\u201c announcing that suggestions are available<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/01\/image-90-1024x500.png\")
<\/figure>\n\n\n\nStep 5: Use the suggestion engine<\/h2>\n\n\n\n
- \n
- Why using the suggestion engine?\n
- \n
- To speed up the dataset creation<\/li>\n\n\n\n
- To quickly assess the machine\u2019s ability to learn<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Go to the Suggestio<\/em>ns view\n
- \n
- Accept\/correct<\/a> the suggested categories then validate the document. It will be added to the dataset with its categorie(s).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Manage suggestions\n
- \n
- Sort them according to their confidence level score<\/strong><\/li>\n\n\n\n
- Filter <\/strong>the list on the label (category) you want to work on<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Please note:\n
- \n
- The suggestion engine is updated after a few validated suggestions<\/li>\n\n\n\n
- The suggestion engine is based on a machine learning algorithm with a fast training time (but which will not necessarily provide the best results)<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/01\/image-92-1024x499.png\")
<\/figure>\n<\/div>\n\n\nStep 6: Review the dataset<\/h2>\n\n\n\n
- \n
- Why reviewing the dataset?\n
- \n
- Dataset quality is essential to create the best possible model<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Go to the Labels <\/em>view\n
- \n
- Make sure the annotations are evenly distributed over the labels \u2026 as much as possible<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/01\/image-62-1024x684.png\")
<\/figure>\n<\/div>\n\n\n- \n
- Go to the Documents <\/em>view<\/li>\n\n\n\n
- Select \u201cLabelled\u201d<\/strong> in the \u201cStatus<\/strong>\u201c filter to access the dataset<\/strong><\/li>\n\n\n\n
- The dataset must be as accurate as possible: without false or missing categories and no inconsistencies<\/a> between categories<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/01\/image-93-1024x498.png\")
<\/figure>\n<\/div>\n\n\nStep 7: Split the dataset<\/h2>\n\n\n\n
- \n
- Why spliting dataset?\n
- \n
- To make sure we will use the same training <\/strong>and test <\/strong>sets to compare different model experiments<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Go to the Model experiments <\/em>view<\/li>\n\n\n\n
- Split the dataset<\/a> by generating train\/test metadata<\/strong> on the dataset<\/li>\n\n\n\n
- Note:\n
- \n
- If new annotations are added to the dataset, the split will be automatically updated when launching a new experiment<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-17.png\")
<\/figure>\n<\/div>\n\n\nStep 8: Train models<\/h2>\n\n\n\n
- \n
- Go to the Model Experiments<\/em> view<\/li>\n\n\n\n
- Edit each predefined experiment, check the training options to use train <\/strong>& test <\/strong>metatada on “train_on” and “test_on” parameters<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2025\/05\/image-1.png\")
<\/figure>\n<\/div>\n\n\n- \n
- Launch<\/a>\u00a0the predefined experiments<\/li>\n\n\n\n
- Check the quality (F-measure) of each experiment and identify the best model<\/li>\n\n\n\n
- Note:\n
- \n
- If the F-measure is below 60%<\/strong> quality<\/strong>, enrich and improve the dataset<\/strong> by iteration (see next steps below)<\/li>\n\n\n\n
- Do not create new experiments to test different algorithms if the F-measure is below 60%, it is useless at this stage<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-18-1024x316.png\")
<\/figure>\n<\/div>\n\n\nStep 9: Iterate steps 4-5-6 above to achieve 60% accuracy<\/h2>\n\n\n\n
- \n
- In the Model experiments <\/em>view\n
- \n
- Identify labels with low quality<\/strong> by ticking the quality box<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Enrich the dataset on these labels either:\n
- \n
- with new manually annotated segments (see above: 4 \u2013 Annotate text)<\/li>\n\n\n\n
- or by using Suggestions<\/em> (see above: 5 \u2013 Use the suggestion engine)<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- In the Model experiments<\/em> view\n
- \n
- Run the experiment again and see if the accuracy of the model has improved for each label<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Iterate<\/strong>\u2026 until achieving at least a 60%<\/strong> accuracy <\/strong>for each label<\/li>\n<\/ul>\n\n\n\n
![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/01\/image-4-1024x498.png\")
<\/figure>\n<\/div>\n\n\nStep 10: Annotate the dataset automatically<\/h2>\n\n\n\n
- \n
- Why annotating the dataset?\n
- \n
- To test the dataset and model quality<\/strong><\/li>\n\n\n\n
- To detect possible discrepancies<\/strong><\/li>\n\n\n\n
- It is only useful if model accuracy is above 60%<\/strong><\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Go to the Documents <\/em>view<\/li>\n\n\n\n
- Run<\/a> an automatic annotation of the dataset with the model<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/01\/image-6.png\")
<\/figure>\n<\/div>\n\n\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/01\/image-5.png\")
<\/figure>\n<\/div>\n\n\nStep 11: Identify discrepancies<\/h2>\n\n\n\n
- \n
- Go to the Documents <\/em>view<\/li>\n\n\n\n
- Open the filter \u201cAgreement: automatic-other<\/strong>\u201d<\/li>\n\n\n\n
- Select “Disagreement<\/strong>“<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-19-1024x498.png\")
<\/figure>\n<\/div>\n\n\n- \n
- Check the origin of the annotation with the letter or the tooltip on the chips<\/li>\n\n\n\n
- If the model is right after all, correct <\/strong>the dataset manually.<\/li>\n\n\n\n
- When corrections are made, remove the automatic annotations from the model.<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-20.png\")
<\/figure>\n<\/div>\n\n\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-21.png\")
<\/figure>\n<\/div>\n\n\n- \n
- Re-train the model. You will improve the model’s precision. <\/li>\n<\/ul>\n\n\n\n
Step 12: Train the final model<\/h2>\n\n\n\n
- \n
- Why selecting a final model?\n
- \n
- To compare different algorithms to judge their accuracy<\/li>\n\n\n\n
- Probably neither the suggestion model nor the pre-packaged experiments result in the best model. In this case, it is necessary to experiment to select the final model .<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Go to the Model experiments<\/em> view<\/li>\n\n\n\n
- Create<\/a> additional model experiments, launch models and compare the quality (F-measure)<\/li>\n\n\n\n
- Note:\n
- \n
- The goal is to achieve an accuracy between 80% and 95% (F-measure)<\/li>\n\n\n\n
- Don\u2019t expect to achieve a 100% accuracy\u2026 but you might reach this in some simple cases<\/li>\n\n\n\n
- Performance might be as important as accuracy, in which case it is not necessarily the highest quality model that is selected as the final model<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n\n\n
\n\n\n\nEntity detection<\/h1>\n\n\n\n
Step 1: Initiate the project<\/h2>\n\n\n\n
- \n
- Create<\/a> the project (type=Entity detection)<\/li>\n\n\n\n
- Upload<\/a> documents.\n
- \n
- If documents are short (a few sentences), it is not necessary to create segments.<\/li>\n\n\n\n
- Otherwise, use the default segmentation engine to start with.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Inspect<\/a> documents and segments\n
- \n
- Go to the Document<\/em>s view and read several documents to see what they look like, how they differ<\/li>\n\n\n\n
- Go to the Segments <\/em>view and check if segmentation is good and appropriate. A different and better segmentation may be necessary. In this case use either an off-the-shelf segmenter<\/a> or build a custom segmentation pipeline<\/a>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n
\n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-22-1024x616.png\")
Document view<\/figcaption><\/figure>\n<\/div>\n\n \n![\"\"](\"https:\/\/kairntech.com\/doc\/wp-content\/uploads\/sites\/2\/2023\/03\/image-23-1024x459.png\")
Segment view<\/figcaption><\/figure>\n<\/div>\n\n\n Step 2: Define labels<\/h2>\n\n\n\n
- Go to the Segments <\/em>view and check if segmentation is good and appropriate. A different and better segmentation may be necessary. In this case use either an off-the-shelf segmenter<\/a> or build a custom segmentation pipeline<\/a>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n
- Go to the Document<\/em>s view and read several documents to see what they look like, how they differ<\/li>\n\n\n\n
- Upload<\/a> documents.\n
- Create<\/a> the project (type=Entity detection)<\/li>\n\n\n\n
- Create<\/a> additional model experiments, launch models and compare the quality (F-measure)<\/li>\n\n\n\n
- Why selecting a final model?\n
- When corrections are made, remove the automatic annotations from the model.<\/li>\n<\/ul>\n\n\n
- Open the filter \u201cAgreement: automatic-other<\/strong>\u201d<\/li>\n\n\n\n
- To detect possible discrepancies<\/strong><\/li>\n\n\n\n
- To test the dataset and model quality<\/strong><\/li>\n\n\n\n
- Why annotating the dataset?\n
- In the Model experiments<\/em> view\n
- Enrich the dataset on these labels either:\n
- Identify labels with low quality<\/strong> by ticking the quality box<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Do not create new experiments to test different algorithms if the F-measure is below 60%, it is useless at this stage<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n
- Edit each predefined experiment, check the training options to use train <\/strong>& test <\/strong>metatada on “train_on” and “test_on” parameters<\/li>\n<\/ul>\n\n\n
- Go to the Model Experiments<\/em> view<\/li>\n\n\n\n
- Go to the Model experiments <\/em>view<\/li>\n\n\n\n
- To make sure we will use the same training <\/strong>and test <\/strong>sets to compare different model experiments<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Select \u201cLabelled\u201d<\/strong> in the \u201cStatus<\/strong>\u201c filter to access the dataset<\/strong><\/li>\n\n\n\n
- Go to the Documents <\/em>view<\/li>\n\n\n\n
- Make sure the annotations are evenly distributed over the labels \u2026 as much as possible<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n
- Why reviewing the dataset?\n
- Filter <\/strong>the list on the label (category) you want to work on<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Manage suggestions\n
- Accept\/correct<\/a> the suggested categories then validate the document. It will be added to the dataset with its categorie(s).<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Continue even after the first appearance of the blue \u201cpop up\u201c announcing that suggestions are available<\/li>\n<\/ul>\n\n\n\n
- Useless labels (categories) can be deleted<\/a> together with their annotations in the Labels <\/em>view<\/li>\n\n\n\n
- Pre-annotate a small number of documents<\/strong> (say 50) to start with, because you all annotations need to be reviewed individually in order to create a high quality dataset<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Pre-annotate<\/a> documents\n
- Pre-annotation with an off-the-shelf model<\/strong> or NLP pipeline<\/strong> saves time because a first version of the training dataset is created automatically<\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Think whether a document could belong to one or several categories<\/strong><\/li>\n\n\n\n
- Create<\/a> labels\n
- Upload<\/a> documents<\/li>\n\n\n\n
- Entity detection (what is entity detection<\/a>?)<\/li>\n\n\n\n