Skip to content

Semantic Segmentation Mode

In semantic segmentation mode, the model assigns each pixel in an image to a predefined class. By defining and annotating these classes, users can train the model to identify different regions of interest.

Dataset Creation

To start creating a dataset for semantic segmentation, follow these steps:

  1. Click the View Collections button on the Home page.
  2. Click the Add a new Segmentation dataset button to create a new dataset.
  3. Enter the dataset name and description, and register its classes.
  4. Upload images to the dataset by clicking on the Upload Images region.
  5. Press Submit and Open Button to proceed with dataset annotation.

The dataset creation page will look like the following:

CVAT

Annotation Process

Once the dataset is created, you can start annotating the images. The annotation interface provides various tools to facilitate the annotation process:

  • Draw mask Tool: Allows users to paint over regions of interest using a brush to assign them to a specific class.
  • Eraser Tool: Enables users to remove parts of the mask that were incorrectly annotated.
  • Polygon/Rectangle Selection Tool: Enables users to create precise masks by outlining the area of interest with polygons or rectangles.

CVAT

Important Note: The Draw mask tool is particularly useful for complex shapes and regions in semantic segmentation.

Dataset Registration

After completing the annotation process, you have to export the dataset using the Segmentation Mask 1.1 export format from CVAT.

The generated segmentation masks can be found under the SegmentationClass/ directory of the exported dataset.

Finally, the dataset should follow the structure below:

DatasetName/
├── images/
│   ├── train/
│   │   ├── image_001.jpg
│   │   ├── image_002.jpg
│   │   └── ...
│   └── val/
│       ├── image_101.jpg
│       ├── image_102.jpg
│       └── ...
├── masks/
│   ├── train/
│   │   ├── image_001.png
│   │   ├── image_002.png
│   │   └── ...
│   └── val/
│       ├── image_101.png
│       ├── image_102.png
│       └── ...
└── labelmap.txt

labelmap.txt defines the mapping between class IDs (colors) and class names. It is automatically exported from CVAT when using the Segmentation Mask 1.1 export format.

Once the dataset structure is finalized:

  1. Compress the DatasetName/ directory into a .zip archive.
  2. Upload the generated .zip file to AmalthAI using the Add a new segmentation dataset button.

Training Initiation

After uploading the dataset, you can train a semantic segmentation model. Follow these steps:

  1. Navigate to the Train an ML model page.
  2. Select the Semantic Segmentation mode.

There are two main fields that need to be filled:

  • Select Model: Choose the model architecture you want to use for training.
  • Select Collection: Choose the dataset to use for training.

To learn about the available architectures, expand Available models. Use a model's information icon to read its description.

Inference Process

Once the model is trained, you can use it for inference on new images. To perform inference, follow these steps:

  1. Navigate to the Inference page.
  2. Select the trained semantic segmentation model from the list.
  3. Upload the image you want to perform inference on.
  4. Click Start Inference to see the segmentation results.

When inference finishes, the same page shows the segmented image with its regions assigned to the predefined classes.

CVAT

Evaluation Metric

For semantic segmentation, the evaluation metric is mean Intersection over Union (mIoU). It is computed by first measuring for each class, how well the predicted pixels match the ground truth pixels in terms of overlap, balancing correct detections against both missed regions and incorrect predictions. This overlap score is calculated separately for every class in the dataset. The final mIoU is obtained by averaging these per-class overlap scores, giving a single value that reflects the overall quality of the segmentation across all categories.

miou_fig