The Cityscapes Dataset

Introduction

Cityscapes is commonly used for semantic segmentation. Its data is divided into 8 categories in total, including one named “void”, and each category contains multiple classes. Cityscapes has 30 classes in all, but once labeled there are 35 kinds of labels in total, which also include labels such as “unlabeled” that are not counted as classes.

The data most commonly used in papers is only a portion of this, namely the finely annotated dataset. Some additional coarsely annotated data is used relatively rarely. Let’s first introduce this commonly used data.

Some of the commonly used data is shown in the figure below:

The top two are the masks after annotation, and the bottom two are the original data, i.e. the 8-bit photos. These two pairs of data are also split into two parts: one part is finely annotated, and the other is additional coarsely annotated data, corresponding to train_extra, which is usually not used for training. For the finely annotated data, there are 5000 photos in total along with their corresponding annotations, of which 2975 are used for train, 500 for val, and 1525 for test and benchmark.

The Annotated Data in Cityscapes

Here we focus on the annotated data in Cityscapes, which is a bit more involved. For each image, there are four kinds of annotation data, also included in the gtFine folder. The four annotations corresponding to each image are: xxx_gtFine_color.png, xxx_gtFine_instanceIds.png, xxx_gtFine_labelIds.png, and xxx_gtFine_polygons.json. Among them, xxx_gtFine_color.png is a colored representation of the annotation, mainly used for visualization; xxx_gtFine_instanceIds.png is the annotation used for the instance-level segmentation task; xxx_gtFine_labelIds.png is the annotation used for pixel-level segmentation; and xxx_gtFine_polygons.json contains the specific information of the annotation, including the image dimensions, the annotated category, and the vertices of the polygons.

Note that the files here are not suitable for use in training directly. Cityscapes assigns a number to every label, including things like sky; we call this number the absolute ID.

For the instance-level task, in xxx_gtFine_instanceIds.png, the grayscale value corresponding to each pixel does not map one-to-one to these IDs; instead, some functions are needed to convert them. These functions can be found in cityscapesscripts. The converted file is xxx_gtFine_instanceTrainIds.png, after which it can be used for instance-level training.

For the pixel-level task, likewise you cannot directly use xxx_gtFine_labelIds.png as the annotation. The grayscale values here represent the number Cityscapes assigns to each label, i.e. the absolute ID, and many of these labels are not used in training or in the final val, such as sky. So these IDs also need to be converted. You can use the createTrainIdLabelImgs function in cityscapesscripts to convert them into xxx_gtFine_labelTrainIds.png, where the grayscale value corresponding to each pixel is a temporary ID, comprising the IDs -1, 0->18, and 255. Neither the -1 nor the 255 IDs will be detected during val, so the two can be grouped into ID 19, which means the final training IDs become the 20 classes of labels from 0 to 19, and therefore there are 20 categories to classify in total. Note that if you want to submit your results to the Cityscapes benchmark, you must convert these classes back to the absolute IDs before uploading. The correspondence between the various IDs and colors can be found in the labels.py file in cityscapesscripts, where the trainId is precisely the temporary ID used for pixel-level segmentation training.

The category annotations in Cityscapes