Superpixel segmentations for thin sections: evaluation of methods to enable the generation of machine learning training data sets

Training data is the backbone of developing either Machine Learning (ML) models or speciﬁc deep learning algorithms. The paucity of well-labeled training image data has signiﬁcantly impeded the applications of ML-based approaches, especially the development of novel Deep Learning (DL) methods like Convolutional Neural Networks (CNNs) in mineral thin section images identiﬁcation. However, image annotation, especially pixel-wise annotation is always a costly process. Manually creating dense semantic labels for rock thin section images has been long considered as an unprecedented challenge in view of the ubiquitous variety and complexity of minerals in thin sections. To speed up the annotation, we propose a human-computer collaborative pipeline in which superpixel segmentation is used as a boundary extractor to avoid hand delineation of instances boundaries. The pipeline consists of two steps: superpixel segmentation using MultiSLIC, and superpixel labeling through a speciﬁc-designed tool. We use a cutting-edge methodology Virtual Petroscopy (ViP) for automatic image acquisition. Bentheimer sandstone sample is used to conduct performance testing of the pipeline. Three standard error metrics are used to evaluate the performance of Mul-⇤

• In image classification (Fig. 1a), the input is usually a tagged image while the goal is to 23 predict the correct class label of the entire image. In the case of petrographic thin section 24 analyses, class labels are usually referring to lithology, rock type or texture (e.g. Marmo  (Russell et al., 2008). 48 Well annotated data sets are being used not only to learn classifiers, but to reliably identify and 49 evaluate the promising methods (Hradiš et al., 2012). In the field of petrographic thin section 50 analyses, such a large annotated data set is not yet available. And as sketched in Fig. 1, a label-51 ing of image domains and grain boundaries requires detailed and careful line draws-a work that 52 quickly becomes infeasible for large thin section data sets. It is also worth noting that labeling of 53 petrographic thin sections is a highly specialized task that cannot be outsourced easily to labeling 54 services: whereas it is possible for almost everyone to identify a tra c light in an image (e.g. 55 Von Ahn et al., 2004), separating quartz from feldspar in a thin section requires specific training 56 and expertise. 57 To overcome the problem of limited data sets for thin section analysis, previous approaches 58 used pre-trained models using the principle of transfer learning (e.g. Zhang et al., 2019) or data 59 augmentation (e.g. Karimpouli and Tahmasebi, 2019). However, these approaches are also limited, 60 as minerals in thin sections show specific characteristics that are di↵erent to most of the images in 61 classical data sets (for example those mentioned above) and the transfer is therefore limited. Also, 62 even if augmentation methods can successfully be used to obtain more robust classification results, 63 the possibility to identify features is still limited to the variability in the initial (small) data set. 64 Based on these preliminary considerations, we derive the premise that fully labeled data sets 65 of thin sections are required to evaluate the full potential of novel machine learning algorithms.

66
However, we also need methods to facilitate labeling by experts, making specific use of the char-67 acteristics in thin sections.

68
In this context, we propose the idea that such a data set including high-quality pixel-wise  High-resolution images at di↵erent rotation angles can be precisely matched on a pixel-level.

106
This matching allows to determine the extinction behaviour at each pixel location, which can be 107 interpolated with a smoothed function to compress information (Virgo et al., 2016). Based on 108 the interpolated extinction information, a phase map can be produced to qualitatively show the 109 mineral axis misorientations. This specific system also contains a dedicated displaying toolbox 110 to allow users to adjust colouring of images and to evaluate extinction angles and behaviour. As The sample data set used in the paper is Bentheimer Sandstone (BS), which is one of the most 116 well-known sandstone types in Europe. Locations with outcrops of this sandstone can be found (b) incomplete contour map generated based on a single layer in the image stack, in contrast, (c) shows the desired result containing the full shape of grains using the methodical approach; (d) Boundary maps delineated by di↵erent human annotators on the border between the Netherlands and Germany. It forms a significant reservoir rock for petroleum reservoirs and is characterized by loose compaction, simple mineral composition and a 119 well-sorted grain and pore space network (Peksa et al., 2015). The sandstone is mainly composed 120 of loosely packed detrital quartz grains, with additional 2-4% of altered Feldspar. Due to its 121 properties and the block-scale homogeneity and lateral continuity, this rock type is widely used as 122 a standard reservoir analogue. In the following, we use a sample of this sandstone which has some 123 detrital chert fragments, which appear as speckled grain composed of coarsely crystalline quartz.

124
Pores in the sample were impregnated with light blue-dyed epoxy resin. Evaluating the quantitative performance of superpixel segmentations for a digital thin section 127 (ViP data set) requires a ground truth, which is manually generated by an expert. In order to 128 obtain comparable ground truth maps from multiple experts, we devised a tracing plan for the 129 procedure to reduce a potential procedural bias. The detailed tracing plan consists of four steps: complete boundary map that includes all boundary information obtained from all image layers.

142
As shown in Fig. 2(d), three di↵erent ground truth boundary maps are generated by three di↵erent  label, which is then assigned to all pixels within this superpixel.

158
A schematic example of the superpixel annotation process is provided in Fig. 3 159 Instead of providing a discrete representation of images, superpixels are better aligned with    in a post-processing step. All of these aspects will be considered and evaluated in the following 228 sections.

229
In a first step, we compare six state-of-art algorithms ( of SLIC to the specific aspects of thin section data sets in the following.

285
The general operating principle of MultiSLIC is practically identical to SLIC. To accommodate the multidimensional input, the distance measure D s di↵ers from SLIC as follows: where m denotes the specific dimension and therefore d m lab expresses the Euclidean distance between Calculate distance D s (Eq: 2) to C k for all pixel from a 2S ⇥ 2S neighbourhood around C k .

6
Assign each pixel to the closest center.

7
Recalculate cluster centers and corresponding residual error E.
8 until E  threshold; 9 Enforce connectivity.  G as the undersegmentation error metrics. For example, this would be (|B out + C out + D out |)/ |G| in Fig. 7.
However, in such a way, superpixels that slightly cover the segments will be overly penalized.

325
As a result, we adopt Neubert (2015)'s formulation expressed in Eq.4: when a superpixel is only 326 slightly crossing a ground truth boundary, it will not a↵ect the UE with the whole superpixel size, 327 but only with the small overlapping part, in the example of Fig. 7, that is (|B in + C out + D in |)/ |G|.
where P (S) is the perimeter of the superpixel. Q s takes a maximum value of one for a circle and value ⇡/4 for a square. Then the compactness of superpixel segmentation is defined as the average of isoperimetric quotient weighted by the size of superpixel compared to the whole image: Superpixel segmentations with a high CO are considered to be more compact. We will now evaluate how these previously described superpixel algorithms perform in a quan- Interesting to note is also that the variation of the input image does not seem to a↵ect the

373
Also interesting to note is that SLIC performs worse on the ppol image, but better at others. This 374 is likely to be related to the fact that MutliSLIC recovers more objects than SLIC and that, 375 therefore, fewer domains without boundaries exist in the SLIC segmentation, leading to more 376 compact superpixels-but at the cost of missing information (see also Fig. 6).

377
Overall, MultiSLIC performs well on both a qualitative and quantitative level and can be 378 used as a pre-segmentation algorithm dealing with the high dimensionality input from digital thin 379 sections.
380 Figure 10: Compactness on four di↵erent information layers of the ViP data set. K represents the number of superpixels, K varies from 200 to 3000 in steps of 200. CO denotes Compactness. A higher CO represents that the shape of superpixel more resembles a circle.

Considerations for practical use 381
The presented superpixel methods, and specifically the developed extension to multiple channels

419
Thin section data sets also pose specific requirements to superpixel segmentation algorithms.

420
The most obvious aspect is that thin section data sets contain more than just a single image, due to to use multiple image layers, resulting in the adapted algorithm MultiSLIC. 425 We evaluated several algorithms with respect to their successful use for the specific requirements 426 of thin section data sets. In Fig. 8 and ETPS, both of them successfully detect more than 99% boundaries simply using a ppol layer.

429
However, an interesting observation can be obtained when analysing Fig. 8, 9, and 10 together:

447
A classical way to simplify the initial segmentation is to merge adjacent regions on the basis of color similarity and spatial proximity (Trémeau and Colantoni, 2000). Region merging can be 449 carried out by constructing a map graph with each node associated with a region, and each pair of 450 adjacent regions are connected by an edge representing the relationship between adjacent regions 451 (Schettini, 1993). This map graph is called Regional Adjacency Graph (RAG) that provides a  We evaluated here the superpixel segmentation with a relatively simple data set, a thin section 471 of the Bentheimer sandstone. The internal structure of this rock type is rather simple, with quartz 472 as the dominating mineral and only low secondary alteration and limited di↵erences grain shapes.
473 Figure 13: Result of RAG merging of initial superpixel segmentation using di↵erent threshold values. Using of higher threshold value will merge more superpixels.
In future work, it would be interesting to apply the best-performing algorithms to a variety of 474 di↵erent rock types, for a detailed evaluation on segmentation of di↵erent rock types. In this paper, we proposed a human-computer collaborative pipeline to speed up the pixel-wise 484 labeling of petrographic thin section images. In order to avoid subjective visual interpretations and 485 hand delineations of the region boundaries, an algorithm will first splits images into superpixels. 486 We have proposed a novel superpixel algorithm to cope with the high input dimensionality of digi-    clustering algorithms for normal and uniform distributions of data points. Journal of computer