FreeMix: Open Vocabulary Domain Generalization of Remote Sensing Images for Semantic Segmentation

1. Introduction

Remote sensing images (RSI) are typically obtained from satellites, aerial platforms, or drones and provide valuable information about the Earth’s surface and its features. Semantic segmentation of these images, where the goal is to classify each pixel in an image into predefined categories, is crucial for various applications including urban planning, environmental monitoring, agriculture, disaster management, and military intelligence. However, classes that are not labeled and not visible during the training process cannot be recognized during the inference stage, which greatly limits the scope of application. For instance, the Potsdam dataset [1], which is widely used for benchmarking semantic segmentation algorithms, only includes 5 classes. We call them base classes. But in reality, RSI often have more than 5 different types of objects. The classes that fall outside the 5 classes are called novel classes, which are presented as clutter or background in Potsdam dataset. To identify novel classes, researchers have introduced different concepts like open-set learning [2,3,4], open world learning [5,6], out-of-distribution detection (OOD) [7,8], zero-shot learning (ZSL) [9,10,11,12,13,14,15,16],and open vocabulary learning (OVL) [17,18,19,20,21,22,23,24,25,26,27]. The difference of these concepts are shown in Figure 1(a). The open-set, open world, and OOD tasks only need to identify novel classes and set them as one label, named ‘unknown’. They do not need to identify the specific class categories in such settings. But zero-shot must classify novel classes into specific categories by using predefined word embeddings [28,29,30]. During training, zero-shot learning model is strictly trained on base classes. In the open vocabulary setting, the model can classify novel classes with the help of pretrained vision language models, which has large language vocabulary knowledge but is not strictly required to contain base classes and novel classes. Compared with zero-shot learning, open vocabulary learning can further extend models’ generalizability on class-level [31]. Despite the resounding success of open vocabulary learning in the computer vision field, its potential application in the context of RS imagery remains relatively unexplored. In addition, open vocabulary learning assumes that samples come from a single known domain, resulting in limited applications in the real world.

Due to the influence of spatial resolution, shooting time, shooting equipment, and geographical location, RSI have different styles. Therefore, there is a significant domain shift between different remote sensing datasets. The model trained on a specific domain exhibits poor performance on the target domains, which have a different but related distribution. To this end, it is necessary to develop a model that can generalize to unseen distributions. As illustrated in Figure 1(b), domain adaptation (DA) focuses on adapting a model from one domain to another, while domain generalization aims to develop models that can generalize well across multiple domains without explicit domain-specific data during training. Besides, domain generalization (DG) involves addressing a difficult scenario where one or multiple distinct yet interconnected domains are provided, with the objective being to train a model capable of generalizing effectively to an unseen test domain. However, existing techniques for domain generalization presuppose the presence of identical classes across all domains, which limits the practical applicability of these methods. Our endeavor seeks to alleviate this constraint by enabling unseen test domains to contain novel classes absent in any training domain. As indicated in Figure 1(c), we introduce this more challenging setting as Open Vocabulary Domain Generalization (OVDG) for semantic segmentation, which, to the best of our understanding, represents the inaugural attempt at tackling this issue. Compared with DG, our OVDG demands not only considering the generalization performance on unseen domains but also identifying novel classes. The closest match to our setting is universal domain adaptation [32,33,34], and the primary distinction of our OVDG lies in involving source domains and identifying novel classes rather than simply labeling them as ’unknown’.

Existing semantic segmentation models [35,36,37,38] lack the capacity to handle OVDG task. For example, conventional semantic segmentation models are trained on closed set, which limits their ability to generalize at the class and domain levels. While there are existing methods [39,40,41,42] that focus on the model’s capacity to generalize to new visual concepts, such as zero-shot learning (ZSL) or generalized zero-shot learning (GZSL), they are dependent on semantic information from novel classes, such as visual attributes [43,44] or Word2Vec vector representations [41,44].

In OVDG, the source data and target data remain isolated from each other. In this setting, aligning the domain distributions becomes unfeasible, unlike traditional methods [9,45,46] that rely on the simultaneous presence of source and target data. This paper introduces FreeMix, an efficient framework for Open Vocabulary Domain Generalization (OVDG). FreeMix includes a dual-branch segmentation module with a Base Segmentation Branch (BSB) and an Entity Segmentation Branch (ESB), along with a CLIP-based recognition module. The dual-branch design generates diverse masks and visual-semantic features for class- and domain-level generalization. The BSB integrates the CMID model [47], adapted for mask and semantic logit decoding as RS_SSL, while the ESB incorporates entity masks from a pre-trained extractor and a custom feature extractor. Outputs from both branches are fused for open-category recognition using CLIP. By freezing key components, we reduce trainable parameters and computation. To enhance performance across domains, we propose Dataset-Aware Sampling (DAS), ensuring uniform domain sampling in each mini-batch. Our contributions are as follows:

• We introduce a new setting for semantic segmentation, i.e., open vocabulary domain generalization (OVDG), which is an important yet unstudied problem. In addition, we propose an effective framework FreeMix for solving OVDG, which focuses on learning a generalized model by integrating entity mask to enhance the diversity and completeness of masks for both base classes and novel classes.
• We propose a dual-branch universal segmentation module by unifying the base segmentation branch (BSB) and the entity segmentation branch (ESB) in an end-to-end trainable framework, where the BSB leverages a self-supervised pre-trained model, CMID, to extract domain-agnostic visual features for decoding masks and semantic logits.
• To integrate and leverage information from various source domains, we propose a simple yet effective training strategy, called dataset-aware sampling (DAS). Extensive experiments on four benchmark datasets reveal that our proposed method outperforms the state-of-the-art methods on the OVL and the OVDG benchmark.

In the following sections of this paper, we will first review the relevant literature in Section 2, which discusses related work in the field. Section 3 introduces our proposed FreeMix method, detailing its innovative approach and underlying principles. In Section 4, we outline the experimental setup, datasets used, and present the results of our experiments, providing a comprehensive analysis of the performance of the FreeMix method. Finally, Section 5 concludes the paper, summarizing our findings and suggesting directions for future research.

2. Related works

2.1. Open Vocabulary Semantic Segmentation

With the development of Visual Language Pre-training Models (VLPMs), such as CLIP [48] and ALIGN [49], models can now localize and recognize classes beyond the annotated label space, no longer confined to identifying predefined classes present in the training set. Many studies have successfully transferred their robust class generalization capabilities to pixel-level classification tasks, such as semantic segmentation. Based on a pretrained CLIP model, Chen et al. [50] use a conditional Unet model to predict segmentation masks and use text descriptions and annotations from OpenStreetMap as auxiliary supervision. Exploration of open vocabulary semantic segmentation in the remote sensing field is relatively limited, while it is more prevalent in natural images. ZegFormer [16], as a simple yet effective zero-shot semantic segmentation model, decomposes the problem into a class-agnostic segmentation task and mask classification tasks. Furthermore, it transfers semantic knowledge from seen classes to unseen classes solely with the assistance of VLPMs. Similarly, ZSSeg [51] proposes a two-stage semantic segmentation framework where the first stage extracts generic candidate masks, and the second stage utilizes CLIP model for open-vocabulary classification of the mask images generated in the first stage. To circumvent the time-consuming process of clipping image patches and computing features from an external pre-trained CLIP model, MaskCLIP [19] designs a Relative Mask Attention (RMA) module, treating segmentation as additional tokens for ViT CLIP models.

Unlike the two-stage segmentation models discussed earlier, SAN [52] attaches a lightweight side network to a pretrained VLM to predict candidate masks and classification outputs. To enhance the versatility of the framework, FreeSeg [24] jointly learns multiple related segmentation tasks, including Open Vocabulary Semantic Segmentation (OVSS), Open Vocabulary Instance Segmentation (OVIS), and Open Vocabulary Panoramic Segmentation (OVPS). Open vocabulary methods achieve impressive results on natural images, but remain relatively unexplored in the field of remote sensing. Existing visual language pre-training models lack participation of remote sensing images in training, resulting in limited generalization capabilities in the field of remote sensing. To address this limitation, some recent studies have specifically constructed large-scale image-text pairs datasets in the remote sensing domain for training visual language models. For example, RS5M [53], SkyScript [54], MMRS [55].

RemoteCLIP transforms heterogeneous annotations of detection boxes and segmentation masks into a unified image-caption data format through Box-to-Caption (B2C) and Mask-to-Box (M2B) strategies for training purposes. In addition, GeoRSCLIP [53] proposes an image-text paired dataset RS5M in the remote sensing domain, consisting of 5 million RS images with English descriptions. Furthermore, building upon the alignment of ground images with text, GRAFT [56] constructs pairs of ground images and remote sensing images to train VLP models, thereby enabling the training of remote sensing image vision-language models without using any text annotations.

2.2. Domain Generalization

Domain generalization (DG) task is one of the key challenges for deep learning models, facing domain shift between training and testing distributions. When the source domain consists of only one dataset, it simplifies to single-source domain generalization [57,58]. Existing domain generalization methods can be categorized into three main types. The first type [57,58] involves utilizing data augmentation to assist the model in learning universal representations, such as randomizing, transforming, or generating diverse data inputs to enhance the dataset. The second type [59,60,61] focuses on representation learning, aiming to learn domain-invariant representations or decompose representations into domain-shared and domain-specific components for better generalization, often incorporating feature alignment techniques to minimize discrepancies between different domains. The third type of methods [62,63,64,65] leverage general learning strategies to improve generalization capabilities, including ensemble learning, meta-learning, gradient manipulation, distributed robust optimization, and self-supervised learning. These three types of methods can complement each other and may be combined to achieve higher performance. Specifically, in the first type of methods, CCDR [57] performs texture and style randomization for simple yet effective auxiliary domain generation to improve the reliability of classification in arbitrary unseen target domains. Recently, several methods [66,67] have utilized Mixup [68] for domain generalization, generating new samples by applying Mixup directly in the original space.

The second category of methods has received considerable attention. A Maximum Mean Discrepancy Depth Reconstruction Classification Network (MMD-DRCN) [60] is proposed for detecting oil palm trees from multi-source high-resolution satellite images in a new environment. The core idea is to utilize the Maximum Mean Discrepancy (MMD) module to learn invariant features across different source domains. Moreover, to acquire domain-invariant features, the frequency-based optimal style mix (FOSMix) [59] model randomizes the styles of images in the source domain. Additionally, language-aware domain generalization network (LDGnet) [61] is proposed to learn cross-domain-invariant representation from cross-domain shared prior knowledge.

The third type of methods leverage general learning strategies to improve generalization capabilities. The approach of leveraging general learning strategies to improve generalization capabilities is simple yet effective, hence it has gained popularity. Segu et al. [69] maintain domain-specific batch normalization (BN) parameters for different source domains while sharing other parameters. Li et al. [62] proposes Meta-Learning for Domain Generalization (MLDG), applying meta-learning strategies to domain generalization. MLDG divides the data in the source domain into meta-training and meta-testing sets to simulate domain transfer scenarios for learning universal representations. Recently, Self-supervised Learning (SSL) has emerged as a popular learning paradigm, constructing self-supervised tasks from large-scale unlabeled data. SSL, as a universal paradigm, can be applied to any existing DG method, particularly for unsupervised domain generalization where labeled data is unavailable in the training domain. Bhattacharya et al. [70] proposes a self-supervised prompt learning approach for remote sensing images, which preserves domain-invariant feature learning while enhancing the representation of visual features. Our approach falls into this category by introducing a self-supervised backbone network to achieve domain generalization.

2.3. Self-Supervised Learning in Remote Sensing

Self-supervised learning (SSL) has garnered significant attention in the remote sensing community and has undergone initial exploration in this field. SSL methods leverage large amounts of unlabeled data to learn generic representations, which can enhance the performance of downstream tasks. Contrastive self-supervised learning involves constructing contrastive learning tasks to train models by leveraging the inherent positive and negative samples within remote sensing data. For instance, SauMoCo [71] utilizes the semantic similarity between nearby geographical locations and the inherent diversity within land cover concepts to train the model. Moreover, Kumar et al. [72] utilizes temporally aligned images as positive sample data and introduce an auxiliary task of predicting the source of the images to enhance pretraining effectiveness. Similarly, Oscar et al. [73] exploits seasonal information inherent in the data to construct tasks for SSL. In addition to temporal information, Dilxat et al. [74] proposes IndexNet, which learns spatiotemporal invariant features by combining image-level contrast and pixel-level contrast.

Compared to contrastive self-supervised learning, Masked Image Modeling (MIM) self-supervised learning has become more popular. Studies have focused on collecting data from various sources such as satellites or aerial platforms to build large-scale datasets covering multiple scenes worldwide for MIM self-supervised pretraining. RingMo [75] optimizes mask strategies for small objects in remote sensing images and employs the MAE model for self-supervised representation learning on a dataset comprising 3 million unlabeled remote sensing images. To better represent robust remote sensing data across various spatial scales, Scale-MAE [76] explicitly learns the relationships between data at different known scales throughout the entire pretraining process. In addition, to handle large-sized images and objects of different orientations in RS images, RVSA [77] introduces a new Rotated Varied-Size Window Attention (RVSA) mechanism, which significantly reduces computational costs and memory usage. Recently, a Transformer-based geographic spatial foundation model named Prithvi [78] has been proposed, pretraining it on over 1TB of multispectral satellite images from the Harmonized Landsat-Sentinel 2 (HLS) dataset.

Despite achieving some success, these methods have prerequisites in terms of underlying architectures; namely, contrastive self-supervised learning relies on CNNs, while most MIM methods are restricted to ViTs. In contrast, the CMID [47] method is not only agnostic to architectures but also combines contrastive and masked generation approaches to learn representations with global semantic separability and local spatial awareness. This ensures that the learned representations have sufficient generalization to meet the requirements of various downstream tasks in remote sensing. What’s more, SMLFR [79] has constructed a large dataset named GeoSense, comprising approximately 9 million diverse remote sensing images, to enhance the robustness and generalization capabilities of foundation models during the pretraining phase. Additionally, they implemented masked image modeling (MIM) based on CNNs architectures.

3. Proposed Method

3.1. Problem Definition

Let X represent visual space, and Y represent label space. We define a domain as the joint distribution of data space, it can be represented as: $D={\{(x_i,y_i)\in X_J\times Y_J\}}_{i=1}^J\sim P_{XY}$, where $x_i$ represents the $i^{th}$ sample, $y_i$ is the corresponding label and J is the size of the domain. We define $Y^B$ as label space with base classes, $Y^N$ as label space with novel classes, and $Y^{B+N}$ as label space with both base and novel classes. As illustrated in Figure 1(c), we have m source domains ${\mathbf{D}}_{\mathbf{src}}={\left\{\left(D_i^{train},D_i^{test}\right)\sim P_{X{Y}^{B+N}}^i\right\}}_{i=1}^m$ and a target domain ${\mathbf{D}}_{\mathbf{tar}}={\{(x_k,y_k)\in X_K\times Y_K^{B_t+N_t}\}}_{k=1}^K\sim P_{X{Y}^{B_t+N_t}}^t$ with K samples. The target domain can be more than one, but for the sake of simplicity in explanation, only one is presented here. Here $D_i^{train}={\left\{(x_j,y_j)\in X_J\times Y_J^{B_i}\right\}}_{j=1}^J\sim P_{X{Y}^{B+N}}^i$ is the training set of the $i^{th}$ source domain, $D_i^{test}={\{(x_l,y_l)\in X_L\times Y_L^{B_i+N_i}\}}_{l=1}^L\sim P_{X{Y}^{B+N}}^i$ is the testing set of the $i^{th}$ source domain, J is the size of the training set of the source domain, and L is the size of the testing set of the source domain. In the open vocabulary domain generalization (OVDG) setting, the base class labels of the target domain are a subset of the aggregated base class labels from the m source domains: $Y^{B_t}\subseteq \left(Y^{B_1}+Y^{B_2}+...+Y^{B_m}\right)$. Additionally, novel class labels in the target domain do not intersect with base class labels, formally expressed as $Y^{N_t}\bigcap Y^{B_t}=\varnothing$. The main objective of OVDG setting is to train a model on the m source domain, and perform well on both target domain and source domains.

3.2. Overview

In this section, we describe our proposed framework for OVDG, named FreeMix. Figure 2 shows the overall architecture of our proposed FreeMix, which comprises two main module: a universal segmentation module and an open vocabulary recognition module. Furthermore, the universal segmentation module consists of a base segmentation branch (BSB) and an entity segmentation branch (ESB). In the OVDG task, our attention is required to encompass not just the OV aspect, but also the DG aspect. When tackling domain generalization challenges, we incorporate a self-supervised base model, CMID, within the BSB branch and efficiently adapt it using mask2former, thereby achieving seamless preservation of pre-trained knowledge. Additionally, to tackle the challenge of open vocabulary, in the ESB, we integrate a diverse range of entity masks and design an extractor specifically for extracting the visual features of these masks. This endeavor seeks to enhances the model’s ability to segment and recognize both base and novel classes while maintaining generalizability. Finally, we utilize an open vocabulary recognition module to classify these masks based on CLIP [48] model. The classification method entails comparing the semantic logits of the masks with the cosine similarity of text embeddings derived from a contrastive language-image pre-trained model.

Next, we first introduce our universal segmentation module (Sec. Section 3.3), and than describe the proposed the dataset-aware sampling training tactic on multi-source domain (Sec. Section 3.4) thoroughly.

3.3. Universal Segmentation Module

The proposed universal segmentation module comprises two branches: a base segmentation branch (BSB) and an entity segmentation branch (ESB). Firstly, the diversity and completeness of extracted masks are pivotal factors influencing model performance in semantic segmentation. However, segmentation models trained on base classes often exhibit category bias, as novel classes are typically perceived as background and thus suppressed. Consequently, the mask proposal generator fails to truly achieve class-agnostic mask extraction, tending instead to extract masks corresponding to base classes. Even when unmatched candidate masks do not contribute to loss computation during training, this tendency persists. Moreover, due to disparities in data style between the target and source domains used for training, segmentation models trained on a specific data domain may exhibit sensitivity to stylistic differences in the target domain, leading to suboptimal outcomes. Hence, we endeavor to enhance model robustness and reduce sensitivity to data styles by incorporating dual branches (BSB and ESB). The BSB excels at segmenting the base classes in supervised learning. However, the single-branch BSB extracts very few masks for novel classes, resulting to low recall rates. Therefore, integrating an ESB is necessary to bolster mask extraction capabilities, especially for novel classes, by generating more universal masks.

1) Base Segmentation Branch (BSB) To make the model applicable across various data domains, we propose adapting self-supervised pre-trained backbone for remote sensing semantic segmentation, named RS_SSL. This backbone has been trained on large-scale remote sensing data. Consequently, the architecture of the BSB comprises a self-supervised pre-trained backbone network, a pixel decoder, and a Transformer decoder as illustrated in Figure 2. Given an image $X\in {\mathbb{R}}^{H\times W\times C}$, the BSB will output mask proposals $M_{\mathrm{b}}\in {\mathbb{R}}^{N\times H\times W}$ and theirs semantic logits $E_{\mathrm{b}}\in {\mathbb{R}}^{N\times d}$, where N is the number of mask proposals, d represents the dimensionality of semantic logits, and H, W and C correspond to image height, image width and number of channels. The backbone network of popular open-vocabulary segmentation models [16,23,24,51] is typically trained in a supervised manner on the ImageNet dataset [80] or trained from scratch. However, the major distinction lies in the fact that our backbone network is initialized using a self-supervised pre-trained model CMID [47] and remains frozen during training, with only the decoders being trainable. CMID combines contrastive learning and masked image modeling to learn robust representations in a self-distillation way. Moreover, it is architecture-agnostic, and is compatible with both convolutional neural networks (CNN) and vision transformers (ViT), allowing it to be easily adapted to our BSB. Compared to mask proposal generator of other models [52], we do not modify the foundational model, allowing us to leverage the powerful generalization capabilities of self-supervised learning in remote sensing images to extract domain-agnostic universal features.

2) Entity Segmentation Branch (ESB) To achieve the elimination of sensitivity across disparate data domains, the key is extracting class-agnostic masks and their corresponding visual features from images. In order to achieve this goal, we first employ CropFormer [81], a tiny entity segmentation model, as an entity mask extractor to generate high-quality entity masks. We then establish a visual feature extractor for these masks, leveraging the Transformer architecture.

Entity masks extractor. An entity refers to each semantically coherent region within an image. Entity segmentation is an emerging task that focuses on open-world, class-agnostic dense image segmentation. It is designed to have superior generalization capabilities for segmenting novel classes [82]. CropFormer [81], trained on a large-scale, high-quality entity segmentation dataset that includes images from various domains, including remote sensing images, is highly suitable for extracting class-agnostic masks. In entity masks extractor, we use NK-dimensional queries $Q\in {\mathbb{R}}^{N\times K}$ to generate entity masks $M_{\mathrm{m}}\in {\mathbb{R}}^{N\times H\times W}$.

Visual feature extractor. The visual feature extractor is based on vision Transformer (ViT) architecture which consists of 12 Transformer block. We denote these blocks as $B=\{b_1,b_2,\dots ,b_i,\dots ,b_{12}\}$. Each block comprises a multi-head attention layer followed by two MLP layers with GELU [83] non-linearity. Layer normalization is applied before each layer, and residual connections are added after each layer.

In the first k Transformer blocks, the visual feature extractor initially encodes the entire image $X\in {\mathbb{R}}^{H\times W\times C}$ to obtain a representation $E\in {\mathbb{R}}^{(hw+1)\times d}$. Here h and w represent the height and width of the attention map in the ViT, the additional 1 corresponds to the semantic logits for the entire image, and d denotes the dimensionality of the features. In the remaining $12-k$ Transformer blocks, to extract the semantic logits $E_{\mathrm{m}}\in {\mathbb{R}}^{N\times d}$ for entity masks $M_{\mathrm{m}}\in {\mathbb{R}}^{N\times H\times W}$, we assign independent classification queries to each entity mask by by repeating the semantic logits of the entire image N times. We then utilize the entity masks as the attention bias (A) in the Multihead Attention mechanism:

$$E_m^{b+1}=\mathrm{softmax}\left({\displaystyle \frac{\mathrm{Q}\left(E_m^b\right)\mathrm{K}{\left(E^b\right)}^{\mathrm{T}}}{\sqrt{d}}}+A\right)\mathrm{V}\left(E^b\right)$$

(1)

where b indicates the block number, $K\left(E^b\right)=W_k{E}^b$ and $V\left(E^b\right)=W_v{E}^b$ are the key and value embeddings of the representation, and $Q\left(E_m^b\right)=W_q{E}_m^b$ is the query embedding of entity masks. Here, $W_q$ , $W_k$ , and $W_v$ are the weights of the query, key and value embedding layer, respectively. To ensure the updating of semantic logits $E_m^b$ for entity masks, we exclusively consider the representation corresponding to entity mask and its own semantic logits, without referencing the semantic logits of other entity masks. We construct a self-attention bias matrix $A\in {\mathbb{R}}^{N\times \left(N+hw\right)}$ as follows:

$$A(x,y)=\left\{\begin{array}{cc}0\hfill & ,\mathrm{if}\overline{M}(x,y)=1\hfill \\ -\infty \hfill & ,\mathrm{if}\overline{M}(x,y)=0\hfill \end{array}\right.$$

(2)

$$\overline{M}=\mathrm{concat}[\mathrm{I}(N,N),f\left(M_m\right)]$$

(3)

where $(x,y)$ is the feature location, $\mathrm{I}(N,N)$ denotes the $(N,N)$ identity matrix, $f\left(M_m\right)$ represents resizing $M_m$ to $(h,w)$ and then flattening it, and $\overline{M}\in {\{0,1\}}^{N\times (N+hw)}$ is the binarized output (thresholded at 0.5) of the flattened mask.

In open vocabulary recognition module, given the base classes $Y^B$ during training, classes prompts $V_y$ are generated using the template: {"semantic segmentation" + learnable vectors + c }, where c represents the filled-in class names. Then the text prompts are then embedded using the pre-trained CLIP text encoder $\phi$:

$$E_t=\phi \left(V_y\right),y\in Y^B$$

(4)

To predict the class of masks $P\in {\mathbb{R}}^{C\times N}$, we compare the similarity between the semantic logits $E=\mathbf{concat}[\psi \left(E_{\mathrm{b}}\right),\psi \left(E_{\mathrm{m}}\right)]$ of mask groups $M=\mathrm{concat}[M_{\mathrm{b}},M_{\mathrm{m}}]$ and the text embedding $E_t\in {\mathbb{R}}^{C\times r}$. Here $\psi \left(\right)$ represents the normalization process, C is the number of classes and r is the dimension of text embeddings. Finally, we compute the semantic segmentation map $S=M\times P^T$.

3.4. Train Tactics: Dataset-Aware Sampling

In order to effectively improve the generalization ability of the model, it is necessary to train on multi-source datasets $D=\{D_1,D_2,...,D_i,...\}$ which exhibit different styles [84,85]. Each dataset has its own distinct label space $Y=\{Y_1,Y_2,...Y_i,...\}$. A straightforward approach to train on multiple datasets is to combine all annotations from these datatsets into a larger dataset $D=D_1\cup D_2\cup ...$, and then relabel and merge their label spaces into $Y=Y_1\cup Y_2\cup ...$. The model is optimized with the same loss on a larger dataset. Fortunately, the number of classes for labeling remote sensing semantic segmentation data is relatively small, making manual label mapping feasible. However, significant variations exist in dataset sizes: GID5 [86] contains $3\times$ more images than Potsdam [1], DeepGlobe [87] is $8\times$ larger than Potsdam [1], and URUR [88] is $79\times$ larger. This imbalance in class distributions and dataset sizes virtually ensures that a mere concatenation of datasets will not work.

To effectively utilize multi-source datasets, we propose a simple yet efficient training strategy: Dataset-aware Sampling (DAS). Re-sampling is a widely used strategy in addressing class-imbalanced or long-tail learning scenarios [89]. Inspired by this, we extend the concept of sampling to the data domain level, introducing dataset-aware sampling. Specifically, we uniformly sample instances from each dataset within every mini-batch. For each dataset, we then compute the loss individually for its samples. Finally, we aggregate these per-dataset losses by averaging them before performing backpropagation:

$$L_{\mathrm{overall}}={\displaystyle \frac{1}{T}}\sum _i{L}_i$$

(5)

where T is the number of datasets, and $L_i$ represents the loss from the $i^{th}$ dataset within a mini-batch. During training, $L_i$ comprises both mask losses and classification loss. Specifically, the mask losses include Dice loss $L_{dic}$ and binary cross-entropy loss $L_{bce}$, while the classification loss $L_{cls}$ uses cross-entropy loss.

$$L_i={\lambda }_1{L}_{\mathrm{dic}}+{\lambda }_2{L}_{\mathrm{bce}}+{\lambda }_3{L}_{\mathrm{cls}}$$

(6)

where ${\lambda }_1$, ${\lambda }_2$, ${\lambda }_3$ are set to 20, 5 and 2, respectively, in our setting. In the proposed DAS, we do not need to design a separate head for each dataset, because we are addressing a single task across multiple datasets [90]. Consequently, there is no need to design distinct loss functions for each dataset or employ complex algorithms to search for the optimal weights of different losses. The experimental results in Section 4 demonstrate that our method, which employs a single model and a unified type of loss function, is both simple and effective for the OVDG task.

4. Experiments

4.1. Experimental Datasets and Processing

We evaluate the proposed methods on four well-known remote sensing datasets: (1) Potsdam [1] dataset is a widely used benchmark in remote sensing, comprising 38 images of size 6,000×6,000 pixels. It includes 6 classes, with ’background’ as one of them. (2) GID5[86] contains 150 pixel-level annotated GF-2 images of 6,800×7,200, sampled from various cities in china. It has a total of 5 classes. (3) DeepGlobe[87] is a large-scale land-cover dataset which contains 803 images (2,448×2,448 pixels). It includes 7 classes of landscape regions, including “unknown” region. (4) URUR[88] features a substantial number of high-resolution images (3,008 images of size 5,120×5,120 pixels) covering a wide range of complex scenes from 63 cities. In open vocabulary setting, we first remove the "clutter" class from Potsdam, the "background" class from DeepGlobe, and the "other" category from URUR by setting the corresponding labels to 255, marking them as invalid classes. Next, we merge similar classes across datasets. For instance, categories such as "built up", "building", and "urban land" are unified and renamed as "building". Finally, we randomly partition the classes into base and novel classes across all datasets. We combine the four datasets into a single dataset, referred as GPDU, comprising a total of 12 classes. These include 7 base classes (building, farmland, forest land, impervious surface, car, range land, greenhouse) and 5 novel classes (meadow, water, tree, bare land, road). The details of class partitioning and the class mapping relationships can be found in Table 1. The class frequency distributions on the training and testing sets of the four datasets are illustrated in Figure 3.

4.2. Implementation Details

In open vocabulary recognition module, we adopt the pre-trained vision-language model CLIP, employing the ViT-B backbone as the text encoder. For the base segmentation branch, we use ResNet50 [91] as the backbone. In the Entity Segmentation Branch (ESB), we employ CropFormer [81] with a Swin-Tiny backbone for entity mask extraction. To stabilize training and leverage pre-trained knowledge, we freeze several components: 1) The text and image encoders of CLIP to preserve the learned multimodal representations. 2) The entity mask extractor in ESB to maintain the robustness of the extracted masks. 3) The backbone of the base segmentation branch after initializing it with self-supervised pre-training weights. During training, we focus on fine-tuning the decoders of BSB and the visual feature extractor of ESB. We conduct training on an NVIDIA GTX 3090 GPU using a mini-batch size of 2 images. Note that, we use $N=100$ queries in both branches and set $k=8$ in visual feature extractor. For a fair comparison, we adopt the same training settings as FreeSeg [24]: The optimizer AdamW is adopted with an initial learning rate of ${10}^{-4}$ and weight decay of ${10}^{-4}$. To avoid over-fitting on training set, the learning rate of image encoder is multiplied by a factor of $\lambda =0.01$. There are totally 40k training iterations. Each dataset is separately divided into training and testing sets. During training, input images are cropped to 512×512 pixels.

4.3. Evaluation Metrics

We evaluate our model and the baselines with Mean Intersection over Union (mIoU), Frequency Weighted IoU (fwIoU), Pixel Accuracy (pACC) and Mean Pixel Accuracy (mAcc). First, mIoU is a widely-used metric for evaluating semantic segmentation models, providing an average measure of segmentation accuracy across all classes. fwIoU adjusts for class imbalance by assigning more weight to frequently occurring classes during evaluation. pACC measures the percentage of correctly classified pixels in the entire image, while mACC calculates the average proportion of correctly classified pixels across all classes.

$$mIoU={\displaystyle \frac{1}{C}}\sum _{c=1}^C{\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c+F{N}_c}}$$

(7)

$$fwIoU={\displaystyle \frac{1}{{\sum }_{c=1}^{C}T{P}_c+F{P}_c+F{N}_c}}\sum _{c=1}^C{\displaystyle \frac{T{P}_c}{T{P}_c+F{P}_c+F{N}_c}}$$

(8)

$$pACC={\displaystyle \frac{{\sum }_{c=1}^{C}T{P}_c}{{\sum }_{c=1}^{C}T{P}_c+F{P}_c+F{N}_c+T{N}_c}}$$

(9)

$$mACC={\displaystyle \frac{1}{C}}\sum _{c=1}^C{\displaystyle \frac{T{P}_c}{T{P}_c+F{N}_c}}$$

(10)

where C is the number of classes, and for each class c: $T{P}_c$ is the number of true positive pixels. $F{P}_c$ is the number of false positive pixels. $F{N}_c$ is the number of false negative pixels. $T{N}_c$ is the number of true negative pixels. Further, we compute mIoU and pACC separately on the base and novel classes, denoted as follows: $mIo{U}_s$: Mean Intersection over Union for base classes, $mIo{U}_{us}$: Mean Intersection over Union for novel classes, $pAC{C}_s$: Pixel Accuracy for base classes, $pAC{C}_{us}$: Pixel Accuracy for novel classes.

4.4. Comparison with SOTA Methods

1) Results of open vocabulary semantic segmentation: Our proposed FreeMix is compared against several existing open vocabulary methods, including ZSSeg [51], ZegFormer [16], MaskCLIP [19], SAN [52], OVSeg [23] , FC-CLIP [92] and FreeSeg [24]. In the open vocabulary setting, the model is trained using only base classes and their corresponding masks. During testing, both base classes and novel classes are evaluated to assess the model’s generalization capability. Table 2 shows the results on Potsdam dataset. For a fair comparison, all models use ResNet50 as the image encoder, with the exception of OVSeg, which uses ResNet101, and FC-CLIP, which employs a large version of ConvNeXt. The proposed FreeMix significantly outperforms other methods across multiple evaluation metrics. Specifically, it achieves the highest mean Intersection over Union (mIoU) of 63.44% and the highest mean Pixel Accuracy (mACC) of 73.87%, both of which are the best among all compared methods. Notably, FreeMix excels in segmenting base classes, achieving an mIoU of 86.46%. This high score underscores its proficiency in accurately delineating familiar object categories. Additionally, for novel classes, FreeMix attains an mIoU of 28.92%. Although this is lower than the performance on base classes, it still represents a commendable achievement considering the inherent challenges of recognizing previously unseen categories. These results collectively demonstrate that FreeMix not only maintains high accuracy for known classes but also generalizes well to novel classes, highlighting its robustness and adaptability in open vocabulary semantic segmentation tasks.

Compared to the second-best method, ZSSeg, our FreeMix achieves significant improvements across all metrics. This is primarily because ZSSeg relies on existing mask proposal networks without any optimization, leading to lower-quality mask generation. In contrast, our approach enhances mask quality through tailored optimizations, resulting in superior segmentation and recognition performance. Notably, as shown in Table 2, our FreeMix outperforms FC-CLIP, which uses a larger image encoder. While FC-CLIP achieves the highest Pixel Accuracy (pACC) on base classes, this is due to its frozen CNN-based CLIP backbone and supervised fine-tuning specifically for base classes, causing it to be more biased toward these categories. Our FreeMix incorporates a frozen self-supervised backbone in the base segmentation branch, contributing to strong performance on both base and novel classes. This design, facilitated by the two-branch architecture of FreeMix, helps maintain robust segmentation capabilities across all categories. In terms of novel classes, FreeMix demonstrates substantial improvements. Specifically, It gains an 11.11% IoU on the tree class and a 46.73% IoU on the meadow class. These improvements are particularly noteworthy because certain models, such as ZSSeg and ZegFormer, struggle to distinguish between similar classes like trees and meadows in remote sensing images. The Entity Segmentation Branch (ESB) of FreeMix plays a crucial role by generating universal masks and extracting corresponding visual features, thereby enhancing the model’s ability to generalize to novel classes. Overall, the proposed FreeMix can generalize well to novel classes while maintaining strong performance on base classes. This balanced approach ensures that FreeMix has the best overall performance, as evidenced by its superior results across multiple evaluation metrics.

In addition, The qualitative results on the Potsdam testing set are depicted in Figure 4, our FreeMix obtains accurate semantic segmentation for both base and novel classes. Notably, when segmenting novel classes, models often struggle due to confusion between similar categories. However, as shown in the top row and third row of Figure 4, our segmentation module excels at distinguishing between challenging classes such as "meadow" and "tree," outperforming any off-the-shelf model. This demonstrates the strong open-vocabulary segmentation ability of our model.

2) Results of OVDG: We also verify the generalization ability of our FreeMix across different datasets. Specifically, the model is trained on one dataset and directly evaluated on another dataset without fine-tuning. This setup presents a challenging task where the model must handle both novel classes and address the domain gap between different datasets. We report the mean Intersection over Union (mIoU) for both base and novel classes. As shown in Table 3, FreeMix consistently achieves the highest average mIoU and mACC across both the source domain and the three target domains, regardless of the training dataset. Specifically, when trained on the GID5 dataset, FreeMix achieves the highest average mIoU of 25.46%, and when trained on the URUR dataset, it attained the highest average mAcc of 39.98%. FreeMix demonstrates strong single-source domain generalization capabilities, with robust performance across different source domains. While the overall performance remains consistently high, we observe minor variations depending on the specific training dataset used. These variations likely stem from the differing characteristics of each domain, which can impact the ease of generalization. Overall, the GID5 dataset proves to be a particularly strong choice for achieving better generalization performance. The consistent high performance across multiple domains underscores the robustness and adaptability of FreeMix in open vocabulary domain generalization tasks.

Notably, when trained on the Potsdam dataset, FreeMix generalizes well to GID5 but underperforms when tested on DeepGlobe and URUR. A similar trend can be observed in other models. This phenomenon may be attributed to the relatively smaller size of the Potsdam dataset compared to others, which renders it more susceptible to overfitting given an equivalent number of training iterations. In contrast, the GID5 dataset—closer in size to Potsdam—demonstrates superior performance relative to both DeepGlobe and URUR. However, this also presents a significant limitation. Training on datasets with a narrow range of classes, such as Potsdam—which predominantly contains urban categories—can lead to suboptimal performance when FreeMix is tested on datasets like DeepGlobe. DeepGlobe includes natural and semi-natural classes that are not present in Potsdam. This mismatch in class distributions underscores the critical importance of dataset composition in domain generalization tasks. Additionally, our FreeMix not only enhances generalization performance but also excels in open-vocabulary segmentation capabilities. When trained on GID5 and tested on other dataset, it attains $mIo{U}_{us}$ of 15.83% on Potsdam, 12.38% on DeepGlobe and 9.79% on URUR. Besides, $mIo{U}_{us}$ reaches 20.26% on GID5 and 6.49% on URUR when FreeMix only trained on DeepGlobe dataset, which outperforms FreeSeg with +4.30% and +3.78%, respectively. These results demonstrate that our proposed FreeMix not only performs excellently within the domain but also exhibits strong generalization capabilities across other data domains. When trained and tested on the single source domain URUR, FreeMix surpasses FreeSeg, achieving mIoU and mACC that are 8.73% and 7.74% higher, respectively. Moreover FreeMix generalizes well to other target domain datasets, with the exception of GID5. Through analysis, we found that the IoU for the easily confused classes, farmland from the base classes and meadow from the novel classes, is lower in FreeMix compared to FreeSeg. Therefore, enhancing the model’s ability to accurately recognize similar classes in cross-domain scenarios presents a more challenging and promising direction for future work.

MaskCLIP shares some similarities with our ESB, as both methods utilize a class-agnostic mask proposal network and extract features from a pretrained CLIP ViT model. However, FreeMix’s design incorporates several enhancements that lead to higher-quality masks and richer semantic features: the Entity Mask Extractor and BSB generate more accurate masks compared to MaskCLIP. And the ESB in FreeMix includes an independent Visual Feature Extractor, which contributes to the richness of the extracted features. This analysis reveals that both mask quality and the semantic richness of the features play a critical role in determining the final segmentation performance. The approach most akin to ours BSB is FC-CLIP [24], which involves directly substituting the backbone network of the mask generator with CLIP’s image encoder and subsequently freezing it during training. This strategy enables maximal retention of the model’s original generalization capabilities while concurrently reducing the memory requirements during training. Despite using a larger image encoder, FC-CLIP is outperformed by FreeMix across all datasets. Notably, FreeMix achieves these results without employing multi-scale test-time augmentation, which other models use. Intuitively, the improved modules in FreeMix can more effectively generate universal masks for both base and novel classes, thereby enhancing its segmentation generalization ability on both class and domain levels. These findings underscore the effectiveness of the proposed method in open domains.

4.5. Experiments on Multi-Source Domain

To evaluate the performance of FreeMix across multiple source domains, we train it on a combined dataset named GPDU. This setup allows us to compare FreeMix with other state-of-the-art models under fair conditions, where all models are trained for 40K iterations. It is important to note that existing models are typically limited to training on a single dataset at a time. To enable a comprehensive comparison, we created the GPDU dataset by performing label mapping for the training of these models. In contrast, FreeMix can be trained on multiple datasets simultaneously, demonstrating its scalability in handling multi-dataset training.

As depicted in Table 4, FreeMix consistently outperforms other models across all datasets. The implementation of the universal segmentation module with a dual-branch architecture and dataset-aware sampling (DAS) has helped FreeMix set a new benchmark, achieving the highest average mIoU. In comparison, FreeSeg and SAN exhibit significantly lower performance, underscoring the superiority of FreeMix in semantic segmentation tasks. Specifically, the use of DAS contributes to an overall improvement in FreeMix’s average mIoU. However, this enhancement reveals a trade-off between base and novel class performance. Significant improvements are observed on datasets like Potsdam and URUR. There is a noticeable decrease, particularly on GID5 and DeepGlobe. This trade-off highlights the need for further optimization to balance the performance between base and novel classes when applying DAS. Addressing this issue could lead to more consistent and robust performance across all classes and datasets.

4.6. Ablation Experiments

In this section, we conduct ablation studies on our FreeMix using the joint GPDU dataset. All experiments utilize a ResNet50 backbone and the CLIP-B/16 VLM, with FreeSeg serving as our baseline for comparison. Table 5 summarizes the effectiveness of three key components: the proposed initialization method with self-supervised learning in remote sensing (RS_SSL), the entity segmentation branch (ESB) and the dataset-aware sampling (DAS) training tactic. These results clearly indicate that each proposed component consistently enhances the overall performance of FreeMix, highlighting their individual and combined contributions to improved semantic segmentation outcomes. Using RS_SSL initialization provides a substantial improvement over the baseline, indicating the value of leveraging self-supervised learning specifically tailored for remote sensing tasks. The introduction of ESB yields consistent performance boosts across all datasets, suggesting its critical role in capturing more accurate and detailed entity representations. DAS not only elevates mIoU and mACC but also demonstrates its ability to balance performance across diverse datasets, thereby ensuring robust generalization.

When using supervised pre-training on ImageNet1K to initialize the image encoder and fine-tuning it on the GPDU dataset, we observe a degradation in performance. However, with our RS_SSL initialization method and freezing the image encoder during training, the model can extract domain-agnostic universal features, thereby accelerating decoder optimization and improving learning efficiency. Furthermore, incorporating the ESB branch to construct a dual-branch network further enhances performance. The ESB effectively extracts entity masks and captures accurate semantic features for classification, contributing significantly to better segmentation results. Additionally, employing DAS improves overall performance, especially on datasets with relatively small amounts of data. For instance, on the Potsdam dataset, mIoU increased from 19.98% to 47.03%, demonstrating substantial improvement. Even on larger datasets like URUR, there was an increase in mIoU from 29.30% to 35.72%. The above findings indicate that our proposed methods—RS_SSL initialization, ESB, and DAS—are highly effective in enhancing the performance of FreeMix. The qualitative results of FreeMix are visualized in Figure 5, providing a clear demonstration of its superior performance in semantic segmentation tasks.

4.7. Additional Experimental Results

1) Performance on different image encoder of BSB: To further demonstrate the adaptability of the proposed RS_SSL adaptation strategy, we conduct experiments on both CNN and ViT architectures. In FreeMix, the backbone of BSB is from CMID [47], which is pre-trained in a self-supervised manner on the remote sensing dataset MillionAID. In contrast, other methods are either trained in a supervised manner on the ImageNet1K dataset or train the backbone network from scratch. Furthermore, we report comparative results under different training tactic: random sampling and the proposed DAS. According to the Table 6 , using ResNet50 as backbone and employing RS_SSL initialization to BSB, our FreeMix achieves average mIoU scores of 34.64% and 40.25% when using random sampling and DAS training tactic, respectively. It should be noticed that, without introducing RS_SSL initialization to BSB, the performance will drop significantly to average mIoU scores of 29.17% (random sampling) and 23.71% (DAS). Besides, applying RS_SSL initialization to Transformer-based backbone network Swin-B, the model gains average mACC of 61.00% and 64.29% when using random sampling and DAS, respectively. Compared to training BSB from scratch, RS_SSL initialization improved average mACC across the four datasets by 1.27% (random sampling) and 7.08% (DAS). These results indicate that RS_SSL initialization is effective for both CNN and ViT architectures. Meanwhile, it demonstrates that FreeMix is architecture-agnostic, allowing further performance improvement by leveraging superior pre-trained models of both convolutional neural networks and vision transformers.

2) Performance on scaling model size of ESB: To evaluate the impact of scaling the model size of the ESB, we train 6 FreeMix models, sweeping over backbone of ESB (Swin-Tiny, Swin-Large, Hornet-Large) and training tactics (random sampling, DAS). The main results are illustrated in Table 7. Swin-Tiny (Swin-T) consistently shows strong performance across the datasets, particularly when paired with the DAS training tactic. It achieves the highest average mIoU (40.25%) and mACC (73.05%). This confirms that smaller backbones like Swin-T perform more efficiently in terms of both segmentation accuracy and generalization across domains. Swin-Large (Swin-L), while larger in size, shows a drop in where its average mIoU falls to 32.51% and mACC to 49.67%. This performance drop may be attributed to Swin-L requiring more data to train effectively, as larger models often need more data to fully capture patterns and avoid overfitting. Hornet-Large (Hornet-L) performs well in certain metrics, such as achieving the highest mACC of 67.61% with random sampling and 66.17% on GID5 with DAS, but it underperforms in terms of average mIoU, especially on DeepGlobe and URUR. The mixed performance indicates that while Hornet-L excels in certain conditions, it may not generalize as effectively across all domains. In conclusion, these results underscore that smaller backbones like Swin-T not only maintain better overall performance but also generalize more effectively across datasets, particularly when using the DAS training tactic. Conversely, larger models like Swin-L and Hornet-L show diminishing returns, especially in terms of mIoU, suggesting that adding a lightweight ESB branch can lead to greater efficiency without sacrificing accuracy.

3) Comparison of the extracted proposal masks: We visualize and compare the proposal masks extracted by FreeSeg and our FreeMix, as shown in Figure 6. FreeSeg processes 100 queries to generate 100 proposal masks. For visualization purposes, we randomly select 10 of these masks. Besides, FreeMix generates 100 proposal masks for each branch (BSB and ESB). And we randomly select 10 masks from each branch for visualization and comparison, indicated in yellow (BSB) and purple (ESB) in Figure 6. Based on the results, it is evident that the proposal masks generated by FreeSeg exhibit low distinguishability and low predicted confidence, as indicated by their gray appearance. In contrast, the masks decoded by FreeMix’s BSB, with the assistance of RS_SSL, exhibit higher confidence and greater distinguishability. Additionally, the ESB of FreeMix offers a richer set of entity masks, capturing more detailed and accurate representations of entities within the images. By combining these high-quality universal masks from both branches, FreeMix demonstrates superior performance on OVDG task compared to other models.

5. Conclusion

In this work, we introduce a new setting for remote sensing image semantic segmentation called open vocabulary domain generalization (OVDG). This novel setting holds great potential in the remote sensing community by addressing the challenge of generalizing models to unseen domains and classes without retraining. To tackle this challenging problem, we propose an effective framework to train a robust model. Firstly, the proposed universal segmentation module has dual-branch: base segmentation branch (BSE) and entity segmentation branch (ESB). Moreover, the remote sensing self-supervised learning (RS_SSL) initialization and adaptation method is introduced to extract domain-agnostic visual feature for decoding masks and semantic logits. Additionally, ESB is proposed to generate entity masks for enhancing the segmentation and recognition of bath base and novel classes. Furthermore, a dataset-aware sampling (DAS) training tactic is designed for multi-source domain learning, aiming to enhance the overall performance of the model. Extensive experiments demonstrate the effectiveness of our proposed universal segmentation module, incorporating RS_SSL initialization, BSB and ESB as well as the DAS training tactic. Our FreeMix achieves state-of-the-art results on open vocabulary benchmarks and OVDG task. Although good semantic segmentation results have been achieved on OVDG, FreeMix relies on class names provided during testing. Future work will focus on designing a more effective training strategy for multi-data set learning and extending the FreeMix framework by incorporating multimodal large language models to generate class predictions and localization.