Abstract:
The task of obtaining informative object representations involves training a model, called an encoder, which constructs informative, compressed representations of the signals it receives as input. One approach to solving this problem is through the use of self-supervised learning (SSL) methods. The advantage of these methods lies in utilizing only unlabeled data, which is significantly more abundant than labeled data. Among self-supervised methods, contrastive approaches are particularly prominent; these are based on bringing representations of semantically similar objects (positive pairs) closer together and pushing representations of different signals (negative pairs) apart. Many modern contrastive self-supervised methods, used for obtaining representations of dependent data-where elements within a sample are semantically related-employ a loss function originally designed for independent data. In this work, we propose a theoretically justified approach for selecting a loss function suitable for continuous dependent data, i.e., data in which neighboring elements within the sample can be considered as a positive pair. The analysis presented introduces various ways to model similarity between objects and corresponding loss functions, explicitly accounting for correlations between objects. To empirically assess the effectiveness of the proposed loss functions, we focused on temperature and drought forecasting tasks, which can be classified as continuous dependent data. The results demonstrate that our model, combined with the proposed loss functions, outperforms approaches based on the assumption of semantic independence between data-i.e., when all elements of the sample are semantically unrelated. These findings highlight the importance of considering such dependencies for developing high-quality encoders.
