|
|
| (80 intermediate revisions by 3 users not shown) |
| Line 1: |
Line 1: |
| | + | __NOTOC__ |
| | | | |
| − | =TESSERA Foundation Model Project= | + | = TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis= |
| − | ==Introduction==
| |
| − | ===Background and motivation===
| |
| | | | |
| − | Earth Observation (EO) from satellites has already generated petabytes of data in the last few decades. The rate of growth of observation data is set to increase as more satellites are launched due to rapidly declining launch costs. Most observation data are freely available. Moreover, due to the repeat patterns of satellite orbits, observation data form a time series for each observed location on Earth. Careful analysis and interpretation of these time series can help address critical issues including biodiversity monitoring, identifying land use and land cover, choosing optimal crop management strategies, and quantifying forest degradation and deforestation.
| + | Satellite remote sensing enables a wide range of downstream applications, including habitat mapping, carbon accounting, and strategies for conservation and sustainable land use. However, satellite time series are voluminous and often corrupted, making them challenging to use: the scientific community's ability to extract actionable insights is often constrained by the scarcity of labelled training datasets and the computational burden of processing temporal data. |
| | | | |
| − | ===Problem===
| + | Our work introduces TESSERA, an open foundation model that preserves spectral-temporal signals in 128-dimensional latent representations at 10-meter resolution globally. It uses self-supervised learning to summarise petabytes of Earth observation data. We compare our work with state-of-the-art task-specific models and other foundation models in five diverse downstream tasks and find that TESSERA closely matches or outperforms these baselines. By preserving temporal phenological signals that are typically lost in conventional approaches, TESSERA enables new insights into ecosystem dynamics, agricultural food systems, and environmental change detection. Moreover, our open-source implementation supports reproducibility and extensibility, while the privacy-preserving design allows researchers to maintain data sovereignty. |
| | | | |
| − | Despite the abundance and free availability of EO data, much of it is “contaminated,” especially in the optical domain, by cloud cover, sensor-specific observation biases, and non-uniform temporal sampling due to the inherent nature of satellite orbital patterns.
| + | To our knowledge, TESSERA is unprecedented in its ease of use, scale, and accuracy: no other foundation model provides analysis-ready outputs, is open, and provides global, annual coverage at 10m resolution using only spectral-temporal features at pixel level. |
| | | | |
| − | ===State of the art===
| + | *[https://arxiv.org/abs/2506.20380 Preprint] |
| | + | **An [https://asxiv.org/pdf/2506.20380 AI explanation] |
| | + | **[https://svr-sk818-web.cl.cam.ac.uk/keshav/wiki/images/9/9d/TESSERA.m4a 15-minute AI-generated podcast] |
| | | | |
| − | Existing approaches for handling contaminated observation time series fall into three categories. The first is multi-temporal compositing, which aggregates data collected across a certain time period so that regions obscured by clouds during one satellite pass are filled in by an earlier/later observation. This approach is effective, but, crucially, it los
| + | *[https://github.com/ucam-eo/tessera Code] |
| − | es the fine-grain temporal signal embedded in the data. For example, a three-month composite of crop observations would lose much of the crop-growth induced reflectance signal. Similarly, composited forest observations lose valuable phenological information.
| |
| | | | |
| − | A second approach is multi-spectral. These techniques extract information from weak signals to restore missing information. However, they work best when the optical signals are only partially affected by clouds.
| + | *[https://anil.recoil.org/notes/geotessera-python GeoTessera Python library] |
| | | | |
| − | Finally, several approaches propose in-painting of cloud-obscured patches. See Section II.B of the PLFM paper for a survey and where they fall short. The most sophisticated in-painting approach in the literature appears to be PLFM, where cloud-penetrating microwave radar data collected by the Sentinel 1 as well as temporal-sequence blending is used to remove clouds from Sentinel 2 optical images. However, a radar sensor cannot provide information on parameters such as chlorophyll absorption, which are of prime importance in the optical signal, because this does not affect the measured microwave backscattering coefficient. Moreover, only very coarse structures are captured by Sentinel 1 and no biochemicals, except water. Finally, the goal of this work seems to be to produce visually reasonable looking outputs rather than exact outputs, as we are.
| + | *[[News]] |
| | | | |
| − | ==Our approach: Barlow Twins SSL===
| + | * [[Acceptable use policy]] |
| | + | ---- |
| | | | |
| − | The key idea in our work is to represent the time series of multi-spectral reflectance patterns from a grid cell (pixel) with a single numerical ‘representation’ that is derived using a self-supervised learning (SSL) algorithm. SSL methods extract meaningful representations of input data by optimising a surrogate objective. Unlike supervised learning approaches that require labelled ‘ground truth’ data created by human experts, which is time-consuming, expensive, and error-prone, SSL needs no labels. Moreover, SSL representations can typically be more easily transferred across time and space.
| + | [[A very simple guide to editing this wiki]] |
| − | The extracted representations can be directly used for downstream tasks, since, although they needn’t do so in general, for data that are inherently ‘sparse’, they implicitly represent a multi-class categorization/clustering of the input data. Alternatively, the semantic meaning associated with the representations can be discovered by training a classifier, such as a random forest, with only a small amount of labelled data.
| |
| − | | |
| − | | |
| − | The specific SSL approach we use is the Barlow Twin (BT) approach, which is a self-supervised, non-contrastive way to train SSL models. Our surrogate task is to ensure that different augmentations (e.g., different cropped versions of the input signal) lead to the same representations, and where the representations across a data batch are more or less uncorrelated to each other. Like other SSL approaches, the Barlow Twin does not need labelled data to create a foundation model. We have recently shown that categorised BT representations achieve high accuracy in crop classification. Specifically, by training a random forest classifier to categorise representations using small amounts of ground truth data, we can assign representations to crop types with high accuracy, when compared with ground truth. Mantle Labs has also used representations for quantification of Above Ground Biomass (AGB in t/ha), land cover mapping, tree species identification, crop type mapping, pasture quality assessment, and selection of counterfactuals and found these to be of good quality.
| |
| − | * [https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Configuration_settings Configuration settings list]
| |
TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis
Satellite remote sensing enables a wide range of downstream applications, including habitat mapping, carbon accounting, and strategies for conservation and sustainable land use. However, satellite time series are voluminous and often corrupted, making them challenging to use: the scientific community's ability to extract actionable insights is often constrained by the scarcity of labelled training datasets and the computational burden of processing temporal data.
Our work introduces TESSERA, an open foundation model that preserves spectral-temporal signals in 128-dimensional latent representations at 10-meter resolution globally. It uses self-supervised learning to summarise petabytes of Earth observation data. We compare our work with state-of-the-art task-specific models and other foundation models in five diverse downstream tasks and find that TESSERA closely matches or outperforms these baselines. By preserving temporal phenological signals that are typically lost in conventional approaches, TESSERA enables new insights into ecosystem dynamics, agricultural food systems, and environmental change detection. Moreover, our open-source implementation supports reproducibility and extensibility, while the privacy-preserving design allows researchers to maintain data sovereignty.
To our knowledge, TESSERA is unprecedented in its ease of use, scale, and accuracy: no other foundation model provides analysis-ready outputs, is open, and provides global, annual coverage at 10m resolution using only spectral-temporal features at pixel level.
A very simple guide to editing this wiki
