Satellite Image Retrieval for Earth Observation

Content-Based Image Retrieval (CBIR) on EuroSAT

Course: Information Retrieval Topic: Satellite Image Retrieval for Earth Observation Submitted to: Prof. Antonio Maria Rinaldi, Prof. Domenico Benfenati Student: Rohan Baidya, D03000192 Dataset link: https://madm.dfki.de/files/sentinel/EuroSAT.zip Site link: https://unina.cc/ir

Abstract

• A CBIR system that finds similar satellite land-cover images in the EuroSAT dataset (from ESA Sentinel-2).
• Compares two ways to describe an image: a classical HSV colour histogram and a deep ResNet18 feature vector.
• Both are indexed with FAISS to search by cosine similarity.
• Tested with a leave-one-out setup using Precision@k, Recall@k, and mAP@50.
• Also supports image-query classification by a k-NN vote over the retrieved neighbours, with simple relevance feedback.

End-to-end Process Flow

• Build the index once, then search it with any query image.
• A query is encoded the same way as the gallery and compared by cosine similarity.

Code Components

Library Reference

Notebook Cells

1. Introduction

• ESA's Copernicus has been collecting Sentinel-2 images since 2015 and passes over the same place every few days.
• CBIR ranks images by how similar they look to a query image, instead of relying on text tags that are often missing.
• This project uses CBIR on EuroSAT with two descriptors: an HSV colour histogram (classical) and a ResNet18 feature vector (deep, already trained on ImageNet).

Problem. Given a query image and N labelled patches, rank the patches so that those of the same land-cover class as the query come first.

Objectives. - Implement a classical and a deep descriptor. - Index both with FAISS for fast search. - Evaluate with Precision@k, Recall@k, mAP@50. - Compare the two and explain the gap.

2. Data and Preprocessing

2.1 EuroSAT

• Land-cover benchmark (Helber et al., 2019): 27,000 labelled 64x64 RGB patches from Sentinel-2, 10 classes.
• Roughly 2,000 to 3,000 patches per class; the labels are used as the correct answers when scoring.

2.2 Source

• Originally, the dataset is used from reference of github link: https://github.com/phelber/EuroSAT, but is downloaded here from torchvision.datasets.EuroSAT(download=True) (torchvision 0.20+ fetches the zip from a Hugging Face mirror).
• Underlying Sentinel-2 data is part of the Copernicus Data Space Ecosystem.

2.3 Preprocessing

• HSV: open with Pillow, convert to HSV. No resize is needed and patches are 64x64.
• ResNet18: official ResNet18_Weights.IMAGENET1K_V1 transform (resize 256, centre-crop 224, ImageNet mean and std).
• Sampling: dev runs use 300 per class; the full run uses all 27,000.

3. Retrieval Methods

3.1 HSV Colour Histogram

• Classic CBIR descriptor (Swain and Ballard, 1991).
• Hue, Saturation, Value each histogrammed over 32 bins, joined to a 96-d vector, L2-normalised.
• Pros:
  • no training, fast, easy to read.
• Cons:
  • ignores where things are in the image,
  • affected by lighting changes,
  • cannot tell apart classes with similar colours (AnnualCrop vs Pasture).

3.2 ResNet18

• 18-layer CNN (with skip connections) already trained on ImageNet; the last classifier layer is swapped for nn.Identity() so we get the 512-d feature vector from the layer before it.
• It picks up texture and shape, which works well on satellite images because classes differ in texture and pattern.
• Pros:
  • much better than colour as it captures texture and shape,
  • can tell apart classes that look colour-similar (Highway vs Residential),
  • compact 512-d vector that works well with cosine similarity,
  • no training needed for our use: the pretrained weights transfer straight to the patches,
  • more stable across lighting and seasonal colour changes,
• Cons:
  • trained on normal photos not satellite data,
  • takes a few minutes to process 27k images on CPU,
  • uses RGB only,

3.3 Similarity and Indexing

• Both descriptors are L2-normalised, so cosine similarity is the same as the dot product.
• FAISS IndexFlatIP checks the query against every stored vector and returns the top-k closest.
• This exact search is fine for our size; a much bigger set (millions) would use a faster approximate index like IVF or HNSW.

4. Implementation and Evaluation

4.1 Feature Extraction

• HSV: np.histogram per channel, normalised with np.linalg.norm.
• ResNet18: DataLoader in eval() under torch.no_grad(), outputs stacked and normalised row by row.

4.2 Evaluation Protocol

• Leave-one-out: each image is a query; itself (rank 1) is dropped. A result is relevant if its label matches the query.
• Precision@k = relevant in top-k / k
• Recall@k = relevant in top-k / total relevant for that class
• Average Precision taken to rank 50; mAP@50 = mean over all queries.

4.3 Results

• Run with 300 images per class (3,000 total), read from results.json.

• Deep descriptor roughly doubles mAP@50 and adds about 16 points of P@1.
• Recall@k is small by design: about 300 relevant per class, so top-10 reaches only a few percent.

4.4 Image Query as Classification (k-NN)

• An image can be used as the query and classified by retrieval: take its nearest neighbours and let their classes vote (retrieval-based, k-NN classification).
• The top-K neighbours each vote with a weight of similarity raised to a power (default 4), so closer matches count more than distant ones.
• The class with the highest total wins, and the normalised totals give a confidence score per class.
• This reuses the same FAISS index and needs no extra training.

4.5 Relevance Feedback

• The predicted class can be confirmed or corrected, and the labelled image is added to the index, so later queries retrieve and vote with it. This is a simple online form of relevance feedback.
• Confirmed corrections are weighted more than ordinary dataset items (about 8x), and a near-duplicate of an already-corrected image (cosine >= 0.97) is trusted strongly, so a single correction is not outvoted by the surrounding dataset.
• Corrections are kept separate from the original EuroSAT data and can later be folded back in to fine-tune the model.

5. Conclusions

• Built a CBIR pipeline for EuroSAT, comparing an HSV colour histogram with a ResNet18 embedding, both indexed with FAISS and scored with leave-one-out IR metrics.
• The deep descriptor clearly beats the colour method (mAP@50 0.648 vs 0.316): learned features pick up land-cover structure that colour alone misses.
• The histogram still does fine on easy classes such as SeaLake and Forest.

Main limitations: - ResNet18 was trained on normal photos, not satellite data. - RGB input drops Sentinel-2's extra bands (NIR, SWIR). - Exact FAISS search checks every image, so it would be slow on very large sets.