Scene classification using a combination of aerial and ground images

• Autor(a) principal: Gabriel Lucas Silva Machado
• Data de Publicação: 2021
• Tipo de documento: Dissertação
• Idioma: eng
• Título da fonte: Repositório Institucional da UFMG
• Texto Completo: http://hdl.handle.net/1843/38082
• topic: Remote sensing; Sensoriamento remoto; Image classification; Classificação de imagens; Multimodal machine learning; Aprendizado de máquina; Computação – Teses.; Sensoriamento remoto – Teses.; Classificação de imagens – Teses.; Aprendizado de máquina – Teses.

It is undeniable that aerial and orbital images can provide useful information for a large variety of tasks, such as disaster relief and urban planing. But, since these images only see the Earth from one point of view, some applications can benefit from complementary information provided by other perspective views of the scene, such as ground-level images. Despite a large number of public image repositories for both georeferenced photographs and aerial images (such as Google Maps and Google Street View), there is a lack of public datasets that allow the development of approaches that exploit the benefits and complementarity of aerial/ground imagery. Because of that, in this dissertation, we present two new publicly available datasets named AiRound and CV-BrCT (Cross-View Brazilian Construction Types). The first one contains triplets of images from the same geographic coordinate with different perspectives, obtained at various places around the world. Each triplet is composed of an aerial RGB image, a ground-level perspective image, and a Sentinel-2 sample. The second dataset contains pairs of aerial and street-level images extracted from the southeast of Brazil. For this dissertation, we conducted a series of experiments involving both proposed datasets with the main objectives of (i) explore the complementary information that aerial and ground images have by using multi-modal machine learning models to enhance results, (ii) compare different feature fusion approaches applied in several state-of-the-art Convolutional Neural Network architectures, and (iii) investigate alternatives to handle missing data in a multi-modal scenario. Experiments show that, when compared to networks trained using only a single view, feature fusion algorithms achieved gains up to 0.15 and 0.20 in F1-Score for the AiRound and CV-BrCT datasets, respectively. Since it is not always possible to obtain the paired aerial/ground samples of a place, we also designed a framework to handle scenarios with missing samples. Comparing the results of a single-view network classification to the use of our framework integrated with a multi-view model, we achieved gains up to 0.03 in F1-Score for both datasets. Thus, our missing data completion framework has proven to be a more effective approach than just classifying images using a single-view model.