On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval
Autor(a) principal: | |
---|---|
Data de Publicação: | 2014 |
Outros Autores: | , , , , , |
Tipo de documento: | Artigo |
Idioma: | eng |
Título da fonte: | Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
Texto Completo: | http://repositorio.inesctec.pt/handle/123456789/7131 http://dx.doi.org/10.1109/tpami.2013.142 |
Resumo: | The problem of cross-modal retrieval from multimedia repositories is considered. This problem addresses the design of retrieval systems that support queries across content modalities, for example, using an image to search for texts. A mathematical formulation is proposed, equating the design of cross-modal retrieval systems to that of isomorphic feature spaces for different content modalities. Two hypotheses are then investigated regarding the fundamental attributes of these spaces. The first is that low-level cross-modal correlations should be accounted for. The second is that the space should enable semantic abstraction. Three new solutions to the cross-modal retrieval problem are then derived from these hypotheses: correlation matching (CM), an unsupervised method which models cross-modal correlations, semantic matching (SM), a supervised technique that relies on semantic representation, and semantic correlation matching (SCM), which combines both. An extensive evaluation of retrieval performance is conducted to test the validity of the hypotheses. All approaches are shown successful for text retrieval in response to image queries and vice versa. It is concluded that both hypotheses hold, in a complementary form, although evidence in favor of the abstraction hypothesis is stronger than that for correlation. |
id |
RCAP_a9c9d518c62b59c1989e1d9878b8011f |
---|---|
oai_identifier_str |
oai:repositorio.inesctec.pt:123456789/7131 |
network_acronym_str |
RCAP |
network_name_str |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
repository_id_str |
7160 |
spelling |
On the Role of Correlation and Abstraction in Cross-Modal Multimedia RetrievalThe problem of cross-modal retrieval from multimedia repositories is considered. This problem addresses the design of retrieval systems that support queries across content modalities, for example, using an image to search for texts. A mathematical formulation is proposed, equating the design of cross-modal retrieval systems to that of isomorphic feature spaces for different content modalities. Two hypotheses are then investigated regarding the fundamental attributes of these spaces. The first is that low-level cross-modal correlations should be accounted for. The second is that the space should enable semantic abstraction. Three new solutions to the cross-modal retrieval problem are then derived from these hypotheses: correlation matching (CM), an unsupervised method which models cross-modal correlations, semantic matching (SM), a supervised technique that relies on semantic representation, and semantic correlation matching (SCM), which combines both. An extensive evaluation of retrieval performance is conducted to test the validity of the hypotheses. All approaches are shown successful for text retrieval in response to image queries and vice versa. It is concluded that both hypotheses hold, in a complementary form, although evidence in favor of the abstraction hypothesis is stronger than that for correlation.2018-01-19T17:33:27Z2014-01-01T00:00:00Z2014info:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/articleapplication/pdfhttp://repositorio.inesctec.pt/handle/123456789/7131http://dx.doi.org/10.1109/tpami.2013.142engJosé Costa PereiraCoviello,EDoyle,GRasiwasia,NLanckriet,GRGLevy,RVasconcelos,Ninfo:eu-repo/semantics/embargoedAccessreponame:Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos)instname:Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informaçãoinstacron:RCAAP2023-05-15T10:20:48Zoai:repositorio.inesctec.pt:123456789/7131Portal AgregadorONGhttps://www.rcaap.pt/oai/openaireopendoar:71602024-03-19T17:53:39.599550Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) - Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informaçãofalse |
dc.title.none.fl_str_mv |
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval |
title |
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval |
spellingShingle |
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval José Costa Pereira |
title_short |
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval |
title_full |
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval |
title_fullStr |
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval |
title_full_unstemmed |
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval |
title_sort |
On the Role of Correlation and Abstraction in Cross-Modal Multimedia Retrieval |
author |
José Costa Pereira |
author_facet |
José Costa Pereira Coviello,E Doyle,G Rasiwasia,N Lanckriet,GRG Levy,R Vasconcelos,N |
author_role |
author |
author2 |
Coviello,E Doyle,G Rasiwasia,N Lanckriet,GRG Levy,R Vasconcelos,N |
author2_role |
author author author author author author |
dc.contributor.author.fl_str_mv |
José Costa Pereira Coviello,E Doyle,G Rasiwasia,N Lanckriet,GRG Levy,R Vasconcelos,N |
description |
The problem of cross-modal retrieval from multimedia repositories is considered. This problem addresses the design of retrieval systems that support queries across content modalities, for example, using an image to search for texts. A mathematical formulation is proposed, equating the design of cross-modal retrieval systems to that of isomorphic feature spaces for different content modalities. Two hypotheses are then investigated regarding the fundamental attributes of these spaces. The first is that low-level cross-modal correlations should be accounted for. The second is that the space should enable semantic abstraction. Three new solutions to the cross-modal retrieval problem are then derived from these hypotheses: correlation matching (CM), an unsupervised method which models cross-modal correlations, semantic matching (SM), a supervised technique that relies on semantic representation, and semantic correlation matching (SCM), which combines both. An extensive evaluation of retrieval performance is conducted to test the validity of the hypotheses. All approaches are shown successful for text retrieval in response to image queries and vice versa. It is concluded that both hypotheses hold, in a complementary form, although evidence in favor of the abstraction hypothesis is stronger than that for correlation. |
publishDate |
2014 |
dc.date.none.fl_str_mv |
2014-01-01T00:00:00Z 2014 2018-01-19T17:33:27Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/article |
format |
article |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://repositorio.inesctec.pt/handle/123456789/7131 http://dx.doi.org/10.1109/tpami.2013.142 |
url |
http://repositorio.inesctec.pt/handle/123456789/7131 http://dx.doi.org/10.1109/tpami.2013.142 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.rights.driver.fl_str_mv |
info:eu-repo/semantics/embargoedAccess |
eu_rights_str_mv |
embargoedAccess |
dc.format.none.fl_str_mv |
application/pdf |
dc.source.none.fl_str_mv |
reponame:Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) instname:Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação instacron:RCAAP |
instname_str |
Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação |
instacron_str |
RCAAP |
institution |
RCAAP |
reponame_str |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
collection |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) |
repository.name.fl_str_mv |
Repositório Científico de Acesso Aberto de Portugal (Repositórios Cientìficos) - Agência para a Sociedade do Conhecimento (UMIC) - FCT - Sociedade da Informação |
repository.mail.fl_str_mv |
|
_version_ |
1799131610520485888 |