Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification
Autor(a) principal: | |
---|---|
Data de Publicação: | 2022 |
Outros Autores: | |
Tipo de documento: | Artigo |
Idioma: | eng |
Título da fonte: | Repositório Institucional da UNESP |
Texto Completo: | http://dx.doi.org/10.3390/f13040632 http://hdl.handle.net/11449/240903 |
Resumo: | Computer vision wood identification (CVWID) has focused on laboratory studies reporting consistently high model accuracies with greatly varying input data quality, data hygiene, and wood identification expertise. Employing examples from published literature, we demonstrate that the highly optimistic model performance in prior works may be attributed to evaluating the wrong functionality—wood specimen identification rather than the desired wood species or genus identification—using limited datasets with data hygiene practices that violate the requirement of clear separation between training and evaluation data. Given the lack of a rigorous framework for a valid methodology and its objective evaluation, we present a set of minimal baseline quality standards for performing and reporting CVWID research and development that can enable valid, objective, and fair evaluation of current and future developments in this rapidly developing field. To elucidate the quality standards, we present a critical revisitation of a prior CVWID study of North American ring-porous woods and an exemplar study incorporating best practices on a new dataset covering the same set of woods. The proposed baseline quality standards can help translate models with high in silico performance to field-operational CVWID systems and allow stakeholders in research, industry, and government to make informed, evidence‐based modality‐agnostic decisions. |
id |
UNSP_a79b648d13d90056fa952a48bcaf5c53 |
---|---|
oai_identifier_str |
oai:repositorio.unesp.br:11449/240903 |
network_acronym_str |
UNSP |
network_name_str |
Repositório Institucional da UNESP |
repository_id_str |
2946 |
spelling |
Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identificationbest practicescomputer visionmachine learningwood identificationXyloTronComputer vision wood identification (CVWID) has focused on laboratory studies reporting consistently high model accuracies with greatly varying input data quality, data hygiene, and wood identification expertise. Employing examples from published literature, we demonstrate that the highly optimistic model performance in prior works may be attributed to evaluating the wrong functionality—wood specimen identification rather than the desired wood species or genus identification—using limited datasets with data hygiene practices that violate the requirement of clear separation between training and evaluation data. Given the lack of a rigorous framework for a valid methodology and its objective evaluation, we present a set of minimal baseline quality standards for performing and reporting CVWID research and development that can enable valid, objective, and fair evaluation of current and future developments in this rapidly developing field. To elucidate the quality standards, we present a critical revisitation of a prior CVWID study of North American ring-porous woods and an exemplar study incorporating best practices on a new dataset covering the same set of woods. The proposed baseline quality standards can help translate models with high in silico performance to field-operational CVWID systems and allow stakeholders in research, industry, and government to make informed, evidence‐based modality‐agnostic decisions.Department of Botany University of WisconsinCenter for Wood Anatomy Research USDA Forest Service Forest Products LaboratoryDepartment of Forestry and Natural Resources Purdue UniversityDepartment of Sustainable Bioproducts Mississippi State UniversityDepartamento de Ciências Biológicas (Botânica) Universidade Estadual Paulista Botucatu, SPDepartamento de Ciências Biológicas (Botânica) Universidade Estadual Paulista Botucatu, SPUniversity of WisconsinForest Products LaboratoryPurdue UniversityMississippi State UniversityUniversidade Estadual Paulista (UNESP)Ravindran, PrabuWiedenhoeft, Alex C. [UNESP]2023-03-01T20:37:44Z2023-03-01T20:37:44Z2022-04-01info:eu-repo/semantics/publishedVersioninfo:eu-repo/semantics/articlehttp://dx.doi.org/10.3390/f13040632Forests, v. 13, n. 4, 2022.1999-4907http://hdl.handle.net/11449/24090310.3390/f130406322-s2.0-85129168788Scopusreponame:Repositório Institucional da UNESPinstname:Universidade Estadual Paulista (UNESP)instacron:UNESPengForestsinfo:eu-repo/semantics/openAccess2023-03-01T20:37:44Zoai:repositorio.unesp.br:11449/240903Repositório InstitucionalPUBhttp://repositorio.unesp.br/oai/requestopendoar:29462024-08-05T23:46:38.275470Repositório Institucional da UNESP - Universidade Estadual Paulista (UNESP)false |
dc.title.none.fl_str_mv |
Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
title |
Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
spellingShingle |
Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification Ravindran, Prabu best practices computer vision machine learning wood identification XyloTron |
title_short |
Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
title_full |
Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
title_fullStr |
Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
title_full_unstemmed |
Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
title_sort |
Caveat emptor: On the Need for Baseline Quality Standards in Computer Vision Wood Identification |
author |
Ravindran, Prabu |
author_facet |
Ravindran, Prabu Wiedenhoeft, Alex C. [UNESP] |
author_role |
author |
author2 |
Wiedenhoeft, Alex C. [UNESP] |
author2_role |
author |
dc.contributor.none.fl_str_mv |
University of Wisconsin Forest Products Laboratory Purdue University Mississippi State University Universidade Estadual Paulista (UNESP) |
dc.contributor.author.fl_str_mv |
Ravindran, Prabu Wiedenhoeft, Alex C. [UNESP] |
dc.subject.por.fl_str_mv |
best practices computer vision machine learning wood identification XyloTron |
topic |
best practices computer vision machine learning wood identification XyloTron |
description |
Computer vision wood identification (CVWID) has focused on laboratory studies reporting consistently high model accuracies with greatly varying input data quality, data hygiene, and wood identification expertise. Employing examples from published literature, we demonstrate that the highly optimistic model performance in prior works may be attributed to evaluating the wrong functionality—wood specimen identification rather than the desired wood species or genus identification—using limited datasets with data hygiene practices that violate the requirement of clear separation between training and evaluation data. Given the lack of a rigorous framework for a valid methodology and its objective evaluation, we present a set of minimal baseline quality standards for performing and reporting CVWID research and development that can enable valid, objective, and fair evaluation of current and future developments in this rapidly developing field. To elucidate the quality standards, we present a critical revisitation of a prior CVWID study of North American ring-porous woods and an exemplar study incorporating best practices on a new dataset covering the same set of woods. The proposed baseline quality standards can help translate models with high in silico performance to field-operational CVWID systems and allow stakeholders in research, industry, and government to make informed, evidence‐based modality‐agnostic decisions. |
publishDate |
2022 |
dc.date.none.fl_str_mv |
2022-04-01 2023-03-01T20:37:44Z 2023-03-01T20:37:44Z |
dc.type.status.fl_str_mv |
info:eu-repo/semantics/publishedVersion |
dc.type.driver.fl_str_mv |
info:eu-repo/semantics/article |
format |
article |
status_str |
publishedVersion |
dc.identifier.uri.fl_str_mv |
http://dx.doi.org/10.3390/f13040632 Forests, v. 13, n. 4, 2022. 1999-4907 http://hdl.handle.net/11449/240903 10.3390/f13040632 2-s2.0-85129168788 |
url |
http://dx.doi.org/10.3390/f13040632 http://hdl.handle.net/11449/240903 |
identifier_str_mv |
Forests, v. 13, n. 4, 2022. 1999-4907 10.3390/f13040632 2-s2.0-85129168788 |
dc.language.iso.fl_str_mv |
eng |
language |
eng |
dc.relation.none.fl_str_mv |
Forests |
dc.rights.driver.fl_str_mv |
info:eu-repo/semantics/openAccess |
eu_rights_str_mv |
openAccess |
dc.source.none.fl_str_mv |
Scopus reponame:Repositório Institucional da UNESP instname:Universidade Estadual Paulista (UNESP) instacron:UNESP |
instname_str |
Universidade Estadual Paulista (UNESP) |
instacron_str |
UNESP |
institution |
UNESP |
reponame_str |
Repositório Institucional da UNESP |
collection |
Repositório Institucional da UNESP |
repository.name.fl_str_mv |
Repositório Institucional da UNESP - Universidade Estadual Paulista (UNESP) |
repository.mail.fl_str_mv |
|
_version_ |
1808129551286403072 |