{"pk":50398,"title":"Measuring the Semantic Consistency of Ordinal Annotations via Text Embedding Spaces and Its Applications","subtitle":null,"abstract":"We propose a method for measuring the consistency of ordinal annotations based on a pre-trained embedding vector space. Intuitively, our method finds a direction in the embedding space along which data points align as closely as possible to their annotated ranks. The proposed approach guarantees a globally optimal solution that is free from approximation errors. Thus, it yields a unique consistency measure given a dataset with human-provided ordinal annotations and a pre-trained embedding model. This feature facilitates a wide range of applications, including not only ordinal prediction but also the unsupervised detection of annotation errors within datasets, as well as consistency assessment of stage-based scales (e.g., whether the transitions \"beginner to intermediate\" and \"intermediate to advanced\" form linear progressions in the embedding space) during dataset construction. We evaluate our method using real-world datasets with ordinal annotations to demonstrate its effectiveness.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[{"word":"Education; Cognitive development; Language understanding; Machine learning; Computer-based experiment"}],"section":"Member Abstracts with Poster Presentation","is_remote":true,"remote_url":"https://escholarship.org/uc/item/9hm9z41w","frozenauthors":[{"first_name":"Yo","middle_name":"","last_name":"Ehara","name_suffix":"","institution":"Tokyo Gakugei University","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"2025-01-01T18:00:00Z","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/50398/galley/38360/download/"}]}