{"pk":49385,"title":"Quantifying Risk Propensities of Large Language Models: Ethical Focus and Bias Detection through Role-Play","subtitle":null,"abstract":"As Large Language Models (LLMs) become more prevalent, concerns about their safety, ethics, and potential biases have risen. Systematically evaluating LLMs' risk decision-making tendencies, particularly in the ethical domain, has become crucial. This study innovatively applies the Domain-Specific Risk-Taking (DOSPERT) scale from cognitive science to LLMs and proposes a novel Ethical Decision-Making Risk Attitude Scale (EDRAS) to assess LLMs' ethical risk attitudes in depth. We further propose a novel approach integrating risk scales and role-playing to quantitatively evaluate systematic biases in LLMs. Through systematic evaluation of multiple mainstream LLMs, we assessed the \"risk personalities\" of LLMs across multiple domains, with a particular focus on the ethical domain, and revealed and quantified LLMs' systematic biases towards different groups. This helps understand LLMs' risk decision-making and ensure their safe and reliable application. Our approach provides a tool for identifying biases, contributing to fairer and more trustworthy AI systems.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[{"word":"Artificial Intelligence; Psychology; Sociology; Social cognition"}],"section":"Papers with Poster Presentation","is_remote":true,"remote_url":"https://escholarship.org/uc/item/97c1h4j6","frozenauthors":[{"first_name":"Yifan","middle_name":"","last_name":"Zeng","name_suffix":"","institution":"Sun Yat-sen University","department":""},{"first_name":"Kairong","middle_name":"","last_name":"Liang","name_suffix":"","institution":"School of Computer Science and Engineering","department":""},{"first_name":"Fangzhou","middle_name":"","last_name":"Dong","name_suffix":"","institution":"Sun Yat-sen University","department":""},{"first_name":"Peijia","middle_name":"","last_name":"Zheng","name_suffix":"","institution":"Sun Yat-sen University","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"2025-01-01T18:00:00Z","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/49385/galley/37347/download/"}]}