{"pk":49273,"title":"Enhancing Objectivity in LLM-as-a-Judge through Perturbation Injection","subtitle":null,"abstract":"LLM-as-a-judge is considered a potential substitute for human evaluation due to its efficiency and cost-effectiveness. However, recent studies indicate that LLM-as-a-judge exhibits systematic biases when comparing candidate answers, including contextual, verbosity, and positional bias. These biases, as we find, mirror human cognitive biases like the anchoring effect and availability heuristic, where intuitive decisions prioritize superficial features over deeper analysis. Inspired by the Dual Process Theory, we propose that LLM evaluations often resemble system 1 thinking, leading to biased judgments. To address this, we introduce PeBC, a Perturbation-Based Calibration framework that shifts LLM evaluations from system 1 to system 2 reasoning through perturbation injection, bias analysis, and rule calibration. Our experiments on the meta-evaluation benchmarks LLMBar-Natural and LLMBar-Adversarial demonstrate that PeBC successfully mitigates evaluation biases, outperforming existing state-of-the-art (SOTA) methods across various test scenarios and achieving better alignment with human judgments.","language":"eng","license":{"name":"","short_name":"","text":null,"url":""},"keywords":[{"word":"Artificial Intelligence; Computer Science; Decision making; Natural Language Processing; Reasoning"}],"section":"Papers with Oral Presentation","is_remote":true,"remote_url":"https://escholarship.org/uc/item/6520s7n6","frozenauthors":[{"first_name":"Zhihao","middle_name":"","last_name":"Zhu","name_suffix":"","institution":"Shanghai Jiao Tong University","department":""},{"first_name":"Haoran","middle_name":"","last_name":"Liao","name_suffix":"","institution":"Shanghai Jiao Tong University","department":""},{"first_name":"Yaohui","middle_name":"","last_name":"Jin","name_suffix":"","institution":"Shanghai Jiao Tong University","department":""}],"date_submitted":null,"date_accepted":null,"date_published":"2025-01-01T12:00:00-06:00","render_galley":null,"galleys":[{"label":"PDF","type":"pdf","path":"https://journalpub.escholarship.org/cognitivesciencesociety/article/49273/galley/37234/download/"}]}