東華大學圖書館 |

Language: English

Help

回圖書館首頁

手機版館藏查詢

Back

Switch To: Labeled | MARC Mode | ISBD

Towards Robust and Scalable Evaluati...

Chiang, Wei-Lin,

Linked to FindBook

Google Book

Amazon

博客來

Towards Robust and Scalable Evaluation for Large Language Models /

Record Type:	Electronic resources : Monograph/item
Title/Author:	Towards Robust and Scalable Evaluation for Large Language Models // Wei-Lin Chiang.
Author:	Chiang, Wei-Lin,
Description:	1 electronic resource (135 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 86-09, Section: B.
Contained By:	Dissertations Abstracts International86-09B.
Subject:	Computer science. -
Online resource:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31641460
ISBN:	9798310103184

Towards Robust and Scalable Evaluation for Large Language Models /
Chiang, Wei-Lin,

Towards Robust and Scalable Evaluation for Large Language Models /Wei-Lin Chiang. - 1 electronic resource (135 pages)

Source: Dissertations Abstracts International, Volume: 86-09, Section: B.

The rapid advancement of Large Language Models (LLMs), driven by scaling laws and substantial computational investments, has revolutionized artificial intelligence, enabling significant breakthroughs across domains such as conversational AI, programming, and complex reasoning. However, traditional benchmarks, being static and narrowly scoped, struggle to comprehensively evaluate LLM capabilities in real-world scenarios. These limitations are further compounded by potential contamination, where overlaps between training and test data inflate performance metrics and obscure true generalization.This dissertation tackles these challenges by introducing robust and scalable evaluation methodologies. First, we develop methods to detect and mitigate benchmark contamination, ensuring reliable and trustworthy assessments. To align evaluations with real-world applications, we present Chatbot Arena, a live, crowdsourced platform that gathers real-time human feedback via pairwise comparisons of model responses. By analyzing millions of interactions, Chatbot Arena offers dynamic insights into model performance, emerging as an industry-standard platform for evaluating human preference in conversational AI.Complementing human evaluation, we introduce LLM-as-a-Judge, an automated framework that leverages advanced LLMs to approximate human judgments, significantly reducing manual evaluation costs while achieving strong agreement with human preferences. Furthermore, we propose the BenchBuilder pipeline, an automated system for creating diverse, high-quality benchmarks from crowdsourced prompts, addressing contamination issues and enabling robust, real-world evaluations. These contributions establish scalable, dynamic, and human-centric evaluation systems, advancing the development of AI systems that are not only powerful but also reliable, and aligned with human values.

English

ISBN: 9798310103184Subjects--Topical Terms:

523869
Computer science.
Subjects--Index Terms:

Benchmarks

Towards Robust and Scalable Evaluation for Large Language Models /
LDR:03314nmm a22004333i 4500 001 2432267
005 20250707081331.5
006 m o d
007 cr|nu||||||||
008 261011s2024 miu||||||m |||||||eng d
020 $a 9798310103184
035 $a (MiAaPQD)AAI31641460
035 $a AAI31641460
040 $a MiAaPQD $b eng $c MiAaPQD $e rda
100 1 $a Chiang, Wei-Lin, $e author. $3 3821423
245 1 0 $a Towards Robust and Scalable Evaluation for Large Language Models / $c Wei-Lin Chiang.
264 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2024
300 $a 1 electronic resource (135 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 86-09, Section: B.
500 $a Advisors: Stoica, Ion Committee members: Gonzalez, Joseph E.; Abbeel, Pieter; Hsieh, Cho-Jui.
502 $b Ph.D. $c University of California, Berkeley $d 2024.
520 $a The rapid advancement of Large Language Models (LLMs), driven by scaling laws and substantial computational investments, has revolutionized artificial intelligence, enabling significant breakthroughs across domains such as conversational AI, programming, and complex reasoning. However, traditional benchmarks, being static and narrowly scoped, struggle to comprehensively evaluate LLM capabilities in real-world scenarios. These limitations are further compounded by potential contamination, where overlaps between training and test data inflate performance metrics and obscure true generalization.This dissertation tackles these challenges by introducing robust and scalable evaluation methodologies. First, we develop methods to detect and mitigate benchmark contamination, ensuring reliable and trustworthy assessments. To align evaluations with real-world applications, we present Chatbot Arena, a live, crowdsourced platform that gathers real-time human feedback via pairwise comparisons of model responses. By analyzing millions of interactions, Chatbot Arena offers dynamic insights into model performance, emerging as an industry-standard platform for evaluating human preference in conversational AI.Complementing human evaluation, we introduce LLM-as-a-Judge, an automated framework that leverages advanced LLMs to approximate human judgments, significantly reducing manual evaluation costs while achieving strong agreement with human preferences. Furthermore, we propose the BenchBuilder pipeline, an automated system for creating diverse, high-quality benchmarks from crowdsourced prompts, addressing contamination issues and enabling robust, real-world evaluations. These contributions establish scalable, dynamic, and human-centric evaluation systems, advancing the development of AI systems that are not only powerful but also reliable, and aligned with human values.
546 $a English
590 $a School code: 0028
650 4 $a Computer science. $3 523869
650 4 $a Systems science. $3 3168411
653 $a Benchmarks
653 $a Large Language Models
653 $a Chatbot Arena
653 $a Evaluation
653 $a Human preferences
690 $a 0984
690 $a 0800
690 $a 0790
710 2 $a University of California, Berkeley. $b Electrical Engineering & Computer Sciences. $e degree granting institution. $3 3819104
720 1 $a Stoica, Ion $e degree supervisor.
773 0 $t Dissertations Abstracts International $g 86-09B.
790 $a 0028
791 $a Ph.D.
792 $a 2024
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31641460