東華大學圖書館 |

語系: 繁體中文

說明(常見問題)

回圖書館首頁

手機版館藏查詢

登入

回首頁

切換: 標籤 | MARC模式 | ISBD

Understanding and Improving Ecologic...

Liu, Nelson F.,

FindBook

Google Book

Amazon

博客來

Understanding and Improving Ecological Validity in Natural Language Processing Evaluations /

紀錄類型:	書目-電子資源 : Monograph/item
正題名/作者:	Understanding and Improving Ecological Validity in Natural Language Processing Evaluations // Nelson F Liu.
作者:	Liu, Nelson F.,
面頁冊數:	1 electronic resource (204 pages)
附註:	Source: Dissertations Abstracts International, Volume: 86-10, Section: A.
Contained By:	Dissertations Abstracts International86-10A.
標題:	Language. -
電子資源:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31974591
ISBN:	9798311950947

Understanding and Improving Ecological Validity in Natural Language Processing Evaluations /
Liu, Nelson F.,

Understanding and Improving Ecological Validity in Natural Language Processing Evaluations /Nelson F Liu. - 1 electronic resource (204 pages)

Source: Dissertations Abstracts International, Volume: 86-10, Section: A.

Benchmark datasets drive natural language processing (NLP) research; they shape research agendas and enable us to evaluate collective progress. In this thesis, we study and improve NLP evaluations through the lens of ecological validity, the extent to which experimental findings on benchmark datasets generalize to real-world settings.First, we argue that constructing ecologically-valid benchmarks is less about collecting data with similar surface-level features to a desired target application (e.g., human-written text vs. synthetically-constructed examples), and more about whether the benchmark necessitates the capabilities required for improving performance in real-world settings. Specifically, we introduce the notion of concurrence, which measures how modeling approaches generalize across datasets. We show that despite different surface-level features, human-constructed question-answering benchmarks have high concurrence among themselves, and that even programmatically-generated benchmarks can have high concurrence with human-constructed benchmarks---these results indicate that focusing on capturing underlying capabilities helps bridge the gap between benchmark performance and real-world outcomes.Second, we develop a more ecologically valid approach to evaluating long-context understanding in language models, which underlies a variety of deployed NLP systems. In particular, a language model's advertised training context length is not necessarily equivalent to its effective context length. In contrast to prior work that evaluates long-context capabilities by analyzing next-word prediction, our experimental setup measures the effectivecontext length of prompted language models, which better aligns with how they are used in practice. We find that improving ecological validity in long-context evaluation leads to qualitatively different conclusions about language model capabilities---while prior work found that language models are recency-biased, our results show that modern language models may be primacy- andrecency-biased.Finally, we design new metrics to evaluate generative search engines, an NLP system that fulfills user information needs by directly generating responses to input queries, along with in-line citations. We develop the citation precision and citation recall metrics and use them to audit four deployed systems. Our results indicate that responses from existing generative search engines are fluent and appear informative, but frequently contain unsupported statements and inaccurate citations: on average, a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence.

English

ISBN: 9798311950947Subjects--Topical Terms:

643551
Language.

Understanding and Improving Ecological Validity in Natural Language Processing Evaluations /
LDR:03972nmm a22003613i 4500 001 2433299
005 20250825104044.5
006 m o d
007 cr|nu||||||||
008 261011s2025 miu||||||m |||||||eng d
020 $a 9798311950947
035 $a (MiAaPQD)AAI31974591
035 $a (MiAaPQD)Stanfordcm489rr5423
035 $a AAI31974591
040 $a MiAaPQD $b eng $c MiAaPQD $e rda
100 1 $a Liu, Nelson F., $e author. $3 3822850
245 1 0 $a Understanding and Improving Ecological Validity in Natural Language Processing Evaluations / $c Nelson F Liu.
264 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2025
300 $a 1 electronic resource (204 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 86-10, Section: A.
500 $a Advisors: Liang, Percy Committee members: Hashimoto, Tatsunori; Manning, Christopher.
502 $b Ph.D. $c Stanford University $d 2025.
520 $a Benchmark datasets drive natural language processing (NLP) research; they shape research agendas and enable us to evaluate collective progress. In this thesis, we study and improve NLP evaluations through the lens of ecological validity, the extent to which experimental findings on benchmark datasets generalize to real-world settings.First, we argue that constructing ecologically-valid benchmarks is less about collecting data with similar surface-level features to a desired target application (e.g., human-written text vs. synthetically-constructed examples), and more about whether the benchmark necessitates the capabilities required for improving performance in real-world settings. Specifically, we introduce the notion of concurrence, which measures how modeling approaches generalize across datasets. We show that despite different surface-level features, human-constructed question-answering benchmarks have high concurrence among themselves, and that even programmatically-generated benchmarks can have high concurrence with human-constructed benchmarks---these results indicate that focusing on capturing underlying capabilities helps bridge the gap between benchmark performance and real-world outcomes.Second, we develop a more ecologically valid approach to evaluating long-context understanding in language models, which underlies a variety of deployed NLP systems. In particular, a language model's advertised training context length is not necessarily equivalent to its effective context length. In contrast to prior work that evaluates long-context capabilities by analyzing next-word prediction, our experimental setup measures the effectivecontext length of prompted language models, which better aligns with how they are used in practice. We find that improving ecological validity in long-context evaluation leads to qualitatively different conclusions about language model capabilities---while prior work found that language models are recency-biased, our results show that modern language models may be primacy- andrecency-biased.Finally, we design new metrics to evaluate generative search engines, an NLP system that fulfills user information needs by directly generating responses to input queries, along with in-line citations. We develop the citation precision and citation recall metrics and use them to audit four deployed systems. Our results indicate that responses from existing generative search engines are fluent and appear informative, but frequently contain unsupported statements and inaccurate citations: on average, a mere 51.5% of generated sentences are fully supported by citations and only 74.5% of citations support their associated sentence.
546 $a English
590 $a School code: 0212
650 4 $a Language. $3 643551
650 4 $a Search engines. $3 869493
650 4 $a Citations. $3 3564655
650 4 $a Documents. $3 3815417
650 4 $a Natural language. $3 3562052
650 4 $a Benchmarks. $3 3548256
690 $a 0679
710 2 $a Stanford University. $e degree granting institution. $3 3765820
720 1 $a Liang, Percy $e degree supervisor.
773 0 $t Dissertations Abstracts International $g 86-10A.
790 $a 0212
791 $a Ph.D.
792 $a 2025
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31974591