東華大學圖書館 |

Language: English

Help

回圖書館首頁

手機版館藏查詢

Back

Switch To: Labeled | MARC Mode | ISBD

Exploring Reasoning and Interactive ...

Prabhakar, Akshara.

Linked to FindBook

Google Book

Amazon

博客來

Exploring Reasoning and Interactive Benchmarking of Language Models.

Record Type:	Electronic resources : Monograph/item
Title/Author:	Exploring Reasoning and Interactive Benchmarking of Language Models./
Author:	Prabhakar, Akshara.
Published:	Ann Arbor : ProQuest Dissertations & Theses, : 2024,
Description:	104 p.
Notes:	Source: Masters Abstracts International, Volume: 85-12.
Contained By:	Masters Abstracts International85-12.
Subject:	Computer science. -
Online resource:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31244374
ISBN:	9798382809557

Exploring Reasoning and Interactive Benchmarking of Language Models.
Prabhakar, Akshara.

Exploring Reasoning and Interactive Benchmarking of Language Models. - Ann Arbor : ProQuest Dissertations & Theses, 2024 - 104 p.

Source: Masters Abstracts International, Volume: 85-12.

Thesis (M.S.E.)--Princeton University, 2024.

This thesis aims to deepen our understanding of the reasoning process undertaken in Large Language Models (LLMs) and introduces a challenging benchmark to evaluate language agents' interactive code-generation capabilities.Part 1 delves into untangling the factors affecting the Chain-of-Thought (CoT) reasoning process in LLMs. Despite CoT prompting having demonstrated significant efficacy in enhancing multi-step reasoning capabilities, it has been found to yield biased answers and produce unfaithful explanations, raising interpretability concerns. Additionally, debates persist regarding whether LLMs truly generalize or rely on heuristics. Focusing on the symbolic reasoning task of decoding shift ciphers, we develop a simple probabilistic approach to identify these factors - the probability of the task's expected output (probabilistic effect), what the model has implicitly learned during pre-training (memorization influenced), and the model's attempt to adopt shorter reasoning steps (noisy reasoning) and show the consequent drastic variability in task accuracy. Through a series of experiments, we conclude that LLM behavior exhibits clear hallmarks of both memorization and true reasoning suggesting that CoT resembles a probabilistic, memorization-influenced form of noisy reasoning.Part 2 transitions from LLMs to the emerging domain of language agents, where the LLM is now grounded in a digital environment to aid in decision-making. We introduce InterCode -- an interactive RL environment to benchmark the interactive code generation abilities of language agents with execution-driven feedback. By converting traditional seq2seq datasets into three interactive code environments (Bash, SQL, and Python), we demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results underscore the benefits of interactive code generation and establish InterCode as a robust, scalable benchmark for advancing code understanding and generation capabilities. Furthermore, we use this framework in the cybersecurity realm; by developing a Capture the Flag task in the InterCode environment, we find that while language agents possess rudimentary cybersecurity knowledge, they are not able to perform multi-step cybersecurity tasks out of the box.Overall, it highlights the evolving landscape of LLMs, emphasizing the need for comprehensive evaluation and understanding of their capabilities across different domains.

ISBN: 9798382809557Subjects--Topical Terms:

523869
Computer science.
Subjects--Index Terms:

Code generation

Exploring Reasoning and Interactive Benchmarking of Language Models.
LDR:03702nmm a2200397 4500 001 2428133
005 20241210085413.5
006 m o d
007 cr#unu||||||||
008 261011s2024 ||||||||||||||||| ||eng d
020 $a 9798382809557
035 $a (MiAaPQ)AAI31244374
035 $a AAI31244374
040 $a MiAaPQ $c MiAaPQ
100 1 $a Prabhakar, Akshara. $3 3815174
245 1 0 $a Exploring Reasoning and Interactive Benchmarking of Language Models.
260 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2024
300 $a 104 p.
500 $a Source: Masters Abstracts International, Volume: 85-12.
500 $a Advisor: Narasimhan, Karthik.
502 $a Thesis (M.S.E.)--Princeton University, 2024.
520 $a This thesis aims to deepen our understanding of the reasoning process undertaken in Large Language Models (LLMs) and introduces a challenging benchmark to evaluate language agents' interactive code-generation capabilities.Part 1 delves into untangling the factors affecting the Chain-of-Thought (CoT) reasoning process in LLMs. Despite CoT prompting having demonstrated significant efficacy in enhancing multi-step reasoning capabilities, it has been found to yield biased answers and produce unfaithful explanations, raising interpretability concerns. Additionally, debates persist regarding whether LLMs truly generalize or rely on heuristics. Focusing on the symbolic reasoning task of decoding shift ciphers, we develop a simple probabilistic approach to identify these factors - the probability of the task's expected output (probabilistic effect), what the model has implicitly learned during pre-training (memorization influenced), and the model's attempt to adopt shorter reasoning steps (noisy reasoning) and show the consequent drastic variability in task accuracy. Through a series of experiments, we conclude that LLM behavior exhibits clear hallmarks of both memorization and true reasoning suggesting that CoT resembles a probabilistic, memorization-influenced form of noisy reasoning.Part 2 transitions from LLMs to the emerging domain of language agents, where the LLM is now grounded in a digital environment to aid in decision-making. We introduce InterCode -- an interactive RL environment to benchmark the interactive code generation abilities of language agents with execution-driven feedback. By converting traditional seq2seq datasets into three interactive code environments (Bash, SQL, and Python), we demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies such as ReAct and Plan & Solve. Our results underscore the benefits of interactive code generation and establish InterCode as a robust, scalable benchmark for advancing code understanding and generation capabilities. Furthermore, we use this framework in the cybersecurity realm; by developing a Capture the Flag task in the InterCode environment, we find that while language agents possess rudimentary cybersecurity knowledge, they are not able to perform multi-step cybersecurity tasks out of the box.Overall, it highlights the evolving landscape of LLMs, emphasizing the need for comprehensive evaluation and understanding of their capabilities across different domains.
590 $a School code: 0181.
650 4 $a Computer science. $3 523869
650 4 $a Logic. $3 529544
653 $a Code generation
653 $a Interactive machine learning
653 $a Interpretability
653 $a Language agents
653 $a Large Language Models
653 $a Reasoning
690 $a 0984
690 $a 0800
690 $a 0395
710 2 $a Princeton University. $b Computer Science. $3 2099280
773 0 $t Masters Abstracts International $g 85-12.
790 $a 0181
791 $a M.S.E.
792 $a 2024
793 $a English
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31244374