東華大學圖書館 |

Language: English

Help

回圖書館首頁

手機版館藏查詢

Back

Switch To: Labeled | MARC Mode | ISBD

Improving Grounding Ability of Visio...

Zhang, Yue,

Linked to FindBook

Google Book

Amazon

博客來

Improving Grounding Ability of Vision and Language Navigation Agents /

Record Type:	Electronic resources : Monograph/item
Title/Author:	Improving Grounding Ability of Vision and Language Navigation Agents // Yue Zhang.
Author:	Zhang, Yue,
Description:	1 electronic resource (113 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 86-10, Section: A.
Contained By:	Dissertations Abstracts International86-10A.
Subject:	Computer science. -
Online resource:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31934827
ISBN:	9798310380820

Improving Grounding Ability of Vision and Language Navigation Agents /
Zhang, Yue,

Improving Grounding Ability of Vision and Language Navigation Agents /Yue Zhang. - 1 electronic resource (113 pages)

Source: Dissertations Abstracts International, Volume: 86-10, Section: A.

Understanding and following human instructions is crucial for intelligent agents to interact with humans in real-world environments. One formal problem setting designed to facilitate advancing this direction of research is Vision-and-Language Navigation (VLN). VLN requires the agent to carry out a sequence of actions in a photo-realistic simulated indoor environment in response to natural language instructions. Although significant progress has been achieved in this direction, navigation agents still have challenges understanding instructions and accurately grounding them in their visual perception.To elaborate, the challenges include 1) lacking explicit learning of spatial semantics in both text and vision modalities, 2) difficulties in handling ambiguous instructions and the lack of explainability, and 3) gaps in language understanding for navigation in realistic environments, such as continuous and 3D spaces. In this thesis, we develop new techniques to address these challenges. First, we explicitly model spatial semantics to improve the navigation agent's grounding by incorporating navigation progress, the alignments between textual landmarks and visual objects, and the corresponding spatial directions. Besides, we design specialized modules to capture distinct semantic aspects through corresponding pre-training tasks, enabling the effective acquisition of the respective skills. Second, to help agents deal with ambiguous instructions, we introduce a translator to convert the original ambiguous instructions into easy-to-following instructions considering recognizable and distinctive landmarks. The designed translator bridges the gap between the instruction given by humans and the agent's visual perception ability. Furthermore, to improve the explainability of the decisions made by the agent, we introduce a language generator for the navigation agent to equip it with the ability to generate explanations about navigation progress, navigation difficulties, and observed visual objects in the selected target view. Such explanations enable the agent to explain the situation from its own perspective, enhancing its ability to interact with humans effectively. Third, to advance navigation in a more realistic setting, we contribute to language grounding in continuous and 3D environments. For navigation in continuous environments, we introduce a dual-action-perception module that integrates a low-level action decoder, jointly trained with high-level action prediction. This design enables the VLN agent to learn and ground the selected visual view to the corresponding low-level controls. Additionally, in 3D environments, we develop techniques to enhance the agent's situated spatial understanding, further improving its navigation capabilities in 3D scenarios. We evaluate our proposed methods across different commonly-used navigation benchmarks and provide comprehensive quantitative results and qualitative analysis. The experimental results demonstrate the effectiveness of our explicit grounding modules, the proposed pre-training tasks, and the synthesized data incorporating recognized and distinctive landmarks, significantly enhance navigation performance, generalizability, and language grounding ability. Additionally, our novel architectures designed for continuous and 3D environments push the boundaries of navigation agent research to real-world scenarios. Notably, these advancements contribute to improved interpretability of the agent's decision-making process, offering deeper insights into the rationale behind its navigational actions.

English

ISBN: 9798310380820Subjects--Topical Terms:

523869
Computer science.
Subjects--Index Terms:

Vision-and-Language Navigation

Improving Grounding Ability of Vision and Language Navigation Agents /
LDR:05055nmm a22004333i 4500 001 2433271
005 20250825104037.5
006 m o d
007 cr|nu||||||||
008 261011s2025 miu||||||m |||||||eng d
020 $a 9798310380820
035 $a (MiAaPQD)AAI31934827
035 $a AAI31934827
040 $a MiAaPQD $b eng $c MiAaPQD $e rda
100 1 $a Zhang, Yue, $e author. $3 3770509
245 1 0 $a Improving Grounding Ability of Vision and Language Navigation Agents / $c Yue Zhang.
264 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2025
300 $a 1 electronic resource (113 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 86-10, Section: A.
500 $a Advisors: Kordjmashidi, Parisa Committee members: Ross, Arun; Tan, Xiaobo; Kong, Yu; Tan, Pang-Ning.
502 $b Ph.D. $c Michigan State University $d 2025.
520 $a Understanding and following human instructions is crucial for intelligent agents to interact with humans in real-world environments. One formal problem setting designed to facilitate advancing this direction of research is Vision-and-Language Navigation (VLN). VLN requires the agent to carry out a sequence of actions in a photo-realistic simulated indoor environment in response to natural language instructions. Although significant progress has been achieved in this direction, navigation agents still have challenges understanding instructions and accurately grounding them in their visual perception.To elaborate, the challenges include 1) lacking explicit learning of spatial semantics in both text and vision modalities, 2) difficulties in handling ambiguous instructions and the lack of explainability, and 3) gaps in language understanding for navigation in realistic environments, such as continuous and 3D spaces. In this thesis, we develop new techniques to address these challenges. First, we explicitly model spatial semantics to improve the navigation agent's grounding by incorporating navigation progress, the alignments between textual landmarks and visual objects, and the corresponding spatial directions. Besides, we design specialized modules to capture distinct semantic aspects through corresponding pre-training tasks, enabling the effective acquisition of the respective skills. Second, to help agents deal with ambiguous instructions, we introduce a translator to convert the original ambiguous instructions into easy-to-following instructions considering recognizable and distinctive landmarks. The designed translator bridges the gap between the instruction given by humans and the agent's visual perception ability. Furthermore, to improve the explainability of the decisions made by the agent, we introduce a language generator for the navigation agent to equip it with the ability to generate explanations about navigation progress, navigation difficulties, and observed visual objects in the selected target view. Such explanations enable the agent to explain the situation from its own perspective, enhancing its ability to interact with humans effectively. Third, to advance navigation in a more realistic setting, we contribute to language grounding in continuous and 3D environments. For navigation in continuous environments, we introduce a dual-action-perception module that integrates a low-level action decoder, jointly trained with high-level action prediction. This design enables the VLN agent to learn and ground the selected visual view to the corresponding low-level controls. Additionally, in 3D environments, we develop techniques to enhance the agent's situated spatial understanding, further improving its navigation capabilities in 3D scenarios. We evaluate our proposed methods across different commonly-used navigation benchmarks and provide comprehensive quantitative results and qualitative analysis. The experimental results demonstrate the effectiveness of our explicit grounding modules, the proposed pre-training tasks, and the synthesized data incorporating recognized and distinctive landmarks, significantly enhance navigation performance, generalizability, and language grounding ability. Additionally, our novel architectures designed for continuous and 3D environments push the boundaries of navigation agent research to real-world scenarios. Notably, these advancements contribute to improved interpretability of the agent's decision-making process, offering deeper insights into the rationale behind its navigational actions.
546 $a English
590 $a School code: 0128
650 4 $a Computer science. $3 523869
650 4 $a Language. $3 643551
650 4 $a Linguistics. $3 524476
653 $a Vision-and-Language Navigation
653 $a Vision modalities
653 $a Natural language instructions
653 $a 3D environments
690 $a 0984
690 $a 0800
690 $a 0679
690 $a 0290
710 2 $a Michigan State University. $b Computer Science - Doctor of Philosophy. $e degree granting institution. $3 3817161
720 1 $a Kordjmashidi, Parisa $e degree supervisor.
773 0 $t Dissertations Abstracts International $g 86-10A.
790 $a 0128
791 $a Ph.D.
792 $a 2025
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31934827