東華大學圖書館 |

Language: English

Help

回圖書館首頁

手機版館藏查詢

Back

Switch To: Labeled | MARC Mode | ISBD

Towards Automatic Testing and Fault ...

Cao, Jialun,

Linked to FindBook

Google Book

Amazon

博客來

Towards Automatic Testing and Fault Localization in Natural Language Processing Systems /

Record Type:	Electronic resources : Monograph/item
Title/Author:	Towards Automatic Testing and Fault Localization in Natural Language Processing Systems // Jialun Cao.
Author:	Cao, Jialun,
Description:	1 electronic resource (157 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 86-05, Section: B.
Contained By:	Dissertations Abstracts International86-05B.
Subject:	Software. -
Online resource:	https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31717885
ISBN:	9798346542360

Towards Automatic Testing and Fault Localization in Natural Language Processing Systems /
Cao, Jialun,

Towards Automatic Testing and Fault Localization in Natural Language Processing Systems /Jialun Cao. - 1 electronic resource (157 pages)

Source: Dissertations Abstracts International, Volume: 86-05, Section: B.

Natural Language Processing (NLP) systems such as machine translation and chatbots become pervasive in daily life. Despite their widespread applications, how to test and debug such systems is still an open question and attracts researchers from both academia and industry. However, unlike conventional software programs, NLP systems, whose core algorithm is the deep neural network (DNN) model, do not make decisions explicitly following the program's control flow. Instead, decisions are made by propagating inputs against the tuned parameters through the trained DNN model. The indirect correlation between the program source code and the resulting DNN impedes the adoption of conventional software testing and debugging approaches. In addition, the shortage of labeled datasets for testing also brings about challenges to automatic testing. While it is possible to feed arbitrary sentences as test inputs to an NLP system, a test oracle that captures their expected test outputs is hard to define automatically.This thesis aims to propose automatic approaches to generate test cases together with test oracles to detect defects in NLP systems, and further propose a fault localization framework to aid the debugging. It consists of the following three studies.The first study proposes a semantic-based testing approach for machine translation systems. Existing methodologies mostly rely on metamorphic relations designed at the textual level (e.g., Levenshtein distance) or syntactic level (e.g., the distance between grammar structures) to determine the correctness of translation results. However, these metamorphic relations do not consider whether the original and the translated sentences have the same meaning (i.e., semantic similarity). To this end, this study proposes an automatic testing approach for machine translation systems based on semantic similarity checking. The insight of this study is that the semantics concerning logical relations and quantifiers in sentences can be captured by regular expressions (or deterministic finite automata) where efficient semantic equivalence/similarity checking algorithms can be applied. By doing so, the accuracy and F-Score of mistranslation detection could be improved.The second study proposes a metamorphic testing approach to test coreference resolution (CR) systems. CR is a task to resolve different expressions (e.g., named entities, pronouns) that refer to the same real-world entity/ event. It is a core natural language processing (NLP) component that underlies and empowers major downstream NLP applications such as machine translation, chatbots, and question-answering. Despite its broad impact, the problem of testing CR systems has rarely been studied. A major difficulty is the shortage of a labeled dataset for testing. While it is possible to feed arbitrary sentences as test inputs to a CR system, a test oracle that captures their expected test outputs (coreference relations) is hard to define automatically. To address this problem, this study proposes a methodology to generate CR-preserving test cases. This is the first study that focuses on effectively generating test inputs that respect coreference, which complements existing works.The third work designs a learning-based fault diagnosis and localization framework for NLP programs. Although various studies have been conducted to understand and detect program faults, these existing studies identify and repair suspicious neurons on the trained DNN, which, unfortunately, might be a detour.

English

ISBN: 9798346542360Subjects--Topical Terms:

619355
Software.

Towards Automatic Testing and Fault Localization in Natural Language Processing Systems /
LDR:04745nmm a22003493i 4500 001 2435939
005 20251014050545.5
006 m o d
007 cr|nu||||||||
008 261011s2024 miu||||||m |||||||eng d
020 $a 9798346542360
035 $a (MiAaPQD)AAI31717885
035 $a AAI31717885
040 $a MiAaPQD $b eng $c MiAaPQD $e rda
100 1 $a Cao, Jialun, $e author. $3 3826428
245 1 0 $a Towards Automatic Testing and Fault Localization in Natural Language Processing Systems / $c Jialun Cao.
264 1 $a Ann Arbor : $b ProQuest Dissertations & Theses, $c 2024
300 $a 1 electronic resource (157 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 86-05, Section: B.
500 $a Advisors: Cheung, Shing-Chi.
502 $b Ph.D. $c Hong Kong University of Science and Technology (Hong Kong) $d 2024.
520 $a Natural Language Processing (NLP) systems such as machine translation and chatbots become pervasive in daily life. Despite their widespread applications, how to test and debug such systems is still an open question and attracts researchers from both academia and industry. However, unlike conventional software programs, NLP systems, whose core algorithm is the deep neural network (DNN) model, do not make decisions explicitly following the program's control flow. Instead, decisions are made by propagating inputs against the tuned parameters through the trained DNN model. The indirect correlation between the program source code and the resulting DNN impedes the adoption of conventional software testing and debugging approaches. In addition, the shortage of labeled datasets for testing also brings about challenges to automatic testing. While it is possible to feed arbitrary sentences as test inputs to an NLP system, a test oracle that captures their expected test outputs is hard to define automatically.This thesis aims to propose automatic approaches to generate test cases together with test oracles to detect defects in NLP systems, and further propose a fault localization framework to aid the debugging. It consists of the following three studies.The first study proposes a semantic-based testing approach for machine translation systems. Existing methodologies mostly rely on metamorphic relations designed at the textual level (e.g., Levenshtein distance) or syntactic level (e.g., the distance between grammar structures) to determine the correctness of translation results. However, these metamorphic relations do not consider whether the original and the translated sentences have the same meaning (i.e., semantic similarity). To this end, this study proposes an automatic testing approach for machine translation systems based on semantic similarity checking. The insight of this study is that the semantics concerning logical relations and quantifiers in sentences can be captured by regular expressions (or deterministic finite automata) where efficient semantic equivalence/similarity checking algorithms can be applied. By doing so, the accuracy and F-Score of mistranslation detection could be improved.The second study proposes a metamorphic testing approach to test coreference resolution (CR) systems. CR is a task to resolve different expressions (e.g., named entities, pronouns) that refer to the same real-world entity/ event. It is a core natural language processing (NLP) component that underlies and empowers major downstream NLP applications such as machine translation, chatbots, and question-answering. Despite its broad impact, the problem of testing CR systems has rarely been studied. A major difficulty is the shortage of a labeled dataset for testing. While it is possible to feed arbitrary sentences as test inputs to a CR system, a test oracle that captures their expected test outputs (coreference relations) is hard to define automatically. To address this problem, this study proposes a methodology to generate CR-preserving test cases. This is the first study that focuses on effectively generating test inputs that respect coreference, which complements existing works.The third work designs a learning-based fault diagnosis and localization framework for NLP programs. Although various studies have been conducted to understand and detect program faults, these existing studies identify and repair suspicious neurons on the trained DNN, which, unfortunately, might be a detour.
546 $a English
590 $a School code: 1223
650 4 $a Software. $2 gtt. $3 619355
650 4 $a Neural networks. $3 677449
690 $a 0800
710 2 $a Hong Kong University of Science and Technology (Hong Kong). $e degree granting institution. $3 3814653
720 1 $a Cheung, Shing-Chi $e degree supervisor.
773 0 $t Dissertations Abstracts International $g 86-05B.
790 $a 1223
791 $a Ph.D.
792 $a 2024
856 4 0 $u https://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=31717885