東華大學圖書館 |

On the Application of Item Response Models and Dependency Parsing to the Assessment of Early-Stage Spanish Acquisition.

Record Type:	Electronic resources : Monograph/item
Title/Author:	On the Application of Item Response Models and Dependency Parsing to the Assessment of Early-Stage Spanish Acquisition./
Author:	Heinrich-Wallace, Glen.
Description:	1 online resource (302 pages)
Notes:	Source: Dissertations Abstracts International, Volume: 82-04, Section: A.
Contained By:	Dissertations Abstracts International82-04A.
Subject:	Linguistics. -
Online resource:	http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=27997170click for full text (PQDT)
ISBN:	9798672161280

On the Application of Item Response Models and Dependency Parsing to the Assessment of Early-Stage Spanish Acquisition.
Heinrich-Wallace, Glen.

On the Application of Item Response Models and Dependency Parsing to the Assessment of Early-Stage Spanish Acquisition. - 1 online resource (302 pages)

Source: Dissertations Abstracts International, Volume: 82-04, Section: A.

Thesis (Ph.D.)--University of California, Davis, 2020.

Includes bibliographical references

In 2016, over 700,000 undergraduates at US universities and colleges enrolled in first- and second-year Spanish as a Foreign Language (SFL) courses. This enrollment was driven by university language requirements and SFL courses counting as general education requirements. In both cases, university policy suggests that language courses are an essential aspect of undergraduate education. Given that these courses are required, it is our responsibility as educators to use state of the art methodology to ensure that students actually meet minimum performance benchmarks when taking these classes. Fortunately, the ubiquitous use of online learning platforms facilitates collecting huge amounts of classroom assessment data, which in turn makes it possible to use data-driven approaches to vet these assessments.This dissertation approaches the question "How can we ensure the reliability of assessments used in first-year, undergraduate SFL courses?" through three coordinated studies. The first analyzes student response to test questions while the other two use data from a written assignment. Both datasets come from the same sample (n = 1,266) of students enrolled in one of three first-year SFL courses. In the first study, longitudinal data from fill-in-the-blank quizzes is analyzed with item response theory (IRT) in a vertical equating design. In addition to being a reliability study, this analysis addresses theoretical questions about the early stages of second language acquisition, including L3 effects. In the second study, a Bayesian hierarchical generalized linear model (HGLM) is used to evaluate copular verb selection in student written production. Third, a Bayesian HGLM is used to evaluate grammatical gender agreement (GGA) in the same writing samples. The dissertation includes both quiz and written data from the same students in order to directly address questions about the nature of errors made on specific content across modalities.The IRT study employs an explanatory model that simultaneously evaluates longitudinal change in item difficulty and the effect of student demographic variables. Results show large growth (0.78 logits) between the first and second course, but small growth (0.10 logits) between the second and third course. Additionally, small (0.20 logits) to moderate (0.41 logits) significant facilitatory effects were found for students with various language backgrounds relative to monolingual speakers of English, providing evidence for an L3 effect. Additionally, the study provides preliminary evidence that items testing semantically determined exceptions to generally robust morphosyntactic rules are not good candidates for placement tests.In the copular verb study of written data, global error rates were relatively low (7.84%), but the semantic category of the verb use had a large, credibly non-zero, effect on accuracy. In particular, the ambiguous case of ESTAR being used to express temporary conditions has the highest error rate across all language groups (40.19%). However, results also suggest that heritage speakers of Spanish have different patterns of errors than students from other language backgrounds, indicating a possible stochastic model of lexical choice for these students.In addition to statistical results, the GGA study also introduces an automated error-coder, which achieved an F1-score of 95.9 and shows potential as a scalable solution to error studies of learner corpora. The results give extremely detailed descriptions of the relative difficulty of different cases of GGA in student writing. One particularly striking result is that heritage speakers do not reliably outperform any other language group on any case of GGA, and actually underperform on some cases relative to L3 learners.In addition to their individual merits, together these studies lead to two preliminary conclusions: (a) semantically determined exceptions to robust rules do not have stable growth curves during the first year of SFL instruction, and (b) on these assessments, heritage speakers do not perform as well as expected. Regarding (b), after these analyses, it is clear that the assessments reviewed do not adequately capture the advantage that heritage speakers have over other demographic groups, despite this group having objectively higher proficiency in oral communication. While the assessments function well for the intended population (i.e. students without previous knowledge of Spanish), this result suggests that enrolling in first-year courses can be counterproductive for heritage speakers, as assessment focuses on content areas that is difficult for these students and deemphasizes oral communication, the area where these students excel. This result adds to a growing body of evidence that heritage speakers should have access to specialized SFL classes where they can capitalize on their strengths without being penalized for weaknesses that the courses do not effectively help them correct.Finally, the use of automated data collection via an online learning platform and the success of the automated coding algorithm employed in the GGA study provide proof-of-concept that modern technology allows for scalable solutions to data collection and data processing for language learning data. Future research can benefit from using larger datasets with measurable error as they allow for more reliable results for more complicated models. In addition to its theoretical merits, this project provides an example of how to effectively collect and analyze large datasets in second language research, promoting a data-driven approach for future work.

Electronic reproduction.
Ann Arbor, Mich. :
ProQuest,
2023

Mode of access: World Wide Web

ISBN: 9798672161280Subjects--Topical Terms:

524476
Linguistics.
Subjects--Index Terms:

Applied LinguisticsIndex Terms--Genre/Form:

542853
Electronic books.

On the Application of Item Response Models and Dependency Parsing to the Assessment of Early-Stage Spanish Acquisition.
LDR:07124nmm a2200409K 4500 001 2358632
005 20230824071929.5
006 m o d
007 cr mn ---uuuuu
008 241011s2020 xx obm 000 0 eng d
020 $a 9798672161280
035 $a (MiAaPQ)AAI27997170
035 $a AAI27997170
040 $a MiAaPQ $b eng $c MiAaPQ $d NTU
100 1 $a Heinrich-Wallace, Glen. $3 3699171
245 1 0 $a On the Application of Item Response Models and Dependency Parsing to the Assessment of Early-Stage Spanish Acquisition.
264 0 $c 2020
300 $a 1 online resource (302 pages)
336 $a text $b txt $2 rdacontent
337 $a computer $b c $2 rdamedia
338 $a online resource $b cr $2 rdacarrier
500 $a Source: Dissertations Abstracts International, Volume: 82-04, Section: A.
500 $a Advisor: Sanchez-Gutierrez, Claudia.
502 $a Thesis (Ph.D.)--University of California, Davis, 2020.
504 $a Includes bibliographical references
520 $a In 2016, over 700,000 undergraduates at US universities and colleges enrolled in first- and second-year Spanish as a Foreign Language (SFL) courses. This enrollment was driven by university language requirements and SFL courses counting as general education requirements. In both cases, university policy suggests that language courses are an essential aspect of undergraduate education. Given that these courses are required, it is our responsibility as educators to use state of the art methodology to ensure that students actually meet minimum performance benchmarks when taking these classes. Fortunately, the ubiquitous use of online learning platforms facilitates collecting huge amounts of classroom assessment data, which in turn makes it possible to use data-driven approaches to vet these assessments.This dissertation approaches the question "How can we ensure the reliability of assessments used in first-year, undergraduate SFL courses?" through three coordinated studies. The first analyzes student response to test questions while the other two use data from a written assignment. Both datasets come from the same sample (n = 1,266) of students enrolled in one of three first-year SFL courses. In the first study, longitudinal data from fill-in-the-blank quizzes is analyzed with item response theory (IRT) in a vertical equating design. In addition to being a reliability study, this analysis addresses theoretical questions about the early stages of second language acquisition, including L3 effects. In the second study, a Bayesian hierarchical generalized linear model (HGLM) is used to evaluate copular verb selection in student written production. Third, a Bayesian HGLM is used to evaluate grammatical gender agreement (GGA) in the same writing samples. The dissertation includes both quiz and written data from the same students in order to directly address questions about the nature of errors made on specific content across modalities.The IRT study employs an explanatory model that simultaneously evaluates longitudinal change in item difficulty and the effect of student demographic variables. Results show large growth (0.78 logits) between the first and second course, but small growth (0.10 logits) between the second and third course. Additionally, small (0.20 logits) to moderate (0.41 logits) significant facilitatory effects were found for students with various language backgrounds relative to monolingual speakers of English, providing evidence for an L3 effect. Additionally, the study provides preliminary evidence that items testing semantically determined exceptions to generally robust morphosyntactic rules are not good candidates for placement tests.In the copular verb study of written data, global error rates were relatively low (7.84%), but the semantic category of the verb use had a large, credibly non-zero, effect on accuracy. In particular, the ambiguous case of ESTAR being used to express temporary conditions has the highest error rate across all language groups (40.19%). However, results also suggest that heritage speakers of Spanish have different patterns of errors than students from other language backgrounds, indicating a possible stochastic model of lexical choice for these students.In addition to statistical results, the GGA study also introduces an automated error-coder, which achieved an F1-score of 95.9 and shows potential as a scalable solution to error studies of learner corpora. The results give extremely detailed descriptions of the relative difficulty of different cases of GGA in student writing. One particularly striking result is that heritage speakers do not reliably outperform any other language group on any case of GGA, and actually underperform on some cases relative to L3 learners.In addition to their individual merits, together these studies lead to two preliminary conclusions: (a) semantically determined exceptions to robust rules do not have stable growth curves during the first year of SFL instruction, and (b) on these assessments, heritage speakers do not perform as well as expected. Regarding (b), after these analyses, it is clear that the assessments reviewed do not adequately capture the advantage that heritage speakers have over other demographic groups, despite this group having objectively higher proficiency in oral communication. While the assessments function well for the intended population (i.e. students without previous knowledge of Spanish), this result suggests that enrolling in first-year courses can be counterproductive for heritage speakers, as assessment focuses on content areas that is difficult for these students and deemphasizes oral communication, the area where these students excel. This result adds to a growing body of evidence that heritage speakers should have access to specialized SFL classes where they can capitalize on their strengths without being penalized for weaknesses that the courses do not effectively help them correct.Finally, the use of automated data collection via an online learning platform and the success of the automated coding algorithm employed in the GGA study provide proof-of-concept that modern technology allows for scalable solutions to data collection and data processing for language learning data. Future research can benefit from using larger datasets with measurable error as they allow for more reliable results for more complicated models. In addition to its theoretical merits, this project provides an example of how to effectively collect and analyze large datasets in second language research, promoting a data-driven approach for future work.
533 $a Electronic reproduction. $b Ann Arbor, Mich. : $c ProQuest, $d 2023
538 $a Mode of access: World Wide Web
650 4 $a Linguistics. $3 524476
650 4 $a Educational tests & measurements. $3 3168483
650 4 $a Bilingual education. $3 2122778
653 $a Applied Linguistics
653 $a Bayesian Statistics
653 $a Language Testing
653 $a Natural Language Processing
653 $a Rasch Modeling
653 $a Second Language Acquisition
655 7 $a Electronic books. $2 lcsh $3 542853
690 $a 0290
690 $a 0288
690 $a 0282
710 2 $a ProQuest Information and Learning Co. $3 783688
710 2 $a University of California, Davis. $b Linguistics. $3 1672057
773 0 $t Dissertations Abstracts International $g 82-04A.
856 4 0 $u http://pqdd.sinica.edu.tw/twdaoapp/servlet/advanced?query=27997170 $z click for full text (PQDT)