Workshop 1（工作坊一）

Workshop title

Methodologies for mapping test scores to external standards in the Chinese context [中国背景下考试分数与外部标准的对接方法]

Workshop description

The introduction of China’s Standards of English Language Ability (CSE) is a major recent development in the conceptualization of Chinese learners’ language proficiency. Released by the Ministry of Education and National Language Commission of China in 2018, the CSE includes comprehensive English proficiency scales covering the full range of EFL learners in China. A test score that is “mapped” (linked or aligned) to the performance levels and the “can do” descriptions of language proficiency in the CSE might also facilitate score-based decisions, e.g., determining admissions to educational institutions, or placement into appropriate classes. Ultimately, alignment is a claim about the interpretation of test scores in relation to external levels of language proficiency. To support such a claim, established procedures should be carefully implemented and multiple sources of evidence should be collected.

This all-day workshop will introduce participants to the development of the CSE levels for speaking and listening and methodologies for mapping the scores of a major international language test to these levels. Specifically, workshop attendees will be offered the opportunity to engage in tasks and activities based on a recent project, during which researchers from China and the United States collaborated to map the scores of theTOEFL iBT®test to the CSE levels. By participating in these tasks and activities, workshop attendees will obtain a deep understanding of the proficiency levels of the CSE and the appropriate steps in mapping test scores to external proficiency levels.

Workshop presenters:

Spiros Papageorgiou, Educational Testing Service, USA [Spiros Papageorgiou，美国教育考试服务中心]

Ching-Ni Hsieh, Educational Testing Service, USA [Ching-Ni Hsieh，美国教育考试服务中心]

Manman Gao, Anhui University, China [高满满，安徽大学]

Shangchao Min, Zhejiang University, China [闵尚超，浙江大学]

Workshop schedule (Friday, July 5, 9:00 AM – 5:30 PM)

· Welcome and overview of the workshop

· Introduction to China’s Standards of English Language Ability (CSE)

· Introduction to theTOEFL iBT®test and the process of score mapping to the CSE levels

· Familiarization with the CSE levels for speaking

· Training on score mapping method for productive language skills

· Familiarization with the CSE levels for listening

· Training on score mapping method for receptive language skills

· Use of test performance data to finalize score mapping

Workshop 2（工作坊二）

Workshop title

Rasch analysis and modeling[Rasch分析与建模]

Workshop description

The Rasch model is a useful probabilistic model in the field of language testing and assessment. The rationality of applying the Rasch model for studies addressing issues of validity lies in two facts: the Rasch model can compare predicted outcomes with observed outcomes of test performance, and the Rasch model predicts outcomes upon several factors: test takers’ ability, difficulty of test items, rater severity, etc.

This workshop focuses on hands-on training, with an emphasis on in-depth Rasch application. It is divided into the following three sections.

The concept of Rasch modeling

Some teachers and researchers in education may find the Rasch model difficult to understand because they might think the Rasch model has nothing to do with their daily research activities. In fact, people unconsciously make all kinds of important judgement and decisions in the same way as the Rasch model does all the time, including judgement in sport competition and decisions for jobs, schools, marriage, etc. In this section, participants will be familiarized with the concept of Rasch modeling through various real life examples. Subsequently, participants will be shown how test item difficulty and test takers’ ability, from a probabilistic perspective, may be estimated merely by eyes observation. Participants will immediately realize that Rasch analysis is no longer a mystery and that its arithmetic calculation is virtually a reflection of common knowledge.

Guided Rasch software practicing (Winsteps and Facets)

Participants will be guided to use the Rasch modeling based software: Winsteps and Facets, both of which are very useful tools to conduct Rasch analysis with different purposes. Participants will be provided with data covering a series of research scenarios. They will learn how to import data into the software and obtain Rasch analysis results with real-time operation following the instructor.

Self-practicing and discussion for assessment purpose

In this section, participants will be invited to answer questions concerning validity based on analysis results obtained from the Rasch software, as a checkout of their understanding of the gist of the Rasch modeling.

With a clear idea of the Rasch model and adequate Rasch software real work practicing, this workshop will hopefully enable participants to conduct Rasch analysis and modeling independently in their own research studies.

Workshop presenters:

Jianda Liu, Guangdong University of Foreign Studies [刘建达，广东外语外贸大学]

JIantao Lu, Guangdong University of Foreign Studies [吕剑涛，广东外语外贸大学]

Workshop schedule (Friday, July 5, 9:00 AM – 5:30 PM)

Workshop 3（工作坊三）

Workshop title

Generalizability Theory designs in language research[语言研究中的概化理论设计]

Workshop description

Generalizability (G-) theory is a statistical theory of measurement that expands classical test theory to include multiple sources of error and explicitly connects measurement operations to the purpose of measurement.This workshop is designed to introduce language researchers to theG-theory approach in language research designs and data analysis.Specific topics for this research seminar include but not limited to: 1) developing and understanding the central concepts ofG-theory in language research, 2) differences betweenG-studies and decision (D-) studies, 3) various (i.e.,fully-crossed and nested)G-study andD-study designs, 4) control cards programming for GENOVA (the program is free for downloading), and 5) interpreting individual tests scores withinG-theory framework.

Workshop presenter:

Jinyan Huang, Hunan University [黄金雁，湖南大学]

Workshop schedule (Friday, July 5, 9:00 AM – 5:30 PM)