• Home
• Conference Details
• Articles & Guidelines
• Conference Archives

Applications of Automated Text Analysis Technology

Automated text analysis has a long history in disciplines such as linguistics, communications, and media studies. This session will discuss applications of this technology to applied measurement tasks, including automated scoring of writing samples, and summary of open-ended survey questions.

Automated Essay Scoring. Automated essay scoring technology is evolving from research and development into operational use. The Educational Testing Service (ETS) uses their “e-rater” software to score computer-administered essays as part of the Graduate Record Examination (GRE). Other research groups, such as Thomas Landauer’s Latent Semantic Analysis (LSA) project at Knowledge Analysis Technologies and Ellis Page’s Project Essay Grade (PEG) at Duke University, have received attention from both the academic and applied measurement communities. Automated scoring for writing assessments offers three benefits: (1) Reduced scoring time. Once an automated system is in place, each essay must be scored by a smaller number of trained raters in addition to the automated scoring system. (2) Reduced scoring expense. Because a smaller number of trained raters is needed to supplement automated scoring, less money must be spent on rater salary and overhead. (3) Increased measurement reliability. An automated scoring procedure performs with perfect reliability. There is no measurement error introduced by rater fatigue, personal bias, and other sources of human error.

Although computer-based scoring is becoming more common, the details of most approaches remain proprietary. Automated content analysis (ACA) has an accessible literature, a number of available software programs, and a well-developed set of statistical techniques. The presenters have developed a general model of the ACA process and a set of techniques to summarize and profile content in large collections of text. This process can be used to determine the general feasibility of automated essay scoring and to make an informed decision about which automated scoring approach is most appropriate. The presenters use this approach to model emphasized content in a set of writing exercises from a Federal selection program. The ten content categories identified by this procedure have three useful properties. They capture content found in most writing samples, they are emphasized differently in high, medium and low scoring writing samples, and they capture patterns of emphasis which distinguish writing samples from other writing. Implications of this technology for rater training and examining adverse impact will also be discussed.

Summary of Open-Ended Question Responses. Automated text analysis has also been used to summarize the responses to open-ended questions. This approach allows researchers to quickly and objectively summarize unanticipated themes in responses to survey questions. The presenters will overview tools currently available for summarizing short text responses and use one such tool, WordStat, to summarize open ended questions from a customer satisfaction survey.

Steven R. Burnkrant is a Personnel Research Psychologist in the Personnel Resources and Development Center of the U.S. Office of Personnel Management. He received a B.A. in Psychology with a minor in Mathematics from Colorado College, and a M.S. in industrial and organizational psychology from Virginia Tech, where he is currently completing his dissertation. Mr. Burnkrant's concentration on the application of technology to I/O problems bridges his many interests, which range from job analysis and statistical methods to organizational surveys and development.

John Ford joined the U.S. Office of Personnel Management in 1998, where he has participated in training evaluation, test development and test validation efforts. John earned a Ph.D. in Instructional Psychology from Brigham Young University in 1993. Before joining OPM, John was a project manager for Alpine Media, where he directed job analysis and test development projects for information technology clients and worked as an instructional designer and interface designer in multimedia development projects. Current professional interests include automated text analysis, computer-based testing and training needs analysis.

Rosemary Miller is a Personnel Research Psychologist with the Office of Personnel Management's Personnel Resources and Development Center. She received a Ph.D. from Hofstra University and has over 15 years of experience working in both private and public sector organizations. Her interests include all aspects of selection testing, job analysis, and survey research.

Alix Roberts attended Rhodes College in Memphis, Tennessee, for her Bachelor of Arts in Psychology and The George Washington University in Washington, D.C., for her Master of Arts in Organizational Sciences. She currently works for the Personnel Resources and Development Center at the U.S. Office of Personnel Management. She has also worked for the Society for Human Resource Management and the University of Tennessee's Criminal Justice Diversion Program. Her research interests include: psycholinguistics, selection, and equity theory.

Henry Thibodeaux is a Personnel Research Psychologist with the Office of Personnel Management's Personnel Resources and Development Center. He received a B.A in Psychology for Louisiana State University, a M.A. in Industrial/Organizational Psychology from the University of West Florida, and is currently completing his dissertation at the University of Southern Mississippi. His research interests include feedback seeking behavior, applicant reactions, and assessment center practices.