<?xml version="1.0" standalone="yes"?>
<Paper uid="N04-4025">
  <Title>Automated Team Discourse Annotation and Performance Prediction Using LSA</Title>
  <Section position="8" start_page="0" end_page="0" type="concl">
    <SectionTitle>
6 Conclusions and Future Work
</SectionTitle>
    <Paragraph position="0"> Overall, the results of the study show that LSA can be used for tagging content as well as predicting team performance based on team dialogues. Given the limitations of the manual annotations, the results from the tagging portion of the study are still comparable to other efforts of automatic discourse tagging using different methods and different corpora (Stolcke et al., 2000), which found performance within 15% of the performance of human taggers. We plan to conduct a more rigorous manual annotation study. We expect that improved human inter-coder reliability would eliminate the need for corrected tags and allow for sequential analysis of tags within turns. It is also anticipated that incorporating additional methods that account for syntax and discourse turns should further improve the overall performance, see also Serafin et al. (2003).</Paragraph>
    <Paragraph position="1"> Even with the limitations of the discourse tagging, our LSA-based approach demonstrates it can be applied as a method for doing automated measurement of team performance. Using automatic methods we were able to duplicate some of the results of Bowers, and colleagues, (1998) who analyzed the sequence of content categories occurring in communication in a flight simulator task.</Paragraph>
    <Paragraph position="2"> They found that high team effectiveness was associated with consistent responding to uncertainty, planning, and fact statements with acknowledgments and responses.</Paragraph>
    <Paragraph position="3"> The LSA-predicted team performance scores correlated strongly with the actual team performance measures. This demonstrates that analyses of discourse can automatically measure how well a team is performing on a mission. This has implications both for automatically determining what discourse characterizes good and poor teams as well as developing systems for monitoring team performance in near real-time. We are currently exploring two promising avenues to predict performance in real time: integration of speech recognition technology, and inter-turn tag sequences.</Paragraph>
    <Paragraph position="4"> Research into team discourse is a new but growing area. However, up to recently, the large amounts of transcript data have limited researchers from performing analyses of team discourse. The results of this study show that applying NLP techniques to team discourse can provide accurate predictions of performance. These automated tools can help inform theories of team performance and also aid in the development of more effective automated team training systems.</Paragraph>
  </Section>
class="xml-element"></Paper>