<?xml version="1.0" standalone="yes"?>
<Paper uid="W04-0902">
  <Title>Solving Logic Puzzles: From Robust Processing to Precise Semantics</Title>
  <Section position="11" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
10 Progress
</SectionTitle>
    <Paragraph position="0"> Using the sculptures puzzle (a set of four questions partly shown in Figure 1) as an initial test case, we have built a prototype end-to-end system. In its present state, the system analyzes and solves correctly all questions in this puzzle, except that there is still no understanding of the phrase &amp;quot;complete list&amp;quot; in question 2. The back-end reasoning module is finished and works for any puzzle formalized in FOL+modals. The probabilistic scope resolution module, trained on 259 two-quantifier sentences extracted from 122 puzzles and tested on 46 unseen sentences, attains an accuracy of about 94% over an 82% linear-order baseline. A preliminary evaluation on another unseen puzzle shows that on 60% of the sentences, the parser's output is accurate enough to support the subsequent computation of the semantics, and we expect this to be better after it is trained on puzzle texts. However, the  are designed to work together, and use the same input syntax.</Paragraph>
    <Paragraph position="1"> system as a whole worked end-to-end on only one of the unseen sentences in that puzzle; key losses come from unhandled semantic phenomena (e.g. &amp;quot;only&amp;quot;, &amp;quot;except&amp;quot;, ellipses), unhandled lexical semantics of words that must be understood (e.g. &amp;quot;complete list&amp;quot;), and unhandled implicit constraint types that need to be filled.</Paragraph>
  </Section>
class="xml-element"></Paper>