<?xml version="1.0" standalone="yes"?>
<Paper uid="P01-1061">
  <Title>Computational properties of environment-based disambiguation</Title>
  <Section position="5" start_page="0" end_page="0" type="evalu">
    <SectionTitle>
4 Evaluation
</SectionTitle>
    <Paragraph position="0"> An implemented system incorporating this environment-based approach to disambiguation has been tested on a set of manufacturersupplied aircraft maintenance instructions, using a computer-aided design (CAD) model of a portion of the aircraft as the environment. It contains several hundred three dimensional objects (buttons, handles, sliding couplings, etc), labeled with object type keywords and connected to other objects through joints with varying degrees of freedom (indicating how each object can be rotated and translated with respect to other objects in the environment).</Paragraph>
    <Paragraph position="1"> The test sentences were the manufacturer's in6This approach is in some sense wedded to a CCG-style syntacto-semantic analysis of quantifier raising, inasmuch as its syntactic and semantic structures must be isomorphic in order to preserve the polynomial complexity of the shared forest.</Paragraph>
    <Paragraph position="2"> structions for replacing a piece of equipment in this environment. The baseline grammar was not altered to fit the test sentences or the environment, but the labeled objects in the CAD model were automatically added to the lexicon as common nouns.</Paragraph>
    <Paragraph position="3"> In this preliminary accuracy test, forest nodes that correspond to noun phrase or modifier categories are dispreferred if they have no potential entity referents, and forest nodes corresponding to other categories are dispreferred if their arguments have no potential entity referents. Many of the nodes in the forest correspond to noun-noun modifications, which cannot be ruled out by the grammar because the composition operation that generates them seems to be productive (virtually any 'N2' that is attached to or contained in an 'N1' can be an 'N1 N2'). Potential referents for noun-noun modifications are calculated by a rudimentary spatial proximity threshold, such that any potential referent of the modified noun lying within the threshold distance of a potential referent of the modifier noun in the environment is added to the composed set.</Paragraph>
    <Paragraph position="4"> The results are shown below. The average number of parse trees per sentence in this set was  a0 before disambiguation. The average ratio of nodes in enumerated tree sets to nodes in shared forests for the instructions in this test set was  a19 a2a4a3 a0 , a nearly tenfold reduction due to sharing.</Paragraph>
    <Paragraph position="5"> Gold standard 'correct' trees were annotated by hand using the same grammar that the parser uses. The success rate of the parser in this domain (the rate at which the correct tree could be found in the parse forest) was a5a7a6a9a8 . The retention rate of the environment-based filtering mechanism described above (the rate at which the correct tree was retained in parse forest) was a1 a8a7a8 of successfully parsed sentences. The average reduction in number of possible parse trees due to the environment-based filtering mechanism described above was a10 a19 a2a11a3 a0 for successfully parsed and filtered forests.7 7Sample parse forests and other details of this application and environment are available at http://www.cis.upenn.edu/a12 schuler/ebd.html.</Paragraph>
    <Paragraph position="6"> # trees nodes in nodes in # trees sent (before unshared shared (after no. filter) tree set forest filter)</Paragraph>
  </Section>
class="xml-element"></Paper>