<?xml version="1.0" standalone="yes"?>
<Paper uid="P05-2006">
  <Title>Automatic Discovery of Intentions in Text and its Application to Question Answering</Title>
  <Section position="4" start_page="32" end_page="33" type="metho">
    <SectionTitle>
3 Learning Model
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="32" end_page="33" type="sub_section">
      <SectionTitle>
3.1 Experimental data
</SectionTitle>
      <Paragraph position="0"> We applied the most frequent syntactic pattern that expresses intentions in text (a0a5a1 a3 to a0a5a1 a6 ) on the first 10,000 sentences of the SemCor2.0 collection and we extracted 1,873 sentences. These sentences contain 115 intentions (manually identified by a graduate student, not the author). The data consisting of these positives and 258 arbitrarily selected negative examples, was randomly divided into a training set that contains 80% of the examples and a test set with the remaining 20% instances. The statistics are shown in Table 2.</Paragraph>
      <Paragraph position="1">  After analyzing our training data, we pinpointed a set of features to help us identify the intentions encoded by the pattern a0 a1 a3 to a0a5a1 a6 . The WordNet senses needed to extract the semantic features were taken from SemCor. We will use Mary intends to revise the paper to show each feature's value.</Paragraph>
      <Paragraph position="2"> The semantic class of the the a0a5a1 a3 verb's agent or specializations of it. Intentions and objectives are specific to humans. Thus, the semantic class of the a0a5a1 a3 agent bears a high importance. We used an in-house semantic parser to retrieve the AGENT of the a0 a1 a3 verb. The feature's value is its WordNet semantic class. Mary names a person. Thus, the semantic class that we are seeking is entity#1.</Paragraph>
      <Paragraph position="3"> We chose this semantic generalization because nouns and verbs belong to open part-of-speech classes. There can be an enormous number of possibilities and any models built using them as feature values will not be able to generalize beyond the training examples. Therefore, we introduce a bias in our learning framework based on the assumption: noun and verb concepts will semantically behave as the concepts that subsume them in the WordNet structures. But, by generalizing concepts, we lose some of their semantic properties. Hence, we specialize the semantic class a24 of a concept a25 by replacing it with its immediate hyponym (a25 ) that subsumes a25 . We can further increase the semantic level by specializing a25 . We note that the number of values is still finite even though we specialized the general concepts. As the specialization level increases, there will be words a25 that cannot be further specialized (entity#1 cannot be specialized even once). In such cases, we add a25 to the set of feature values.</Paragraph>
      <Paragraph position="4"> The semantic class of the a0a5a1 a3 verb or its specializations. The intention phrase is subordinated to a verb (a0a5a1 a3 ). The semantic class of this verb is the system's second feature. In our example, a0 a1 a3 (intend#1) semantic class is wish#3.</Paragraph>
      <Paragraph position="5"> The semantic class of the a0a2a1a2a6 verb's agent, if this agent differs from the a0a5a1 a3 verb's agent; otherwise, a common value (equal) is given. We identify the AGENT of the a0a5a1 a6 verb. The specializations of its semantic class will be used if the top noun proves to be too general. In the sample sentence, the agent of revise is Mary. We can have a different agent for  the a0a2a1a7a6 verb (Mary intends John to revise the paper). Let's assume that Mary is John's supervisor and she can make him revise the document. The sentence expresses Mary's intention of persuading John to revise the paper, but this objective is not encoded by the pattern we considered.</Paragraph>
      <Paragraph position="6"> The semantic class of the a0a5a1 a6 verb or its specializations. The a0a5a1 a6 verb expresses the future action or behavior that the agent intends. We extract this feature using WordNet hierarchies. Revise#1 belongs to the act#1 semantic class.</Paragraph>
      <Paragraph position="7"> A flag indicating if the a0 a1 a3 verb has an affirmative or a negative form. We want to differentiate between sentences like John wants to go for a walk and John doesn't want to go for a walk. The first sentence expresses John's intention, while, in the second one, no intention can be identified.</Paragraph>
      <Paragraph position="8"> The type of the analyzed sentence. This feature is primarily concerned with questions. A question like Where do you plan to go for a walk? indicates the intention of going for a walk, unlike the question Do you plan to go for a walk? which might express an intention if the answer is &amp;quot;yes&amp;quot;. This feature's values are the wh-words that begin a question or n/a for the other types of English sentences.</Paragraph>
      <Paragraph position="9"> We did not analyze the affirmative versus the negative form of the a0a2a1 a6 verb because it does not affect the objective attribute of the intention. The sentence John intends not to go for a walk expresses a negative intention. This sentence is much stronger than John doesn't intend to go for a walk. In the former context, John has set a goal for himself , while in the second sentence, the objective does not exist.</Paragraph>
    </Section>
  </Section>
  <Section position="5" start_page="33" end_page="34" type="metho">
    <SectionTitle>
4 Experimental Results
</SectionTitle>
    <Paragraph position="0"/>
    <Section position="1" start_page="33" end_page="33" type="sub_section">
      <SectionTitle>
4.1 Impact of specialization
</SectionTitle>
      <Paragraph position="0"> The first experiment was performed using the LIB-SVM package6 and the WordNet semantic classes.</Paragraph>
    </Section>
    <Section position="2" start_page="33" end_page="33" type="sub_section">
      <SectionTitle>
6http://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html
</SectionTitle>
      <Paragraph position="0"> These features yield an accuracy of 87.67%. Trying to improve the performance, we specialized the semantic classes. When the a0 a1 a6 's agent semantic class was specialized, the accuracy remained constant. If we replace the a0a2a1 a6 's semantic class with its direct hyponyms, the accuracy drops 5.48%. But, the specialization of the a0a5a1 a3 agent's semantic class brings an improvement of 1.37% and the specialization of the a0a2a1 a3 's class produces an increase in accuracy of 2.74%. Given this fluctuation in performance, we performed 81 different experiments which create SVM models using the same training data annotated with more general or more specific feature values. For each feature, we analyzed the first two semantic specialization levels.</Paragraph>
      <Paragraph position="1"> From our experiments, we noticed that the specialization of the a0 a1a33a6 's agent semantic class does not influence the performance. Out of the 27 experiment triplets in which this specialization level changes, in only 4, it influences the result and, in 3 of them, the accuracy increases with the specialization level. Thus, our third feature is the second specialization level of the a0a5a1 a6 's agent class. Table 3 shows the results obtained when the values of the radial kernel parameters were chosen to optimize the 5-fold-cross-validation on the training data. The best models are described in Table 4.</Paragraph>
      <Paragraph position="2"> Model Level of specialization for the features A semantic class of the a8a53a9 a11 agent, a0a13a1 a3 level of specialization for the a8a10a9 a11 's semantic class, and semantic class of the a8a10a9 a13 verb</Paragraph>
      <Paragraph position="4"> semantic level for the a8a53a9a53a11 agent class, a0a7a1 a3 level of the a8a53a9 a11 's semantic class, and the semantic class of the a8a10a9 a13 verb</Paragraph>
      <Paragraph position="6"> level of the a8a12a9 a11 agent's semantic class and a0a7a1 a3 specialization levels for the a8a12a9a53a11 and a8a10a9a14a13 semantic classes</Paragraph>
    </Section>
    <Section position="3" start_page="33" end_page="34" type="sub_section">
      <SectionTitle>
4.2 Learning curves
</SectionTitle>
      <Paragraph position="0"> We further analyzed our data and models and tried to see how many training examples are needed to reach 90.41% accuracy. We varied the training data  size and validated the new models using our previous test set. Figure 1 shows the performance variation of three models that use feature sets identical in terms of specialization levels to the ones of the A, B, and C classifiers. All three models exhibit a similar behavior with respect to the change in the training set size. Therefore, our features create a stable algorithm. The highest accuracy models use all 300 training examples. Thus, we did not reach the saturation point, but, considering the performance curve, this point is not very far.</Paragraph>
    </Section>
    <Section position="4" start_page="34" end_page="34" type="sub_section">
      <SectionTitle>
4.3 Feature impact on the SVM models
</SectionTitle>
      <Paragraph position="0"> All our previous experiments used the entire set of features. Now, we investigate the relative contribution of each feature. We performed experiments that use only five out of the six features. In Table 5, we list the accuracy increase that is gained by the inclusion of each feature. The most influential attribute is the a0a5a1 a3 verb's semantic class or its specializations.</Paragraph>
      <Paragraph position="1"> The intention's description verb does not influence the classification result. Because intentions consist of a future action and verbs express actions, there are very few verbs, such as dream or snore (involuntary actions) that cannot occupy the a0a2a1 a6 verb's position. The syntactic features bring an average increase in accuracy of 3.50%.</Paragraph>
    </Section>
    <Section position="5" start_page="34" end_page="34" type="sub_section">
      <SectionTitle>
4.4 Impact of word sense disambiguation
</SectionTitle>
      <Paragraph position="0"> Perfect word sense disambiguation might be a too strong assumption. In this section, we examine the effects of weaker disambiguation. Table 6 shows the accuracies of the best three models when each concept is tagged with its first WordNet sense (No WSD) and when the senses are given by an in-house WSD system with an accuracy of 69% computed on the</Paragraph>
    </Section>
    <Section position="6" start_page="34" end_page="34" type="sub_section">
      <SectionTitle>
4.5 C5 results
</SectionTitle>
      <Paragraph position="0"> After examining the SVM results, we applied the C5 machine learning algorithm (Quinlan, 2004) to the same training data annotated with the same feature set, in a similar manner. Again, we specialized the four semantic classes, independently, and tested the decision trees against the testing data. Table 7 shows their accuracy. The highest values were obtained for the first level of specialization of the a0a2a1 a3 verb semantic class. The specialization levels of the other semantic classes do not influence the accuracy of the decision trees. The most tested attribute is the a0a5a1 a3 verb. This further substantiates our observation, made during our SVM models analysis, that this feature has the greatest importance in the intention classification process. Our error analysis of the C5 results indicates that, because of the relatively small numbers of training instances, C5 ignores some of the features and makes wrong decisions.</Paragraph>
    </Section>
  </Section>
  <Section position="6" start_page="34" end_page="35" type="metho">
    <SectionTitle>
5 Application to Question Answering
</SectionTitle>
    <Paragraph position="0"> Questions involving intentions cannot be answered only by keyword-based or simple surface-level matching techniques. Table 8 lists two questions for  which finding the correct answer primarily depends on the discovery of the INTENTION relation.</Paragraph>
    <Paragraph position="1"> The answer type for the question a49 a3 is the INTENTION argument itself. The question processing module will detect that the answer being sought is Putin's intention. The semantic relations module processes a50 a3 's text and discovers the INTENTION relation. The question is searching for the intent of Putin with regards to North Korea and the answer text reveals Putin's intention to restore Russia's influence in the area. Question a49 a6 is searching for a location as its answer type and the correct answer is one which involves al Qaeda intending to purchase weapons of mass destruction. The candidate answer text (a50 a6 ) reveals the organization's past intent to buy (synonym with purchase) weapons in Russia. Because the two intentions have the same agent, future action and theme, the two semantically enhanced logic forms can now be unified and we can pin down the location of the intent (Russia).</Paragraph>
  </Section>
class="xml-element"></Paper>