<?xml version="1.0" standalone="yes"?>
<Paper uid="W06-0205">
  <Title>Automatic Knowledge Representation using a Graph-based Algorithm for Language-Independent Lexical Chaining</Title>
  <Section position="7" start_page="43" end_page="45" type="evalu">
    <SectionTitle>
5 Evaluation
</SectionTitle>
    <Paragraph position="0"> The evaluation of Lexical Chains is generally difficult. Even if they can be effectively used in many practical applications, Lexical Chains are seldom desirable outputs in a real-world application, and it is unclear how to assess their quality independently of the underlying application in which they are used (Budanitsky and Hirst, 2006). For example, in Summarization, it is hard to determine whether a good or bad performance comes from the efficiency of the lexical chaining algorithm or from the appropriateness of using Lexical Chains in that kind of application. It is also true that some work has been done in this direction (Budanitsky and Hirst, 2006) by collecting Human Lexical Chains to compare against automatically built Lexical Chains. However, this type of evaluation is logistically impossible to perform as we aim at developing a system that does not depend on any language or topic. So, in this section, we will only present some results generated by our architecture (like (Barzilay and Elhadad, 1997; Teich and Fankhauser, 2004) do), althoughweacknowledgethatothercomparativeeval- null uations (with WordNet, with Human Lexical Chains or within independent applications like Text Summarization) must be done in order to draw definitive conclusions.</Paragraph>
    <Paragraph position="1">  Wehavegeneratedfourtaxonomiesfromfourdifferent domains (Sport, Economy, Politics and War) from a set of documents of the DUC 200415. Moreover, we have extracted Lexical Chains for all four  domains to show the ability of our system to switch from domain to domain without any problem.</Paragraph>
    <Section position="1" start_page="44" end_page="44" type="sub_section">
      <SectionTitle>
5.1 Quantitative Function
</SectionTitle>
      <Paragraph position="0"> Four texts from each domain of the DUC 2004 corpus have been used to extract Lexical Chains based on the four knowledge bases built from all texts of DUC 2004 for each one of the four following domains: Sport, Economy, Politics and War. However, in this section, we will only present the results from the Sport Domain as results show similar behaviors for the other domains. In particular, we present in Table 1 the characteristics of each document.</Paragraph>
      <Paragraph position="1">  The first interesting conclusion shown in Table 2 is that the number of Lexical Chains does not depend on the document size but rather on the nominal units distribution. Indeed, for example, the number of words in Document 1 is twice as big as in Document 2. Although, we have more Lexical Chains in Document 2 than in Document 1, as Document 2 has more distinct nominal units.</Paragraph>
      <Paragraph position="2">   Thesecondinterestingconclusionisthatouralgorithm does not gather words that belong to only one cluster and take advantage of the automatically built lexico-semantic knowledge base. This is illustrated in Table 3. However, it is obvious that by increasing the constant c the words in a chain tend to belong to only one cluster as it is the case for most of the best</Paragraph>
    </Section>
    <Section position="2" start_page="44" end_page="45" type="sub_section">
      <SectionTitle>
Lexical Chains with c = 8.
5.2 Qualitative Evaluation
</SectionTitle>
      <Paragraph position="0"> Inthissection, asitisdonein(BarzilayandElhadad, 1997; Teich and Fankhauser, 2004), we present the  fivehighest-scoringchainsforthebestthresholdthat we experimentally evaluated to be c = 7 for each domain (See Tables 4, 5, 6, 7). It is clear that the obtained Lexical Chains show a desirable degree of representativeness of the text in analysis.</Paragraph>
      <Paragraph position="1"> Domain=Sport, Document=3, c=7 - #0, 1 cluster and score=1.0: {United States, couple, competition} - #6, 3 clusters and score=1.0: {boats, Sunday night, sailor, Sword, Orion, veteran, cutter, WinstonChurchill, SoloGlobe, Challenger, navy, Race, supposition, instructions, responsibility, skipper, east, Melbourne, deck, kilometer, masts, bodies, races, GMT, Admiral's, Cups, Britain, Star, Class, Atlanta, Seattle, arms, fatality, sea, waves, dark, yacht's, Dad, Guy's, son, Mark, beer, talk, life, Richard, Winning, affair, canopy, death} - #9, 1 cluster and score=1.0: {record, days, hours, minutes, rescue} - #16, 3 clusters and score=1.0: {Snow, shape, north, easters, thunder, storm, change, knots, west, level, maxi's, search, Authority, seas, helicopter, night vision, equipment, feet, rescues, Campbell, suffering, hypothermia, safety, foot, sailors, colleagues, Hospital, deaths, bodies, fatality} - #19, 2 clusters and score=1.0: {challenge, crew, Monday, VC, Offshore, Stand, Newcastle, mid morning, Eden, Rescuers, aircraft, unsure, whereabouts, killing, contact} Table 4: 5 best Lexical Chains for Sport Domain=Economy, Document=5, c=7 - #88, 4 clusters and score=1.0: {sign, chance, Rio, Janeiro, Grande, Sul, uphill, promise, hospitals, powerhouse, success, inhabitants, victory, pad, presidency, contingent, exit, legislature} - #50, 1 cluster and score=1.0: {transactions, taxes, Stabilization, spate, fuel, income, fortunes, means} - #77, 1 cluster and score=1.0: {proposal, factory, owners, Fund, Rubin's} - #126, 1 cluster and score=1.0: {disaster, control, investment, review} -#12, 2clustersandscore=0.99: {issue, order,University, population,question, timing, currencies} Table 5: 5 best Lexical Chains for Economy For instance, the Lexical Chain #16 in the domain of Sport clearly exemplifies the tragedy of climbers that were killed in a sudden change of weather in the mountains and who could not be rescued by the authorities.</Paragraph>
      <Paragraph position="2"> However, some Lexical Chains are less expressive. For instance, it is not clear what the Lexical Chain #40 expresses in the domain of Politics. Indeed, none of the words present in the chain seem  Domain=Politics, Document=3, c=7 - #5, 1 cluster and score=1.0: {report, leaders, lives, information} - #33, 1 cluster and score=1.0: {past, attention, defenders, investigations} - #28, 2 clusters and score=0.95: {investigators, hospital, ward, wounds, neck, description, fashion, suspects, raids, assault, rifles, door, further details, surgery, service, detective, Igor, Kozhevnikov, Ministry} - #40, 2 clusters and score=0.92: {security, times, weeks, fire} - #24, 3 clusters and score=0.85: {enemies, Choice, stairwell, assailants, woman, attackers, entrance, car, guns, Friends, relatives, Mrs. Staravoitova, founder, movement, well thought, Sergei, Kozyrev, Association, Societies, supporter, Stalin's, council, criminals, Yegor, Gaidar, minister, ally, suggestions, measures, smile, commitment} Table 6: 5 best Lexical Chains for Politics Domain=War, Document=1, c=7 - #25, 2 clusters and score=1.0: {lightning, advance, Africa's, nation, outskirts, capital Kinshasa, troops, Angola, Zimbabwe, Namibia, chunk, routes, Katanga, Eastern, Kasai, provinces, copper} - #53, 1 cluster and score=1.0: {Back, years, Ngeyo, farm, farmers, organization, breadbasket, quarter, century, businessman, hotels, tourist, memory, rivalry, rebellions} - #56, 1 cluster and score=1.0: {political, freedoms, Hutus, Mai-Mai, warriors, Hunde, Nande, militiamen, Rwanda, ideology, weapons, persecution, landowners, ranchers, anarchy, Safari, Ngezayo, farmer, hotel, owner, camps} - #24, 2 clusters and score=0.87: {fighting, people, leaders, diplomats, cause, president, Washington, U.S, units, weeks} - #51, 2 clusters and score=0.82: {West, buildings, sight, point, tourists, mountain, gorillas, shops, guest, disputes} Table 7: 5 best Lexical Chains for War to express any idea about Politics. Moreover, due to the small number of inter-related nominal units within the Lexical Chain, this one can not be understood as it is without context. In fact, it was related to problems of car firing that have been occurring in the past few weeks and provoked security problems in the town.</Paragraph>
      <Paragraph position="3"> Although some Lexical Chains are understandable as they are, most of them must be replaced in their context to fully understand their representativeness of the topics or subtopics of the text being analyzed. As a consequence, we deeply believe that Lexical Chains must be evaluated in the context of Natural Language Processing applications (such as Text Summarization (Doran et al., 2004)), as comparing Lexical Chains as they are is a very difficult task to tackle which may even lead to inconclusive results.</Paragraph>
    </Section>
  </Section>
class="xml-element"></Paper>