<?xml version="1.0" standalone="yes"?>
<Paper uid="P06-1094">
  <Title/>
  <Section position="5" start_page="746" end_page="747" type="metho">
    <SectionTitle>
3 Computational Model
</SectionTitle>
    <Paragraph position="0"> Below we describe a model of relative proximity that uses (1) the distance between objects, (2) the size and salience of the landmark object, and (3) the location of other objects in the scene. Our model is based on first computing absolute proximity between each point and each landmark in a scene, and then combining or overlaying the resulting absolute proximity fields to compute the relative proximity of each point to each landmark.</Paragraph>
    <Section position="1" start_page="746" end_page="746" type="sub_section">
      <SectionTitle>
3.1 Computing absolute proximity fields
</SectionTitle>
      <Paragraph position="0"> We first compute for each landmark an absolute proximity field giving each point's proximity to that landmark, independent of proximity to any other landmark. We compute fields on the projection of the scene onto the 2D-plane, a 2D-array ARRAY of points. At each point P in ARRAY , the absolute proximity for landmark L is</Paragraph>
      <Paragraph position="2"> In this equation the absolute proximity for a point P and a landmark L is a function of both the distance between the point and the location of the landmark, and the salience of the landmark.</Paragraph>
      <Paragraph position="3"> To represent distance we use a normalised distance function dist normalised (L, P, ARRAY ), which returns a value between 0 and 1.  The smaller the distance between L and P, the higher the absolute proximity value returned, i.e. the more acceptable it is to say that P is close to L. In this way, this component of the absolute proximity field captures the gradual gradation in applicability evident in Logan and Sadler (1996).  We normalise by computing the distance between the two points, and then dividing this distance it by the maximum distance between point L and any point in the scene. We model the influence of visual and discourse salience on absolute proximity as a function salience(L), returning a value between 0 and</Paragraph>
      <Paragraph position="5"> is computed using the algorithm of Kelleher and van Genabith (2004). Computing a relative salience for each object in a scene is based on its perceivable size and its centrality relative to the viewer's focus of attention. The algorithm returns scores in the range of 0 to 1. As the algorithm captures object size we can model the effect of landmark size on proximity through the salience component of absolute proximity. The discourse salience (S disc ) of an object is computed based on recency of mention (Hajicov'a, 1993) except we represent the maximum overall salience in the scene as 1, and use 0 to indicate that the landmark is not salient in the current context.</Paragraph>
      <Paragraph position="7"> tered in a 2D plane, points ranging from plane's upper-left corner (&lt;-3,-3&gt;) to lower right corner(&lt;3,3&gt;).</Paragraph>
      <Paragraph position="8"> Figure 4 shows computed absolute proximity with salience values of 1, 0.6, and 0.5, for points from the upper-left to the lower-right of a 2D plane, with the landmark at the center of that plane. The graph shows how salience influences absolute proximity in our model: for a landmark with high salience, points far from the landmark can still have high absolute proximity to it.</Paragraph>
    </Section>
    <Section position="2" start_page="746" end_page="747" type="sub_section">
      <SectionTitle>
3.2 Computing relative proximity fields
</SectionTitle>
      <Paragraph position="0"> Once we have constructed absolute proximity fields for the landmarks in a scene, our next step is to overlay these fields to produce a measure of  relative proximity to each landmark at each point. For this we first select a landmark, and then iterate over each point in the scene comparing the absolute proximity of the selected landmark at that point with the absolute proximity of all other landmarks at that point. The relative proximity of a selected landmark at a point is equal to the absolute proximity field for that landmark at that point, minus the highest absolute proximity field for any other landmark at that point (see Equation 3).</Paragraph>
      <Paragraph position="1">  The idea here is that the other landmark with the highest absolute proximity is acting in competition with the selected landmark. If that other landmark's absolute proximity is higher than the absolute proximity of the selected landmark, the selected landmark's relative proximity for the point will be negative. If the competing landmark's absolute proximity is slightly lower than the absolute proximity of the selected landmark, the selected landmark's relative proximity for the point will be positive, but low. Only when the competing landmark's absolute proximity is significantly lower than the absolute proximity of the selected landmark will the selected landmark have a high relative proximity for the point in question.</Paragraph>
      <Paragraph position="2"> In (3) the proximity of a given point to a selected landmark rises as that point's distance from the landmark decreases (the closer the point is to the landmark, the higher its proximity score for the landmark will be), but falls as that point's distance from some other landmark decreases (the closer the point is to some other landmark, the lower its proximity score for the selected landmark will be).</Paragraph>
      <Paragraph position="3"> Figure 5 shows the relative proximity fields of two landmarks, L1 and L2, computed using (3), in a 1-dimensional (linear) space. The two landmarks have different degrees of salience: a salience of 0.5 for L1 and of 0.6 for L2 (represented by the different sizes of the landmarks). In this figure, any point where the relative proximity for one particular landmark is above the zero line represents a point which is proximal to that landmark, rather than to the other landmark. The extent to which that point is above zero represents its degree of proximity to that landmark. The overall proximal area for a given landmark is the overall area for which its relative proximity field is above zero.</Paragraph>
      <Paragraph position="4"> The left and right borders of the figure represent the boundaries (walls) of the area.</Paragraph>
      <Paragraph position="5"> Figure 5 illustrates three main points. First, the overall size of a landmark's proximal area is a function of the landmark's position relative to the other landmark and to the boundaries. For example, landmark L2 has a large open space between it and the right boundary: Most of this space falls into the proximal area for that landmark. Landmark L1 falls into quite a narrow space between the left boundary and L2. L1 thus has a much smaller proximal area in the figure than L2. Second, the relative proximity field for some landmark is a function of that landmark's salience.</Paragraph>
      <Paragraph position="6"> This can be seen in Figure 5 by considering the space between the two landmarks. In that space the width of the proximal area for L2 is greater than that of L1, because L2 is more salient.</Paragraph>
      <Paragraph position="7"> The third point concerns areas of ambiguous proximity in Figure 5: areas in which neither of the landmarks have a significantly higher relative proximity than the other. There are two such areas in the Figure. The first is between the two landmarks, in the region where one relative proximity field line crosses the other. These points are ambiguous in terms of relative proximity because these points are equidistant from those two landmarks. The second ambiguous area is at the extreme right of the space shown in Figure 5. This area is ambiguous because this area is distant from both landmarks: points in this area would not be judged proximal to either landmark. The question of ambiguity in relative proximity judgments is considered in more detail in SS5.</Paragraph>
      <Paragraph position="9"/>
    </Section>
  </Section>
  <Section position="6" start_page="747" end_page="749" type="metho">
    <SectionTitle>
4 Experiment
</SectionTitle>
    <Paragraph position="0"> Below we describe an experiment which tests our approach (SS3) to relative proximity by examining  the changes in people's judgements of the appropriateness of the expression near being used to describe the relationship between a target and landmark object in an image where a second, distractor landmark is present. All objects in these images were coloured shapes, a circle, triangle or square.</Paragraph>
    <Section position="1" start_page="748" end_page="748" type="sub_section">
      <SectionTitle>
4.1 Material and Procedure
</SectionTitle>
      <Paragraph position="0"> All images used in this experiment contained a central landmark object and a target object, usually with a third distractor object. The landmark was always placed in the middle of a 7-by-7 grid.</Paragraph>
      <Paragraph position="1"> Images were divided into 8 groups of 6 images each. Each image in a group contained the target object placed in one of 6 different cells on the grid, numbered from 1 to 6. Figure 6 shows how we number these target positions according to their nearness to the landmark.</Paragraph>
      <Paragraph position="2">  used in the experiment.</Paragraph>
      <Paragraph position="3"> Groups are organised according to the presence and position of a distractor object. In group a the distractor is directly above the landmark, in group b the distractor is rotated 45 degrees clockwise from the vertical, in group c it is directly to the right of the landmark, in d it is rotated 135 degrees clockwise from the vertical, and so on. The distractor object is always the same distance from the central landmark. In addition to the distractor groups a,b,c,d,e,f and g, there is an eighth group, group x, in which no distractor object occurs.</Paragraph>
      <Paragraph position="4"> In the experiment, each image was displayed with a sentence of the form The is near the , with a description of the target and landmark respectively. The sentence was presented under the image. 12 participants took part in this experiment. Participants were asked to rate the acceptability of the sentence as a description of the image using a 10-point scale, with zero denoting not acceptable at all; four or five denoting moderately acceptable; and nine perfectly acceptable.</Paragraph>
    </Section>
    <Section position="2" start_page="748" end_page="749" type="sub_section">
      <SectionTitle>
4.2 Results and Discussion
</SectionTitle>
      <Paragraph position="0"> We assess participants' responses by comparing their average proximity judgments with those predicted by the absolute proximity equation (Equation 1), and by the relative proximity equation (Equation 3). For both equations we assume that all objects have a salience score of 1. With salience equal to 1, the absolute proximity equation relates proximity between target and landmark objects to the distance between those two objects, so that the closer the target is to the landmark the higher its proximity will be. With salience equal to 1, the relative proximity equation relates proximity to both distance between target and landmark and distance between target and distractor, so that the proximity of a given target object to a landmark rises as that target's distance from the landmark decreases but falls as the target's distance from some other distractor object decreases.</Paragraph>
      <Paragraph position="1"> Figure 7 shows graphs comparing participants' proximity ratings with the proximity scores computed by Equation 1 (the absolute proximity equation), and by Equation 3 (the relative proximity equation), for the images in group x and in the other 7 groups. In the first graph there is no difference between the proximity scores computed by the two equations, since, when there is no distractor object present the relative proximity equation reduces to the absolute proximity equation.</Paragraph>
      <Paragraph position="2"> The correlation between both computed proximity scores and participants' average proximity scores for this group is quite high (r = 0.95). For the remaining 7 groups the proximity value computed from Equation 1 gives a fair match to people's proximity judgements for target objects (the average correlation across these seven groups in Figure 7 is around r = 0.93). However, relative proximity score as computed in Equation 3 significantly improves the correlation in each graph, giving an average correlation across the seven groups of around r = 0.99 (all correlations in Figure 7 are significant p &lt; 0.01).</Paragraph>
      <Paragraph position="3"> Given that the correlations for both Equation 1 and Equation 3 are high we examined whether the results returned by Equation 3 were reliably closer to human judgements than those from Equation 1.</Paragraph>
      <Paragraph position="4"> For the 42 images where a distractor object was present we recorded which equation gave a result that was closer to participants' normalised aver- null age for that image. In 28 cases Equation 3 was closer, while in 14 Equation 1 was closer (a 2:1 advantage for Equation 3, significant in a sign test: n+ = 28, n[?] = 14, Z = 2.2, p &lt; 0.05). We conclude that proximity judgements for objects in our experiment are best represented by relative proximity as computed in Equation 3. These results support our 'relative' model of proximity.</Paragraph>
      <Paragraph position="5">  It is interesting to note that Equation 3 overestimates proximity in the cases (a, b and g)  Note that, in order to display the relationship between proximity values given by participants, computed in Equa- null of 0 and a standard deviation of 1. This normalisation simply means that all values fall in the same region of the scale, and can be easily compared visually.</Paragraph>
      <Paragraph position="6"> where the distractor object is closest to the targets and slightly underestimates proximity in all other cases. We will investigate this in future work.</Paragraph>
    </Section>
  </Section>
  <Section position="7" start_page="749" end_page="751" type="metho">
    <SectionTitle>
5 Expressing spatial proximity
</SectionTitle>
    <Paragraph position="0"> We use the model of SS3 to interpret spatial references to objects. A fundamental requirement for processing situated dialogue is that linguistic meaning provides enough information to establish the visual grounding of spatial expressions: How can the robot relate the meaning of a spatial expression to a scene it visually perceives, so it can locate the objects which the expression applies to? Approaches agree here on the need for ontologically rich representations, but differ in how these are to be visually grounded. Oates et al. (2000)  and Roy (2002) use machine learning to obtain a statistical mapping between visual and linguistic features. Gorniak and Roy (2004) use manually constructed mappings between linguistic constructions, and probabilistic functions which evaluate whether an object can act as referent, whereas DeVault and Stone (2004) use symbolic constraint resolution. Our approach to visual grounding of language is similar to the latter two approaches.</Paragraph>
    <Paragraph position="1"> We use a Combinatory Categorial Grammar (CCG) (Baldridge and Kruijff, 2003) to describe the relation between the syntactic structure of an utterance and its meaning. We model meaning as an ontologically richly sorted, relational structure, using a description logic-like framework (Baldridge and Kruijff, 2002). We use OpenCCG for parsing and realization.</Paragraph>
    <Paragraph position="2">  Example (2) shows the meaning representation for &amp;quot;the box near the ball&amp;quot;. It consists of several, related elementary predicates (EPs). One type of EP represents a discourse referent as a proposition with a handle: @ {b:phys[?]obj} (box) means that the referent b is a physical object, namely a box. Another type of EP states dependencies between referents as modal relations, e.g. @ {b:phys[?]obj} &lt;Location&gt; (r : region &amp; near) means that discourse referent b (the box) is located in a region r that is near to a landmark. We represent regions explicitly to enable later reference to the region using deictic reference (e.g. &amp;quot;there&amp;quot;). Within each EP we can have semantic features, e.g. the region r characterizes a static location of b and expresses proximity to a landmark. Example (2) gives a ball in the context as the landmark. We use the sorting information in the utterance's meaning (e.g. phys-obj, region) for further  http://www.sf.net/openccg/ interpretation using ontology-based spatial reasoning. This yields several inferences that need to hold for the scene, like DeVault and Stone (2004). Where we differ is in how we check whether these inferences hold. Like Gorniak and Roy (2004), we map these conditions onto the energy landscape computed by the proximity field functions. This enables us to take into account inhibition effects arising in the actual situated context, unlike Gorniak &amp; Roy or DeVault &amp; Stone.</Paragraph>
    <Paragraph position="3"> We convert relative proximity fields into proximal regions anchored to landmarks to contextually interpret linguistic meaning. We must decide whether a landmark's relative proximity score at a given point indicates that it is &amp;quot;near&amp;quot; or &amp;quot;close to&amp;quot; or &amp;quot;at&amp;quot; or &amp;quot;beside&amp;quot; the landmark. For this we iterate over each point in the scene, and compare the relative proximity scores of the different landmarks at each point. If the primary landmark's (i.e., the landmark with the highest relative proximity at the point) relative proximity exceeds the next highest relative proximity score by more than a predefined confidence interval the point is in the vague region anchored around the primary landmark. Otherwise, we take it as ambiguous and not in the proximal region that is being interpreted.</Paragraph>
    <Paragraph position="4"> The motivation for the confidence interval is to capture situations where the difference in relative proximity scores between the primary landmark and one or more landmarks at a given point is relatively small. Figure 8 illustrates the parsing of a scene into the regions &amp;quot;near&amp;quot; two landmarks. The relative proximity fields of the two landmarks are identical to those in Figure 5, using a confidence interval of 0.1. Ambiguous points are where the proximity ambiguity series is plotted at 0.5. The regions &amp;quot;near&amp;quot; each landmark are those areas of the graph where each landmark's relative proximity series is the highest plot on the graph. Figure 8 illustrates an important aspect of our model: the comparison of relative proximity fields naturally defines the extent of vague proximal regions. For example, see the region right of L2 in Figure 8. The extent of L2's proximal region in this direction is bounded by the interference effect of L1's relative proximity field. Because the landmarks' relative proximity scores converge, the area on the far right of the image is ambiguous with respect to which landmark it is proximal to.</Paragraph>
    <Paragraph position="5"> In effect, the model captures the fact that the area is relatively distant from both landmarks. Follow- null proximity fields for landmarks L1 and L2, with confidence interval=0.1 and different salience scores for L1 (0.5) and L2 (0.6). Locations of landmarks are marked on the X-axis. ing the cognitive load model (SS1), objects located in this region should be described with a projective relation such as &amp;quot;to the right of L2&amp;quot; rather than a proximal relation like &amp;quot;near L2&amp;quot;, see Kelleher and Kruijff (2006).</Paragraph>
  </Section>
class="xml-element"></Paper>