Answering it with charts
— Dialogue in natural language and charts —
Tsuneaki KATO
Graduate School of Arts and Sciences, The University of Tokyo
3-8-1 Komaba, Meguroku, Tokyo 153-8902 Japan
kato@boz.c.u-tokyo.ac.jp
Mitsunori MATSUSHITA and Eisaku MAEDA
NTT Communication Science Laboratories, NTT Corp.
2-4 Hikaridai, Seika-cho, Soraku-gun, Kyoto 619-0237 Japan
a0 mat, maeda
a1 @cslab.kecl.ntt.co.jp
Abstract
A methodology is proposed for taking queries and
requests expressed in natural language as input and
answering them in charts through organizing that in-
teraction into felicitous dialogue. Charts and graph-
ics, as well as languages, are important modes of
communication. This is especially true of those
which are used frequently when people analyze
huge amount of data interactively, in order to find
out its characteristics or to resolve questions about
it. This paper raises the problem that in such situ-
ations the correctness of the charts depends on the
context, and proposes a framework to resolve it. The
core of the framework is a logical form that includes
the specifications of the user’s perspective and the
proper treatment of the logical form for handling
utterance fragments. The framework has been im-
plemented and confirmed to be appropriate.
1 Introduction
Charts and graphics, as well as languages, are im-
portant modes of communication. Considering this
importance, the automatic design of charts and
graphics suitable for achieving a given communica-
tive purpose has been studied actively (Maybury and
Wahlster, 1998). It has been demonstrated that the
characteristics of data drawn on the chart (Mackin-
lay, 1986), the intention to be achieved through the
chart (Roth and Mattis, 1990), and the task accom-
plished using the chart (Casner, 1991) play impor-
tant roles in designing appropriate charts. The au-
tomatic design of multimedia documents in which
those charts coordinate with natural language text
has also been studied (Roth et al., 1991; Kerpedjiev
et al., 1998).
In that research, systems take an assertion to be
conveyed or a communicative goal to be achieved,
and design the most appropriate multimedia pre-
sentation for that purpose. The purpose of draw-
ing charts and graphics, however, is not restricted
just to use in such presentations. In particular, as
the drawing of quantitative charts helps to analyze
huge amount of data and to find out its characteris-
tics, they can be a useful means for interactive ex-
ploratory data analysis. An analyst, led by an inter-
est or question, draws a chart, then a new interest
or question comes up and she draws another chart.
Through this process, the analyst finds out a char-
acteristic behind the data or understands the reason
for it.
This paper discusses automatic quantitative chart
design that supports such interactive exploratory
data analysis. That is, a methodology is proposed
for taking queries and requests expressed in natu-
ral language as input and answering them in charts
through organizing that interaction into felicitous
dialogue. The objectives of our research are an auto-
matic chart design that considers dialogue context,
and a dialogue mechanism that uses charts as its out-
put mode.
2 Context sensibility of chart
appropriateness
Let us suppose an analyst, planning sales of her
company’s products, gets interested in its sales in
a particular district. So she requests the following
from a system in front of her:
(1) Show me the sales in the Shikoku district for ’93
and ’94.
The system answers this request by drawing an
appropriate chart. She continues and utters a frag-
ment of a request:
(2) By prefecture.
a2
a3 a2
a4 a2
a5 a2
a6a7 a8 a6a7 a9
a10 a11 a12a13 a14 a13 a15
a16 a17 a18 a16 a19
a16 a17 a18 a20 a19 a16 a17 a18 a20 a21 a19
a2
a3 a2
a4 a2
a5 a2
a6a7 a8 a6a7 a9
a22 a14 a13 a15 a23a11 a12a24 a25
a26 a14 a15 a27 a11 a12
a26 a25 a28 a25 a29 a25
a30 a11 a12a24 a31
a2
a4a32
a33a34
a3 a2
a6a7 a8 a6a7 a9
a30 a11 a12a24 a31
a26 a25 a28 a25 a29 a25
a26 a14 a15 a27 a11 a12
a22 a14 a13 a15 a23a11 a12a24 a25
a16 a17 a18 a35 a19
a2
a3 a2
a4 a2
a5 a2
a7 a8 a7 a9 a7 a36 a7 a37 a7 a38 a7 a39
a22 a14 a13 a15 a23a11 a12a24 a25
a26 a14 a15 a27 a11 a12
a26 a25 a28 a25 a29 a25
a30 a11 a12a24 a31
a30 a11 a12a24 a31
a26 a25 a28 a25 a29 a25
a26 a14 a15 a27 a11 a12
a22 a14 a13 a15 a23a11 a12a24 a25
a2
a4
a32
a33
a34
a3 a2
a3a4
a7 a8 a7 a9 a7 a36 a7 a37 a7 a38 a7 a39
a16 a17 a18 a35 a21 a19
Figure 1: Examples of Context Sensibility of Chart
Correctness (1)
The system understands this fragment and draws
a new chart. This dialogue might be followed by an
utterance like:
(3) Through ’98.
The correct chart made in the response to utter-
ance (1) is like that shown in Figure 1-(1). Follow-
ing this chart, the response to utterance (2) should
be made by the stacked column chart, as shown in
Figure 1-(2), in which each column is subdivided in
order to show the sales by prefecture. Here, these
two consecutive utterances have a combined mean-
ing similar to:
(2’) Show me the sales in each prefecture of the
Shikoku district for ’93 and ’94.
It is interesting that when utterance (2’) is given
without a specific context, the response should be
made by the grouped column chart, as shown in Fig-
ure 1-(2’), rather than as in Figure 1-(2). The pref-
erence for Figure 1-(2) as the response to utterance
(2) may come from the perspective represented in
utterance (1) that she wants to look at the sales of
the district in total or the similarity in shape between
Figure 1-(2) and 1-(1). In any case, it is important
that an appropriate chart form depends on what ut-
terance or series of utterances was used to express a
request and what chart has been drawn previously.
This context sensibility of chart appropriateness oc-
a40a41 a42 a40a41 a43
a44 a45 a46a47 a48 a47 a49
a50 a45 a49 a51 a48 a47 a49
a52a53 a54
a52a53 a55
a56
a57 a56
a58 a56
a54 a56
a59 a60 a61a62 a63 a62 a64 a65 a60 a64 a66 a63 a62 a64
a52a53 a54
a67 a68 a69 a67 a70 a67 a68 a69 a67 a71 a70
a67 a68 a69 a72 a70
a56
a57 a56
a58 a56
a54 a56
a59 a60 a61a62 a63 a62 a64 a65 a60 a64 a66 a63 a62 a64
a56
a57 a56
a58 a56
a54 a56
Figure 2: Examples of Context Sensibility of Chart
Correctness (2)
curs constantly. For utterance (3) in the previous di-
alogue, the chart shown in Figure 1-(3) is preferred
when Figure 1-(2) is used to answer the previous re-
quest. The chart of Figure 1-(3’) is preferred, how-
ever, as the response to utterance (3’), which com-
bines requests (1), (2) and (3) into one;
(3’) Show me the sales in each prefecture of
Shikoku district from ’93 through ’98.
The chart form is not the only dimension sensitive
to dialogue context. Consider the following exam-
ple.
(4) Show me the sales in Shikoku and Chugoku for
’93.
(5) Add the one for ’94.
(5’) Show me the sales in Shikoku and Chugoku for
’93 and ’94.
The preferred response to utterance (5) must be
in the chart shown in Figure 2-(2) when preceding
utterance (4) was answered by the chart shown in
Figure 2-(1), while the chart of Figure 2-(2’) would
be used when the request is just utterance (5’). It
is clear that the decision on axis assignment is also
context sensitive.
In conventional ellipsis handling (Hendrix et al.,
1978; Carbonell and Hayes, 1983), the interpreta-
tion of an utterance fragment, such as utterance (2)
following utterance (1), is the same as the interpre-
tation of utterance (2’). When a response based
on this interpretation is made, the data plotted on
the chart may be correct, but the chart form and
style cannot be. This implies that something ex-
tra is needed for handling dialogue in charts, and it
is insufficient just to combine two mechanisms for
non-interactive automatic chart design and natural
language dialogue understanding.
3 Handling dialogue in natural language
and charts
This section proposes a methodology for handling
dialogue in natural language and charts. First, a log-
ical form that represents the interpretation of utter-
ances is proposed. Then, how to represent the per-
spectives from which the user wants to look at the
data and how to relate them to chart realization are
described. Last, a way of handling utterance frag-
ments is discussed.
3.1 A logical form
A logical form that describes the content of a given
utterance must be able to represent what region
of data the user is interested in and what type of
analysis she wants to conduct in order to obtain
the data to be plotted. Moreover, it must convey
the intention of the user, that is, what informa-
tion she wants to obtain through that chart. The
logical form we propose, which satisfies those re-
quirements, takes the form, a73 Description, a74a75a74a75a74a75a76 Ac-
tiona77 , where Action specifies the main speech act
in a given utterance, and Descriptions describe con-
straints or conditions that the objects related to the
action should satisfy. Action can be a request to
display charts or a request for information con-
veyed through charts. In this paper, however, it
only covers requests to display charts, which takes
the form a78a80a79a82a81a82a83a85a84a87a86a89a88a85a90a87a91a93a92a95a94 ListofVars, ListofAspectsa96 ,
where ListofVars is the list of variables plotted on
the chart. ListofAspects is the list of aspects of the
data the user is focusing on and represents the per-
spectives from which she wants to look at the data.
Descriptions describe constraints or conditions that
the objects related to the action should satisfy,
which has the form, a73 Quantifier, Var/Class, Restric-
tiona77 , where Quantifier is a generalized quantifier,
Var is the variable of quantification, and the quan-
tification ranges over the objects each of which is
a member of Class and satisfies Restriction. That
is, this logical form is a flattened version of Woods’
MRL (Woods, 1978), and as in Woods’ MRL, Class
can be a function. Moreover, Classes, each of which
each variable and object is associated with, are hier-
archically organized and represent not only the do-
main an object is classified into, but also its gran-
ularity. An object that belongs to the area domain,
for example, belongs to one of classes: district, pre-
fecture, or city, according to its granularity. The
subsumption relation is defined between objects that
belong to classes with different granularities and the
same domain. In Restriction, implicit coercion be-
tween granularities is allowed and aggregation such
as summation is represented implicitly using this
mechanism. 1
For example, utterance (2’) is interpreted into the
logical form:
a97a75a97a75a98a100a99a89a98a100a101a75a102a104a103a106a105a108a107a109a102a89a98a100a110a100a101a104a103a112a111a93a98a113a111a89a114a89a98a100a101a87a115a116a105a80a103a75a117a119a118a121a120a100a120a75a122a89a107a109a102a89a98a100a110a100a101a80a103a113a118a109a120a75a120a121a123a108a107a109a102a89a98a75a110a121a101a75a124a108a125a121a126a87a103
a97a75a98a100a99a89a98a100a101a75a102a104a103a106a127a108a107a113a128a89a101a75a98a100a129a89a98a75a130a121a131a121a132a89a101a75a98a80a103a106a127a89a133a75a134a109a135a108a136a121a137a75a138a100a137a100a132a108a107a121a139a89a136a75a134a109a131a75a101a89a136a75a130a109a131a89a126a87a103
a97a100a131a100a135a108a98a80a103a116a140a89a107a100a134a100a110a100a141a75a98a75a134a82a115a116a105a80a103a142a127a108a125a82a103a106a131a75a101a121a132a108a98a100a126a80a143
a101a89a98a100a144a100a145a93a136a75a134a109a128a89a141a75a110a121a102a80a115a75a117a116a140a100a124a80a103a75a117a146a135a108a147a100a138a109a111a75a128a89a148a100a102a80a115a142a105a93a125a93a103a142a139a89a147a100a138a109a111a75a128a89a148a100a102a80a115a142a127a93a125a113a124a93a125a109a126
The first description states that variable a149 ranges
over two objects of year class, 1993 and 1994. The
second description states variable a150 ranges over ob-
jects with prefecture granularity that are subsumed
by Shikoku, which is itself an object with district
granularity. In this case, the equality in the restric-
tion coerces into the subsumption relation. In the
third description, a86a82a91a87a90a87a79a80a86a151a94a109a149a153a152a113a150a154a96 is a function from
time and area to sales amounts. The perspective is
discussed in the next section.
3.2 Perspectives and chart realization
The second argument of the a78a80a79a82a81a82a83a85a84a87a86a89a88a85a90a87a91a93a92 action
specifies the user’s perspective, which is the list of
aspects of the data she is focusing on. The follow-
ing basic aspects are covered for the present. Sup-
pose Var1 is an element of the first argument of
the action, that is, a variable to be plotted. The
a155a82a156a108a157
a88a80a158a87a92a95a94 Var2a96 specifies that the user is interested
in comparison of the values of Var1 by each pos-
sible Var2 instance. In other words, she is focus-
ing on the difference in the values of Var1 which
came from the difference of Var2 instances. The
a159
a156
a159
a91a87a90a80a160a89a161a153a94 Var2a96 specifies that the user also wants
to know the total for the values of Var1 in all
Var2 instances. The a88a104a78
a156
a88a80a162
a156
a94 Var2a96 specifies that
she is interested in the proportion of each value
of Var1 to the total in all Var2 instances. The
a155a89a163
a91a108a161a104a164a80a79a93a165a85a84
a159
a163
a94 Var2a96 specifies that she is interested
in the change of value of Var1 with the change of
Var2 instances. The a155a82a156a108a157 a88a80a158a87a92a95a94 Var2a96 is, in a sense,
a default since the quantitative chart, in principle,
shows variation of the values of a dependent vari-
able by the difference of the values of independent
variables.
1The way to create a data table to be plotted on a chart from
raw material data using this type of logical form was discussed
in (Matsushita et al., 2000).
From now on, our discussion is restricted to the
two-dimensional chart forms for up to two indepen-
dent variables and one dependent variable. These
forms of charts are widely used and still have plenty
of variety. In chart realization under this restric-
tion, one of the independent variables, which are
the arguments of the function in the logical form,
is assigned to the horizontal axis and the other to
the hidden or depth axis. For example, variable a149
is assigned to the horizontal axis and variable a150 is
assigned to the depth axis in the realization of the
logical form of the previous example into Figure 1-
(2’). One of the realization parameters is this axis
assignment of variables.
Let us think about how the perspective guides
chart realization. In two-dimensional charts,
changes of values can be displayed along the hor-
izontal axis using, say, the line chart, while total
and proportion is easy to show on a depth axis by
using, say, the stacked chart. Comparison can be
exhibited on either axis. Therefore, a155a82a156a108a157 a88a80a158a87a92 and
a155a89a163
a91a108a161a104a164a80a79a93a165a85a84
a159
a163 are possible for the variable assigned
to the horizontal axis, and a155a82a156a108a157 a88a80a158a87a92 , a159 a156 a159 a91a87a90a80a160a89a161 , and
a88a104a78
a156
a88a80a162
a156 are possible for the variable assigned to
the depth axis. Next, let us think about what com-
binations of aspects are possible and sufficient for
the perspective. Since we cannot compare the dif-
ference nor see the changes regarding a variable
that has only one instance in the range specified
by the restriction, the aspect for such a variable,
which we call a uniquely instantiated variable, is
meaningless. Excluding that variable, each remain-
ing variable takes only one aspect. Their combi-
nation should result in those aspects being prop-
erly assigned to the axes. Here, by subcategorizing
a155a82a156a108a157
a88a80a158a87a92 , which can be assigned to either axis, into
a163a104a166a80a156a108a157
a88a80a158a87a92 and a167
a166a80a156a108a157
a88a80a158a87a92 , which should be assigned
to the horizontal and depth axis respectively, you
can assign an axis to the variables only by assign-
ing one aspect to each. Thus, when two indepen-
dent variables, a149 and a150 , are not uniquely instanti-
ated, one of them, say a149 , takes either a163a104a166a80a156a108a157 a88a80a158a87a92a95a94a109a149a85a96
or a155a89a163 a91a108a161a104a164a80a79a93a165a85a84
a159
a163
a94a109a149a154a96 , and the other, say a150 , takes ei-
ther a167
a166a80a156a108a157
a88a80a158a87a92a95a94a109a150a154a96 , a159 a156 a159 a91a87a90a80a160a89a161a153a94a109a150a154a96 , or a88a104a78
a156
a88a80a162
a156
a94a109a150a154a96 . In
this case, variable a149 is assigned to the horizontal
axis and variable a150 to the depth axis. Both vari-
ables are not allowed to take aspects assigned to the
same axis. The combination of a167
a166a80a156a108a157
a88a80a158a87a92a95a94a109a149a154a96 and
a159
a156
a159
a91a87a90a80a160a89a161a153a94a109a150a154a96 is forbidden, for example. Figure 3
summarizes the relationship between combinations
of aspects and realized chart forms.
Now, chart realization is reduced to aspect assign-
ment. In other words, you can realize an appropri-
ate chart from a given user utterance by obtaining
its logical form, and, if some of the aspects are im-
plicit in the utterance, by completing the perspective
using the characteristics of the data concerned, the
current context and so on.2 Obtaining and complet-
ing the perspective, which is vital for chart realiza-
tion, is carried out as follows.3
a168 A portion of the perspective is explicit in the
utterance. For example, it is obvious that the
user is focusing on the change over time when
she says that she wants to see the change in
sales. Such perspectives are acquired through
the interpretation of her utterance.
a168 Characteristics of statistics returning a depen-
dent variable sometimes suggest the perspec-
tive. For example, share and profitability sug-
gest her interest in the proportion to the total.
Such characteristics are used for completing
the perspective.
a168 The class of independent variables also sug-
gests the perspective. A class whose in-
stances constitute an interval scale, such
as a time series, suggests changes over it
(i.e. a155a89a163 a91a108a161a104a164a80a79a93a165a85a84
a159
a163 ) for its aspect to be fo-
cused on when the variable ranges over a lot
of instances. Even when it has a few in-
stances, comparisons by it (i.e. a163a104a166a80a156a108a157 a88a80a158a87a92 ) are
preferred. This criterion for selecting between
a155a89a163
a91a108a161a104a164a80a79a93a165a85a84
a159
a163 and a163a104a166a80a156a108a157
a88a80a158a87a92 is used throughout
this paper.
a168 When variables with more than one instance
are left with no aspect, a default is used.
When just one such variables is left, it takes
a155a89a163
a91a108a161a104a164a80a79a93a165a85a84
a159
a163 or a163a104a166a80a156a108a157
a88a80a158a87a92 if possible, other-
wise it takes a167
a166a80a156a108a157
a88a80a158a87a92 . When two such vari-
2Chart realization has dimensions other than those dis-
cussed here. For example, while the independent variable is
always assigned to the vertical axis in our discussion, it can be
assigned to the horizontal axis. The rank of instances on an
axis, the scales of axes, and visual prompts such as labels and
arrows are also dimensions which should be considered (Mit-
tal, 1998; Fasciano and Lapalme, 1996). Although discussion
of those dimesions exceeds the scope of this paper, we believe
that a natural extension of perspective would cover them.
3In implementation, perspective completion based on these
features is more sophisticated, referring to heuristics from a
textbook for drawing charts (Zelazny, 1996) and knowledge
acquired from a chart corpus using a machine learning tech-
nique (Yonezawa et al., 2000).
a169
a170a169
a171a169
a172a169
a173a169
a174a169
a175a169
a176a169
a177a175 a177a176 a177a178 a177a177
a169
a171a169
a173a169
a175a169
a178a169
a170a169a169
a170a171a169
a170a173a169
a170a175a169
a177a175 a177a176 a177a178 a177a177
a169a179
a170a169a179
a171a169a179
a172a169a179
a173a169a179
a174a169a179
a175a169a179
a176a169a179
a178a169a179
a177a169a179
a170a169a169a179
a177a175 a177a176 a177a178 a177a177
a169
a171a169
a173a169
a175a169
a178a169
a170a169a169
a170a171a169
a170a173a169
a170a175a169
a177a175 a177a176 a177a178 a177a177
a180
a181a180
a182a180
a183a180
a184a180
a185a180a180
a185a181a180
a185a182a180
a185a183a180
a186a183 a186a187 a186a184 a186a186 a188
a189
a188
a190
a188
a191
a188
a192
a188
a193
a188
a194
a188
a195
a188
a196a194 a196a195 a196a197 a196a196
a169a179
a170a169a179
a171a169a179
a172a169a179
a173a169a179
a174a169a179
a175a169a179
a176a169a179
a178a169a179
a177a169a179
a170a169a169a179
a177a175 a177a176 a177a178 a177a177
a169
a171a169
a173a169
a175a169
a178a169
a170a169a169
a170a171a169
a170a173a169
a170a175a169
a177a175 a177a176 a177a178 a177a177
a169
a170a169
a171a169
a172a169
a173a169
a174a169
a175a169
a176a169
a178a169
a177a169
a170a169a169
a177a175
a169a179
a170a169a179
a171a169a179
a172a169a179
a173a169a179
a174a169a179
a175a169a179
a176a169a179
a178a169a179
a177a169a179
a170a169a169a179
a177a175
a198 a199 a200 a201 a202 a203 a204 a205 a206 a207
a208
a198 a209 a210 a211 a212 a213 a214 a215 a198 a205 a206 a207
a216 a199 a200 a201 a202 a203 a204 a205 a217 a207 a215 a200 a215 a209 a218 a219 a210 a205 a217 a207 a202 a220 a200 a202 a221 a200 a205 a217 a207
a169
a170a169
a171a169
a172a169
a173a169
a174a169
a175a169
a176a169
a178a169
a177a169
a170a169a169
a177a175
a210 a200 a210 a212
a210 a200 a210 a212
a222 a223
a224
a223 a225 a225 a226 a227 a228 a223 a229 a230 a231 a229 a232
a225 a226 a233 a234 a231 a235 a233 a236 a231 a223 a229
Figure 3: Relationship between Perspectives and Chart Forms
ables are left, one takes a163a104a166a80a156a108a157 a88a80a158a87a92 , the other
takes a167 a166a80a156a108a157 a88a80a158a87a92 .
3.3 Utterance fragments and chart
appropriateness
Utterance fragments in a specific context should be
interpreted not as a logical form, but as a request to
revise the logical form given as the context. In or-
der to correctly handle chart appropriateness, which
is sensitive to the context, the perspective in the log-
ical form should be revised appropriately according
to that request. First, utterance fragments are classi-
fied into the following categories according to what
part of the logical form should be revised as a re-
sult of those fragments. Examples are shown, which
are assumed to have been uttered after the utterance,
“Show me the sales in Shikoku and Chugoku from
’93 through ’95.”
Domain alteration: The Restriction part of an in-
dependent variable is revised. Examples, “Just
Shikoku.”, “Through ’98.”
Granularity alteration: The Class part of an inde-
pendent variable is revised to one with a differ-
ent granularity. Examples, “By quarter.”, “By
prefecture.”
Statistic alteration: The function returning the de-
pendent variable, which locates its Class part,
is revised. Examples, “Show me the number of
the dealers.”
Perspective alteration: The perspective is revised.
Examples, “Show me the change.”, “How
about the total?”
The type of content words, clue words, and spe-
cific phrases contained are exploited for interpret-
ing utterance fragments and for classifying them
into one of the above categories. Using those, we
can identify what part of the logical form should
be changed and how. In addition to the revisions
identified, appropriate revision of the perspective is
needed for correct chart realization. Revisions of
perspective are summarized as follows.4
a168 As a result of domain alteration on variable X,
if the number of instances of X turns into more
than one and the current perspective includes
no aspect relating to X, that is, X is a uniquely
instantiated variable, check a155a89a163 a91a108a161a104a164a80a79a93a165a85a84
a159
a163
a94 Xa96 ,
a163a104a166a80a156a108a157
a88a80a158a87a92a95a94 Xa96 , and a167
a166a80a156a108a157
a88a80a158a87a92a95a94 Xa96 in this order,
and add the first possible one to the perspec-
tive. On the other hand, if X turns into a
uniquely instantiated variable, delete the as-
pect related to X from the perspective.
a168 As a result of granularity alteration on vari-
able X, if the number of instances of X
turns into more than one and the cur-
rent perspective includes no aspect related
to X, check a155a89a163 a91a108a161a104a164a80a79a93a165a85a84
a159
a163
a94 Xa96 , a163a104a166a80a156a108a157 a88a80a158a87a92a95a94 Xa96 ,
a159
a156
a159
a91a87a90a80a160a89a161a153a94 Xa96 , and a167
a166a80a156a108a157
a88a80a158a87a92a95a94 Xa96 in this order,
and add the first possible one to the perspec-
tive. Here, a159 a156 a159 a91a87a90a80a160a89a161a153a94 Xa96 is possible only when
the statistics concerned use summation for ag-
gregation. On the other hand, if X turns into
4Revisions of perspective for statistic alterations and per-
spective alterations are omitted, because the space is limited
and our concern is to trace the changes of the user’s perspective
especially when she does not mention them explicitly.
a uniquely instantiated variable, delete the as-
pect related to X from the perspective.
4 Examples
This section demonstrates how our proposal ad-
dresses the problems raised. First, let us consider
series of utterance (1), (2) and (3). The interpreta-
tion of utterance (1) is
a97a75a97a75a98a100a99a89a98a100a101a75a102a104a103a106a105a108a107a109a102a89a98a100a110a100a101a104a103a112a111a93a98a113a111a89a114a89a98a100a101a87a115a116a105a80a103a75a117a119a118a121a120a100a120a75a122a89a107a109a102a89a98a100a110a100a101a80a103a113a118a109a120a75a120a121a123a108a107a109a102a89a98a75a110a121a101a75a124a108a125a121a126a87a103
a97a75a98a100a99a89a98a100a101a75a102a104a103a106a127a108a107a109a139a108a136a100a134a121a131a75a101a89a136a75a130a109a131a104a103a106a127a89a133a75a134a109a135a108a136a121a137a75a138a100a137a121a132a93a107a121a139a89a136a75a134a109a131a75a101a89a136a75a130a109a131a89a126a87a103
a97a100a131a100a135a108a98a80a103a116a140a89a107a100a134a100a110a100a141a75a98a75a134a82a115a116a105a80a103a142a127a108a125a82a103a106a131a75a101a121a132a108a98a100a126a80a143
a101a89a98a100a144a100a145a93a136a75a134a109a128a89a141a75a110a121a102a80a115a75a117a116a140a100a124a80a103a75a117a146a135a108a147a100a138a109a111a75a128a89a148a100a102a80a115a142a105a93a125a113a124a93a125a121a126
As for the perspective, variable a149 , which repre-
sents time series and ranges over only two instances,
obtains not a155a89a163 a91a108a161a104a164a80a79a93a165a85a84 a159 a163 a94a109a149a154a96 , but a163a104a166a80a156a108a157 a88a80a158a87a92a95a94a109a149a154a96 .
Variable a150 , which ranges over areas, is uniquely in-
stantiated and no aspect is given to it. The chart
realized from this logical form is the column chart
of Figure 1-(1). The chart form is determined from
the perspective by referring to Figure 3. When
utterance (2) is given in this context, it is inter-
preted as a granularity alteration on variable a150 , and
a237
a79a93a238a104a79a82a78a82a92a153a152a113a150a104a239a82a167a80a84a87a86
a159
a78a104a84
a155
a159
a152a119a150a80a240a104a86
a163
a84a108a241
a156
a241a82a242a104a239a82a167a80a84a87a86
a159
a78a104a84
a155
a159a104a243 of
the logical form is revised to a237 a79a93a238a104a79a82a78a82a92a153a152a244a150a104a239
a88a104a78a80a79a93a245a104a79
a155
a159
a242a104a78a80a79a246a152a113a150a80a240a80a86
a163
a84a89a241
a156
a241a82a242a85a239a93a167a104a84a87a86
a159
a78a104a84
a155
a159a80a243 . In addi-
tion, a159 a156 a159 a91a87a90a80a160a89a161a153a94a109a150a154a96 is added to the perspective, since
no aspect related to a150 was in it and sales is a
statistic for which summation is used for aggrega-
tion. The perspective ends up with a247 a163a104a166a80a156a108a157 a88a80a158a87a92a95a94a109a149a154a96a154a152
a159
a156
a159
a91a87a90a80a160a89a161a153a94a109a150a154a96a89a248 , and the new chart is the stacked col-
umn chart of Figure 1-(2). Utterance (3) in this con-
text is interpreted as a domain alteration on variable
a149 . Since the perspective remains the same as before,
while the restriction of variable a149 is revised accord-
ing to the utterance, the chart obtained is of the same
form, which is shown in Figure 1-(3).
On the other hand, for utterance (2’), as nei-
ther its expression nor its statistic implies a spe-
cific aspect, the perspective is determined according
to the characteristics of the independent variables.
First, variable a149 representing time series obtains
a163a104a166a80a156a108a157
a88a80a158a87a92a95a94a109a149a154a96 , and then variable a150 ranging over ar-
eas is given a167
a166a80a156a108a157
a88a80a158a87a92a95a94a109a150a154a96 . The perspective ends up
with a247
a163a104a166a80a156a108a157
a88a80a158a87a92a95a94a109a149a154a96a154a152a249a167
a166a80a156a108a157
a88a80a158a87a92a95a94a109a150a154a96a108a248 , and the chart
realized is the grouped column chart shown in Fig-
ure 1-(2’). For utterance (3’), almost the same cri-
teria are applied, but the aspect given to variable a149
is a155a89a163 a91a108a161a104a164a80a79a93a165a85a84
a159
a163
a94a109a149a85a96 as it has many instances. As a
result, the grouped line chart of Figure 1-(3’) is re-
alized.
Let us move to series of utterance (4) and (5).
For utterance (4), since variable a149 representing time
series is uniquely instantiated, no aspect is given.
Then variable a150 ranging over districts can obtain
a163a104a166a80a156a108a157
a88a80a158a87a92a95a94a109a150a154a96 . By utterance (5) following it, domain
alteration on a149 is specified, and a149 obtains the possi-
ble aspect a167 a166a80a156a108a157 a88a80a158a87a92a95a94a109a149a154a96 , since a163a104a166a80a156a108a157 a88a80a158a87a92a95a94a109a150a85a96 already
exists. On the other hand, for utterance (5’), vari-
able a149 representing time series priors variable a150 , ob-
taining a163a104a166a80a156a108a157 a88a80a158a87a92a95a94a109a149a154a96 . As a result, Figure2-(2), the
response to utterance (5), and Figure 2-(2’) , the re-
sponse to utterance (5’), are different in their axis
assignments.
5 Discussion
Appropriateness of charts is known to be a func-
tion of several factors. This paper revealed that
discourse context is one of those factors in ad-
dition to those already known such as character-
istic of data and user intentions, and proposed a
methodology for addressing that factor properly. To
our knowledge, there are few studies on automatic
chart design for such interactive situations as dis-
cussed here. There are many studies on interactive
graph drawing of course (Roth et al., 1994). Those,
however, are concerned with tools for producing a
graph interactively that achieves the user’s inten-
tion. Their standpoint differs from ours, and the
mode used for their interactions is direct manipu-
lation not natural language.
Our proposal partially overlaps with recent stud-
ies on automatic chart design. Our logical form
has a lot in common with the content language
in (Green et al., 1998). The objective of their re-
search, however, is to describe communicative goals
to be achieved through generating graphics and text,
and differs from ours, which is to describe the user’s
requests in order to respond to them using charts.
Our perspective plays a similar role to that of inten-
tion in PostGraphe (Fasciano and Lapalme, 1996).
However, there is a crucial difference in that, while
their intention is given as input, our perspective is
acquired from the user’s utterances, data character-
istics and dialogue context.
Most of the framework proposed in this paper
has been implemented. The prototype system ac-
cepts such a wide range of Japanese expressions
that the appropriateness of our proposal can be con-
firmed, though the understanding of those expres-
sions is driven by a simple pattern-based mecha-
nism. The matter worth special mention is the pro-
totype’s chart redraw mechanism. As suggested in
this paper, a correct chart in a certain context is one
that can be realized by minimum change to the chart
as the context, and loses minimum information from
it. We supposed that changes and loss of informa-
tion mattered because considerable mental loads are
needed to relate new information in the new chart to
old information in the context. In order to reduce
those loads, we made the process of change visu-
ally understandable. Specifically, the system shows
animations that represent which component of the
current chart moves and changes to which compo-
nent of the new one. This feature has received a
favorable reception in demonstrations.
A lot remains for future work. First, we will
consider answering wh- and yes-no interrogatives
in charts in addition to answering requests to show
charts. Answering such questions fluently requires
collaboration between charts and text. Then, we
will examine richer chart realization. We should
consider not only increasing the kinds of chart form
covered, but also assembling more than one chart to
achieve a certain goal. In addition, dimensions such
as visual prompts should be incorporated. Lastly,
in the context of information visualization, the ef-
fects of animation introduced in the implementation
of the prototype should be measured quantitatively
to prove that it really can reduce mental load rather
than merely make itself conspicuous.

References

Jaime G. Carbonell and Phillip J. Hayes. 1983.
Recovery strategies for parsing extragrammatical
language. American Journal of Computational
Linguistics, 9(3-4):123–146.

Stephen M. Casner. 1991. A task-analytic approach
to the automated design of graphic presentations.
ACM Trans. on Graphics, 10(2):111–151.

Massimo Fasciano and Guy Lapalme. 1996. Post-
graphe: A system for the generation of statis-
tical graphics and text. In Proc. 8th Interna-
tional Workshop on Natural Language Genera-
tion, pages 51–60.

Nancy Green, Giuseppe Carenini, Stephan Kerped-
jiev, Steven Roth, and Johanna Moore. 1998.
A media-independent content language for in-
tegrated text and graphics generation. In Proc.
Workshop on Content Visualization and Interme-
dia Representations, pages 69–75.

Gary G. Hendrix, Earl D. Sacerdoti, Daniel Saga-
lowicz, and Jonathan Slocum. 1978. Develop-
ing a natural language interface to complex data.
ACM Trans. on Database, 3(2):105–147.

Stephan Kerpedjiev, Giusappe Carenini, Nancy
Green, Johanna Moore, and Steven F. Roth.
1998. Saying it in graphics: from intentions to
visulaizations. In Proc. of the Symposium on In-
formation Visualization, pages 97–101.

Jock Mackinlay. 1986. Automating the design of
graphical presentations of relational information.
ACM Trans. on Graphics, 5(2):110–141.

Mitsunori Matsushita, Hayato Yonezawa, and
Tsuneaki Kato. 2000. A frame representation of
user requirements for automated data visualiza-
tion. In Proc. 14th European Conference on Ar-
tificial Intelligence, pages 631–635.

Mark T. Maybury and Wolfgang Wahlster, editors.
1998. Readings in Intelligent User Interface.
Morgan Kaufmann Publishers.

Vibhu O. Mittal. 1998. Visual prompts and graph-
ical design: A framework for exploring the de-
sign space of 2-d charts and graphs. In Proc.
14th National Conference on Artificial Intelli-
gence, pages 57–63.

Steven F. Roth and Joe Mattis. 1990. Data charac-
terization for intelligent graphic presentation. In
Proc. Conference on Human Factors in Comput-
ing Systems, pages 193–200.

Steven F. Roth, Joe Mattis, and Xavier Mesnard.
1991. Graphics and natural language as compo-
nents of automatic explanation. In J. W. Sullivan
and S. W. Tyler, editors, Intelligent User Inter-
faces. ACM Press.

Steven F. Roth, John Kolojejchick, Joe Mattis, and
Jade Goldstein. 1994. Interactive graphics de-
sign using automatic presentation knowledge. In
Proc. Conference on Human Factors in Comput-
ing Systems, pages 112–117.

William A. Woods. 1978. Semantics and quantifi-
cation in natural language question answering. In
M. Yovits, editor, Advances in Computers, vol-
ume 17. Academic Press.

Hayato Yonezawa, Mitsunori Matsushita, and
Tsuneaki Kato. 2000. Criteria to choose appro-
priate graph-types. In Proc. Pacific Rim Interna-
tional Conference on Artificial Intelligence, page
823.

Gene Zelazny. 1996. Say It with Charts. McGraw-
Hill, third edition.
