MultilingualinteractiveexperimentswithFlickr
PaulClough
Departmentof
InformationStudies
UniversityofSheffield
Sheffield,UK
p.d.clough@sheffield.ac.uk
JulioGonzalo
DepartamentodeLenguajes
ySistemasInform´aticos
UNED
Madrid,Spain
julio@lsi.uned.es
JussiKarlgren
SwedishInstituteof
ComputerScience
Stockholm
Sweden
jussi@sics.se
Abstract
Thispaperpresentsa proposalforiCLEF
2006, the interactive track of the CLEF
cross-languageevaluationcampaign. In
thepast,iCLEFhasaddressedapplications
such as informationretrieval and ques-
tion answering. However, for 2006 the
focushas turnedto text-basedimagere-
trievalfromFlickr. WedescribeFlickr,the
challengesthiskindofcollectionpresents
tocross-languageresearchers,andsuggest
initialiCLEFtasks.
1 InformationRetrievalEvaluationby
UserExperiment
Informationretrieval systems,especiallytext re-
trieval systems, have benefitedgreatly from a
fairlystrictandstraight-lacedevaluationscheme,
whichenablessystemdesignersto run tests on
versionsof theirsystemusinga testcollectionof
pre-assesseddata. Theserelevance-orientedex-
perimentsshedlighton comparative systemper-
formanceandenablebothintroductionofnew al-
gorithmsandincrementaloptimization.However,
batch-orientedsystemevaluationbasedon large
amountsofdata,abstractedawayfromsituational
constraints,variationinusage,andinteractiveness
issuesonlyaddressessomeof the bottlenecksto
buildasuccessfulsystem.
The CLEF1 Interactive Track(iCLEF2) is de-
votedto the comparative studyof userinclusive
cross-languagesearchstrategies. Over the past
5 years,iCLEFhas studiedthreecross-language
searchtasks:retrieval ofdocuments, answersand
1http://www.clef-campaign.org/
2http://nlp.uned.es/iCLEF/
annotatedimages(GonzaloandOard,2002;Gon-
zaloet al., 2005). All tasksinvolve the userin-
teractingwithinformationsystemsin a language
different from that of the documentcollection.
AlthoughiCLEF experimentscontinueproduc-
ing interestingresearchresults, whichmayhave
a substantialimpacton the way effective cross-
languagesearchassistantsare built,participation
inthistrackhasremainedlowacrossthefiveyears
ofexistenceofthetrack.Interactive studies,how-
ever, remainas a recognizednecessityin most
CLEFtracks.
Therefore,toencouragegreaterparticipationin
2006our focushas turnedto FLICKR3, a large-
scale,web-basedimagedatabasewiththepoten-
tialforofferingbothchallengingandrealisticmul-
tilingualsearchtasksforinteractive experiments.
Ouraimin selectinga primarilynon-textualtar-
getto studytextualretrieval is basedon someof
the multi-lingualand dynamiccharacteristicsof
FLICKR. Wewilloutlinethembelow.
2 TheFlickrsystem
The majorityof Web imagesearchis text-based
and the success of such approachesoften de-
pendsonreliablyidentifyingrelevanttext associ-
atedwitha particularimage. FLICKR is an on-
linetoolformanagingandsharingpersonalpho-
tographsandcurrentlycontainsover five million
freelyaccessibleimages.Theseareavailablevia
theweb,updateddailybya largenumberofusers
and availableto all web users(userscan access
FLICKR forfree,althoughlimitedtotheuploadof
20MBofphotospermonth).
3http://www.flickr.com/
70
2.1 Photographsinthecollection
ItisestimatedthatthecompleteFLICKR database
contains37 millionphotos with approximately
200,000imagesaddeddailyby1.2millionmem-
bers4. FLICKR provides both private and pub-
lic imagestorage,and photoswhichare shared
(around5 million)canbe protectedundera Cre-
ativeCommons(CC)licensing5 agreement(anal-
ternative to fullcopyright). Imagesfroma wide
varietyoftopicscanbeaccessedthroughFLICKR,
includingpeople,places,landscapes,objects,ani-
malsandevents. Thismakesthecollectiona rich
resourceforimageretrievalresearch.
2.2 Annotations
In FLICKR, photosareannotatedbyauthorswith
freelychosenkeywordsinanaturallymultilingual
manner:mostauthorsusekeywordsintheirnative
language;somecombinemorethanonelanguage.
In addition,photographshave titles,descriptions,
collaborative annotations, andcommentsinmany
languages. Figure5 providesan examplephoto
with multilingualannotations;Figure 5 shows
whatthequery“cats”retrievesfromthedatabase,
comparedwithwhatthequery“chats”retrieves.
Annotationsareusedbytheauthorstoorganize
theirimages,andbyany userto searchon. Key-
words assignedto the imagescan includeplace
namesand subjectmatter, and photoscan also
be submitted to online discussiongroups. This
providesadditionalmetadatato the imagewhich
can also be used for retrieval. An exploreutil-
ity providedby FLICKR makes use of this user-
generateddata (plus other informationsuch as
Clickthroughs)todefinean”interestingness”view
ofimages6.
3 FlickratiCLEF2006
Many imagesare accompaniedby text, enabling
theuseof bothtext andvisualfeaturesforimage
retrieval and its evaluation(M¨uller et al., 2006,
seee.g.). Imagesarenaturallylanguageindepen-
dent and oftensuccessfullyretrieved with asso-
ciatedtexts. This has been exploredas part of
ImageCLEF(Cloughet al., 2005)for areassuch
as informationaccessto medicalimagesandhis-
toricphotographs.Thewayinwhichuserssearch
4These figures are accurate as of October 2005:
http://www.wired.com/news/ebiz/0,1272,68654,00.html
5http://creativecommons.org/image/flickr,
http://flickr.com/creativecommons/
6http://www.flickr.com/explore/interesting
forimagesprovidesaninterestingapplicationfor
user-centereddesignandevaluation.AsaniCLEF
task,searchingforimagesfromFLICKR presents
a new multilingualchallengewhich,to date,has
notbeenexplored.Challengesinclude:
• Differenttypesof associatedtext, e.g. key-
words, titles, comments and description
fields.
• Collective classificationand annotationus-
ingfreelyselectedkeywords(knownasfolk-
sonomies)resultinginnon-uniformandsub-
jectivecategorizationofimages.
• Annotationsinmultiplelanguages.
Given the multilingualnatureof the FLICKR
annotations,translatingthe user’s searchrequest
would provide the opportunityof increasingthe
numberof imagesfoundand make moreof the
collectionaccessibleto a wider range of users
regardlessof their languageskills. The aim of
iCLEFusingFLICKR will be to determinehow
cross-languagetechnologiescould enhanceac-
cess, and explore the user interactionresulting
fromthis.
4 Proposedtasks
For iCLEF, participantsto this evaluationcam-
paignwillbeprovidedwiththefollowing:
• AsubsetoftheFlickrcollectionincludingan-
notationsandphotographs7.
• Example(realistic)search tasks. Ideally
thesesearchtaskswillreflectrealuserneeds
whichcouldbederivedfromlogfiles,studies
orsimilarretrievaltasks.
• Aframeworkinwhichtorunanevaluation.
5 Summary
Flickr will allow us to create an extremelyin-
terestinginteractive task based on truly hetero-
geneousannotations(that will in turn hopefully
attract more participants). Using imagesfrom
withina Web environmentis a realisticandcon-
temporarysearchchallengeandallowsmany im-
portantresearchquestionsto be addressed from
7We are currentlyin negotiationswithYahoo! (owners
of Flickr)andFlickrto provideresearcherswithlegitimate
accesstoasubsetofthecollection.
71
a quicklydevelopingfield. User-centeredstudies
arerequiredwithinbothtext andimageretrieval,
butareoftenneglectedastheyrequiremoreeffort
andtimefromparticipatinggroupsthana system-
centeredcomparisonthat can oftenbe run with-
outhumanintervention.Still,user-centeredeval-
uationcannotbereplaced andtheinfluenceofthe
useron theresultsis in generalstrongerthanthe
influenceofthesystemitself.

References
Paul Clough,HenningM¨uller, and MarkSanderson.
2005.Theclef2004crosslanguageimageretrieval
track. In Carol Peters, Paul Clough,Julio Gon-
zalo, GarethJones,MichaelKluck,and Bernardo
Magnini,editors,MultilingualInformationAccess
for Text, Speech and Images: Resultsof the Fifth
CLEFEvaluationCampaign, number3491/2005in
LectureNotesinComputerScience,pages597–613.
Springer, Heidelberg,Germany.
Julio Gonzaloand Doug Oard. 2002. The clef
2002 interactive track. In Advancesin Cross-
Language InformationRetrieval, number2785 in
Lecture Notes in Computer Science. Springer-
Verlag,Berlin-Heidelberg-NewYork.
Julio Gonzalo,Paul Clough, and A Vallin. 2005.
Overview of the clef 2005 interactive track. In
WorkingnotesoftheCLEFworkshop, Vienna,Aus-
tria,September.
HenningM¨uller, PaulClough,WilliamHersh,Thomas
Deselaers,ThomasLehmann,and AntoineGeiss-
buhler. 2006. Usingheterogeneousannotationand
visualinformationfor the benchmarkingof image
retrieval systems. In SPIE conference Photonics
West,ElectronicImaging, specialsessiononbench-
markingimage retrievalsystems, SanDiego,Febru-
ary.
