Network of Excellence Peer-to-Peer Tagged Media

Social Event Detection Task Dataset

This task requires participants to discover events and detect media items that are related to either a specific social event or an event-class of interest.

Use scenario

By social events we mean that the events are planned by people, attended by people and that the social media are captured by people. Much multimedia on the Internet was captured during an event or otherwise is related to events. However, this multimedia is scattered, i.e., disassociated from the related event. Associating multimedia with events allow more natural browsing and search for users.

 

Guidelines from the task organizers

The SED task comprises two challenges. There will be a common test dataset for both, comprising images + typical metadata for them (time-stamps, tags, geotags for a small subset of them, etc.), and participants are welcome to submit results to either one of the challenges, or to both of them. In all cases, the image metadata that can be used by the participants for completing this task are only those provided to them as part of the task. The use of additional information (e.g. some specific geotags) that may be available on the internet for a given image of the development or test dataset but are not included in the XML files supplied by the task organizers is prohibited. Please be reminded that MediaEval works on the "honor system", i.e., we trust participants to not make use of such external sources of information for the images of the development and test datasets. Note, however, that generic external resources (such as Wordnet, Wikipedia, or even visual concept detectors trained on external collections) can be employed, providing that they do not relate to specific images of the test dataset, and that their development and use did not benefit from any knowledge of the dataset and challenge definitions of this task.

 

Challenge 1

The first challenge consists in detecting all soccer games that took place in a couple of specified cities within a specified period of time (e.g. one month). The evaluation criteria will be the number of events detected (out of all the relevant events included in the testset) and the number of correct/incorrect media detected for these events. If all images were tagged and geo-tagged, this would be a trivial task; but, since most images will not be geo-tagged (similarly to what is the case for most images out there on the internet), participants will need to also consider tag and/or visual information for finding the most complete set of relevant events and images. As a required (baseline) run, the participants are asked to use any combination of the available image metadata they see fit, but no visual information, for finding the relevant events and images; the use of visual information in addition to the various provided image metadata is encouraged in subsequent runs.

 

Challenge query: "Find all soccer events taking place in Barcelona (Spain) and Rome (Italy) in the test collection. For each event provide all photos associated with it."

Soccer events, for the purpose of this task, are actual soccer games involving the usual number of players and taking place in a soccer stadium (as opposed to, for example, a single person playing with a soccer ball out in the street, which is not a soccer event under our definition).

Finding the events, in this task, does not mean finding some textual descriptions or metadata of the events. What we are looking for is a set of photo clusters, each cluster comprising only photos associated with a single soccer event (thus, each cluster defining a retrieved soccer event).
The "photos associated with a soccer event" that we are looking for are all photos of the test collection that directly relate (in content, and also in terms of place/time) to an actual, specific soccer game. These photos are seen as part of the same "social event" as the corresponding soccer game. E.g., photos of game x being played, photos of fans inside the stadium during/a bit before/a bit after game x, photos of fans leaving the stadium after the end of game x, are all related to the "game x" soccer event. On the contrary, photos that miss the above relations to an actual game (e.g. a photo showing part of the stadium where the fans gather, with no fans visible or otherwise any relation to a specific game), are not considered as relevant to a soccer event. A few indicative examples of images that are judged to be relevant to soccer events are given below (beware: these indicative images do NOT relate to soccer events that took place in any of the specific two cities named in the above challenge definition), together with a couple of examples of images that are judged to be NON-relevant.

Example of relevant images:
http://www.flickr.com/photos/fabiogarigiola/4318829784/
http://www.flickr.com/photos/the_cha/3477276248/
http://www.flickr.com/photos/jorge-11/3257905870/
http://www.flickr.com/photos/pjacobs13/4745735768/

Example of NON-relevant image (despite its title and possibly also other metadata suggesting that it is related to a soccer game):
http://www.flickr.com/photos/psgmag/3588983822/

Example of NON-relevant image (despite involving a human and a soccer ball, this photo does not relate to an actual soccer game):
http://www.flickr.com/photos/xuxin/3522516020/

 

Challenge 2

The second challenge consists in finding all photos related to past events that took place in a couple of specific venues. For specifying these events, the venue names and some exemplary images will be provided (but these exemplary images will not have time-stamps). The evaluation criteria will be the number of events detected (out of all the relevant events included in the testset) and the number of correct/incorrect media detected for these events. Again, participants will probably need to consider different kinds of information for finding the most complete set of relevant images. Similarly to the first challenge, a required (baseline) run in which the participants use any combination of the available image metadata they see fit, but no visual information, for finding the relevant events and images is foreseen; the use of visual information in addition to the various image metadata is encouraged in subsequent runs.

 

Challenge query: "Find all events that took place in May 2009 in the venue named Paradiso (in Amsterdam, NL) and in the Parc del Forum (in Barcelona, Spain). For each event provide all photos associated with it."

 

For both venues, more than one event took place in May 2009. What we are looking for is again a set of photo clusters, each cluster comprising only photos associated with a single event.

 

Example of relevant images for the Paradiso:
http://www.flickr.com/photos/hydraulix16aa/3572159259

http://www.flickr.com/photos/blue_funk/3693762094

http://www.flickr.com/photos/elisevanherwaarden/3525133483

http://www.flickr.com/photos/francescograssotti/3506308497

 

Example of relevant images for the Parc del Forum:

http://www.flickr.com/photos/grauschleier/3747107954

http://www.flickr.com/photos/alterna2/3581663911/

http://www.flickr.com/photos/crlsblnc/3583142370

http://www.flickr.com/photos/haretz/3759469109/

http://www.flickr.com/photos/buckyrocks/3609348975/

 

 

Ground truth and evaluation

Ground truth information will record the true media-event associations and will be generated by the organizers. The results of event-related media item detection will be evaluated with the use of the ground truth annotations. Two evaluation measures will be used:

  • Harmonic mean (F-score) of Precision and Recall for the retrieved images (measures only the goodness of the retrieved photos; not the number of retrieved events, or how accurate the correspondence between retrieved images and events is). See, for instance, http://en.wikipedia.org/wiki/Precision_and_recall for a definition of this measure.
  • Normalized mutual information (compares two sets of photo clusters, each cluster comprising the images of a single soccer event; this measure jointly considers the goodness of the retrieved photos and the retrieved events). See, for instance, http://www.gabormelli.com/RKB/Normalized_Mutual_Information_Metric for a definition of this measure.

 

Data

The data set comprises URL of images available on Flickr (~70k images), together with their accompanying metadata such as tags, time and location information, etc. (in an XML format).

NOTE: This data is the test data (i.e., the data which you use to generate your runs). Due to the nature of the SED task, it did not make sense to release development data as with other tasks. Instead, the SED this readme contains examples (above) of target media sorts that provide you with information about the relevance criteria used by the task.

 

Contact

Martha Larson

m.a.(lastname)@tudelft.nl

 

Contact: Martha Larson, m.a.(lastname)@tudelft.nl