Introduction

First Cycle Evaluation

Second Cycle Evaluation

Third Evaluation

Download Zone

Contact & Feedback

F.A.Q.

Here are some questions you may have about ETISEO evaluation. If you need more information please contact us at etiseo@silogic.fr

General Questions

  • What are the evaluation cycles?

    Here is a short overview of the schedule:

    First evaluation is intended to verify protocols, technical details as well as the pertinence of criteria and metrics. It is based on a specific data set (Video data corpus 1). A first comparison of algorithm performances is achieved at the end of this cycle. First evaluation starts in January and ends in July. Participants can submit their results until end of March, after that period we will analyse the results, the metrics, and the evaluation cycle.
    Second evaluation is the real evaluation cycle. It will start in June 2006. Participants can submit their results until end of September.
  • Is ETISEO evaluation having a mandatory set of data for all participants?

    There are no mandatory sequences. But as the dataset contains several sequences, we decided to classify them in three categories:
    - priority,
    - wished,
    - and other sequences.
    Then, we are asking to participants to process at first sequences from the "priority" subset, then those of the "wished" subset and then the other sequences. We invite you do process in that order but of course with the capabilities of your system. For example: if your system only addresses people, you should not process sequences containing vehicles…
  • My system does not process multiple camera views. How can I submit results?

    For multiple camera views, you can submit one result file per camera without merging them.
    Or if your system supports multiple views, you can submit one file containing all results per view.
  • The evaluation seems to be after both tracking and recognition. Can we run these as separate processes in your evaluation? or do we need to have an integrated system?

    You can run your system in separate processes, but then you should merge the results in one file.
    The result file should contain all data corresponding to the different tasks your system can process (detection, localisation, classification, tracking, event recognition).
  • What should be contained in the results?

    The results should contain the output of your system formatted in an XML file. We give you the XML schema we are waiting for and corresponding examples.
  • What kind of activities do we expected to detect?

    Examples of events are:
    Walking/running, People/vehicles entering or leaving a zone, abandoned baggage…
    For each sequence, you will find on the DVDs, the list of expected scenarios. 
  • How can I use VIPER to see the context?

    VIPER-GT can be downloaded from: http://viper-toolkit.sourceforge.net/products/gt/

    You can visualise the 2D context file with Viper-GT.

    Load the context xml file from the DVD directory (context/...) : Open->Open Existing Viper file (except tiff sequence files).
    If Viper-GT does not find the path of the media file, browse the .info file of the sequence or change the path in the xml file.

First evaluation Questions

  • I don't find camera 1, 2 and 3 for sequence ETI-VS1-RD-16.
    The README file on the DVDs contains wrong camera numbers for sequence ETI-VS1-RD-16. It says for this sequence: cameras 1 (priority), 2 and 3 (wished) where it should be cameras 4 (priority), 5 and 6 (wished).

Real evaluation Questions

  • Should I register a second time for the real evaluation process?

    For those who have registered into the first evaluation cycle, returning the signed agreement, no extra registration is required.
  • Can we submit several times for the real evaluation as for the first evaluation ?

    Only one submission per participant is allowed for this set. You can submit several times, but only the latest submission will be taken into account and you will not receive any feed back about your submission during the evaluation period.
  • Can I participate to the second evaluation even if I did not participate to the first evaluation?

    Yes you can.

    First round was dedicated for evaluation validation (video corpus, file formats, metrics...). Second round is the real evaluation, and is open to every participants.

© SILOGIC