Main menu
Home
Databases
Evaluation Protocols
Submission Instructions
Schedule
Registration
Evaluation Protocols

  

Evaluation Protocols and taks for Participants

 

Generic Protocol: For each individual enrolled, 5 genuine signatures of Session 1 will be used as Reference Signatures. Testing will be carried out on the remaining 10 genuine signatures of Session 1, 10 genuine signatures of Session 2, 20 skilled forgeries acquired during both sessions, and 30 random forgeries (randomly chosen).

Following the three main objectives of the BSEC, the Generic Protocol will be used by participants in two Evaluation Tasks:

 

-TASK 1, following OBJECTIVE 1 (impact of mobility acquisition conditions on algorithms’ performance):

  • Participants will only use the Development Subset DS2-50 [experiment task1-ds2];
  • Only pen coordinates will be considered in this Task;
  • Participants will submit the best possible system giving an Output Similarity Score.

EVALUATION PROTOCOL 1: The Organizer will test the submitted system in this Task on the whole DS2-382 and DS3-382 Test Subsets to study the impact of the mobile platform on system performance, only on Session 1. Ten random samplings of the 5 enrolment signatures will be considered to compute confidence intervals.

-TASK 2, following OBJECTIVE 2 (impact of time variability according to input time functions):

  • Participants will use both Development Subsets DS3-50 [experiment task2-ds3] and DS2-50; for DS2-50, participants will consider two configurations of input time functions: one with only coordinates and pressure [experiment task2-ds2-1], the other with all time functions [experiment task2-ds2-2];
  • Participants will submit, for each Development Subset, the best possible system resisting to time variability, giving an Output Similarity Score following the Generic Protocol.

EVALUATION PROTOCOL 2: The Organizer will test the 3 submitted systems in this Task (one on the whole DS3-382 and the two other on the whole DS2-382) in order to study the impact of time variability on systems’ performance, according to the Generic Protocol. Ten random samplings of the 5 enrolment signatures will be considered to compute confidence intervals.

 

- Following OBJECTIVE 3 (impact of information content in signatures on algorithms’ performance):

EVALUATION PROTOCOL 3: The Organizer will test all submitted systems of both Tasks 1 and 2 on DS2-382 and DS3-382 Test Subsets, separately on different categories of users depending of the quality of their signatures. This will allow evaluating the impact of information content in signatures on both systems’ robustness to mobility conditions, and to time variability. Ten random samplings of the 5 enrolment signatures will be considered to compute confidence intervals.