Evaluation Protocols |
Evaluation Protocols and taks for Participants
Generic Protocol: For each individual enrolled, 5 genuine signatures of Session 1 will be used as Reference Signatures. Testing will be carried out on the remaining 10 genuine signatures of Session 1, 10 genuine signatures of Session 2, 20 skilled forgeries acquired during both sessions, and 30 random forgeries (randomly chosen). Following the three main objectives of the BSEC, the Generic Protocol will be used by participants in two Evaluation Tasks:
-TASK 1, following OBJECTIVE 1 (impact of mobility acquisition conditions on algorithms’ performance):
EVALUATION PROTOCOL 1: The Organizer will test the submitted system in this Task on the whole DS2-382 and DS3-382 Test Subsets to study the impact of the mobile platform on system performance, only on Session 1. Ten random samplings of the 5 enrolment signatures will be considered to compute confidence intervals.
-TASK 2, following OBJECTIVE 2 (impact of time variability according to input time functions):
EVALUATION PROTOCOL 2: The Organizer will test the 3 submitted systems in this Task (one on the whole DS3-382 and the two other on the whole DS2-382) in order to study the impact of time variability on systems’ performance, according to the Generic Protocol. Ten random samplings of the 5 enrolment signatures will be considered to compute confidence intervals.
- Following OBJECTIVE 3 (impact of information content in signatures on algorithms’ performance): EVALUATION PROTOCOL 3: The Organizer will test all submitted systems of both Tasks 1 and 2 on DS2-382 and DS3-382 Test Subsets, separately on different categories of users depending of the quality of their signatures. This will allow evaluating the impact of information content in signatures on both systems’ robustness to mobility conditions, and to time variability. Ten random samplings of the 5 enrolment signatures will be considered to compute confidence intervals. |