3. Results
3.1 Finger chosen to enroll
Before seeing any of the readers, 100% of the test subjects chose the index finger (either right or left). This is most likely for the same reason that the index finger is naturally used when pointing, when reading Braille or when following a line of text while reading.
The extended index finger is the closest hand gesture to the relaxed position of the fingers of the human hand. Extending any other finger requires more excessive use of hand muscles. Confounded by additional early language developmental factors (Abry et al. 2004), humans therefore have a natural tendency to point with the index finger.
Of the forty test subjects (visually impaired and not visually impaired), n = 37 were right handed. Of these, n = 36 (97%) chose the right index finger. The remaining right-handed test subject chose the left index finger as he uses this finger to read Braille. Similarly, two out of three left-handed test subjects chose the left index finger. The remaining left-handed test subject chose the right index finger; the subject did not provide an explanation for this. Therefore it can be concluded that the choice of the right versus the left index finger closely relates to right- versus left-handedness.
3.2 Task analysis
A complete summary of the descriptive statistics for each task is provided in Tables 2 - 5.
3.2.1. Task 1: Did the user automatically know what finger motion to use when enrolling a fingerprint?
It should firstly be noted that the Hamster and Microsoft readers required the same finger motion: place finger on top of reader. The IBM reader required a different motion: swiping the finger across the reader.
3.2.1.1. Readers analysed individually
All (100%) VI test subjects were able to guess how to position their finger when using the Hamster reader (Binomial test result: higher than expected, p < 0.001). Most (96%) VI test subjects were able to guess how to position their finger when using the Microsoft reader (Binomial test result: higher than expected, p < 0.001). The one VI test subject who did not guess the correct finger motion had not yet used the other readers and he guessed that the finger should be rolled from left to right, similar to the motion used when an ink fingerprint is taken.
All (100%) test subjects with good vision were able to guess how to position their finger when using the Hamster and Microsoft readers (Binomial test result: higher than expected, p < 0.001 for both).
No VI test subject guessed the correct finger motion to use when using the IBM reader (Binomial test result: lower than expected, p < 0.001). Most (75%) good vision test subjects did not guess the correct finger motion to use when using the IBM reader (Binomial test result: n.s.)
3.2.1.2. Readers compared for Task 1
A significantly lower number of VI (Cochrane Q = 54.1, df = 2, p < 0.001) and good vision (Cochrane Q = 18.0, df = 2, p < 0.001) test subjects guessed what finger motion was required to use the IBM reader (the swiping motion) when compared to the Hamster and Microsoft readers (press finger on top).
3.2.1.3. Task 1 results summary
In summary, having never used a fingerprint reader before, the majority of test subjects assumed that the finger should simply be placed on the reader. Only test subjects who were able to see the IBM reader could tell that it required a different finger motion than expected.
3.2.2. Task 2: Did the user automatically know how to interact with the reader?
This task included the simple hand motion required to complete Task 1, plus the additional gestures needed to successfully interact with the reader.
The Hamster and Microsoft readers required the same interaction: place and hold finger still on reader until an image has been taken. The IBM reader required the following interaction: place the fingertip onto the tactile indentation above the scanner area, and moving the finger towards your body, swipe the finger across the scanner area.
3.2.2.1. Readers analysed individually
Most of the VI test subjects correctly guessed how to interact with the Hamster and Microsoft readers (89% and 96% respectively; Binomial test result: higher than expected, p < 0.001). None (0%) of the VI test subjects correctly guessed how to interact with the IBM reader (Binomial test result: lower than expected, p < 0.001).
All (100%) of the test subjects with good vision correctly guessed how to interact with the Hamster reader (Binomial test result: higher than expected, p < 0.001). Most (92%) of the test subjects with good vision correctly guessed how to interact with the Microsoft reader (Binomial test result: higher than expected, p < 0.001). The one test subject who did not guess correctly had just used the IBM reader, and guessed that the finger had to be swiped. Most (75%) of the test subjects with good vision did not guess how to interact with the IBM reader correctly (Binomial test result: n.s.).
3.2.2.2. Readers compared for Task 2
A significantly lower number of VI (Cochrane Q = 48.5, df = 2, p < 0.001) and good vision (Cochrane Q = 16.2, df = 2, p < 0.001) test subjects correctly guessed how to interact with the IBM reader (the swiping motion) when compared to the Hamster and Microsoft readers (press finger on top).
3.2.2.3. Task 2 results summary
In summary, as highlighted in Task 1 (see Section 3.2.1), the majority of test subjects assumed that the finger should simply be placed on the scanner. When the next required action is simply to hold the finger in place, the majority of test subjects were able to figure this out for themselves. In contrast, only test subjects who were able to see the IBM reader knew that it required a different interaction than expected. But the majority of these good vision test subjects could not guess what the correct interaction was.
3.2.3. Task 3: Could the user correctly interact with the scanner once the manufacturer's instructions were read or called out?
3.2.3.1. Readers analysed individually
All (100%) VI test subjects could correctly interact with the Microsoft reader once the manufacturer's instructions were read out (Binomial test result: higher than expected, p < 0.001).
Most (93%) of the VI test subjects could correctly interact with the Hamster reader once the manufacturer's instructions were read out (Binomial test result: higher than expected, p < 0.001).
Approximately half (52%) of the VI test subjects could correctly interact with the IBM reader once the manufacturer's instructions were read out (Binomial test result: n.s.).
All (100%) test subjects with good vision were able to correctly interact with all three scanners once the manufacturer's instructions were read (Binomial test result: higher than expected, p < 0.001).
3.2.3.2. Readers compared for Task 3
A significantly lower number of VI (Cochrane Q = 48.5, df = 2, p < 0.001) test subjects understood how to interact with the IBM reader after the manufacturer's instructions were read out, when compared to both the Hamster and Microsoft readers.
3.2.3.3. Task 3 results summary
In summary, the manufacturer's instructions were not sufficient to explain to VI test subjects how to interact with the IBM reader.
3.2.4. Task 4: Could the user correctly interact with the scanner once more detailed instructions were provided by the evaluator?
3.2.4.1. Readers analysed individually
All (100%) of the remaining VI test subjects were able to correctly interact with the IBM reader once further instructions were provided (Binomial test result: higher than expected, p < 0.001). The remaining VI test subjects were able to correctly interact with the Hamster reader once further instructions were provided (statistics not performed: insufficient sample size).
3.2.4.2. Task 4 results summary
By simplifying and adding more detail to the manufacturer's instructions, all users were able to understand how to interact with all three scanners.
3.2.5. Task 5: Did the user have success on the very first scan (success = the first image was successfully taken, but the following images were not necessarily successfully taken)? [Please note: this category includes users who tried their first scan after the manufacturer's instructions were read or called out and those who requested more detailed instructions before they tried their first scan.]
3.2.5.1. Readers analysed individually
A majority (71%) of VI test subjects had a successful scan on their first try of the Hamster reader (Binomial test result: higher than expected, p < 0.05). Approximately half (54%) of VI test subjects had a successful scan on their first try of the Microsoft reader (Binomial test result: n.s.). None (0%) of the VI test subjects had a successful scan on their first try of the IBM reader (Binomial test result: lower than expected, p < 0.001).
A majority (83%) of the test subjects with good vision had a successful scan on their first try of the Microsoft reader (Binomial test result: higher than expected, p < 0.05). Half (50%) of the test subjects with good vision had a successful scan on their first try of the Hamster reader (Binomial test result: n.s.). One third (33%) of the test subjects with good vision had a successful scan on their first try of the IBM reader (Binomial test result: n.s.).
3.2.5.2. Readers compared for Task 5
Significantly more VI test subjects had a successful scan on their first try of the Hamster reader when compared to the IBM reader (Cochrane Q = 28.2, df = 2, p < 0.001). Significantly more test subjects with good vision had a successful scan on their first try of the Microsoft reader when compared to the IBM reader (Cochrane Q = 6.2, df = 2, p < 0.05).
3.2.5.3. Task 5 results summary
In summary, on first use, the Hamster reader was most successful with VI test subjects and the Microsoft reader was most successful with test subjects with good vision. The IBM reader was the least successful in both user groups.
3.2.6. Task 6: Did the user experience at least one successful scan (i.e. a single image successfully taken) after the manufacturer's instructions were provided?
3.2.6.1. Readers analysed individually
The majority (86%) of VI test subjects had at least one successful scan taken with the Hamster reader, once the manufacturer's instructions were read out (Binomial test result: higher than expected, p < 0.001). A majority (68%) of VI test subjects had at least one successful scan taken with the Microsoft reader, once the manufacturer's instructions were read out (Binomial test result: n.s.). A minority (21%) of VI test subjects had at least one successful scan taken with the IBM reader, once the manufacturer's instructions were read out (Binomial test result: lower than expected, p < 0.01).
All (100%) test subjects with good vision had at least one successful scan taken with the Microsoft reader, once the manufacturer's instructions were read (Binomial test result: higher than expected, p < 0.001). The majority (83%) of test subjects with good vision had at least one successful scan taken with the Hamster reader, once the manufacturer's instructions were read (Binomial test result: higher than expected, p < 0.05). Under half (42%) of the test subjects with good vision had at least one successful scan taken with the IBM reader, once the manufacturer's instructions were read (Binomial test result: n.s.).
3.2.6.2. Readers compared for Task 6
Significantly fewer VI test subjects had at least one successful scan on the IBM reader when compared to both the Hamster and Microsoft readers (Cochrane Q = 24.7, df = 2, p < 0.001). Significantly fewer test subjects with good vision had at least one successful scan on the IBM reader when compared to the Microsoft reader (Cochrane Q = 9.8, df = 2, p < 0.01).
3.2.6.3. Task 6 results summary
In summary, the number of test subjects who had at least one successful scan on the IBM reader was lower than for the Hamster and Microsoft readers.
3.2.7. Task 7: Did the user experience at least one successful scan (i.e. a single image successfully taken) after more detailed instructions were provided?
3.2.7.1. Readers analysed individually
All (100%) VI test subjects had at least one successful scan taken with the Microsoft and Hamster readers, once more detailed instructions were provided (Binomial test result: higher than expected, both p < 0.001). All (100%) test subjects with good vision had at least one successful scan taken with the Microsoft and Hamster readers, once more detailed instructions were provided (Binomial test result: higher than expected, both p < 0.001).
Approximately two thirds (65%) of VI test subjects had at least one successful scan taken with the IBM reader, once more detailed instructions were provided (Binomial test result: n.s.). Approximately two thirds (67%) of test subjects with good vision had at least one successful scan taken with the IBM reader, once more detailed instructions were provided (Binomial test result: n.s.).
3.2.7.2. Readers compared for Task 7
Significantly fewer VI (Cochrane Q = 20.0, df = 2, p < 0.001) and good vision (Cochrane Q = 8.0, df = 2, p < 0.05) test subjects had at least one successful scan on the IBM reader when compared to both the Hamster and Microsoft readers.
3.2.7.3. Task 7 results summary
In summary, all test subjects experienced at least one successful scan with the Microsoft and Hamster readers. Whereas, approximately one third of all test subjects (VI and good vision) did not have even one successful scan with the IBM reader.
3.2.8. Task 8: Did the user automatically know that more than one scan is taken for enrolment?
3.2.8.1. Readers analysed individually
None (0%) of the users knew prior to the evaluation that more than one scan was required for the registration process (Binomial test result: all lower than expected, p < 0.001).
3.2.9. Task 9: Was the registration process eventually successful (i.e. did the user do everything required for the fingerprint enrolment process to be a success at least once)?
3.2.9.1. Readers analysed individually
A majority (89%) of VI test subjects successfully enrolled a fingerprint with the Hamster reader (Binomial test result: higher than expected, p < 0.001). A majority (82%) of VI test subjects successfully enrolled a fingerprint with the Microsoft reader (Binomial test result: higher than expected, p < 0.01). A minority (29%) of VI test subjects successfully enrolled a fingerprint with the IBM reader (Binomial test result: lower than expected, p < 0.05).
All (100%) test subjects with good vision successfully enrolled a fingerprint with both the Hamster and Microsoft readers (Binomial test result: higher than expected, p < 0.001). Approximately two thirds (67%) of test subjects with good vision successfully enrolled a fingerprint with the IBM reader (Binomial test result: n.s.).
3.2.9.2. Readers compared for Task 9
Significantly fewer VI (Cochrane Q = 23.5, df = 2, p < 0.001) and good vision (Cochrane Q = 8.0, df = 2, p < 0.05) test subjects successfully enrolled at least once on the IBM reader when compared to both the Hamster and Microsoft readers.
3.2.9.3. Task 9 results summary
In summary, the majority of test subjects successfully enrolled with the Microsoft and Hamster readers. Approximately two thirds of test subjects with good vision and less than a third of visually impaired test subjects successfully enrolled on the IBM reader.
3.2.10. Task 10: Did the test subjects recognise when a scan was unsuccessful?
Please note: Of the VI test subjects, 27, 13 and 8 (out of n = 28) experienced at least one unsuccessful scan with the IBM, Microsoft and Hamster readers respectively. Of the test subjects with good vision, 8, 2 and 5 (out of n = 12) experienced at least one unsuccessful scan with the IBM, Microsoft and Hamster readers respectively. Therefore, the percentages in Task 10 and Task 11 refer only to these test subjects.
An unsuccessful scan on the Microsoft reader was accompanied by a visual and audio message. An unsuccessful scan on the Hamster and IBM readers was accompanied by a visual message only.
3.2.10.1. Readers analysed individually
Just over half (54%) of VI test subjects recognised when a Microsoft scan was unsuccessful (Binomial test result: n.s.). Only one (13%) VI test subject recognised when a Hamster scan was unsuccessful (Binomial test result: n.s.). The majority (93%) of VI test subjects did not recognise when an IBM scan was unsuccessful (Binomial test result: lower than expected, p < 0.001).
All of the test subjects with good vision recognised when a scan was unsuccessful on the Hamster and Microsoft readers (Binomial test result: n.s.; insufficient sample size).
3.2.10.2. Task 10 results summary
The test subjects who could see the computer screen knew when a scan was unsuccessful on all three readers. Most VI test subjects could not see the screen and therefore failed to recognise when a scan was unsuccessful on the IBM and Hamster readers. The Microsoft reader produced an audio signal when a scan was unsuccessful and, accordingly, half of the VI test subjects did recognise when a scan was unsuccessful.
3.2.11. Task 11: When a scan was unsuccessful, did the test subject know why?
3.2.11.1. Readers analysed individually
The majority (85% and 75%) of VI test subjects could not provide a possible reason why a scan was unsuccessful on the Microsoft (Binomial test result: lower than expected, p < 0.05) and Hamster readers respectively (Binomial test result: n.s.). None (0%) of the VI test subjects could explain why a scan was unsuccessful on the IBM reader (Binomial test result: lower than expected, p < 0.001).
Of the test subjects with good vision, 100% and 75% could not provide a possible reason why a scan was unsuccessful on the Microsoft and IBM readers respectively. Three of the five (60%) test subjects with good vision were able to guess why a scan was unsuccessful on the Hamster reader. The sample sizes were too small for these results to be analysed.
3.2.11.2. Task 11 results summary
The majority of test subjects did not know why a scan was unsuccessful.
Table 2. Descriptive statistics of the task analysis for the three fingerprint readers showing the percentages of success and failure in each task for visually impaired test subjects.
Task |
N |
IBM |
Hamster |
Microsoft |
|||
% Yes |
% No |
% Yes |
% No |
% Yes |
% No |
||
1. Automatically knew which finger motion to use |
28 |
0 |
100 |
100 |
0 |
96 |
4 |
2. Automatically knew how to interact with the reader |
28 |
0 |
100 |
89 |
11 |
96 |
4 |
3. Knew how to interact with reader after manufacturer's instructions were provided |
27 |
52 |
48 |
93 |
7 |
100 |
0 |
4. Knew how to interact with reader after more detailed instructions were provided |
28 |
100 |
0 |
100 |
0 |
100 |
0 |
5. Success on first go of first scan |
28 |
0 |
100 |
71 |
29 |
54 |
46 |
6. Success after manufacturer's instructions were provided |
28 |
21 |
79 |
86 |
14 |
68 |
32 |
7. Success after more detailed instructions were provided |
28 |
65 |
35 |
100 |
0 |
100 |
0 |
8. Automatically knew to provide more than one scan for enrolment procedure |
28 |
0 |
100 |
0 |
100 |
0 |
100 |
9. Once the procedure was understood, were the following scans successful? |
28 |
29 |
71 |
89 |
11 |
82 |
18 |
10. When a scan was unsuccessful, did the test subject recognise this? |
- |
7a |
93a |
13b |
88b |
54c |
46c |
11. When a scan was unsuccessful, did the test subject know what the problem was? |
- |
0a |
100a |
25b |
75b |
15c |
85c |
aN = 27, bN = 8, cN = 13.
Table 3. Descriptive statistics of the task analysis for the three fingerprint readers showing the percentages of success and failure in each task for the test subjects with good vision.
Task |
N |
IBM |
Hamster |
Microsoft |
|||
% Yes |
% No |
% Yes |
% No |
% Yes |
% No |
||
1. Automatically knew which finger motion to use |
12 |
25 |
75 |
100 |
0 |
100 |
0 |
2. Automatically knew how to interact with the reader |
12 |
25 |
75 |
100 |
0 |
92 |
8 |
3. Knew how to interact with reader after manufacturer's instructions were provided |
12 |
100 |
0 |
100 |
0 |
100 |
0 |
4. Knew how to interact with reader after more detailed instructions were provided |
12 |
100 |
0 |
100 |
0 |
100 |
0 |
5. Success on first go of first scan |
12 |
33 |
67 |
50 |
50 |
83 |
17 |
6. Success after manufacturer's instructions were provided |
12 |
42 |
58 |
83 |
17 |
100 |
0 |
7. Success after more detailed instructions were provided |
12 |
67 |
33 |
100 |
0 |
100 |
0 |
8. Automatically knew to provide more than one scan for enrolment procedure |
12 |
0 |
100 |
0 |
100 |
0 |
100 |
9. Once the procedure was understood, were the following scans successful? |
12 |
67 |
33 |
100 |
0 |
100 |
0 |
10. When a scan was unsuccessful, did the test subject recognise this? |
- |
88a |
13a |
100b |
0b |
100c |
0c |
11. When a scan was unsuccessful, did the test subject know what the problem was? |
- |
25a |
75a |
60b |
40b |
0c |
100c |
aN = 8, bN = 5, cN = 2
Table 4. Table showing the VI test subject results for each task. The data are summarised as 'Yes' (indicating a majority of successes), 'No' (indicating a majority of failures) or 'Equal' (indicating an equal number of successes and failures).
Task |
N |
IBM |
Hamster |
Microsoft |
1. Automatically knew which finger motion to use |
28 |
No* |
Yes* |
Yes* |
2. Automatically knew how to interact with the reader |
28 |
No* |
Yes* |
Yes* |
3. Knew how to interact with reader after manufacturer's instructions were provided |
27 |
Equal |
Yes* |
Yes* |
4. Knew how to interact with reader after more detailed instructions were provided |
28 |
Yes* |
Yes |
Yes |
5. Success on first go of first scan |
28 |
No* |
Yes* |
Equal |
6. Success after manufacturer's instructions were provided |
28 |
No* |
Yes* |
Yes |
7. Success after more detailed instructions were provided |
28 |
Equal |
Yes* |
Yes* |
8. Automatically knew to provide more than one scan for enrolment procedure |
28 |
No* |
No* |
No* |
9. Once the procedure was understood, were the following scans successful? |
28 |
No* |
Yes* |
Yes* |
10. When a scan was unsuccessful, did the test subject recognise this? |
- |
No* |
No |
Equal |
11. When a scan was unsuccessful, did the test subject know what the problem was? |
- |
No* |
No |
No* |
* Statistically higher or lower than expected (expected = 50%)
Table 5. Table showing the good vision test subject results for each task. The data are summarised as 'Yes' (indicating a majority of successes), 'No' (indicating a majority of failures) or 'Equal' (indicating an equal number of successes and failures).
Task |
N |
IBM |
Hamster |
Microsoft |
1. Automatically knew which finger motion to use |
12 |
No |
Yes* |
Yes* |
2. Automatically knew how to interact with the reader |
12 |
No |
Yes* |
Yes* |
3. Knew how to interact with reader after manufacturer's instructions were provided |
12 |
Yes* |
Yes* |
Yes* |
4. Knew how to interact with reader after more detailed instructions were provided |
- |
- |
- |
- |
5. Success on first go of first scan |
12 |
No |
Equal |
Yes* |
6. Success after manufacturer's instructions were provided |
12 |
Equal |
Yes* |
Yes* |
7. Success after more detailed instructions were provided |
12 |
Equal |
Yes* |
Yes* |
8. Automatically knew to provide more than one scan for enrolment procedure |
12 |
No* |
No* |
No* |
9. Once the procedure was understood, were the following scans successful? |
12 |
Yes |
Yes* |
Yes* |
10. When a scan was unsuccessful, did the test subject recognise this? |
- |
Yes |
Yes |
Yes |
11. When a scan was unsuccessful, did the test subject know what the problem was? |
- |
No |
Equal |
No |
* Statistically higher or lower than expected (expected = 50%)
3.3. Evaluation of the instructions
A comparison of the success rates before and after instructions were provided can be seen in Tables 6 and 7.
Table 6: Results of pairwise analysis comparing the scan success rates before and after instructions were provided to the visually impaired test subjects.
Successive Tasks |
N |
IBM |
Hamster |
Microsoft |
% Yes |
% Yes |
% Yes |
||
Successful scan without instructions |
28 |
0a |
0c |
0e |
Successful scan after manufacturer's instructions read out |
28 |
21ab |
8cd |
68ef |
Successful scan after more detailed instructions provided |
28 |
65b |
100d |
100f |
aMcNemar Test: p < 0.05
bMcNemar Test: p < 0.01
cMcNemar Test: p < 0.001
dMcNemar Test: n.s.
eMcNemar Test: p < 0.001
fMcNemar Test: p < 0.01
Table 7: Results of pairwise analysis comparing the scan success rates before and after instructions were provided to the test subjects with good vision.
Successive Tasks |
N |
IBM |
Hamster |
Microsoft |
% Yes |
% Yes |
% Yes |
||
Successful scan without instructions |
12 |
0a |
0c |
0e |
Successful scan after manufacturer's instructions read out |
12 |
42ab |
83cd |
100e |
Successful scan after more detailed instructions provided |
12 |
67b |
100d |
100 |
aMcNemar Test: n.s.
bMcNemar Test: n.s.
cMcNemar Test: p < 0.01
dMcNemar Test: n.s.
eMcNemar Test: p < 0.001
< Contents page