Evaluation Criteria/Measures

Given the little experience with HCI-type evaluations represented in the group, we have decided that this version of the benchmark will not include any such evaluation criteria, e.g., learning costs or usability by the disabled, which tend to be harder in defining precisely. We thus confined ourselves to two criteria, which are given below together with their corresponding quantitative measures.

Efficiency: This is measured by the following quantities:

• Number of errors performed by the user.
• Time spent by the user to perform the task.
• Time spent by the system to return the visual answer. (Note: This should be clarified more so that issues unrelated to visualization (like network costs) do not affect what is being measured, if so desired. On the other hand, if someone is simply interested to see if they will use a system or not, they should try it within the precise environment of interest and all such costs should be taken into account. For research prototypes, users should be warned that slow performance may be due to the exploratory nature of the tool and may go away with a more industrial-strength implementation.)

Quality of answer: This is measured by the following quantities:

• Some user-oriented version of ``precision''.
• Some user-oriented version of ``recall''.

  The use of the terms is not in the traditional way. The intention here is to capture the level of satisfaction expressed by users with respect to whether or not they found what they were looking for (assuming they know that), and level of user confidence that they have succeeded based on system feedback. Either the terms should be carefully defined, or some other terms should be used.