As shown above, there were three main reasons for rejection. First, some participants produced messages that were clearly not about healthy eating, but for example about physical exercise (noted in the table as ‘Different domain’). Second, there were messages where participants had not provided information in the format requested (for example, in Figure 1, the participant is asked to complete the phrase ‘the goal of the user is to’, and a participant may have written a full message instead of completing the phrase (this is noted in the table as ‘Not followed instructions’). Third, there were messages that were identical to the sample messages provided with the scheme (noted in the table as ‘Copied’ if they followed instructions, and ‘Copied and not followed instructions’ if, for example, they copied parts of the sample message as answers for the wrong question).
The table below shows the distribution of the number of messages produced with the 14 argumentation schemes used in the system. The ‘total approved’ is calculated by combining the ‘approved’ and ‘considered to be approved’ messages. The table does not include all rejected messages, as most were copied or a different domain (so, unrelated to difficulty with using a particular argumentation scheme, but rather to the instructions for the system as a whole), however, the number of cases where instructions were not followed may point towards a difficulty with a particular scheme. Overall, the proportion of messages for which people managed to follow the instructions of the argumentation schemes was 84% (86% if excluding copied messages). The proportion was worst for ‘Argument from memory with goal’, where it was 76%. Though the system was quite easy to use, the experimental setup was not clear enough with some participants copying the example message or producing messages which were not about healthy eating.