Personalising Messages to User Characteristics

Study 1: Message Validation in Relation to Cialdini’s Principles

Introduction

Food contributes to obesity and other serious health conditions. Ideally, to combat growing levels of obesity, people should receive individual support from a therapist to improve their eating habits, but this is hard to resource. Hence, there has been much work on developing digital behaviour change interventions to encourage healthy eating, which often take the form of a virtual healthy eating coach. This project investigated how to design such a coach which personalises user characteristics, such as personality.

A coach can use different persuasive message types. In this project, we investigated messages based on Cialdini’s principles of persuasion (see Table 1). Additionally, we investigated different message framings. For example, “Most people believe that eating a healthy breakfast contributes to a longer lifespan” and “Most people believe that eating an unhealthy breakfast contributes to a shorter lifespan” are messages of the same type but use positive and negative framing respectively. People’s characteristics such as personality are expected to influence the relative persuasiveness of messages. Therefore, this paper will also investigate the effect of personality on message persuasiveness and consider the interaction with type and framing.

Table 1

Objective

The research questions we address in this project are the following:

1. how do message types based on Cialdini’s principles impact on the perceived persuasiveness of messages?

2. how does framing impact on the perceived persuasiveness of messages based on Cialdini’s principles?

3. how does personality impact on the perceived persuasiveness of messages?

4. how do gender and age impact on the perceived persuasiveness of messages?

Work

Study 1: Message Validation in Relation to Cialdini’s Principles

We created 36 messages which instantiated each of the six principles of persuasion. For each principle, there were three message pairs; each pair consisted of one positively and one negatively framed variant of the same message content. In the study, we validated which messages could be reliably classified as adhering to one Cialdini principle. For our further studies, we needed both messages in the pair to be validated to investigate the impact of framing.

Participants. Participants were recruited through Amazon Mechanical Turk, a crowd-sourcing Internet marketplace. They needed to be in the US, possess a 90% acceptance rating (denotes that 90% of their work is acknowledged as good value), and pass a Cloze Test for English fluency. They were paid $1 for completing the study. 29 people participated: 19 males (4 aged 18–25; 8: 26–40; 7: 41–65); 9 females (5: 26–40; 4: 41–65); and 1 undisclosed (aged 26–40).

Procedure. Participants were introduced to the six Cialdini principles and their definitions. Next, the 36 messages were presented in random order. For each message, participants choose one of the six principles while viewing its definition, or ’other’ if they felt the message did not follow any of the principles. They were informed that there were no correct or incorrect answers.

Validation Measure. The Free-Marginal Kappa was utilised as a standard for demonstrating how effectively our messages were classified into the six principles. The Kappa measures the compliance between raters as follows: 1 denotes complete agreement, 0.7−1 exceptional agreement and 0.4−0.7 reasonable agreement. A message’s Kappa had to be greater than 0.4 for a reasonable classification.

Results: In this study, Kappa ≥ 0.4 was achieved by 29 out of 36 messages, of which 16 scored a Kappa ≥ 0.7. The messages with the highest Kappa for four principles will be used in follow-on studies, which also shows how many participants selected each principle for each message.

REC denotes Reciprocation, COM denotes Commitments and Consistency, CON denotes Consensus, LIK denotes Liking, AUT denotes Authority and SCA denotes Scarcity

Table 2

We decided to exclude reciprocity and scarcity from the follow-on studies. Only 2 reciprocity messages were validated with Kappa ≥ 0.4, and these were positive and negative framings of different message contents, making them hard to use for comparison in follow-up studies. On reflection, we also felt that reciprocity is hard to apply in a system, as it requires a plausible favour (a message that validated was “We have spent a lot of effort and money in organising this “How to Eat a Healthy Breakfast” workshop. We will be disheartened if you don’t eat fresh foods for breakfast.”). Whilst 4 scarcity messages were validated with the reasonable agreement (Kappa ≥ 0.4), none validated with Kappa ≥ 0.7. Additionally, scarcity is also hard to use by an interactive healthy eating coach in a persuasive message, as messages such as - This is your last chance to eat your cereal with fruit and nuts today and replenish your body with important nutrients, may not be plausible in real life1.

Study 2: Adaptive Message Selection

Our next study investigates the relationship between message properties (Cialdini principle and framing) and people’s personality on the one hand and message persuasiveness on the other. We used a mixture of a within and between study design. Each participant saw all messages, using the four Cialdini principles and the two framing types. Personality was used as a between subject variable.

Participants. Participants were recruited through Amazon Mechanical Turk. As before, they had to be based in the US, have an acceptance rate of 90%, and have passed a Cloze test for English fluency. Participants were paid $1.5. 152 participants took part: 89 males (22 aged 18–25; 48: 26–40; 19: 41–65); 62 females (8: 18–25; 35: 26–40; 18: 41–65; 1 over 65); and 1 undisclosed (aged 26–40).

Procedure. Participants provided their gender and age (which were optional). In addition, their attitude and behaviour towards food were examined by two questions. Next, we used a brief personality test applying personality sliders intended for the Five-Factor Model. The participants were displayed stories created by Dennis et al. (2012) that portray two extreme personalities (low and high) for each trait (see Figure 1). The participants specified how similar they were to these personalities by moving the slider. This produced a score for each trait out of 180. This personality measuring tool was previously validated as correlating with the 40-item mini markers test for the Five-Factor Model as previously used by Smith et al. (2016, 2019).

Figure 1

Next, participants imagined a fictional person ‘Alex’ who resembled their personality. They rated the 16 messages in random order on how persuasive they would be for Alex using 4 criteria: motivational, effectiveness, appropriateness, and convincing which were averaged for analysis (See Figure 2). They were told that the messages would be used to encourage Alex to eat healthy breakfasts.

Figure 2

Results

Participants’ Personality Distribution. The sample comprised of different ranges of traits. We divided the trait scores into high (above 90), and low (below 90) categories. The few who left the slider at the midpoint (90) were excluded from the between-subjects analysis but were included in the correlation analysis. This resulted in the following high/low totals for each personality trait: Extraversion: 68/82; Agreeableness: 113/38; Conscientiousness: 118/31; Emotional Stability: 99/51; and Openness: 107/39.

Table 3

Effect of Cialdini’s Principles on Perceived Persuasiveness. AUT was the highest-rated principle while LIK was the lowest. A one-way repeated measures ANOVA on principles showed a significant effect, F(2.76, 416.10) = 158.75, p < 0.001. AUT was significantly more persuasive than the other principles (p < 0.001) and LIK was significantly less persuasive (p < 0.001). There was no significant difference between COM and CON. This supports the hypothesis (H1) that perceived persuasiveness ratings differ for the different principles.

Effect of Framing on Perceived Persuasiveness. To test H2, we compared the persuasiveness of both framings of all the message pairs. On average, positively framed messages were rated higher (M = 3.03, SD = 1.15) than negatively framed ones (M = 2.56, SD = 1.17). This difference was significant, t(1215) = 17.78, two-tailed p < 0.001. This supports the hypothesis, i.e., H2 that peoples’ ratings of perceived persuasiveness differ for different framing.

Effect of Personality on Perceived Persuasiveness of Cialdini Principles. A one-way repeated measures ANOVA with principles as the within-subjects variable and personality traits as between-subjects variables showed that there is a significant effect of trait level for Conscientiousness (F(1, 147) = 6.73, p = 0.01), with higher persuasiveness for the high conscientious group than for the low conscientious group. There was no interaction effect with principles, though participants with high Conscientiousness seemed to rate COM messages higher. There were no significant main effects of trait level for the other traits. However, there was a trend for Openness (p = 0.07) and Extraversion (p = 0.06), with the low openness group and the high Extraversion group having higher persuasiveness ratings. There was a significant interaction effect of Cialdini’s principles with the Openness trait level on persuasiveness (F(2.79, 400.10), p = 0.047). Participants with low Openness rated both CON and LIK messages higher than participants with high Openness, as illustrated in Fig. 5(d). This provides some support for hypothesis H1a that perceived persuasiveness ratings for different principles vary depending on personality.

We also conducted a correlation analysis using the original numerical values for the traits (see Table 3). It shows many significant but small correlations between personality traits and Cialdini’s principles. This further supports H1a.

Table 4

For each trait, for the low and high values, a one-way repeated measures ANOVA showed a significant main effect of Cialdini’s principles (p < 0.001). Pairwise comparisons showed that AUT was significantly more persuasive than the other principles (p < 0.001) and LIK significantly less (p < 0.001). There was no difference between COM and CON. This indicates that the results of the analyses of the whole dataset (related to H1) are still valid independent of personality.

Effect of Personality on Perceived Persuasiveness of Framing. A one-way repeated measures ANOVA with framing as the within-subject variable and trait level as the between-subjects effects showed that there is a significant effect for Conscientiousness (F(1, 147) = 6.73, p = 0.01). Participants with high Conscientiousness rated both the framed messages much higher than those with low Conscientiousness. This is similar to what we found for Conscientiousness above, so highly conscientious people are more easily persuaded.

For both, low and high trait values, a t-test showed a significant effect of framing (two-tailed, p < 0.001). Participants tended to rate the persuasiveness of positively framed messages higher than negatively framed ones. This supports our earlier findings for H2.

It is interesting to note that high Emotional Stability participants rated negatively framed messages a little lower than low stability ones, but overall, both groups rated positively framed messages higher than negative ones.

There were significant interaction effects between personality and framing for the trait level of Conscientiousness (F(1, 147) = 5.30, p = 0.02) and Emotional Stability (F(1, 148) = 5.12, p = 0.03), see Fig. 6. There was a trend for participants with low Openness rating both positively and negatively framed messages higher than participants with high Openness, however, this was not significant. This supports hypothesis (H2a) that peoples’ ratings of perceived persuasiveness for framings vary depending on the personality of participants.

Interaction Effects between Personality, Cialdini’s Principles and Framing. All positively framed messages for all Cialdini’s principles were rated higher than the corresponding negatively framed ones. A two-way repeated measures ANOVA on Cialdini’s principles and framing showed that there is a significant interaction effect between Cialdini’s principles and framing, F(2.85, 429.91) = 4.24, p < 0.01. This supports hypothesis H3a.

A two-way repeated measures ANOVA on Cialdini’s principles and framings with personality traits as Between-Subjects Effects showed that there is a significant 3-way interaction effect between Emotional Stability, Cialdini’s principles and framing, F(3, 444) = 2.74, p = 0.04. Figure 5(c) shows hardly any difference in the messages rated by participants with high and low Emotional Stability on Cialdini’s principles, but the statistics indicate that there is a difference when framing is also taken into account. This supports hypothesis H3b.

Best message type. Authority (AUT)messages were favoured over Commitment and Consistency (COM), Consensus (CON) and Liking (LIK). A chi- square goodness-of-fit test showed there was a significant effect for the best message type, x² (2)= 132.74, p < 0.05. This supports hypothesis H4.

Best message framing. Positively framed messages were favoured over negatively framed ones. A chi-square goodness-of-fit test showed there was a significant effect for the best message type, x² (2)= 44.11, p < 0.05. This supports hypothesis H4.

Interaction between best message type and best message framing. A Fisher’s exact probability test resulted in a borderline significant relationship between best message type and the selection of framing, two-tailed Fisher’s exact yielded p = 0.05. This supports hypothesis H6.

Impact of personality on best message type and best message framing. A Fisher’s exact probability test resulted in a significant overall effect of personality on best message type for Agreeableness, two-tailed Fisher’s exact p < 0.05. This supports hypothesis H4.

Impact of gender on best message type and best message framing. A Fisher’s exact probability test resulted in no significant overall impact of gender on best message type, two-tailed Fisher’s exact p > 0.05. In addition, a chi-square test showed that males and females did not differ on preference for the best message framing, x² (2)= 1.71, p > 0.05. This does not support hypothesis H5.

Impact of age on best message type and best message framing. A Fisher’s exact probability test resulted in no significant overall impact of age on best message type, two-tailed Fisher’s exact p > 0.05. In addition, a chi-square test showed that males and females did not differ on preference for the best message positive or negative framing, x² (2)=0.76, p > 0.05. This does not support hypothesis H5.

Interaction between age and gender. A chi-square test resulted in a significant effect of gender on best message framing for the ages between 26 to 40, x² (2)= 6.97, p <0.05. This supports hypothesis H6a.

Interaction between personality and age. A Fisher’s exact probability test resulted in a significant effect of age on best message type for participants with high Extraversion (see figure 5) and high Openness, two-tailed Fisher’s exact p < 0.05. This supports hypothesis H6c.

Interaction between personality and gender. A Fisher’s exact probability test resulted in a significant effect of gender on the best message type for the highly agreeable participants, two-tailed Fisher’s exact p < 0.05 (see figure 4). While there was a significant effect of gender on best message framing for the low conscientious and low emotionally stable participants, two-tailed Fisher’s exact p < 0.05. This supports hypothesis H6b.

Interaction between personality, age, and gender. Overall, there were no significant effects between personality with best message type and best message framing for gender and age. This does not support hypothesis H6d.

Reflection

We created messages in the healthy eating domain for the persuasive principles of Reciprocity, Authority, Commitment and Consensus, Liking and Scarcity. On the whole, positively framed messages were favoured over negatively framed ones; while Authority messages were preferred to other message types. These messages may also be useful to other researchers. Overall, our work has applications in the area of virtual agents — by identifying the personality of a user they may be interacting with, such agents can tailor their persuasive techniques and messages to improve the outcomes of their interactions, and our work is a first step towards achieving this. We have provided some insights into the process for the selection of persuasive messages. These can then be incorporated into a heuristic to enable an intelligent agent system to deploy these adaptations. We can further develop this approach by considering the user’s previous records on healthy eating behaviour such as including healthy foods in their diets as well as existing attitudes. For example, a person who regularly eats three portions of fruits and vegetables a day may require a different message in comparison to one who regularly eats only one portion a day. Further work is also needed on testing persuasive messages in the real world, including the longitudinal effects on behaviour (i.e., actual rather than perceived persuasiveness) and the effect of message sequences.

Page updated

Google Sites

Report abuse