Publication #

On the Limited Impact of Visualizing Encryption: Perceptions of E2E Messaging Security
Christian Stransky, Dominik Wermke, Johanna Schrader, Nicolas Huaman, Yasemin Acar, Anna Lena Fehlhaber, Miranda Wei, Blase Ur and Sascha Fahl.
Seventeenth Symposium on Usable Privacy and Security (SOUPS 2021), August 8–10, 2021.

View PDF Cite Conference Page

Overview #

We conducted five user studies with a total of 683 participants on Amazon’s Mechanical Turk and Prolific. Each study had distinctive goals and were conducted sequentially to allow for the findings of preceding studies to inform the design of later studies.

Our goal was to investigate whether adding encryption visualizations to E2E-encrypted messaging app’s UI would increase perceptions of trust, security, and privacy without sacrificing usability.

Studies #

In our first study, we investigated why participants use particular messaging tools. Based on these initial findings, we aimed to improve the visibility of E2E encryption in a messaging app. Across the four subsequent studies, we compared six different text disclosures, three different icons, and three different animations of the encryption process.

Study 1: Tool Usage #

• Participants: 149
• Platform: Amazon Mechanical Turk
• Goal: Establish common communication tools for both day-to-day and confidential conversations.

We conducted this first study as a way to gain insight into the selection of communication tools for both day-to-day and confidential conversations. We wanted to learn which communication tools our participants use and prefer for everyday and confidential conversations, as well as to learn about the decisions they made when using specific communication tools for particular conversations.

We asked our participants about their current use of communication tools for day-to-day and confidential conversations, as well as decisions they make when they choose one of the tools they have available for communicating with a single person or with groups of people. The questionnaire consisted of both closed- and open-ended questions.

The top three reasons to use a certain communication tool for day-to-day conversation are ease of use (61.49%) followed by the availability of contacts in this tool (49.32%) and convenience (28.38%).

E-mail was an outlier as a preferred communication tool in many ways. Some participants (17.86% of e-mail users) prefer e-mail over other tools because they did not feel forced to reply to e-mails immediately. E-mail has a professional reputation as it is often used in the workplace, which 16.06% of e-mail users noted. For 26.79%of e-mail users, a key reason to use e-mail is the support for large attachments and long text.

We asked participants to elaborate on their preference for a specific tool for sensitive or confidential conversations and how they can tell that a specific tool keeps conversations confidential:

• 45.54% mentioned a gut instinct that leads to a security belief as their main reason to prefer a specific tool for confidential conversations, e. g. “I feel that it is safe.".
• 25.00% assumed a tool to be confidential when they send messages directly to their intended contact and had their own name and the name(s) of the communication partner(s) being shown in the user interface.
• 16.96% mentioned access control and strong passwords as reasons to prefer a particular tool: “I have a secure E-mail that is guarded by a good strong password."
• 26.79% assumed a tool to be acceptable for confidential conversations because they thought it uses some form of encryption. However, 14.29% made wrong assumptions and thought encryption was being deployed on unencrypted channels (e.g., for SMS/text messages).

Interestingly, only a few (8.04%) referenced “secret mode” or “secure chat” options in their decision. For a few (3.57%), visual indicators like colors or icons earned trust even if they did not directly relate to security or privacy, e.g. “If the message is blue it should be encrypted." (In the iOS messenger, a blue message indicates a message via iMessage, while a green message indicates that it was sent as a Text Message).

Based on these results, previous work, and the visual design of modern secure messaging apps, we implemented potential encryption visualizations in a modern secure messaging app.

Key insights: Tool Usage

• E-mail and SMS/text messages are the most popular tools for both day-to-day and confidential conversations.
• 53.69% of participants use a communication tool with E2E encryption enabled by default.
• 62.50% of participants who use E2E-encrypted tools prefer less secure alternatives for confidential conversations.
• Participants reported a gut instinct that made them believe a tool to be secure.

Study 2: Disclosures #

• Participants: 196
• Platform: Amazon Mechanical Turk
• Goal: Compare six textual framings (disclosures) in perception of trust, privacy, security, and usability.

Current secure messaging apps use specific textual framing (disclosures) to inform users that conversations are E2E-encrypted. Whats-App for example displays “Messages to this chat and calls are now secured with end-to-end encryption.". We comparatively tested six different versions in a between-group design:

We were specifically interested in the participants’opinions on usability and their perceptions of trust, security, and privacy. As a usability metric we compared the distribution of UMUX Lite answer categories between our conditions.

• For perceived usability, we could not determine apparent differences between the conditions, suggesting no observable effect (positive or negative).
• For participants' perception of trust, privacy, and security, we generated linear regression models based on combined score averaged from their answers to our set of 10 likert-item questions. The final model selected by best AIC shows that the “encrypted” condition has the best positive coefficient with about 0.7 score points compared to the [no text] control.

Participants seemed to prefer the encryption text, likely due to not fully understanding the term “end-to-end”, or regarding it as a subset (i.e., less secure) of being “just” encrypted. Based on these results, we proceeded with the “encrypted” text for our subsequent disclosures.

Key insights: Text Disclosures

• Participants felt most secure and private within the “encrypted” condition.
• The different disclosures did not have a significant impact on usability and satisfaction.

Study 3: Icons #

• Participants: 86
• Platform: Amazon Mechanical Turk
• Goal: Compare three different icons in perception of trust, privacy, security, and usability.

Aside from text disclosures, another common visualization is the use of security icons (e.g., lock symbols) to indicate the presence of encryption or other security mechanisms. Similar to Study 2, we based our analysis on current secure messaging apps' security icons and icons discussed in previous work. We investigated three different icons, studying their impact on perceived trust, security and privacy, and usability.

We found no significant differences across conditions. Our final linear regression model shows all three icon conditions performing worse than the baseline. In addition, the optional computer science education factor is significant with a negative coefficient in the model.

This is in line with previous work and additional evidence for a very limited effect of security icons on perceived trust, security, and privacy. We chose to proceed to the next study using the control (no icon) due to the negative coefficients of all other conditions.

Key insights: Icons

• We found a negative effect of security icons on perceived trust, security, and privacy compared to the baseline.
• Participants with a computer science background particularly disliked the security icons we investigated.
• The icons had no impact on usability and satisfaction.

Study 4: Animations #

• Participants: 107
• Platform: Amazon Mechanical Turk
• Goal: Compare three different animations in perception of trust, privacy, security, and usability.

We implemented and studied three animations of encryption. Prior work and the results of Study 1 implied that dynamic animations of the encryption process (e.g., disappearing messages or animations of plaintext turning into ciphertext) might increase perceptions of trust, security, and privacy.

Based on the results from the previous studies, each condition included one of the encryption anima-tions and a small text hint with the “Encrypted” disclosure.

We found no impact of the different animation conditions on usability. The final linear regression model includes a somewhat increased, but not significant, coefficient for the “Progress Circle Animation” compared to the baseline, and almost non-existent positive or negative effects for the other two animations.

Key insights: Animations

• No factors were significant, but the progress circle animation had a weak positive effect on perceived trust, security, and privacy.
• Security animations did not impact usability or satisfaction.

Study 5: Validation #

• Participants: 145
• Platform: Prolific
• Goal: Validate previous results with a different populations.

To validate and clarify the findings from the previous studies, we conducted a validation study on the Prolific platform. For this study, we added an additional control condition and a number of qualitative questions to further investigate previous results. To remove a potential confound suggested by the results of Study 2–4, we changed the messaging app from WhatsApp to a fictitious app we called Erebus.

Key insights: Validation

• Study 5 confirmed the findings of Study 2–4.
• Visualizing encryption in any way, even a basic text disclosure, improves perceptions compared to not mentioning it at all.
• Most participants saw the animations and felt they communicated “security”, yet this did not change their perceptions any more than a text disclosure did.
• An app’s reputation greatly impacts perceptions.

Considerations #

• Bias: Self-report studies may suffer from several biases, including over- and under-reporting, sampling bias, and social desirability bias. While we utilize self-report data, our central claims are not about the accuracy of respondents’ answers to a given question, but rather about whether and how responses from different conditions differ from each other. Consequently, the threats to validity caused by those biases should apply equally across all conditions.
• Platforms: Conducting user studies on Amazon MTurk and Prolific is a widely used and accepted procedure for this type of research. However, MTurkers are known to be younger and more tech-savvy than the average population. Additionally, our study focuses on the responses of U.S. Internet users, and thus, we can offer no insight into the generalizability of results for international participants.
• Variations: Studies 2–5 test a small set of different text disclosures, icons, and animations. While we based our designs on previous work and the results of our studies, we cannot guarantee that there are no other variations that perform better.
• Multiple Studies: Individual studies transpired in a somewhat isolated context, potentially missing certain effects of long-time exposure. We deliberately focused on multiple shorter studies, instead of one single in-depth, long-term study, to gather wider insights with different elements.

Summary #

We conducted five user studies with a total of 683 participants on Amazon’s Mechanical Turk and Prolific. In Study 1, we investigated why participants usage of messaging tools in everyday and sensitive context. In subsequent Study 2–5, we compared six text disclosures, three icons, and three animations of the encryption process. Study 5 was specifically designed to be a validation study with a separate participant pool.

• Impact of Text Disclosures: We were surprised to find that the simple “encrypt” disclosure outperformed most others (aside from the progress circle) in terms of perceived trust, security, and privacy. As expected, however, all disclosures performed better than the baseline of having no disclosure at all.
• Impact of Icons: We were also surprised to see that security icons had a negative effect, rather than increasing perceptions of trust, security, and privacy. This negative effect was particularly distinct for people with a computer science background.
• Impact of Animations: Previous work suggested that encryption visualizations might positively impact perceived trust, security, and privacy. Those suggestions were based primarily on qualitative data. Our studies, which combined quantitative and qualitative data, reached a somewhat different conclusions.
  • Based only on the qualitative data we collected, one might have reached conclusions similar to those of prior work. For example, nearly half of participants indicated that the animations of encryption indicate an increased level of security.
  • Based on our quantitative analyses, animations did not have a larger impact on perceptions of trust, security, and privacy of E2E-encrypted messaging tools than a basic text disclosure.

Our findings suggest that highlighting the use of encryption in basic ways (e.g., “Messages to this chat are now encrypted”) significantly increases perceived security, privacy, and trust in messaging applications.

While we had hypothesized that richly visualizing the process of encryption would emphasize E2E-encrypted messaging apps’ security properties, our results suggest that the existing practice of disclosing the use of encryption in a straightforward text disclosure may be sufficient if the text disclosure is displayed prominently.

Recommendations #

• Trust in Company: In the qualitative answers of tool usage and validation studies, participants reported that they trust the brand and that the company would keep their data secure. Tool providers could focus on generally improving trust in the brand.
• Convenience: Several participants mentioned using a specific app to communicate with their peer groups that decided onthat app. Introducing an app or feature that is not compatible with their peer groups leads to them switchingback to another channel. Tool providers could make sure that E2E encryption features do not lead to inconveniences for their users.
• Functionality: Our participants mentioned that they switched the tools when a messenger did not support a required feature, for example with large attachments that they send via e-mail. Making E2E encryption available in communication tools should not limit existing functionality.
• Mental Models: Our participants showed a number of incorrect mental models, most strikingly:
  • Around 25% of our participants assumed that their conversations are free of eavesdroppers if the user interface shows only the names of their intended communication partner(s), showing a lack of understanding of man-in-the-middle attacker capabilities.
  • Also, 14.29% of participants falsely assumed channels that are generally not encrypted by default (e.g., SMS) to be encrypted.

Cite This Work #

Feel free to cite this publication as:

@inproceedings{conf/soups/stransky21,
	title = {On the Limited Impact of Visualizing Encryption: Perceptions of E2E Messaging Security},
	author = {Christian Stransky and
		Dominik Wermke and
		Johanna Schrader and
		Nicolas Huaman and
		Yasemin Acar and
		Anna Lena Fehlhaber and
		Miranda Wei and
		Blase Ur and
		Sascha Fahl},
	booktitle = {Seventeenth Symposium on Usable Privacy and Security, SOUPS 2021, August 8-10, 2021},
	isbn = {978-1-939133-25-0},
	pages = {437--454},
	url = {https://www.usenix.org/conference/soups2021/presentation/stransky},
	publisher = {{USENIX} Association},
	month = {Aug},
	year = {2021},
}

Stransky et al. "On the Limited Impact of Visualizing Encryption: Perceptions of E2E Messaging Security." Proceedings of the Seventeenth Symposium on Usable Privacy and Security. 2021.