A Life of Brian comment that's been edited, and assigned a ribbon to show it's the top pick from a moderator.

Highlighting High Quality Comments

Improve the quality of users’ comments

Our confidence rating

Likely

Share This Intervention

What It Is

In a comments section of an online publication, editors select a high quality comment to be highlighted and acknowledged, for example as an "Editors' Pick".

Civic Signal Being Amplified

Understand
:
Promote thoughtful conversation

When To Use It

Interactive

What Is Its Intended Impact

By acknowledging a commenter for their high quality contribution - and in doing so, giving them a social reward - that commenter is incentivized to continue creating similar high quality comments. Editors may also signal to other users and commenters (whose comments were not highlighted) the types of comments that are valued and, subsequently, incentivize others to aim for similar levels of comment quality.

Evidence That It Works

Evidence That It Works

Wang & Diakopolous (2022) compared the subsequent comments of readers who received a NYTimes "Editor's Pick" with those of readers who wrote comments of similar quality (to the "pick"). From a natural experiment (so lacking a randomized control), the results are persuasive that recipients of "picks" continue to create higher quality content than they would otherwise. There was less evidence, however, that commenters who merely viewed a "pick" went on to create higher quality comments.

Why It Matters

Special Considerations

  • This approach might yield different results in communities where quality of responses is less valued; in other words, the effectiveness of this intervention might be mediated by the culture and values of a community. The existence of shared values might be a prerequisite for this intervention to work.
  • This intervention may have limited impact as there may be a ceiling to the number of commenters who can receive a pick; that number is constrained by the number of users who create adequately high comments as well as the threshold beyond which being "picked" is no longer a prized acknowledgement. In the Wang & Dialopoulos paper, for example, 10% of all commenters received a pick.
  • The highlighting effect is rather short-lived (i.e., lasting for around three subsequent comments) This is not only notable from a purely practical perspective but also because it might tell us something about how this intervention works. Discussions with researcher Yixue Wang suggest that this intervention is very externally rewarding focused. That is, the highlighting is less about internally stirring up people’s perception and feeling of individual social responsibility in a comment section. More so the observed short-term effect speaks to the interpretation that picked commentators want to be picked again and, therefore, produce high-quality content. This is important to note because external reward does not just fade quickly (in the study users lost interest in high-quality comments after having not received highlighting for their next 2 or 3 comments) but can also turn sour when commentators grow frustrated because they do not get highlighted again even though they invest more time in a comment (to make it high-quality).
  • Being transparent regarding the procedure by which a comment has been picked may be considered; in other contexts procedural transparency has been shown to promote prosocial behavior.

Examples

This intervention entry currently lacks photographic evidence (screencaps, &c.)

Citations

Highlighting High-quality Content as a Moderation Strategy: The Role of New York Times Picks in Comment Quality and Engagement

Yixue Wang, Nicholas Diakopoulos
ACM Transactions on Social Computing
January 29, 2022
10.1145/3484245

Is this intervention missing something?

You can help us! Click here to contribute a citation, example, or correction.

Further reading

Back to Prosocial Design Library