How the Prosocial Design Network determines where interventions go: a flowchart

What determines where interventions live in the Prosocial Design Library?

With the number of behavioral and new media studies proliferating everyday, its critical for us to sort findings based on our confidence in them. To help, we've created this flowchart. It helps us decide where interventions belong within our Prosocial Design Library.

How should I use this flowchart?

Each question on the right column asks for a yes or no response. If the answer is no, it yields a citation grade on the right column. The further down the document, the more verified the document is deemed to be.

Let's start at the upper left:

1. Is this a specific design element for a prosocial outcome? Are you looking at something that is tangible and specific? If it's more of a general principle, or if it does not engender empathy, collaboration, and so forth—or the cessations of antisocial behaviors—in users, then it's not meant for the library. But if it is...‍
2. Do any documents directly support it? Are the documents intervention specific. If not, then are they about an analogous intervention (e.g. an experiment that was about an intervention in the real world that lacks a digital version) or about the broader context of the issue? If so, then it receives the grade of Inference.
   If neither of those are true, then it's not meant for the library.
   However, if it is intervention specific, then...‍
3. Are said intervention specific documents not only quantitative but also open with statistical methods & other supplemental materials? If the research is entirely qualitative, or if supplemental data is behind a non-disclosure agreement, or otherwise absent, then the intervention receives a grade of Emergent. However, if the supplementals are readily available, then not only are we entering the realm of Tested Interventions (congratulations!) but also...
4. Did the research use an experiment with randomized control groups? Was it an experiment on the intervention itself? If not, as may happen with a cohort study or other quasi experimental research, then the intervention receives a grade of Tentative. If, however it was an experiment, then...
5. Was the experiment conducted without any vested interests? If there's a conflict of interest, or if the research is of corporate origin, then it receives the grade of Tentative. If, however, the experiment is independently conducted, then...
6. Were the experiments findings peer reviewed? Were they published in a journal? If not, then by process of elimination it would be a preprint, and it receives the grade of Likely. If, however, it is peer reviewed, then...
7. Has someone independent of the first trial repeated the experiment? Did researchers unaffiliated with the original research repeat the experiment to see if it worked? If not, if the document stands alone and is peer reviewed, then it receives the penultimate grade of Convincing. If, however, it has been independently repeated, then...
8. Were the results replicated? If it was not independently replicated, then the grade is bumped down to Tentative. But should the results be replicated, then the intervention receives the highest grade that we can award: Validated. As of February 2021, no such intervention has received the Validated grade.

Are citations grades permanent?

No. We update citation grades according to new and emerging findings.

What occurs if multiple pieces of research conflict about an intervention?

Let's say that a Validated entry is later unable to be replicated, then said entry would—if the new research is found to be well executed—get bumped down to Tentative. If it is later replicated, and we conclude that the new findings are conducted with tighter controls, and with of higher accuracy and precision than the prior research, we would likely restore the intervention to Validated status.

If dozens upon dozens of studies were run on an intervention, with no clear pattern of replicability arising, then we would likely place said intervention as Tentative until further notice.

About the Prosocial Design Network

The Prosocial Design Network researches and promotes prosocial design: evidence-based design practices that bring out the best in human nature online. Learn more at prosocialdesign.org.

###