Robert Kraut

Using Social Psychology to Motivate Contributions to Online Communities

Biography
Robert Kraut is Herbert A. Simon Professor of Human-Computer Interaction at Carnegie Mellon University. He received his Ph.D. in Social Psychology from Yale University in 1973, and has previously taught at the University of Pennsylvania and Cornell University. He was a research scientist at AT&T Bell Laboratories and Bell Communications Research for twelve years, and directed a group conducting research on interpersonal communication technologies. Dr. Kraut has broad interests in the design and social impact of computing and conducts research on everyday use of the Internet, technology and conversation, collaboration in small work groups, computing in organizations and contributions to online communities. His most recent work examines factors influencing the success of online communities and ways to apply psychology theory to their design. More information is available at http://www.cs.cmu.edu/~kraut.

Position Paper
Most social science researchers study online communities as natural phenomena (e.g., Baym, 1999), with the goal of understanding how the communities and the individuals within them behave. Yet online communities are socio-technical systems, and the behavior exhibited in them is strongly shaped by the engineering and design decisions on which they are based. Most recommendations for the design of online communities come from intuition and observations from history {Kim, 2000 #2296}. Although some scholars have identified social science theory as a basis for the design of online communities {e.g., Kollock, 1996 #2235; Preece, 2000 #2248}, the guidelines they develop tend to be broad and nonspecific. In this paper, we describe an attempt to root the design of online communities more firmly in social science theory {see Ling, In press #2340 for a fuller discussion }.

Under-contribution is a problem for many online communities. Although not everyone needs to contribute for a community to be successful, those with a large proportion of non-contributors have difficulty providing needed services to members. In this paper, we attempt to tackle the problem of under-contribution in an online community called MovieLens {Cosley, 2003 #2263}. MovieLens is a web-based movie recommender community where members can rate movies, write movie reviews, and receive recommendations for movies. Currently a large fraction of movies in MovieLens have too few ratings for the system to make valid recommendations for subscribers.

We have conducted six experiments in which we applied social science theory to the redesign of the MovieLens online community and empirically evaluated our success. We briefly describe the theories used and major results in the bullets below. The remainder of this white paper will reflect on the lessons we have drawn from attempting to apply theory to the design of online communities.

We have conducted the following experiments:

Similarity and uniqueness as a basis for conversation in online discussion groups {Ludford, 2004 #2334}. This experiment formed online discussion groups among moviegoers, manipulating whether groups consisted of people with similar or dissimilar tastes in movies and whether they were prompted to discuss issues in which their opinions and experiences were unique or ones in which they agreed with others in the group. Consistent with a collective effort model of contribution to groups {Karau, 1997 #661}, individuals posted more when they thought their ideas and experiences were unique. However, inconsistent with similarity models of attraction {Byrne, 1997 #2338}, individuals participated less when they were in groups that were similar to them, even though generally people like those most similar to themselves.

Uniqueness and benefits as a basis for increasing contributions to an online group {Beenen, 2004 #2260}. This experiment invited members of MovieLens to rate more movies, making salient who would benefit from their ratings (self versus others) and whether they were evaluating movies others could also evaluate or obscure movies. Consistent with the collective effort model {Karau, 1997 #661}, individuals rated more movies when they thought their movie tastes were unique. However, inconsistent with the collective effort model and other self-interest models of human behavior, individuals rated fewer when the invitation made salient the benefit that they personally or the community as a whole would receive from their ratings.

Making the social visible as a basis for increasing contributions to an online group {Cosley, 2004 #2342}. The collective effort model and other economically oriented theories of social behavior predict that people will contribute more to groups if the benefits of their contributions are salient to them. In this experiment, the MovieLens website was redesigned to make more visible the presence of others. However, this redesign did not increase contributions that members made to the MovieLens system.

Benefits and intrinsic interest as a basis for increasing contributions to an online group {Ling, In press #2340}. To follow up experiment 3, the invitation in this experiment made salient intrinsic motivations for rating movies (i.e., the enjoyment people receive from remembering and evaluating movies) or an extrinsic motive (the benefit that either others or the rater receives from ratings.) {Deci, 1999 #2294}. As in experiment 3, making salient extrinsic motives reduced ratings, but making salient the intrinsic motive neither increased ratings nor reduced the effects of the extrinsic motives.

Individual and group goals as a basis for increasing contributions to an online group {Beenen, 2004 #2260}. This experiment invited members of MovieLens to rate more movies, urging them to rate as many movies as possible or assigning them specific goals for the number of movies to rate. The goals were assigned either to each individual or to a nominal group. Consistent with goal-setting theory {Locke, 2002 #2241}, people rated more movies when given specific, challenging goals. However, inconsistent with the collective effort model {Karau, 1993 #2231}, they rated more movies when assigned the group goals than the individual ones.

Oversight to encourage quality of online contributions {Cosley, 2005 #2339}. Theories of deterrence {Gibbs, 1985 #2341} suggest that oversight promotes pro-social behavior, including the quality of contributions to an online group. In contrast, a derivation from the collective effort model {Karau, 1993 #2231} suggests that people might be less careful in their contribution if they believe that others will fix their mistakes. In this experiment, members of the MovieLens community entered information into a movie database, believing that their contributions would not be reviewed, would be reviewed by an expert, or would be reviewed by peers. Oversight increased both the quantity and quality of contributions while reducing antisocial behavior, and peers were as effective at oversight as experts.

We use three criteria for applying social psychology theory to design in order to evaluate these attempts to base the design of an online community on social psychology theory. First, does the theory generate design options that are not obvious from current design practice? Second, does applying the design options generated from theory lead to the desired communal outcomes? Third, does the theory specify how to craft and implement the design options to lead to the designed communication outcomes? For shorthand, we can label these criteria as inspiration, prediction, and implementation respectively. We discuss both failures of implementation and adequacies in underlying social science theory itself.

Inspiration. According to the inspiration criterion, our attempt to drive design from theory was successful in the sense that the theories led to design innovations that are rarely seen in existing online communities. For example, one key insight from the collective effort model is that people will be more likely to contribute to a group task if they think their contribution does not duplicate what others can provide and is thus needed for accomplishing the group’s goal. Many online communities provide feedback on the number or assessed quality of their contributions, like the “top reviewer” designations given to some contributors on the www.epinions.com website. However, we know of no online community that provides feedback to contributors about the uniqueness of their contributions. Similarly, the key insight from Locke’s theory of goal-setting is that people work hard to achieve specific, challenging goals.

Predication. According to the prediction criterion, our attempts to drive design from theory were partially successful. Applying some of the design principles inspired by theory led to increased contributions in MovieLens. Reminding people of their uniqueness caused them to post more messages in an online discussion (Experiment 1) and to rate more movies (Experiment 2). Without the collective effort model, it would not be obvious whether emphasizing uniqueness or commonality would be more effective. Experiment 5 showed that a simple email message assigning recipients specific ratings goals led recipients to rate more movies than a comparable message just urging them to rate more. Without the prior research on goal-setting, it would not be obvious whether a specific goal would be helpful, since it could plausibly discourage contributions above the goal.

However, not all the design ideas derived from the theories led to increased contributions. Results from Experiment 1 were inconsistent with a fundamental prediction from attraction theories in social psychology, that people will contribute more to a group they are similar to. Results from Experiment 5 were inconsistent with a fundamental prediction of the collective effort model, that people would exert less effort when they believed their output would be pooled rather than being individually identified. Finally, the prediction that people would contribute most when the beneficiaries of their contributions are salient was not supported in Experiments 2, 3 and 4.

Why did the design choices inspired by social psychology theories sometime fail to increase contribution? The inconsistencies with theory may have been partly a failure of prediction and partly a failure of implementation. Here we consider both classes of explanation.

Implementation. It is unlikely that the social psychology theories we used as the basis for design were fundamentally wrong, in the sense that they incorrectly explain people’s motivation in the domains in which they were developed. For example, similarity models of attraction, goal-setting theory and the collective effort model are robust theories of human motivation, consistent with and able to explain a wide variety of field and experimental data, as evidenced by recent literature reviews {Byrne, 1997 #2338; Locke, 2002 #2241; Karau, 1993 #2231}

It is possible, however, that some of our manipulations of group connection, uniqueness, and benefit may simply have been too weak to effectively motivate subscribers. Weak manipulations, however, cannot explain reliable disconfirmations of predictions derived from the theories. For example, the collective effort model predicts that people will contribute less when they think their contributions will be aggregated with those of others than when they think their contributions stand apart; but Experiment 5 appears to find the opposite. Similarly, a basic premise of the expectancy-value model, underlying both the collective effort model and goal-setting theory, is that people contribute to a group when they perceive they will personally benefit, either directly or indirectly, because their contributions are helping a group that they value. Yet results from Experiments 2 and 3 seem to show the opposite; participants contributed less when benefit either to the participant or to the group as a whole was made salient.

Although we may have started with correct and applicable theory, our implementations may have failed to capture the design principles appropriately. In Experiments 2, 4 and 5, in which we implemented the design principles extracted from theory as short, single electronic mail messages, the messages themselves may have been poorly worded. For example, the messages used to manipulate self- and other-benefit in Experiment 2 may have been too weak to manipulate perceptions of benefit, but instead may have undermined intrinsic motivation. The test of this idea in Experiment 4, however, did not support the explanation that mention of benefits undermined intrinsic motivation. Alternatively, these messages that were intended to make benefit salient may have stimulated feelings of psychological reactance (Brehm, 1966), which would lead people to try to assert their autonomy.

Incomplete theories. In some cases, the social psychological theories may simply not be up to the task when multiple features vary simultaneously, as they do in tests of real designs. Figure 1 illustrates one problem in attempting to use social psychological theory to drive design. Figure 1 is a variant of the familiar input-process-output model often used to analyze group behavior (McGrath, 1984; Hackman, 1987). The designer desires multiple outcomes from an online community. In the case of MovieLens, for example, its designers want the site to provide useful predictions for its users and therefore want subscribers to contribute ratings. They also want subscribers to have a satisfying experiment on each visit and to return often. Typically, these outcomes of groups—known generically as production, member support, and group maintenance (McGrath, 1984)—are only weakly correlated. To achieve these outcomes, the designer can modify features of the technology in an attempt to influence’s subscribers’ psychological states and group process. In the experiments described here, for example, email messages providing goals and emphasizing the uniqueness and benefits of contributions were designed to influence the recipients’ perceptions of the norms and benefits for contribution.

The problem is that each design feature can potentially influence multiple intervening psychological states and group processes. Each of these intervening states and processes, in turn, can have multiple determinants. Finally, each intervening psychological state and group process can have multiple and potentially inconsistent effects on the outcomes the designer wants to influence. Concretely, it is possible, for example, that participants perceive benefit indicators as attempts to manipulate their behavior. Therefore, this design feature could influence their perceptions that their contributions were valuable, increasing willingness to contribute, but also influence their perception that they are being manipulated. In the latter case, they may refrain from contributing to reassert their autonomy. Similarly, assigning participants a group goal may have made them think they were redundant, thus reducing their perceptions of the value of their contribution. Simultaneously, though, the group assignment may have increased participants’ attachment to MovieLens overall, thereby increasing their motivation to help their group.

Figure 1: Multiply Determined Outcomes

In summary, design features have multiple consequences, and intervening states and behavioral outcomes are multiply determined. These are general phenomena. They apply to the experimental manipulations deployed in the experiments reported here, but more generally whenever one tries to leverage social science theory as a basis for design. For example, there is abundant evidence that conversation is one mechanism for getting people to like each other, a route by which people develop attachment to the community as a whole (Sassenberg, 2002) and a mechanism to increase contributions to public goods. On the other hand, members can develop attachment to a group because of their commitment to what the group stands for, without liking or even knowing particular members. This attachment to particular members and to the group as a whole should encourage members to contribute to the group. However, conversation often frequently leads to cognitive overload, which in turn can drive people from the group {Jones, 2004 #2315}.

The failure of social psychology to produce theory that is sufficiently complete for the purposes of design stems from fundamental differences in the goals and values of social psychologists and of HCI and CSCW researchers. Social psychology is primarily a behavioral science, whose goal is to unambiguously determine the causes for and explanations of social phenomena. The conventional analytic approach is to examine the influence of a small number of variables through measurement or experimental manipulation, while holding other variables constant through statistical techniques or experimental control. In addition, a value in the social science tradition is to produce theory that is as general as possible. While social psychologists recognize the importance of context, they attempt to create theories that abstract across “irrelevant” differences in context. For example, if they were interested in the degree to which group members’ liking for each other influenced their willingness to contribute to the group, they would prefer a theory of contribution that allowed them to generalize across different bases for liking—e.g., similarity, familiarity, attractiveness, or reciprocity—unless data said these bases caused a difference in the amount or type of contribution. A consequence of this scientific orientation is that many social psychological theories are sparse, describing the relationships among a small set of variables.

In contrast, HCI and CSCW are primarily engineering disciplines, where the primary goal is problem-solving. In trying to build an online community where members liked each other, for example, the bases of liking, which might be irrelevant to a psychological theory of contribution, would be very relevant to the design of the online community. The designer would want to know which source of likely is most powerful, which sources are compatible to each other, and how they hold up over time. The designer would also want to know about the interaction among variables implicated in different theories. For example, does uniqueness, which the collective effort model suggests can be a basis of contribution, conflict with perceived similarity to other group members, which can lead to liking for the group, which is another basis of contribution? Because psychologists have not set themselves the task of designing the optimum configuration of features for contribution, they provide little guidance about which effect will predominate when a single design feature can have opposite effects.

The net result is that designers get insufficient guidance from social science theory, because the theory isn’t generally complete enough. This lack of detail forces the designer to improvise when attempting to apply social psychological knowledge to solve design problems. While social science theory can inspire design by suggesting options, it does not eliminate the need for the creativity that designers bring to the process.

Back to Social Computing Symposium 2005