Examining pol.is as a Survey Methodology

This is a guest post for the pol.is blog from Lynette Shaw, PhD researcher in computational sociology. Her research interests include cognition and culture, economic sociology, social theory, computational modeling, complex systems and statistical methodologies.

Social research often involves a tension between richness in observations and the ability to systematically summarize and synthesize the information that has been collected. For instance, if a researcher wants to study the political views and attitudes of individuals regarding say, a refugee crisis, she is immediately faced with the issue of choosing how she will go about collecting that information from people.

Traditionally, there have been two general classes of approaches: “qualitative” and “quantitative.”

QUALITATIVE

Qualitative approaches include methodologies such as:

• interviews
• focus groups
• participant observation

These methods have the unique benefit of allowing researchers to gather data that is nuanced, rich, and surprising. The data can be gathered in a relatively organic fashion. There are a couple of significant downsides to using these tools:

• The amount of labor and time required to conduct these sorts of studies usually places a practical limit on the number of subjects who can be involved, which introduces the risk of overfitting our understandings on the viewpoints of just a few people.
• Not only is the synthesizing of qualitative data often a quite labor and time intensive in its own right, it is also a process that hinges upon the preconceptions and orientations of researchers to a degree that makes the interpretation of findings particularly vulnerable to researcher bias.

QUANTITATIVE

In historic opposition to qualitative approaches have been quantitative methods which rely on data gathered via standard social surveys. There are several advantages to quantitative methods:

• The relative ease with which surveys can be administered on a per person basis means that such studies can involve a much larger, and potentially much more statistically representative, group of participants than is usually feasible with qualitative approaches.
• There are a wide number of sophisticated statistical tools available to researchers that allow them to adeptly but easily investigate and summarize the results of such surveys.

These benefits come with their own costs too, however. From the outset, reliance on standard survey instruments requires researchers to translate their interests into a set of limited, narrow questions that can be answered via a choice from a provided set of simplified answers (e.g. “yes/no”, “Republican/Democrat/Other,” Likert scales, etc.).

Such necessary reductions inherently limits the depth of understanding that can be achieved via such instruments. Quantitative methods essentially force, from the outset, complex and nuanced social matters into flat, comparatively shallow statements about the world. The rigidity of standard surveys also entails that researchers are essentially stuck with the initial set of questions they thought of asking, even when in the course of their research they discover new, more important issues that should be captured but are not via their survey design.

In spite of past divisions between quantitatively and qualitatively oriented approaches, many contemporary researchers now acknowledge the need for “mixed-method” strategies for understanding social processes and phenomena.

It is in this arena that I believe Polis represents a truly innovative and potentially transformative new instrument for both private and academic social research.

BRIDGING THE DIVIDE

pol.is is capable of providing much of the same functionality to researchers that other standard online survey platform do via its ability to present specific ideas or questions to participants and record their agreement/disagreement with them.

Whereas this is the limit of other surveying approaches, however, it is only a starting point for pol.is.

Though researchers may begin with seeding a conversation with particular prompts and opinions related to said prompt, the capacity of pol.is to then go several steps further inviting participants to react with their own words and then allow other participants to respond and react to those reactions entails that the researchers will not be confined to capturing only the information they thought to collect from the group before they began their study.

This same functionality also allows researchers to, in real time, ask further follow up questions to the group on important but unexpected new themes that emerge from the conversation. In this respect, pol.is essentially allows researchers to have, in one tool, both the systematicness of a survey and the sorts of rich, organic observations that are usually found through focus groups or interviews.

Beyond this unusual capacity to bridge breadth and depth of individually produced information, pol.is is also able to provide a streamlined way for researchers to quickly look at and digest higher order information on the relationships between individuals and their responses to each other. Specifically, by using patterns of agreement/disagreement to map the space of collective thinking and opinion and then using the actual content of conversations to allow researchers to interpret that space, pol.is proves uncommonly objective in summarizing large amounts of rich social data quickly and easily.

Though this “out of the box” functionality does not represent anywhere near the extent of analysis that can potentially be done with the data pol.is captures, it is an unexpectedly sophisticated level of ready-made analysis.

These are a few of the most important potentials I see in pol.is with regard to advancing the state-of-the-art in social research, but I expect that there are probably still others to be discovered in practice. As with all new technologies, I also expect that there will be a minor amount of time and effort required train both participants and researchers in the best ways of using this novel instrument.

Nonetheless, with its unique ability to integrate many of the best aspects of quantitative and qualitative research into a single tool, pol.is is of significant enough interest to social researchers that such early adoption learning curves should be well worth undertaking.