Quick disclaimer/TLDR.

This article is written for fairly technical people who love detail. If you don’t, here are a few key things you can take away:

  • Traditional survey text analysis is basically limited to word clouds (which were cool in 2001) and yield little to no actual insight from your hard earned opinion data.
  • If you think machine learning will fix it, you’re close, but survey text alone isn’t rich enough to yield a useful output.
  • Networked surveys are a new kind of survey that create way richer insights from your open-ended responses.
  • Networked surveys multiply text data per-respondent 27x on average and provide higher ROI on your panel investment.
  • Networked surveys automatically discover respondent personas based on their behavior and reactions (not just text).
  • Networked surveys mean change, and if you don’t like change, stick to what you know.
  • If you want to get started, or learn more about networked surveys in a more market-y way, click on this.

Alright, game on.

Survey text analysis is limited.

Are you a technical/analytical thinker? Do you run surveys often? If you are, and you do, then you probably already know where they are strong and where they have limits.

One of those limits is text analysis, and as markets move to contextual insights to comply with GDPR, this barrier can be paralyzing. At the center of the challenge is the very nature of survey text data collection.

You invest cash to get a representative sample of your market, collect answers to open-ended questions, and then invest again in time spent tagging each of the ~3,000 opinions you collected for keywords and themes. If you spend one minute tagging each opinion, that’s 50 billable hours ($7,500 @ $150/hr) spent on tagging alone. You spent all of that time tagging text and what you got back was… the number of times you tagged your text… the depth ends there.

Survey text machine learning is limited.

Now, you might say, ‘but what about natural language processing (NLP) and machine learning (ML) approaches to analyzing text for sentiment?’ The problem with survey opinion data is that it’s flat text, with no metadata. Because you’re feeding the machine learning algorithm short-length, flat text with no additional descriptors, there’s just not enough training data for the ML algorithm to yield useful or relevant insights.

If you’ve ever made use of these machine learning text analysis tools for surveys, you’ve probably already been disappointed by the obviousness of the conclusions they draw. You’ve likely also had to manually adjust outputs, because your human perspective is more relevant to the problem you’re solving. That human perspective is what makes networked surveys so powerful. But, before we get into what they are and why they’re useful, let’s travel back in time to simpler days.

Remember when search engines sucked?

If you’re ancient enough to remember Lycos, Ask Jeeves, and Yahoo! search, you remember that they were pretty basic. Put a keyword in the box, and if the keyword appeared most often in the text of a result, that result displayed highest for your search. That was it – flat text-level thinking. This led SEOs of yester-year to stuff keywords into content in hopes that it would rank higher, and it worked… This made for a frustrating search experience and was the de facto standard until Google entered the market with a simple and obvious discovery, links.

They realized that text alone wasn’t enough to determine that a result was relevant to your search. There was a trove of additional non-text human behavioral signals in the form of links between web content that formed a massive network. This network dramatically improved the quality of search results and set a new bar for what search engines needed to be.

Networked surveys go beyond text-level thinking.

Ok, back to 2018. Traditional survey text analysis is a lot like Lycos right now. You tag your text, and those tags that show up most often get ranked higher in your report (or worse, your word cloud, but don’t get me started there). That’s where your insight really ends – frequency – making your text analysis output about as high-quality as a circa-‘95 search result.

Now let’s talk about networked surveys. Networked surveys work exactly the same way as a traditional survey in terms of sampling and survey distribution. You log into the software, create a survey, get a link, and share it with your panel. They integrate with panel recruiters like Research Now/SSI, and panel market places like Luc.id. They support Google analytics URL tracking parameters for digital survey recruitment attribution. They support demographic exclusion rules to filter out bad sample fits and save you budget. All of the basics are covered, including basic survey question types (like Likert scales, multiple choice single-answer, multiple choice multiple-answer, open-ended, etc.). They also support a new question type: the networked question.

Networked questions spin-up miniature disposable social networks inside of your survey. Sounds badass, but what does it mean? More specifically, the simplest version of a networked question takes an open-ended text response, and lets other respondents react to it along a rating scale (from positive to negative, agree to disagree, not interesting to interesting, etc.).

Example of a networked survey in action:


These non-text human rating signals between opinions and respondents create a large network of opinions (the opinion network) and have a huge multiplying effect on text data. Now, in addition to each respondent’s open-ended text, you get on average an extra 27 open-ended data points with that respondent’s reaction to each on a scale.

Here’s an example of an opinion network (nodes are opinions, edges are shared respondents).

That many more qualitative data points per respondent means a significantly increased depth of knowledge when multiplied across your sample and a much bigger return on your survey investment. But the increased quantity of qualitative data isn’t the only advantage networked surveys give you. The extra data enables much richer discovery of patterns that only occur in the network of opinions that forms (e.g. Craig likes pizza, but not spicy pizza, but spicy tacos are yum).

Here’s a comparison between traditional and networked (you’ll want to scroll over it)

 

Networked surveys let you cluster your tags together to form respondent segments.

Now here’s something cool you can do with all of your tags. Because networked surveys track respondent scores for each opinion, and those scores can be aggregated by tags, we can ask questions like “which tags tend to be scored similarly by the same people?” If tags have high score similarity and respondent overlap, we group them together into segments and find useful patterns that are not text-dependent. For example, what do you think pool goers care about in parks? Take a minute.

Ok, times up, were canopies on the list? It might seem pretty obvious once you hear it and it makes sense, but, I know it at least wasn’t top of mind for me. Did you know people who care about fitness stations also care about the same things? Not to mention other seemingly unrelated things like “invasive control.”

Here’s an example of a single respondent segment (“the pool goer”). The bars represent levels of agreement with each tag/factor.

Networked surveys also measure complex behavior like persuasion.

More advanced versions of networked surveys can even measure persuasion. Take the Net Promoter Score for example. If you ask an NPS scale question, followed by an open-ended, networked surveys make it possible to segment customer loyalty feedback along an extra dimension that traditional surveys can’t access. You get to find out which opinions were exclusively those of your promoters (your promoter tribe), which were written by your promoters and got your detractors to agree with them (positive persuasive), which opinions were exclusively those of your detractors (your staunch opposition), and which were written by your detractors that got your promoters to agree with them (negative persuasive/leaks in your dam). This means you can tell which opinions persuade your market to (and not to) buy your product, recommend your brand, vote for your candidate, use your service, engage in your workplace, donate to your cause… etc.

Here’s an example of how networked surveys can be used to increase donations.

Networked surveys are actually pretty straight forward to learn and run with but they’re not for people who don’t like change. If you’re cool with word clouds and basic validation of your assumptions, there’s nothing wrong with a traditional survey. But, if you’re an innovator, and angling for a way to get past word clouds, you have little to lose and a lot (27x) to gain.

This has been fun! If you’re planning research for yourself or a client and want a trial, contact us below and we’ll help you figure out the right setup.

~ Alan