Don’t forget to collect free text answers in your customer experience survey

(Time: approx. 5 minutes read)

In our previous blog post we recommended that you start measuring your customer experience through two KPI’s; the Customer Satisfaction Score (CSAT) and the Customer Effort Score (CES). These two measurements will give you quantitative data that answer the questions HOW satisfied your customers are with a specific product, service or interaction and HOW much effort is required for them to get their job done, i.e. to get an issue resolved, a question answered, or a product purchased/returned.

However, adding a simple qualitative layer to your surveys will allow you to better understand WHY your customers feel as they do. Therefore, we recommend you include open-text questions in your customer experience surveys. Open-text questions are basically questions that enable the respondents to write out their response within a text box. This enables the customers to use their own voice instead of the company’s pre-written option responses.

For example, CSAT surveys are not limited to one single question. You can use multiple questions and combine open-ended and closed-ended questions in the same survey.

Examples of open-text questions:

  • “In your own words, describe how you feel about (insert company name or product here)”
  • “What’s working for you and why?”

To keep your survey questions as straightforward as possible, you might want to consider using just one word: “Why?”

We recommend that you make the open-ended questions optional since they take up more of the customers (and your) time. Thereby, customers who lack time or desire to leave a written response can still be part of the survey.

Last but not least, another way of showing your customers that you genuinely care about them is to include additional questions. These are questions unrelated to the survey questions such as for example:

  • “Do you have any questions for us?”
  • “What else would you like us to know?”

Analyzing free-text answers

So, you have done your homework and included open-ended questions to complement your survey. Now you are faced with perhaps hundreds or thousands of answers to summarize. How could that be managed? They key lies in grouping the answers into categories using what is called “coding”; assigning codes to the answers so the text can be analyzed just like numerical data.

Source: Thematic (2018)

First, decide if you should use manual or automated code. You can do automated coding using text analysis software. We will explain how you do manual coding, because even if you choose to use automated coding this knowledge will help you choose an effective approach. Word of caution, if you use open-ended questions and chose to analyze the answers manually, you need to be prepared to read all the answers.

5 steps to manual coding of free-text answers

1. Choose coding frame

If you choose manual coding, the first step is to choose between a flat and a hierarchical coding frame. Codes are put in something called a coding frame that represents structure and effects how useful the code results will be. The code frames can be either flat or hierarchical, the first being easier and faster to use and the second more powerful.

  • Flat (easier and faster to use): All codes have the same level and are equally important. But if you include a large number of categories it will become difficult to organize and navigate within it.
  • Hierarchical (more powerful): Include how the codes are related to each other and allows a higher level of granularity in the coding and analysis of the result. The hierarchical coding frame can support difference in sentiment, e.g. if the code in the top supports what the issue is about, the mid-level code can specify if it is a positive or a negative response and the last code can specify what made it positive or negative.

Example of a hierarchical coding frame:

Source: Thematic (2018)

2. Make sure your coding frames are flexible

Since coding manually takes a bit of time and therefore is costly, make sure your coding frame is flexible so it also can be used in later contexts. For example, your survey might have had the purpose of knowing how your customers experience your customer service, but the responses also include information about your products that can be used for another purpose.

3. Create high-quality codes

Three aspects to consider:

  1. Coverage: It is important to make sure that the codes cover all the wordings that have the same meaning, as respondents tend to use different words to describe the same thing. The coder needs a good understanding of the code’s coverage. For example, if the code is meant to cover the theme “clean” it should also cover words like “tidy” and “spotless” but also, expressions like “could eat of the floor”. You have to find the right balance between covering too much or too little. For example, using the code “product” might be too broad and the code “product stops working after using it three times” is probably too specific and will not cover enough responses.
  2. They capture positive and negative aspects separately. Make sure to have separate codes for positive and negative responses. For example, “useful product features” and “unnecessary product features” should be two different codes.
  3. They reduce the amount of data. The purpose of the codes is to reduce the total amount of data points in order to make the analysis useful. Therefore, avoid having too many codes or too few.

4. Choose between a deductive and an inductive approach

Deductive: In the deductive coding approach a pre-existing frame is used with predefined set of codes. It is an approach that intends to report back on specific questions. For example, if you already know that you are interested in “call waiting times” this can be one of your codes. The benefit of this approach is that it guarantees that you cover the items you are interested in. However, you need to be careful with bias, you might miss important themes with this approach.

Inductive: The inductive approach is iterative and includes sampling and re-coding. This approach is also called “grounded”, and basically means that you start from scratch and find codes from the sample responses and adjust them as you work your way through the responses. The process is as follows: 1. Read the sample 2. Create codes that cover the sample 3. Read the sample and apply the codes 4. Read a new sample and apply the codes again but note where they didn’t match 5. Create new codes 6. Go back and recode all the responses again 7. Repeat from step 4.

5. Ensure consistency

Whether or not you choose a deductive or inductive approach, it will be a challenge to ensure consistency. The coders state of mind and past experiences will affect the interpretation of the responses, and different people are likely to disagree on what the proper codes should be. Therefore, we recommend that you review how a change will affect all responses and log your line of thoughts connected to a code and change of code. A more time-consuming approach is to test the codes using either the same coder redoing the coding or another coder to remake it -and compare the results.

 

Leave a Reply

Your email address will not be published. Required fields are marked *