I'm @PixieJenni. As you may know I've been being vocally critical of #GamerGate on twitter these past few days, but I hope the fact I've also criticised people on the other 'side' will mean you give me the benefit of the doubt here. What is GamerGate? GamerGate is an internet-based movement that claims to be about protecting ethics in gaming journalism, but in reality the entire thing started when Eron Gjoni got dumped and decided to slander his ex-girlfriend on a blog.
© 2021 Valve Corporation. All rights reserved. All trademarks are property of their respective owners in the US and other countries. Welcome to the latest internet controversy, starring misogynists, high-profile video game websites, and female gamers. Imagine for a moment that you are a gamer. You love video games (not.
With the possibility of running a mobile survey within your app, you can take your product or its development process to a new level, building on a stable foundation of user insights. As a smart product manager or developer, you should cherish a chance to delve deep into the customer feedback.
Mobile app development is a tricky process. It is costly and time-consuming and requires proper planning, testing, and marketing strategies. Hours poured in the development and testing of an app are always going to be insufficient, and at early stages, updates and fixes are going to be rolled frequently.
Even the most extensive testing will often not account for the end-users older device, compatibility with other systems and permissions, or just plain system errors. And when that negative feedback from users will eventually reach you, it will do so in the most natural way for the user, but the most dreadful for the company: Appstore ratings.
There is a looming gap between users and developers that stems from the very nature of how mobile apps work and are distributed. Users will tend to look at an app as a unit, far from realizing you can reach the developers by email, uninstalling an app and finding a better one is the quicker option.
And they just might, taking into account that over 2000 apps are added every day to the google play store, and almost twice as much to the iOS app store. Your potential competitors are likely to pop up quickly. To indeed keep ahead of them and build a reliable app, you have to look closely to the user feedback.
In this guide, you will learn:
Benefits of running surveys in a mobile application
For starters, they can be indispensable for product managers in the early stages of a mobile app’s life. Siphoning the users’ feedback can speed up the development process, helping you locate potential issues and making improvements based on the insights provided.
Additionally, a product roadmap becomes more evident once you can tap into the user input.
Once the app is defined, stable, and running, you can use surveys to keep the service healthy and your app ratings high.
Giving your users a possibility to voice their negative opinions lets you further act upon it, locate issues early and keep your users happy.
You can also use surveys as additional functionality of your app. A system of ratings, open-ended questions, and multiple-choice surveys can be actually disguised as features.
What kind of feedback you can collect with mobile app surveys
Improve the product development process with Surveys
If your app is early in development than using well-targeted surveys can be very beneficial. You can ask your users if everything works as intended or if they met any issues. Likewise, you can ask about the app design or navigation. You can target the surveys to appear at those parts of your app you feel are the most vulnerable, or to ask a more general question after some time has passed or after the user used a function a specific number of times. You can target your surveys to appear individually on particular functions you are still developing.
You should also run a survey after major updates. Again, give your users a couple of days with the new version before you ask them to voice their opinion. Maybe you need testers before another update goes live? You can survey them in the current version with a contact form survey.
Free-to-Use Voice of Customer Survey Template
No Credit Card Required • Cancel Any Time
Prioritize your product roadmap
So you used surveys to understand how users feel about your current functions. Now use surveys further to scout for additional features. Don’t be afraid of asking your users directly in which direction they would rather you take your application. Ask an open-ended question to some of them after some time has passed, asking them what additional feature should be added. Once you collect a bunch of responses, select the few most popular and run another survey, this time a single answer one, asking the users to choose one of the selected features. With only two surveys you now have confirmation from your users about the future of your app they want to see.
Ask promoters to leave reviews
There are some types of surveys any application can benefit from. One of the most common uses you probably have seen yourself would be the “Do you like the app?” question, that follows with a request to rate it in the app store. This can be useful if that rating is something important to you though it should not be crucial.
Still, if you have a quota of ratings to meet, I advise you first to run a Net Promoter Score survey asking the user if he would recommend the app. You can check out our guide on measuring Net Promoter Score. Follow up with a call to action to those respondents who give you high scores. Ask them politely if they would not mind rating your app.
Free-to-Use Net Promoter Score Survey Template
No Credit Card Required • Cancel Any Time
Catch negative response before reviews come
Those who rate you poorly in the app will obviously not rate you well in the app store. Instead of following up with a request to rate your app, ask them instead to leave their constructive feedback. “What can we do improve the app for you?” It’s a general good practice in surveying – when you get negative feedback don’t be scared to ask the respondent to elaborate.
Use Customer-oriented surveys for more than just ratings.
If your users can make purchases through the app, you can run a customer satisfaction or Net Promoter Score survey regarding the purchasing process. Just make sure the purchase process is completed first to keep the validity of your survey results. If your customer made their first purchase through your app, ask them to rate the experience after a week. Remember to include an open-ended question at the end to make that feedback qualitative.
Free-to-Use Post-Purchase Survey Template
No Credit Card Required • Cancel Any Time
Surveys can be used as additional functionality
We can always take that technical feedback a step further – surveys can enhance your help section. Our tip is to place an open-ended survey at the bottom of the section asking if there is any additional help that the user did not find in the app. Maybe a function you have can be further enhanced with a rating system? Or, you can use a contact form survey for those moments when you need your users to be contacted? Remember that a good surveying tool is, in essence, a channel of communication, you can use it for more than just statistics.
Mobile app survey questions
In the early days of your app’s life, a series of well-targeted mobile surveys asking users questions can be extremely beneficial to your time and resources put into your app development.
Knowing what features and elements need improvement or do not work correactly, or understanding if a feature is obsolete to your users is an obvious benefit. We can divide these questions into four groups.
- Early development technical
- Early development features scouting
- Prioritizing mobile app features
- To keep app-store ratings high
Early development technical survey questions
#1. How does the app run after the update?
You may have tested extensively before rolling a new update, but the user might just have a unique issue you were not able to test for. Or they run a system you didn’t have time to test.
Use additional tools to filter responses based on the device used to locate why the update might not work as expected.
#2. How do you like the app design?
Your users’ esthetic preferences might be a subjective matter… Unless they form the majority. Your company’s style and design might not exactly fit your user base.
If you have doubts about the design then you should better find out early and avoid chasing those esthetic-savvy users away.
Early development features scouting survey questions
#3. Can you describe a situation in which our app is most useful?
Get an idea for that perfect scenario between your app and your users. It might differ from your initial projections but it is the ideal situation you should strive for.
This question will give you insights into both where you should focus your resources but also what functions you should be communicating more clearly.
#4. Is our app helping you achieve your goals?
If your app is a tool that is getting a job done for your user then it is definitely on the right path. If your app is helping your user solve a problem then you are a step closer to building a lasting relationship with them.
Identifying your users’ goals is crucial so follow-up the positive answer with the question below:
#5. What goals are we helping you achieve?
This follow-up will help you identify the job and the user who is aiming to perform it. You may be surprised at what goals your users actually have, and what they use your app for.
Knowing this will give you a chance to target new user groups you didn’t think to target before.
#6. Are there any functions you would like us to add?
If your user has an idea of a function that would help them achieve their tasks, all they need is a chance to voice it. Don’t be afraid to ask your users what function they see as a perfect fit for your app.
#7. How would you rate this new feature?
Once you implement what your users are asking for, make sure it is indeed working as intended and meeting your users’ expectations.
Use this question in the early days after releasing a new feature. Should you get negative responses – act quickly.
Prioritizing mobile app features survey questions
#8. How often do you use the following features?
This will give you an idea of what feature is the most popular or most often used. Identifying the most useful feature for your users can be extremely beneficial.
Always make sure this feature works properly and is readily accessible. To get a wider overview of how your users feel about the feature of your app, follow up with the question below:
#9. Which of the following features do you use least?
You might be spending your resources on a feature that is not in fact significant. Once you get an idea of your least popular features, you will be able to decide if they are indeed necessary for your app, or if your resources would better be spent on a different one entirely.
#10. Should this feature be above (on top)?
The users might want access to a feature to be quicker, even if it is not the most popular function. Compare your answers to this question and design your user experience and menus according to the user input.
To keep app-store ratings high survey questions
App-store ratings are tricky. For starters, they are not entirely reliable as a review platform as people rate apps for non-obvious reasons. Negative experiences with an app tend to attract more ratings than positive ones. On the other hand, positive ratings are often not very objective.
Nonetheless, ratings play an essential part in the decision-making process for a potential user. To encourage your users to give positive reviews, and to protect your ratings from negative feedback, we advise you to use the following set of questions in the order specified below. Start from the first one:
#11. How would you rate our app?
If you released your app recently and you don’t have an estimation on how well it will be rated, run this survey and target users who have spent at least a couple of hours in your app.
The answers can give you a general idea about the app store rating you can expect from users. In order to keep your ratings protected, wait for some more time and survey the user with the question below:
#12. Would you recommend this app to your friends?
Yes, it is crucial for it to be an NPS. This type of survey gives you a rating from 1 to 10. Everything up to 6 is a detractor, 7 and 8 is neutral but still worth querying for more feedback.
People who answer 9 and 10 are your promoters. So, to those promoters follow up with a question and call to action:
Free-to-Use Survey Template
No Credit Card Required • Cancel Any Time
#13. Great! Would you like to give us a Review?
A positive answer to this question leads the user to the app store to give you a rating. Note that you should only show this question to promoters.
To keep your ratings high you want to avoid detractors and neutrals giving the rating. But just avoiding them is a terrible and dishonest practice. Show the following question to the neutrals instead:
#14. What is the reason for your score?
This one should be asked to the neutrals, the ones that answered 7 or 8. This should give you an idea on how to convert those neutrals to promoters in the future. As for the detractors, phrase the follow-up question like this:
#15. What can we do to improve?
Never be scared to collect that negative feedback. This question not only shows your users that you care about them, but it will also give you the most actionable feedback possible – direct instruction on how you can keep your users satisfied.
How to use Survey channel properly
One way to bridge the user-developer communication gap is to gather user feedback with targeted surveys. Unfortunately, the only touchpoint you can often reach them through is within the app itself. Redirecting them out of your app will hurt your response rates significantly.
Running your surveys entirely within your mobile application addresses this challenge directly. This is what mobile app survey tools have been created for. Additionally, collecting responses through the app itself allows for some exciting targeting options.
Several points to address here but let’s ask a simple question first. Why shouldn’t you link to a survey on your website? How significant is the potential damage of a survey taking your users away from the app?
Response rates suffer significantly every time you have to redirect your respondent. If you put yourself in their shoes, you have just been transported out of the app you are using into a browser window. You are essentially forcing them to take an extra step. Once you pull your user out of your app, you have no guarantee they will come back to it.
Mobile means a shorter attention span. Your respondents naturally have a much smaller tolerance towards time-wasting. This is often disregarded, but it is a crucial factor you cannot ignore. Wasting time is almost a literal pain to an app user. Valuable seconds are consumed for a browser window to open, a page to load, and even if the user intends to skip it, they now have to navigate back to the app. If you waste their time, they will feel it under their skin and swipe out of your survey, possibly uninstalling your app shortly after.
If they need your app or it is crucial to their life or work – you will only annoy them by making their routine actions randomly longer.
Since your users are right within your application, you target using a slightly different logic than with website surveys. If you run a business and a website at the same time, think if your survey needs to be in the app itself. If the survey is not app related then maybe it is better to run it where you have the most traffic. So, adjust the survey to your needs and situation. Mobile app surveys are best used to gather feedback relating to the app itself, its design and its functions. It is indispensable in the developing process, it will help you locate issues or design flaws.
If you plan to run standard Net Promoter Score surveys or Customer Satisfaction surveys than you have to target them with the app in mind. Since you will be showing the survey in the app, you have to be careful about not interrupting your customers’ experience. App surveys are, however, significantly more useful to gather qualitative feedback.
Just as with adequately targeted website surveys the idea is to reduce friction and improve ease of use, help the validity of your results by targeting the surveys well and keeping everything concise and in one place.
Now both that qualitative and quantitative insight can be further enhanced with proper targeting and proper questions. Just as website surveys are most effective when they fully take into account the platform, time and place they appear, so can mobile app surveys provide you with the most tangible and instantly usable information.
According to research, an average person spends a total of 4 hours a day completely on their smartphone, 90% of which is using mobile apps. You can capitalize on this by keeping the survey completely attached to the user experience.
How to enhance mobile App’s functionality
A good mobile SDK for surveying will provide you with a range of tools that you can actually use as additional functionality for your mobile application.
A survey tool is, in essence, a channel of communication and you can use it for more than just statistics.
Here are some examples:
Contact forms
If your tool has a contact form survey than you just got a free extension for lead hunting. You can also use contact forms to scout for beta testers or those loyal users who are willing to be of more assistance to you and your product.
How about using contact forms for sales or invoice billing? You can generally save some time and space in your app this way.
Ratings
Be it a star rating, smiley scale or an X out of 10 surveys, you don’t have to limit yourself to NPS, Customer satisfaction or service ratings. You can use ratings as a function of your actual application’s job.
If your app is a social hub of sorts the users can use these rating systems in relation to each other, processes between them, submissions, actions, or services. This section is only limited by your imagination and ingenuity.
Open-ended questions
Open-ended questions can be used in an endless amount of ways limited only by your creativity. Maybe you are collecting love letters from users?
Do so in an open-ended survey disguised as a message. Are you worried your help section might be missing something?
Place an open-ended survey at the end asking the user needs any other assistance and if they found what they were looking for.
Maybe you didn’t create a commenting system in time for the launch. Just use open-ended surveys in the meantime.
Conclusion
As a product manager, or working closely with your product team, establish exactly what role your app has and what goals it aims to achieve.
Then compare how it is used currently and how well it reaches those goals. Surveys should be used to bridge the gap between one and the other.
Wherever your analytical and benchmarking tools cannot provide, surveys can – both qualitatively and quantitatively. So don’t stop at using ready-made mobile app survey questions, like the ones provided by our free to use survey template library.
Survey researchers employ a variety of techniques in the collection of survey data. People can be contacted and surveyed using several different modes: by an interviewer in-person or on the telephone (either a landline or cellphone), via the internet or by paper questionnaires (delivered in person or in the mail).
The choice of mode can affect who can be interviewed in the survey, the availability of an effective way to sample people in the population, how people can be contacted and selected to be respondents, and who responds to the survey. In addition, factors related to the mode, such as the presence of an interviewer and whether information is communicated aurally or visually, can influence how people respond. Surveyors are increasingly conducting mixed-mode surveys where respondents are contacted and interviewed using a variety of modes.
Survey response rates can vary for each mode and are affected by aspects of the survey design (e.g., number of calls/contacts, length of field period, use of incentives, survey length, etc.). In recent years surveyors have been faced with declining response rates for most surveys, which we discuss in more detail in the section on the problem of declining response rates.
In addition to landline and cellphone surveys, Pew Research Center also conducts web surveys and mixed-mode surveys, where people can be surveyed by more than one mode. We discuss these types of surveys in the following sections and provide examples from polls that used each method. In addition, some of our surveys involve reinterviewing people we have previously surveyed to see if their attitudes or behaviors have changed. For example, in presidential election years we often interview voters, who were first surveyed earlier in the fall, again after the election in order to understand how their opinions may have changed from when they were interviewed previously.
Cellphone surveys
Telephone surveys have traditionally been conducted only by landline telephone. However, now that almost half of Americans have a cellphone but no landline telephone service, more surveys are including interviews with people on their cellphones. For certain subgroups, such as young adults, Hispanics and African Americans, the cell only rate is even higher. Research has shown that as the number of adults who are cell only has grown, the potential for bias in landline surveys that do not include cellphone interviews is growing.
Cellphone surveys are conducted in conjunction with a landline survey to improve coverage. The data are then combined for analysis. In addition to the issues associated with sampling cellphones, there are also unique challenges that arise when interviewing people on their cellphones.
One of the most important considerations when conducting cellphone surveys is that the costs are substantially higher than for a traditional landline survey. The cost of a completed cellphone interview is one-and-a-half to two times more than a completed landline interview. Although some of the fixed costs associated with landline surveys are not duplicated when a cellphone sample is added (such as programming the questionnaire), other costs are higher (data processing and weighting are more complex in dual-frame surveys).
Cellphone surveys are more expensive because of the additional effort needed to screen for eligible respondents. A significant number of people reached on a cellphone are under the age of 18 and thus are not eligible for most of our surveys of adults. Cellphone surveys also cost more because federal regulations require cellphone numbers to be dialed manually (whereas auto-dialers can be used to dial landline numbers before calls are transferred to interviewers). In addition, respondents (including those to Pew Research surveys) are often offered small cash reimbursements to help offset any costs they might incur for completing the survey on their cellphone. These payments, as well as the additional time necessary for interviewers to collect contact information in order to reimburse respondents, add to the cost of conducting cellphone surveys.
Most cellphones also have caller identification or other screening devices that allow people to see the number that is calling before deciding to answer. People also differ considerably in how they use their cellphones (e.g., whether they are turned on all the time or used only during work hours or for emergencies). The respondents’ environment also can have a greater influence on cellphone surveys. Although people responding to landline surveys are generally at home, cellphone respondents can be virtually anywhere when receiving the call. Legal restrictions on the use of cellphones while driving, as well as concerns about safety, also have raised the issue of whether people should be responding to surveys on their cellphones while driving. In addition, people often talk on their cellphones in more open places where they may have less privacy; this may affect how they respond to survey questions, especially those that cover more sensitive topics. These concerns have led some surveyors (including Pew Research Center) to ask cellphone respondents whether they are in a safe place and whether they can speak freely before continuing with the interview. Lastly, the quality of connection may influence whether an interview can be completed at that time, and interruptions may be more common on cellphones.
Response rates are typically lower for cellphone surveys than for landline surveys. In terms of data quality, some researchers have suggested that respondents may be more distracted during a cellphone interview, but our research has not found substantive differences in the quality of responses between landline and cellphone interviews. Interviewer ratings of respondent cooperation and levels of distraction have been similar in the cell and landline samples, with cellphone respondents sometimes demonstrating even slightly greater cooperation and less distraction than landline respondents.
Related publications
- The Growing Gap between Landline and Dual Frame Election Polls: Republican Vote Share Bigger in Landline-Only Surveys Nov. 22, 2010
- Cell Phones and Election Polls: An Update Oct. 13, 2010
- Assessing the Cell Phone Challenge May 20, 2010
- Accurately Locating Where Wireless Respondents Live Requires More Than a Phone Number July 9, 2009
- Calling Cell Phones in ’08 Pre-Election Polls Dec. 18, 2008
- Cell Phones and the 2008 Vote: An Update Sept. 23, 2008
- Cell Phones and the 2008 Vote: An Update July 17, 2008
- Research Roundup: Latest Findings on Cell Phones and Polling May 22, 2008
- The Impact of “Cell-Onlys” On Public Opinion Polling Jan. 31, 2008
- How Serious is Polling’s Cell-Only Problem? June 20, 2007
- The Cell Phone Challenge to Survey Research May 15, 2006
Internet surveys
The number of surveys being conducted over the internet has increased dramatically in the last 10 years, driven by a dramatic rise in internet penetration and the relatively low cost of conducting web surveys in comparison with other methods. Web surveys have a number of advantages over other modes of interview. They are convenient for respondents to take on their own time and at their own pace. The lack of an interviewer means web surveys suffer from less social desirability bias than interviewer-administered modes. And web surveys also allow researchers to use a host of multimedia elements, such as having respondents view videos or listen to audio clips, which are not available to other survey modes.
Although more surveys are being conducted via the web, internet surveys are not without their drawbacks. Surveys of the general population that rely only on the internet can be subject to significant biases resulting from undercoverage and nonresponse. Not everyone in the U.S. has access to the internet, and there are significant demographic differences between those who do have access and those who do not. People with lower incomes, less education, living in rural areas or ages 65 and older are underrepresented among internet users and those with high-speed internet access (see our internet research for the latest trends).
There also is no systematic way to collect a traditional probability sample of the general population using the internet. There is no national list of email addresses from which people could be sampled, and there is no standard convention for email addresses, as there is for phone numbers, that would allow random sampling. Internet surveys of the general public must thus first contact people by another method, such as through the mail or by phone, and ask them to complete the survey online.
Because of these limitations, researchers use two main strategies for surveying the general population using the internet. One strategy is to randomly sample and contact people using another mode (mail, telephone or face-to-face) and ask them to complete a survey on the web. Some of the surveys may allow respondents to complete the survey by a variety of modes and therefore potentially avoid the undercoverage problem created by the fact that not everyone has access to the web. This method is used for one-time surveys and for creating survey panels where all or a portion of the panelists take surveys via the web (such as the GfK KnowledgePanel and more recently the Pew Research Center’s American Trends Panel). Contacting respondents using probability-based sampling via another mode allows surveyors to estimate a margin of error for the survey (see probability and non-probability sampling for more information).
Pew Research Center also has conducted internet surveys of random samples of elite and special populations, where a list of the population exists and can be used to draw a random sample. Then, the sampled persons are asked to complete the survey online or by other modes. For example, see the scientist survey reported in “Public and Scientists’ Views on Science and Society.”
Another internet survey strategy relies on convenience samples of internet users. Researchers use one-time surveys that invite participation from whoever sees the survey invitation online, or rely on panels of respondents who opt-in or volunteer to participate in the panel. These surveys are subject to the same limitations facing other surveys using non-probability-based samples: The relationship between the sample and the population is unknown, so there is no theoretical basis for computing or reporting a margin of sampling error and thus for estimating how representative the sample is of the population as a whole. (Also see the American Association for Public Opinion Research’s (AAPOR) Non-Probability Sampling Task Force Report and the AAPOR report on Opt-In Surveys and Margin of Error.) Many organizations are now experimenting with non-probability sampling in hopes of overcoming some of the traditional limitations these methods have faced. One example of this is sample matching, where a non-probability sample is drawn with similar characteristics to a target probability-based sample, and the former uses the selection probabilities of the latter to weight the final data. Another example is sample blending, whereby probability-based samples are combined with non-probability samples using specialized weighting techniques to blend the two. Here at the Pew Research Center we are closely following experiments with these methodologies, and conducting some of our own, to better understand the strengths and weaknesses of varying approaches.
Related publications
- Public and Scientists’ Views on Science and Society Jan. 29, 2015
- Women and Leadership Jan. 14, 2015
- Public Sees U.S. Power Declining as Support for Global Engagement Slips Dec. 3, 2013
- What the Public Knows – In Words, Pictures, Maps and Graphs Sept. 5, 2013
- A Survey of LGBT Americans June 13, 2013
The problem of declining response rates
As Americans are now faced with more demands on their time, they are exercising more choice over when and how they can be contacted. The growth in the number of unsolicited telephone calls has also resulted in people employing more sophisticated technology for screening their calls (e.g., voice mail, caller identification, call blocking and privacy managers). This has resulted in fewer people participating in telephone polls than was the case when telephone surveys first became prevalent. As a consequence, response rates have continued to decline over the past decade.
Pew Research Center has conducted several survey experiments to gauge the effects of respondent cooperation on the validity of the results. These experiments compare responses from a standard survey, conducted with commonly utilized polling procedures over a five-day field period, with a survey conducted over a much longer period that employed more rigorous techniques aimed at obtaining a higher response rate and interviewing more difficult to reach respondents.
Findings from the 2012 study “Assessing the Representativeness of Public Opinion Surveys,” the 2003 study “Polls Face Growing Resistance, But Still Representative” and the 1997 study “Conservative Opinions Not Underestimated, But Racial Hostility Missed” indicate that carefully conducted polls continue to obtain representative samples of the public and provide accurate data about the views and experiences of Americans. These results are also reported in Public Opinion Quarterly.
Related publications
- Assessing the Representativeness of Public Opinion Surveys May 15, 2012
- Gauging the Impact of Growing Nonresponse on Estimates from a National RDD Telephone Survey (2006) Public Opinion Quarterly 70: 759-779.
- Polls Face Growing Resistance, But Still Representative April 20, 2004
- Consequences of Reducing Nonresponses in a National Telephone Survey (2000) Public Opinion Quarterly 64:125-148.
- Conservative Opinions Not Underestimated, But Racial Hostility Missed March 27, 1998
Mixed-mode surveys
Over the past decade, there has been a rise in mixed-mode surveys where multiple modes are used to contact and survey respondents. The increase in mixed-mode surveys has been driven by several factors, including declining response rates, coverage problems in single-mode surveys and the development of web surveys. Because there are now a variety of methods available, survey researchers can determine the best mode or combination of modes to fit the needs of each particular study and the population to be surveyed. However, when multiple modes are used for data collection, factors related to each mode, such as the presence of an interviewer and whether information is communicated aurally or visually, may affect how people respond.
Although Pew Research Center primarily conducts telephone surveys, we also occasionally conduct mixed-mode surveys, where people are surveyed by more than one mode. For example, we have conducted mixed-mode surveys of foreign policy experts and journalists, where respondents can complete the survey via the web or by telephone.
Related publications
- U.S. Seen as Less Important, China as More Powerful Dec. 3, 2009
- Financial Woes Now Overshadow All Other Concerns for Journalists March 17, 2008
- Bottom-Line Pressures Now Hurting Coverage, Say Journalists May 23, 2004
- Self Censorship: How Often and Why April 30, 2000
Reinterviews
Reinterviews are typically used to examine whether individuals have changed their opinions, behaviors or circumstances (such as employment, health status or income) over time. Survey designs that include reinterviews are sometimes called panel surveys. The key feature of this survey design is that the same individuals who were interviewed at the time of the first survey are interviewed again at a later date. Pew Research Center sometimes conducts reinterviews, especially to learn more about whether and how voters’ opinions change during the course of a presidential election campaign. For an example from the 2012 presidential campaign, see “Low Marks for the 2012 Campaign.” For an example comparing foreign policy opinions before and after the events of Sept. 11, 2001, see “America’s New Internationalist Point of View.”
Some of the reports listed below used reinterviews primarily to ask follow-up questions about respondents’ opinions rather than to analyze opinion change on the same issues. Survey reports of this sort include “Beyond Red vs. Blue” and “Voters Like Campaign 2004, But Too Much ‘Mud-Slinging.'”
Related publications
- Low Marks for the 2012 Campaign, Nov. 15, 2012
- High Marks for the Campaign, a High Bar for Obama Nov. 13, 2008
- Beyond Red vs. Blue May 10, 2005
- Voters Liked Campaign 2004, But Too Much ‘Mud-Slinging’ Nov. 11, 2004
- Swing Voters Slow to Decide, Still Cross-Pressured Oct. 27, 2004
- America’s New Internationalist Point of View Oct. 24, 2001
- Campaign 2000 Highly Rated Nov. 16, 2000
- Voters Side with Bush for Now Nov. 14, 2000
- Retro-Politics Nov. 11, 1999
- Popular Policies and Unpopular Press Lift Clinton Ratings Feb. 6, 1998
- News Attracts Most Internet Users Dec. 16, 1996
- Campaign ’96 Gets Lower Grades from Voters Nov. 15, 1996
- The People, the Press & Politics Sept. 21, 1994
- Voters Say ‘Thumbs Up’ To Campaign, Process & Coverage Nov. 15, 1992
- Perot is Back Oct. 26, 1992
Panel surveys
A survey panel is a sample of respondents who have agreed to take part in multiple surveys over time. Pew Research Center has used panels on a number of occasions and now has its own nationally representative survey panel known as the American Trends Panel.
Panels have several advantages over alternative methods of collecting survey data. Perhaps the most familiar use of panels is to track change in attitudes or behaviors of the same individuals over time. Whereas independent samples can yield evidence about change, it is more difficult to estimate exactly how much change is occurring – and among whom it is occurring – without being able to track the same individuals at two or more points in time.
A second advantage of a panel is that considerable information about the panelists can be accumulated over time. Because panelists may respond to multiple surveys on different topics, it is possible to build a much richer portrait of the respondents than is feasible in a single survey interview, which must be limited in length to prevent respondent fatigue.
Related to this is another advantage. Additional identifying information about respondents (such as an address) is often obtained for panelists, and this information can be used to help match externally available data, such as voting history, to the respondents. The information necessary to make an accurate match is often somewhat sensitive and difficult to obtain from respondents in a one-time interview.
A fourth advantage is that panels can provide a relatively efficient method of data collection compared with fresh samples because the participants have already agreed to take part in more surveys. The major effort required with a fresh sample – making an initial contact, persuading respondents to take part and gathering the necessary demographic information for weighing – is not needed once a respondent has joined a panel.
A fifth advantage is that it may be possible to survey members of a panel using different interviewing modes at different points in time. Contact information can be gathered from panelists (e.g., mailing addresses or email addresses) and used to facilitate a different interview mode than the original one, or to contact respondents in different ways to encourage participation.
Surveygames.net
But panels have some limitations as well. They can be expensive to create and maintain, requiring more extensive technical skill and oversight than a single-shot survey. A second concern is that repeated questioning of the same individuals may yield different results than we would obtain with independent or “fresh” samples. If the same questions are asked repeatedly, respondents may remember their answers and feel some pressure to be consistent over time. Respondents might change their behavior because of questions you’ve asked; for example, questions about voting might spur them to register to vote. Respondents also become more skilled at answering particular kinds of questions. This may be beneficial in some instances, but to the extent it occurs, the panel results may be different from what would have been obtained from independent samples of people who have not had the practice in responding to surveys. A final disadvantage is that panelists may drop out over time, making the panel less representative of the target population as time passes if the kinds of people who drop out are different from those who tend to remain. For example, young people may move more frequently and thus be more likely to be lost to the panel when they move.
Probability and non-probability panels
Survey panels comprise many different types of samples. A fundamental distinction is between panels built with probability samples and those built with non-probability, or “opt-in” samples (click here for a discussion of what makes a probability sample).
Among both types of survey panels, the samples may be intended to represent the entire population or only a portion of it. Pew Research Center’s American Trends Panel (described below) is a nationally representative probability sample of U.S. adults. Another nationally representative panel is GfK’s KnowledgePanel. An example of a panel representing a subgroup of the population is the National Longitudinal Survey of Youth 1979. It is a nationally representative sample of young men and women who were 14-22 years old when they were first surveyed in 1979. This panel is a product of the U.S. Bureau of Labor Statistics.
Surveygame.net
There are numerous non-probability or opt-in panels in operation. The methods used to build the samples for these panels differ, but in most cases the panelists have volunteered to join the panel and take surveys in exchange for some type of modest reward, either for themselves or for a charity. These panels tend to be used more for market research than for opinion and policy research, but the distinction is not a sharp one. Two well-known opt-in panels are YouGov and SurveyMonkey Audience. In addition to the debate surrounding non-probability samples, opt-in panels are typically limited to people who already have internet access, and thus do not represent the entire population of the U.S.