Open-ended survey inquiries to elicit folks’s first-order considerations

The general public debate is dominated by controversial and polemic points in politics, science, and well being. With rising proof that Individuals maintain strongly polarised views about insurance policies (Nyhan 2020, Alesina et al. 2020), it’s changing into of vital significance for researchers and policymakers to develop new and rigorous strategies to hearken to residents of various backgrounds and to raised perceive their views. 

Surveys: A novel analysis device 

To this finish, surveys are a singular device to grasp what is going on in society and in folks’s minds – what they suppose, how they motive, and what their considerations are. Few different analysis instruments enable researchers to take such a snapshot of individuals’s minds. When designed effectively, calibrated correctly, and focused to related samples of curiosity, surveys allow researchers to conduct on-line, large-scale investigations in real-time. They’re intuitive, clear, and adaptive to members.

Surveys have to date helped make clear many related coverage subjects. One notably fruitful line of analysis has leveraged survey information to discover folks’s perceptions and preferences about tax coverage and redistribution (Cruces et al. 2013, Karadja et al. 2017, Alesina et al. 2018, Roth and Wohlfart 2018, Fisman and Kuziemko 2020, Hvidberg et al. 2021). 

The worth of open-ended survey questions

The spine of surveys typically consists of closed-ended questions that present a hard and fast set of reply choices. The benefits of these questions are that the reply choices are standardised and streamlined throughout respondents and so they simply lend themselves to quantitative evaluation. Nevertheless, in some settings, we could unintentionally prime respondents to consider reply choices that they’d in any other case not have thought of. Conversely, we could omit related choices that we have no idea about. In open-ended questions, respondents usually are not provided reply choices, however moderately an empty textual content entry discipline wherein they’ll write freely. Open-ended survey questions can due to this fact circumvent a number of the above-mentioned points. By being much less guided, they could educate us issues that we could in any other case have missed and that we will not be used to fascinated by. On this sense, solutions to open-ended questions could assist to make clear the first-order issues that come to folks’s minds with out constraining them to consider a restricted set of reply choices.

What are some greatest practices when designing open-ended questions? Open-ended questions can vary from very broad to slim and particular. Broader open-ended questions are helpful to elicit first-order, intrinsic considerations that folks have earlier than they’re prompted to consider a selected coverage side with extra directed questions. Thus, it is smart to begin by asking folks massive image questions such because the ‘essential issues’ that come to their minds when they give thought to a difficulty (e.g. revenue or property taxes). It’s then helpful to slim down the main target by asking folks what they give thought to particular facets of a coverage, its benefits and drawbacks, or much more focused questions on its penalties. Ideally, open-ended questions ought to be complemented with closed-ended questions for cross-validation.

Solutions to open-ended questions could seize two various things. First, the solutions of respondents who haven’t beforehand thought concerning the matter could also be ‘intestine reactions’. These reactions are informative, as they mirror what a respondent thinks and can maintain considering, absent extra studying or centered reflection. Second, the solutions of respondents who’ve already thought concerning the matter or who take time to consider it in the course of the survey earlier than answering could mirror extra profound views. Both manner, solutions to open-ended questions seize the first-order issues that matter to folks and the facets of a difficulty which might be on the prime of their minds.

Strategies of study

In a current mission (Ferrario and Stantcheva 2022), we apply these open-ended query strategies to information from two surveys on revenue and property taxes carried out in 2019 on massive and consultant samples of US residents aged 18 to 70.

We give attention to solutions to the broad query, “What are your essential issues?”, when fascinated by revenue or property taxes, respectively, and we suggest three totally different strategies of study. 

Phrase clouds are greatest used as a primary step in visualising the information and for scanning solutions shortly. Figures 1 and a couple of present the phrase clouds derived from the responses. For revenue taxes, respondents categorical disagreement with the present ranges of taxes, views on the route wherein to alter them (“decrease tax” or “greater tax,” relying on the teams they discuss with), and concern concerning the impacts on the “center class.” For property taxes, respondents’ first-order response centres across the situation of “double taxation” and the equity of property taxes, adopted intently by considerations concerning the “center class” and “household.” 

Determine 1 Fundamental issues about revenue and property taxes

a) Revenue tax

b) Property tax

Notes: Phrase clouds based mostly on solutions to open-ended questions on respondents’ essential issues on revenue and property taxes.

Determine 2 Fundamental subjects respondents take into consideration relating to the revenue and property taxes

a) Revenue tax

b) Property tax

Notes: The determine reveals the distribution of subjects talked about within the solutions about the primary issues on revenue and property taxes. The bars characterize the variety of occasions a subject was talked about out of the overall mentions of any matter by political group.

It’s then notably attention-grabbing to review whether or not folks belonging to totally different teams (e.g. of various political affiliations, ages, and incomes) use a systematically totally different lexicon. We focus our consideration on the heterogeneity by political affiliation and reduce our pattern utilizing a mixture of the presidential candidate supported in 2016 (Clinton versus Trump) and of the self-reported diploma of conservatism versus liberalism.

We think about two methods of investigating heterogeneity: keyness evaluation and matter evaluation. Keyness evaluation is predicated on a relative frequency evaluation that compares using phrases between two teams of individuals. This technique attaches a rating to every time period, which measures how attribute the time period is of 1 group relative to the opposite. Matter evaluation is predicated on a keywords-count mannequin, the place subjects are outlined by units of key phrases. 

First-order considerations about revenue and property taxes

For revenue tax, we determine eight distinct subjects: Distribution, Equity, Authorities Spending, Social Security, Effectivity, Loopholes, Flat Tax, Public Items, and Do not Know. The Distribution matter, for instance, incorporates key phrases comparable to “center class”, “low revenue,” and “millionaire.” The Equity situation incorporates the phrases “honest” and “unfair.” Public Items captures “infrastructure”, “training”, and “well being care.” Effectivity is represented by phrases comparable to “damage economic system”, “work much less”, “competitors”, and “spend much less”, amongst others. The ultimate class is for respondents who categorical that they have no idea sufficient concerning the coverage to provide a significant reply.

The subjects of Distribution and Equity, in addition to Authorities Spending and Loopholes, are on the prime of many individuals’s minds. The Effectivity matter doesn’t seem like as salient. These outcomes echo Stantcheva (2021), who finds that distributional and equity issues dominate effectivity considerations in shaping folks’s tax coverage views. 

For property tax in Panel b, the subjects are much like these from the revenue tax survey. Particular to property tax is the Double Tax matter, captured by key phrases comparable to “already taxed”, and the Grieve matter, which is captured by phrases comparable to “grieve”, “bury”, and “funeral”. The distribution of subjects is once more closely centred round problems with Distribution and Equity, however Double Tax can also be vital. A lot rarer are mentions of Authorities Spending, Effectivity, Loopholes, Grieve, and Public Items. Moreover, extra folks categorical a lack of expertise about property tax than revenue tax.

Partisan variations in first-order considerations

There are clear political variations within the subjects talked about. Points associated to Distribution are far more prevalent on the left aspect of the political spectrum than on the appropriate, whereas Authorities Spending is a extra urgent concern for respondents on the appropriate. Effectivity just isn’t ceaselessly talked about by any political group. Equity mentions are fairly evenly distributed throughout the political spectrum. Nevertheless, despite the fact that everybody cares about equity, the which means of this idea tremendously differs throughout respondents; equity is within the eye of the beholder.

Determine 3 reveals the key phrases which might be most particular to Democrats and Republicans. According to the subject distribution, the key phrases on the Democrat aspect centre round problems with Distribution (“poor wealthy,” “decrease class,” “center class,” and “tax rich”). Republicans have a tendency to emphasize Authorities Spending, Authorities Waste, and the Financial Prices (“folks work”, “economic system spend”). For property tax, there are additionally clear variations within the subjects talked about by political leanings. Distribution points are most prevalent amongst Clinton liberals and diminish quickly and monotonically in direction of Trump conservatives. Conversely, Double Taxation considerations are prevalent amongst Trump conservatives, however fairly uncommon amongst Clinton liberals. The mentions of Equity are once more evenly distributed throughout the political spectrum. These patterns are confirmed by the keyness evaluation in Panel b of Determine 3.

Determine 3 Key phrases talked about by democrats and republicans relating to revenue and property taxes

a) Property tax

b) Revenue tax

Notes: The determine reveals key phrases amongst Democrats and Republicans in solutions to the query about respondents’ essential issues on the revenue and property taxes. The rating reported for a set of two phrases is the χ^2-test statistic, testing the null speculation that the incidence of the given key phrases is similar amongst Democrats and Republicans. * p<0.1, ** p<0.05, *** p<0.01.

Open-ended questions as a information to in-depth analysis

The solutions to open-ended questions can information subsequent analysis. Within the case of revenue and property taxes, they reveal that folks care concerning the distributional impacts of taxes and equity. This prompts us to dig deeper into these points utilizing detailed (closed-ended) survey questions and experiments to raised perceive these considerations (Stantcheva 2021). Many analysis areas in economics may benefit from using open-ended survey inquiries to elicit folks’s first-order considerations and information extra in-depth analysis. 

References

Alesina, A, A Miano, and S Stantcheva (2020), “The Polarization of Actuality”, AEA Papers and Proceedings 110: 324-28.

Alesina, A, S Stantcheva, and E Teso (2018), “Intergenerational mobility and preferences for redistribution”, American Financial Overview 108(2): 521-554.

Cruces, G, R Perez-Truglia, and M Tetaz (2013), “Biased Perceptions of Revenue Distribution and Preferences for Redistribution: Proof from a Survey Experiment”, Journal of Public Economics 98(C): 100-112.

Ferrario, B and S Stantcheva (2022), “Eliciting Individuals’s First-Order Issues: Textual content Evaluation of Open-Ended Survey Questions”, AEA Papers and Proceedings 112: 1-8.

Fisman, R, Okay Gladstone, I Kuziemko, and S Naidu (2020), “Do Individuals Wish to Tax Wealth? Proof from On-line Surveys”, Journal of Public Economics 188: 104207.

Hvidberg, Okay, C Kreiner, and S Stantcheva (2020), “Social Place and Equity Views”, NBER Working Paper 28099.

Karadja, M, J Mollerstrom, and D Seim (2017), “Richer (and Holier) Than Thou? The Impact of Relative Revenue Enhancements on Demand for Redistribution”, Overview of Economics and Statistics 99(2): 201-212.

Nyhan, B (2020), “Information and Myths about Misperceptions”, Journal of Financial Views 34(3): 220-36.

Roth, C and J Wohlfart (2018), “Skilled Inequality and Preferences for Redistribution”, Journal of Public Economics 167: 251-262.

Stantcheva, S (2021), “Understanding Tax Coverage: How Do Individuals Cause?”, The Quarterly Journal of Economics 136(4): 2309-2369.