CNN Transparency Questionnaire for Polling Standards
CNN has created the “CNN transparency questionnaire for polling standards”, to make it easier for readers and viewers to understand what a poll says. Although ActiVote is new to polling, we are very much aware of the pitfalls involved in polling and especially the self-selection bias that can skew a poll like ActiVote’s poll. The self-selection bias for ActiVote is not likely to be found in characteristics that we correct for (such as gender, age, party affiliation, ethnicity, education and income) but in characteristic differences that we do not correct for. A likely candidate for such a characteristic is that our survey respondents are likely more politically engaged than the general electorate as they went through the effort to download a political app and rank a candidate for the presidential election on November 3rd, 2020. While ActiVote’s polling results seemed to have a slight liberal bias in the early days of our polling with small sample sizes (less than 1000 participants), that bias seems to have evaporated since our sample size has increased to (significantly) over 2000.
We are providing the answers to the 16 questions of the CNN transparency questionnaire to help readers of our website assess what value can be assigned to our weekly presidential poll.
1. What survey firm conducted the poll? The poll was conducted by ActiVote Inc., a non-partisan organization focused on helping people vote through our app ActiVote.
2. How were respondents interviewed – by live interviewers on the phone, IVR, online, self-administered questionnaire or another method? Respondents are users of ActiVote who have ranked presidential candidates in the app within the last 28 days and have linked themselves to their voter registration record in the app.
3. Who paid for the survey and why was it done? ActiVote Inc. is a passion project of its founders who are paying for the development and operation of the app. The survey itself, is an automated anonymized aggregation of user rankings and as such has no separate cost.
4. How many people were interviewed for this survey? The survey is a rolling poll (all rankings of the previous 28 days are used) and as of October 1st, 2020 has at least 2000 respondents in each poll, with the number of respondents trending upwards.
5. In what language(s) were respondents interviewed? The ActiVote app is currently only available in English.
6. Please provide a copy of the full text and interviewer instructions/programming for all questions included in this survey release. The screenshot below shows the ranking screen that users see in ActiVote:
7. When was your survey conducted? Every poll we publish contains the rankings entered or updated in the past 28 days.
8. What is the source of the sample for your survey, and by what method were respondents selected? Please be as specific as possible, and if via web panel(s), please include a description of how the panelists were recruited. If your study was conducted online and included respondents chosen via routers, approximately what percentage of respondents were directed to surveys via routers? Our sample is entirely self-selection of people who download ActiVote and decide to rank their candidate for president.
9. If any quotas were assigned to sampling or interviewing, at what stage were they applied, what variables and targets were used, and what is the source of your estimate of the target quota? No targets were assigned.
10. What is the universe of people that you are trying to survey, and what makes you confident that the sample source represents that universe? We aim to represent the approximately 150 Million people who are expected to vote on November 3rd, 2020. We are confident that with increased usage of our app in combination with our weighing algorithm, our poll will become representative of that universe. At this point in time (October 2020) we believe the poll correctly represents the universe, as evidenced by its overall outcome tracking within 2% of national polling averages.
11. If surveys were conducted by telephone, what percentage of interviews were conducted via calls to cellphones? If surveys were conducted online, were respondents allowed to complete the survey via mobile browers, and approximately what share of respondents did so? All rankings were made on the ActiVote app running on a smartphone.
12. If surveys were conducted by telephone, how many callback attempts did a sampled number receive before being retired? Not applicable.
13. If surveys were not conducted by a live interviewer, what do you do to ensure your respondents are real people and are paying attention to the survey? We perform automated checks on each user profile to ensure that it is genuine before we include the ranking in our survey.
14. What is your estimate of this survey’s error, how is it calculated, and why is this an appropriate error estimation of your survey? If you are reporting a margin of sampling error, has it been adjusted for design effects. If the survey would be based on a perfectly randomized sample, the size of the survey suggests that there is a 3% error with 95% certainty. However, there could also be a self-selection bias, that is not fully corrected for. At this time we no longer see evidence of such a self-selection bias in our polls with sample size of 2000 and larger.
15. If your survey has been weighted, please list the weighting variables and the source of the weighting parameters. If your survey has not been adjusted for education, please explain why and provide an unweighted frequency for education distribution among your respondents. ActiVote weighs the poll for 8 variables: region (5), gender (2), age (4) , party affiliation (4), ethnicity (5), education (3), income (3), and voting chance (calculated as chance between 0% and 100% based on cohort and voting history). The target for each combination of variables is derived from the national voter file with approximately 200 million voters. After weighing each of the 3 education subgroups are represented correctly to within 1% of their target.
16. Is there a minimum unweighted sample size you require before releasing any subset estimates and if so, what is it? We show breakdowns in every poll for 7 of the 8 categories (all but region), no matter what. The smallest subset is in the ethnicity breakdown (Asian) with a sample size > 50.
