1. Why does ActiVote publish Election Polls?

There are three key reasons why we publish election polls.
1) Credibility for Policy Polls. Our voter education app includes approximately 500 policy questions where we provide instant polling feedback on what the country thinks about a particular topic. To give our users confidence that our policy polling is of high quality, we also publish election polls which are rated after an election season by organizations like ABCNews/538. ActiVote is relatively new to polling and will receive its first pollster rating after the 2024 election cycle.
2) Public Service. Through the increasing usage of our voter education app, we have the data available to produce polls for many races, including races that are not polled by other pollsters. We believe that providing information on those races is a useful public service.
3) Having your voice heard. We have heard regularly from our users that “they have never been called by a pollster” and that they feel that their voice is therefore not represented in the polls. By publishing our polls, our users know that their voice is being included in the polls.

2. Why do you think that an opt-in app can possibly create a reliable poll?

That is a good question, and when we started our polling journey in 2020, we were not sure whether it could.
Evidence over the past four years, however, has shown that we can. You can read more about our analysis of our polling accuracy here. So, we know we produce accurate polls, and the more polls we produce the more anyone can analyze our polling accuracy and polling bias to determine how accurate we are.

Also, the challenge traditional pollsters face is that they attempt to call over 100,000 people and then with a response rate of less than 1% use the answers of those people that they do reach. Our challenge is that of the 200 million or so registered voters in the US we have an opt-in rate in our app of less than 1% and use the answers of those people that we do reach. There is not necessarily so much difference between an implicit opt-in rate of 1% (by the other 99% explicitly opting out) as an explicit opt-in rate of 1% (and the other 99% implicitly opting out).

Also, after the 2024 election cycle we will get a 538 pollster rating which will independently determine how we compare to traditional pollsters.

3. Isn’t your sample hopelessly biased to Democrats and/or political junkies?

Actually, no it isn’t. Our app is used by people who like the help it provides in educating themselves about their possible choices in upcoming elections. Not many people need help in selecting the candidate for president, as they are all over the media with ads for months before the election. But, for local races, such as state representative or district attorney or sheriff, many people do need the help. And among those voters there are also many non-political junkies.

So, our users include newly registered voters, first-time voters, irregular voters, all the way up to people who never miss an election. Now, our unweighted samples are not a perfect representation of the electoral: slightly more rural, younger, more male, more White, higher income, higher educated than average, but in many of these categories not by much.

We weigh our samples to correctly represent the electorate, and our results show that that works just fine.

4. There is no way you can weigh yourself out of bad data!

As a general statement, this is true, but it fortunately does not apply to ActiVote, as the quality of our individual data items (obtained from genuine users) is excellent, and we believe it is the foundation of our high quality polls.

In the ActiVote app, a user will set a preference for an election at the time of their choosing. When they get to the election screen they will see all candidates, including profile pictures and party affiliation. They can zoom in on a candidate and easily research the candidate through the links to candidate information. Thus, they are fully informed when making their choices.

With traditional polling, a participant is approached at a time that is often not of their choosing. They are typically asked questions for an extended period of time and for many of those questions they may not have a considered opinion yet, but they have to make a choice on the spot.

Therefore, we think our raw data is of excellent quality, for which reason we don’t need to weigh us “out of bad data”. Instead, we need to weigh excellent data to fit the electorate. And that is what we do.

5. You don’t seem to do any verification of your users, so anyone can manipulate your polls!

No, that is not true at all and you may want to check out the article written by ABC News/538 about exactly this question.

We do extensive verification of our users and our polls; we just don’t give any feedback on that to our users. There are various reasons for that. If someone wants to use our app to prepare for their election, they can give us as little information as they want, or as much information as they feel comfortable with. The more information they share, the more we can help them be prepared for their next elections. For instance, if users share their email address with us, we will remind them of specific election deadlines (absentee ballot, early voting, election day, etc.). If they don’t: fine with us. So, from the perspective of the user, they can use the app any way they want, and we will not stop them doing so.

From the perspective of ActiVote when preparing polls, our aim is to include every genuine user, while we wish to exclude any disingenuous user. We believe that our detection mechanisms to distinguish between the two types is rather sophisticated, works well and continues to be fine-tuned whenever we find an increase of disingenuous users in our app.

Obviously, we do not reveal all the checks we perform, nor do we alert users that they are excluded from our polls, as that would only encourage them to try harder.

Please note that this problem is not unique to ActiVote: any pollster aims to exclude disingenuous users. For example, some participants in traditional polls are incentivized to participate (e.g.: by offering a monetary reward). And some therefore participate without paying attention, which leads to nonsensical answers (e.g.: picking the first answer to every question, or answering the same question, which has been inserted twice to check for consistency, differently the second time).

Thus, every pollster takes actions to make sure that their data quality is as good as possible, and ActiVote is no exception.

Ultimately, the proof of the pudding is the evaluation of our polls in each election cycle and our last poll in the only race we polled in 2020 was better than the polling averages, our 2022 polling errors were exactly as we predicted them to be, and we are confident that our 2024 polling will be another step forward in our journey to be a top-rated pollster.

6. I just downloaded your app from London, said I lived in Texas, voted for both Trump and Allred and there was no attempt to verify ID whatsoever!

The app is only available in the US app store, but as an American you probably have a US Apple ID. Other than that, we are totally happy that you wanted to see what the user experience is for a Texas voter, and the fact that you made a few choices in races that you cannot vote in, is not a problem for us at all.
Please note that those choices will not be included in our Texas polls. In our logic to determine whether you are a genuine user or not, you were easily flagged as someone to be excluded from our polls. Thus, do not mistake the fact that the app can be used by anyone does not mean that just anyone makes it through the verification checks to determine whether their responses are sufficiently trustworthy to be included in our polls.

7. How can you produce new polls almost daily?

Our users select preferences for many races whenever they are preparing for their elections. Every day, we run our poling engine to see for all those races what our current sample size of genuine users is for that race. As soon as the sample size hits the threshold for that race (400 likely voters for state polls in 2024, 1000 likely voters for national polls in 2024), we perform quality checks on the poll to ensure that it meets our quality criteria, and if it passes those checks, then we publish it.
Thus, we are entirely dependent on our users: if they are active in our app when the election season heats up, we will be able to publish many polls. If not, all we can do is sit and wait. The table below shows the number of polls in each month of 2024 so far.

We expect that especially in October 2024 we will be able to publish a record number of polls.

8. You have more Democrats in your sample than Republicans, so how can we trust your polls?

It is true that our unweighted samples often contain too many Democrats and too few Republicans. In our September 2nd National Presidential Poll, we had the following statistics:

Thus, we wanted to have 40% Democrats but had 45%, which is 5% too many. We wanted to have 35% Republicans but had 30%, which is 5% too few. And we wanted to have 23% Independents and had almost that with 22%.

This means that on average, any vote from a Democrat in our unweighted sample counted for 40%/45%= 0.89 vote in our weighted sample, while any vote from a Republican in our unweighted sample counted for 1.17 vote in our weighted sample.

This kind of weighing of the original sample is what all pollsters do all the time. In a perfect world, all weights are 1.0. The more weights deviate from 1.0, the larger the variance in the result will be, and typically pollsters frown upon weights that are smaller than 0.2 or larger than 5.0. Our weights fall well within those bounds and in this example, they are both close to 1.0, which makes the under sampling of Republicans and over sampling of Democrats not a problem for the quality of our polls.

9. Why are you weighing your polls to D+5? There are about as many Republicans as Democrats in the country.

There are two conflicting definitions of “Democrat” and “Republican” and the issue is that our definition, leading to D+5 and your definition, leading to D=R are at odds. Let’s spell them out:

Definition 1: (“Identify As”): A Democrat resp. Republican is someone who “identifies” as a Democrat resp. Republican when asked by their family, friends or a pollster
Definition 2: (“Voter file”): A Democrat resp. Republican is someone who is registered in the voter file as a Democrat resp. Republican.

Anyone who uses the “Identify As” definition should probably aim to have the number of Democrats closer to the number of Republicans. When using the “Voter File” definition at this time (2024), we end up with D+5
To elaborate on party affiliation in voter files, we must distinguish between two types of states: there are 30 states where party affiliation is included in voter registrations. Thus, for those states we know exactly how people are affiliated. Details can be found here. For those 30 states, the split as of March 2024 is D-38%, R-30%, and I-28%, thus D+8.

For the other states, voter file providers use sophisticated methods to try to deduce for each voter which party they are affiliated with. These algorithms are not perfect but do a good job. When we add all states together and then include the voting chance for every person, it adds up to that 40% of the likely voters are affiliated with the Democratic party and 35% with the Republican party in the voter file, independent of what they currently identify as.

To more explicitly see the difference between the two definitions, we can look at West Virginia. In 2020 Trump won West Virginia by 39% (D=30%, R=69%), thus R+39. Even in March 2024 as can be seen here, West Virginia was R+9 (D=31%, R=40%). Thus, according to the “Identify As” definition WV is R+39, and according to the “Voter File” definition,  WV is R+9.

Thus, if a pollster uses the Identify As definition for party affiliation (by asking the participant “what they identify as” and they have a sample of 40% Republicans and 31% Democrats, they have way too many Democrats. If a pollster like ActiVote looks participants up in the voter file, and uses the Voter File definition, then having a R=40% and D=31% split in the sample, would be exactly right.

10. Some of your crosstabs have counterintuitive results, undermining trust in the poll as a whole. Why is that?

The first thing to recognize is that in general natural random variation is larger than most people expect. That means that random sequences produced by repeatedly throwing a dice, or observing the patterns of red/black in a casino often make people suspicious that “something is off” even when what they are observing is natural random variation.
This principle also becomes apparent when people read crosstabs and expect each number to be in line with their expectations, instead of showing significant random variation. However, that random variation can be large, mostly dependent on the size of the subsample analyzed: the smaller the sample, the larger the natural random variation.
The following table contains some key statistical values applicable to our 400 Likely Voter polls:

Let’s focus on the 1st line: any subsample of 133 participants, where the expected support for a candidate lies in the range of 40%-60%, the 95% confidence level has a Margin-of-Error (MoE) of 8%, while the 99% confidence level has a MoE of 11%.

This line applies to most of the categories with 3 elements such as region (rural, suburban, urban), income (low, medium, high) and education (High School or less, some college and BSC+), where each category is approximately 1/3 of the overall sample of 400 likely voters.

Suppose now that we expect that the Republican candidate should get about 60% of the rural vote, then in approximately 1 in 20 cases (95% confidence) we should expect that the result is more than 8% off, which could mean that we measure just 51% support for the Republican candidate from rural voters even though the actual support among rural voters is till 60%.

In 1 in 100 cases (99% confidence), we should expect the result to be more than 11% off, thus, perhaps just 48% support found for the Republican candidate among rural voters in the poll, even though in the overall electorate the actual support was still 60%.

Let’s now look at the 2nd line: any subsample of just 40 likely voters, where the actual support  for a candidate is 85%, the 95% confidence MoE is 11% and the 99% confidence MoE is 15%.

This line applies to the situation where Black voters are about 10% of the electorate of a state, and 85% of them support the  Democratic candidate. The table shows that occasionally finding just 70%-75% support can be entirely the result of natural statistical variation.

Besides natural statistical variation, these deviations from expected values can be increased further by smaller unweighted samples (for instance, a sample of only 25% rural voters or a sample of only 5% Black voters).

In summary, crosstabs for small sample sizes should be interpreted while keeping in mind the significant statistical variability that may occur. That variability is not an indication of a badly executed poll: it is an indication that natural variability is a key part of polling.

11. If ActiVote’s crosstabs for small sample sizes can have so much natural variability, why publish them at all?

Transparency is key in the polling industry. That is why AAPOR has the Transparency initiative, where members like ActiVote are required to answer 11 questions for every poll they produce. ABC News/538 includes transparency as a key part of their pollster rating methodology, which means that pollsters should answer 10 questions for every poll.

One of the requirements is to provide clarity about subsamples and weighting, for which reason we include crosstabs in every poll. They are there to help readers understand how (un)balanced the unweighted sample was and what electoral targets were set for each subcategory, thus indicating by how much we had to weigh our polls. It also serves to get a general indication of where the core support from each candidate comes from.

E.g., the crosstabs may show as that Republicans are typically more supported by rural voters, men, White voters, and those with the lowest education, while Democrats are typically more supported by urban voters, women, Black voters, and those with the highest education. If the poll shows results that are consistent with these assumptions, it often reinforces confidence in the poll as a whole. However, with smaller sample sizes, large variations can be observed, where these patterns are less clear than readers of the polls expect.

The main finding of a poll, however, is the topline, which will have less unavoidable natural variation as the overall sample is significantly larger than the subsamples.

12. Your polls are published with results accurate to 1 decimal. Given the statistical uncertainty in these polls, that amount of accuracy makes no sense. Why don’t you stick to showing full precentages instead?

When we started publishing polls, we published them without decimals, as you suggest in your question.

However, in a 2-way race, that means that the reported gap will always be a multiple of 2%. The difference between a 50.4%-49.6% race (rounded to 50-50) and a 50.6%-49.4% race (rounded to 51-49), is in reality only 0.4%, but will go from from “tie” to “+2”, while both polls are really a “+1%” as they are resp. +0.8% and +1.2% result.

As pollsters are evaluated on accuracy, we believe that reporting our polls potentially almost 1% away from our actual findings is undesirable. Therefore, we report polls with 1 decimal which will be interpreted by most polling aggregators as “+1%”, These two polls will be shown in their overview as resp. 51%-49% (+1%) and 50%-50% (+1%).

While that may sometimes lead to questions from puzzled readers who feel that 51-49=2 and 50-50=0, it ensures that the reported gap between candidates is in line with the actual findings of the poll.