ActiVote’s Polling Accuracy

ActiVote’s polling methods are somewhat unconventional in a number of ways:
• We include rankings from several weeks in a single poll, instead of polling a large sample of people in just 2-3 days.
• Participants in our polls are self-selected in that people chose to download our app and rank elections, instead of being randomly selected by us.
A reasonable question therefore is whether we can provide any information about the expected accuracy of our polls.
On this page we provide a historical analysis of the accuracy of our general elections polls for 2020-2023, which will show that our polls are surprisingly accurate.

How we measure polling accuracy In any general election race, we represent the outcome of the election as the name of the winner of the race and the margin that winner had with the runner-up. Thus, in the 2020 national election for president, where Joe Biden received 51.31% of the vote and Donald Trump received 46.85% of the vote, we would report it as Biden +4.46%.

We calculate the error as the difference between the poll’s margin and the margin in the election. The final poll we published in 2020 had Biden +6%., thus, our polling error was 1.54%.
We calculate the bias as a positive number if the poll leaned more democratic than the election result, and a negative number if the poll leaned more republican. Thus, our 2020 national poll had a polling bias of +1.54%.
The goal is to minimize both error and absolute bias.

Expected Error and Excess Error
Even with perfect polling, on average, polls cannot have a polling error approaching 0%, unless samples are very large, which in practice they never are. This can easily be seen by doing a simple thought experiment.

Suppose that we have an “election” consisting of flipping a perfect coin 100 million times. The outcome of the “election” will, obviously, be a (near) tie between Heads and Tails, something like Heads +0% or Tails +0%.

We now want to take a small “poll” to predict this outcome by flipping a coin just N times, where N is some small number. If we would take repeated polls of size N, how good would the “polls” be on average?

For polls of size N=1, a single coin flip, we would either get Heads +100% or Tails +100% as polling result. Thus, in every poll of N=1 we will be off by 100%!

For N=2, two coin flips, we will either Get Heads-Heads, Heads-Tails, Tails-Heads or Tails-Tails which would mean resp. Heads +100%, tie, tie, or Tails +100%. Thus, on average, over many such two-flip polls we would still be 50% off.

For N=4, four coin flips, for every 16 polls we get on average 1 time Heads +100%, 4 times Heads +50%, 6 times a tie, 4 times Tails + 50% and one time Tails +100%. Thus, on average we will be 37.5% off.

Therefore, no matter how good our polling is (or in this case, how diligently we flip coins perfectly), there is a certain amount of expected error, which diminishes with the sample size per poll.

The approximate formula for the expected error for sample size N is 1.6 * sqrt(p1*p2/N) where p1 and p2 are the fractions of the electorate obtained by the winner and runner-up in the election. The following table shows the expected error for a close election (p1=p2=50%).

The formula is slightly inaccurate for very small N, (N=1 gives almost 80% instead of 100%, N=2 gives 56% instead of 50% and N=4 gives 40% instead of 38%) but it is accurate for large N.
The important message here is that if we conduct a very large number of perfect polls with sample size 1000, then on average we would still expect an error of 2.5% and there is nothing we can do about that.

Therefore, when we measure the error of our polls, we split the error in expected error (the part we cannot avoid) and excess error, which is the part that we are responsible for, and should aim to minimize.

Excess Error
The following table shows for 547 ActiVote general election polls taken between 2020 and 2023 of various sample sizes the overall error as well as the expected error and excess error:

The main finding from this table is that ActiVote’s overall excess error is between 0.8% and 2.0% and trending lower for larger samples sizes. For all polls together, excess error is about 13% of the overall error. Stated otherwise: of the error in our polls 87% is expected and dependent on the sample size, and only 13% can be attributed to the specific way of polling. Thus, the best way to make our polls more accurate is by increasing our sample size, which is what we are doing in 2024.

Bias
For bias, there is no expected non-zero minimum: over a large enough series of polls, bias should trend towards 0. Any non-zero number suggests that the pollster can do better.
The following table shows for the same 547 polls that, on average, our historic polls had a democratic bias equal to about 12% of the overall error of the polls.

Top-level pollsters typically have a bias (in either direction) of about 25% of their overall error. In this historical sample, ActiVote is overperforming on that metric. As a result, our main goal is to keep our bias, as percentage of our overall error, as low as it is today, while increasing our sample size.

Conclusion
ActiVote’s polling error is close to what is theoretically achievable, at least for current sample sizes, and bias is as good as it gets. Therefore, ActiVote’s main goal is to keep up this performance with larger sample sizes.

For any questions and comments on this analysis of ActiVote’s polling capabilities, please contact
victor@activote.net.