Probability vs. Non-Probability Polling in 2024

For years, a central question in public opinion research has been whether traditional probability sampling - the method long considered the scientific gold standard - outperforms non-probability sampling techniques in producing accurate results.

Quick Summary

• The 2024 ActiVote MVP rankings show that non-probability-leaning pollsters were especially strong at the very top of the leaderboard: the #1 pollster was non-probability, and no probability-leaning pollster reached the Top 10.
• Across the full field of 136 pollsters, the broad differences between methodology groups were much smaller. Pure probability and pure non-probability performed similarly overall, while the “mostly probability” category was the main (negative) outlier on average rank.
• Most broad differences were not statistically significant. The clearest statistical signal appears in the Top 10, where the non-probability surge was unlikely to be random under a simple no-relationship baseline.
• The practical takeaway for 2026 is not to judge polls by sampling methodology. Track record, transparency, weighting, and calibration matter much more than whether a poll begins from a probability or non-probability sample.

Probability sampling refers to surveys where each member of the target population has a known, non-zero chance of selection - often via random digit dialing (RDD), address-based sampling (ABS), or random draws from voter files. In the textbook version of this approach, random selection supports design-based inference: researchers can quantify sampling error (the “margin of error”) under the assumptions of the sampling design [17]. For generations, academic orthodoxy and major polling institutions treated this method as the undisputed “gold standard” of research.

Non-probability sampling, by contrast, includes opt-in online panels, digital “river” sampling (people who click on ads), text-to-web invitations, and app-based self-selected respondents. These approaches do not give every voter an equal chance of selection, so they rely heavily on statistical modeling - such as multilevel regression and post-stratification (MRP), quota weighting, and calibration to voter-file data or past election results [10, 11, 21] - to adjust the final estimates toward known population benchmarks [19].

For decades, academic and professional consensus, reflected in early AAPOR task-force reports [7], Pew Research Center guidelines [14, 18], and textbooks [20], favored probability methods as clearly superior. Popular polling aggregators often discounted or excluded non-probability polls on principle.

However, response rates for traditional telephone surveys have fallen into the low single digits (often 2-9%) [14, 15]. When a “probability” poll yields such low response rates, it loses much of its theoretical guarantee, as the small fraction who respond may not represent the whole and can introduce severe nonresponse bias [12, 13, 16].

At the same time, non-probability pollsters have dramatically improved their modeling capabilities. The prevailing sentiment today is pragmatic: execution, modeling quality, and transparency matter at least as much as the sampling label [6, 8, 9].

Methodological debates about coverage error and nonresponse bias can become abstract. Election polling has a rare advantage: on Election Day we observe a concrete benchmark - the actual vote totals - so we can directly compare pre-election estimates to real outcomes [2, 3, 4]. That does not make elections a perfect laboratory (opinions can shift late, and pollsters choose different races), but it makes them one of the clearest large-scale tests available for assessing practical performance.

To examine the current state of the art, we turn to the ActiVote Most Valuable Pollster (MVP) rankings [1]. This dataset evaluated 136 pollsters based on 1,349 polls released between October 7 and November 5, 2024. The scoring combines pinpoint accuracy in the final 30 days (adjusted for race difficulty) with prolificacy across national, state, and district races. Importantly, the rankings are cycle-specific and forward-looking - no historical reputation is baked in - making them an especially clean lens for comparing methodologies in one recent, high-visibility election.

Understanding the Data Landscape

Before diving into the results, it helps to define the methodology classification used in this analysis. Each firm was assigned a rating from 1 to 5 based on its publicly described approach to sampling and fieldwork. This rating is not a measure of “quality” by itself; it is a structured way to categorize methodology, and some categories - especially hybrids - can include a variety of designs. (Note: a complete list of all 136 pollsters and their specific methodology descriptions can be found in the Appendix at the end of this article.)

When analyzing the distribution of these methods, a clear baseline emerges. Non-probability-leaning pollsters (ratings 1 and 2) are the majority, accounting for 80 out of the 136 total pollsters, or 58.8% of the dataset. Conversely, probability-leaning pollsters (ratings 4 and 5) account for 50 firms, or 36.8%. Just 6 pollsters (4.4%) fall into the strictly mixed rating 3. Because non-probability approaches are the more common baseline in the dataset, any analysis must examine over- or under-representation relative to these shares rather than raw counts alone. Lower MVP rank numbers indicate better performance.

Figure 1. Overall distribution of the 136 pollsters across the five methodology categories.

Five Key Matchups in the 2024 Data

To make the leaderboard digestible without losing substance, we focus on five comparisons that are both intuitive for general readers and informative for technical readers.

1. The Absolute Top of the Leaderboard

The #1 spot naturally draws attention in any ranking. In the 2024 cycle, the number one overall rank belongs to AtlasIntel, a firm rated as a 1 (pure non-probability) that utilizes river sampling and MRP weighting. By contrast, the highest-ranking pure probability pollster (rating 5) is Suffolk University, appearing at #12. Notably, there are zero probability-leaning pollsters (ratings 4 or 5) in the Top 10.

The full Top 10 list, with each pollster’s numeric rating and the corresponding methodology category, is:

2. The Elite Tier (Top 20)

Looking beyond the Top 10, the Top 20 provides a broader “elite” tier. In the Top 20, non-probability-leaning pollsters (ratings 1-2) occupy 14 of 20 slots (70%), compared with their 58.8% baseline share in the full dataset - an over-representation. By rating, the Top 20 contains 7 rated “1,” 7 rated “2,” 2 rated “3,” 1 rated “4,” and 3 rated “5.”

Figure 2. Composition of the Top 20 using the three broad methodology groupings.

3. Overall Performance Across All Five Methodology Groups

Looking at average and median ranks across the entire field of 136 pollsters gives a complete picture that includes both stars and laggards.

Ratings 1, 2, 3, and 5 cluster tightly in the mid-60s. Rating 4 (“mostly probability”) stands out as the clear outlier with the worst mean and median performance. This suggests that hybrid approaches attempting to blend probability frames with substantial non-probability elements may, in this cycle, have combined the drawbacks of both worlds without consistent gains.

4. Performance in the Top 50 Tier

The Top 50 contains the pollsters whose results are most visible to the public and most likely to influence media coverage and campaign strategy. Non-probability-leaning pollsters (Ratings 1-2) occupy 30 of the 50 spots (60%), slightly above their 58.8% baseline share. Probability-leaning pollsters (Ratings 4-5) occupy 18 spots (36%), with the 2 mixed pollsters filling the remainder. The non-probability edge that is very pronounced in the Top 10 and Top 20 is still present but noticeably more moderate once we reach the broader group of high-performing pollsters.

5. Representation in the Bottom Half (Ranks 69-136)

To check whether any methodology is disproportionately likely to produce poor results, we examined the bottom 68 pollsters (the lower 50% of the ranking). In the grouped comparison, non-probability-leaning pollsters (ratings 1-2) account for 39 of the 66 pollsters once the two mixed firms are set aside (59.1%), essentially matching their 58.8% baseline share. Rating 4 (“mostly probability”) is modestly over-represented at 15 pollsters (22.1% vs. 19.1% baseline), while pure probability (rating 5) accounts for 12 pollsters (17.6%), essentially matching its overall share.

Within this bottom half, the average rank for non-probability-leaning groups (ratings 1-2 only) is 102.0 (median 102.0); for probability-leaning groups (ratings 4-5 only), it is 101.4 (median 99.0). This distribution reinforces that no methodology is immune to misses and that non-probability approaches are not clustered among the weakest performers.

The Statistical Reality Check

It is tempting to look at the complete absence of probability pollsters in the Top 10 and declare traditional probability polling obsolete. However, a rigorous statistical interpretation provides a more nuanced picture of the industry.

For readers less familiar with statistics: when analysts test “significance,” they are asking whether an observed pattern is unlikely under a simple baseline where there is no relationship between methodology and outcomes. When comparing overall rank distributions across all five methodology groups, a Kruskal-Wallis test yields p ≈ 0.837, which provides no evidence of a systematic difference across the five categories in the full dataset. A Mann-Whitney U test comparing non-probability-leaning pollsters (Ratings 1-2) against probability-leaning pollsters (Ratings 4-5) yields p ≈ 0.480, also not statistically significant. Comparing the pure extremes (Rating 1 vs. Rating 5) produces p = 1.00, indicating that their overall rank distributions are not distinguishable in this dataset (see also Yeager et al., 2011 [22] for an earlier large-scale comparison).

In other words, while non-probability-leaning approaches are prominent at the very top of the leaderboard, the broader differences across the full 136-pollster distribution are not statistically significant. That is consistent with an industry in which performance depends heavily on execution details - weighting, likely-voter modeling, and calibration - not only on the sampling label.

There is, however, one area where the statistics signal a clear shift. A Fisher exact test analyzing specifically who made it into the Top 10 (comparing Groups 1-2 against Groups 4-5) returns a p-value of 0.023, which is statistically significant. This supports the conclusion that the surge of non-probability polling at the very peak of the leaderboard is a meaningful trend, unlikely to have occurred under a baseline assumption of no relationship. The dominance at the absolute top is real, likely driven by highly effective, firm-specific modeling techniques rather than a universal superiority of all non-probability surveys.

A parallel Fisher exact test on Top-20 membership (Groups 1-2 vs. Groups 4-5) returns a two-tailed p-value of approximately 0.191 - which is not statistically significant at conventional levels. This indicates that the non-probability advantage is most pronounced at the absolute peak and gradually moderates as we move further down the leaderboard. These p-values are best read as exploratory signals within one election cycle rather than definitive proof of a universal effect.

Looking Ahead: Interpreting Polls for the 2026 Midterms

The 2024 ActiVote MVP data provide a vital roadmap for how ordinary voters, journalists, and campaign professionals should interpret polling data heading into the 2026 midterms. First and foremost, it is time to retire the dogma that assigns the phrase “gold standard” exclusively to traditional, probability-based, live-caller polling. Labels like “RDD live phone” do not automatically confer reliability. The 2024 data suggest that river sampling, opt-in panels, and heavy data modeling are not intrinsically inferior; in the modern era of low response rates, this analysis shows that they can be superior at the elite level.

Going forward, sampling still matters, but it increasingly interacts with adjustment - weighting, turnout modeling, validation against voter files, and other calibration steps. Whether a poll uses live calling, SMS-to-web, panels, or river recruitment, modern election polling is fundamentally a process of statistical estimation under real-world constraints, and the quality of that estimation is not captured by a single label.

When judging polling quality in upcoming cycles, treat a pollster’s stated methodology as useful context, but treat their recent, demonstrated election-cycle track record as the primary verdict. The most defensible standard is radical transparency in weighting, turnout modeling, and likely-voter screens [5], regardless of sampling frame, combined with repeatable performance on independent scorecards.

The 2024 data do not declare a permanent winner in the probability versus non-probability debate. Instead, they illustrate a polling industry that has become more pragmatic and results-oriented. Non-probability approaches, when executed with sophisticated modeling, showed they can compete - and sometimes lead - at the highest level. Traditional probability methods continue to deliver solid results for many firms. The real differentiator is the skill with which any pollster turns raw responses into calibrated estimates of the electorate.

As we head into the 2026 midterms, the most reliable guide will be empirical performance in recent cycles, combined with clear methodological disclosure. The era of relying on labels is giving way to an era of results - and that is ultimately healthy for both the polling industry and the public that depends on it.

References

[1] ActiVote. (2025). 2024 Most Valuable Pollster (MVP) Rankings. ActiVote.

[2] American Association for Public Opinion Research (AAPOR). (2017). An evaluation of the 2016 election polls in the United States.

[3] American Association for Public Opinion Research (AAPOR). (2021). Task Force on 2020 pre-election polling: Final report.

[4] American Association for Public Opinion Research (AAPOR). (2025). 2024 election polling review.

[5] American Association for Public Opinion Research (AAPOR). (n.d.). Transparency Initiative.

[6] Bailey, M. A. (2023). The death of the random-sampling paradigm? Harvard Data Science Review, 5(3).

[7] Baker, R., Brick, J. M., Bates, N. A., Battaglia, M., Couper, M. P., Dever, J. A., Gile, K. J., & Tourangeau, R. (2013). Report of the AAPOR Task Force on Non-Probability Sampling. Journal of Survey Statistics and Methodology, 1(2), 90-143.

[8] Cornesse, C., et al. (2020). A review of conceptual approaches and empirical evidence on probability and nonprobability sample survey research. Journal of Survey Statistics and Methodology, 8(1), 4-36.

[9] Dutwin, D., & Buskirk, T. D. (2017). Apples to oranges or gala versus golden delicious? Comparing data quality of nonprobability internet samples to low response rate probability samples. Public Opinion Quarterly, 81(S1), 213-239.

[10] Elliott, M. R., & Valliant, R. (2017). Inference for nonprobability samples. Statistical Science, 32(2), 249-264.

[11] Gelman, A., & Little, T. C. (1997). Poststratification into many categories using hierarchical logistic regression. Survey Methodology, 23(2), 127-135.

[12] Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 70(5), 646-675.

[13] Groves, R. M., & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167-189.

[14] Keeter, S., Hatley, N., Kennedy, C., & Lau, A. (2017). What low response rates mean for telephone surveys. Pew Research Center.

[15] Kennedy, C., & Hartig, H. (2019). Response rates in telephone surveys have resumed their decline. Pew Research Center.

[16] MacInnis, B., Krosnick, J. A., Ho, A. S., & Cho, M.-J. (2018). The accuracy of measurements with probability and nonprobability survey samples: Replication and extension. Public Opinion Quarterly, 82(4), 707-744.

[17] Pew Research Center. (2016). Understanding the margin of error in election polls.

[18] Pew Research Center. (2019). For weighting online opt-in samples, what matters most?

[19] Rivers, D. (2006/2007). Sample matching: Representative sampling from internet panels.

[20] Tourangeau, R., Conrad, F. G., & Couper, M. P. (2013). The Science of Web Surveys. Oxford University Press.

[21] Wang, W., Rothschild, D., Goel, S., & Gelman, A. (2015). Forecasting elections with non-representative polls. International Journal of Forecasting, 31(3), 980-991.

[22] Yeager, D. S., et al. (2011). Comparing the accuracy of RDD telephone surveys and internet surveys conducted with probability and non-probability samples. Public Opinion Quarterly, 75(4), 709-747. View online

Appendix: MVP Pollster Methodology Descriptions

Disclaimer on Methodology Ratings

The 1-5 methodology ratings assigned to each of the 136 pollsters are based on the most detailed public descriptions of their sampling frames, recruitment methods, fieldwork modes, and adjustment procedures available as of early 2026. We made every reasonable effort to locate accurate information from pollster websites, methodological appendices, white papers, press releases, and other disclosures. However, transparency levels vary widely across firms, and some descriptions are brief, incomplete, or subject to change. Classification of hybrid or mixed-mode approaches requires interpretive judgment. Reasonable experts may reach different conclusions about how to categorize the same pollster depending on the weight given to various elements (for example, the presence of a probability frame versus the extent of non-probability supplementation). These ratings are therefore best understood as a good-faith, informed interpretive framework designed to facilitate analysis, not as an official or indisputable judgment. Readers are encouraged to consult the original methodology descriptions provided in this appendix and to form their own assessments.