538 Pollster Ratings Disclosures

538’s pollster ratings are used by many organizations to judge the quality of individual pollsters. In order to score well according to the criteria set by 538, a pollster needs to achieve two goals: (1) produce accurate polls (with minimal error and bias), and (2) be transparent. The disclosures in this section aim to answer the 10 questions that make up the transparency test of 538.

1. Did the pollster publish the exact trial-heat question wording used in this poll?

Every election poll published by ActiVote is based on visually selecting a candidate in a series of candidates in a race. All user screens are available in the app and remain accessible to anyone who wishes to analyze the exact presentation of each survey. In this generic text we are providing as examples what the screens look like for the presidential election poll.

The next three screenshots show the “ranking screen”.

The screen starts with the name of the race (President of the US), then the selected preference(s), followed by the list of candidates in alphabetical order (by first name), with profile picture, colored by their party color.
By selecting the “+” in front of a candidate tile, the selected candidate moves to the top of the screen as preferred candidate.
• The left screen shows the screen before any selection has been made.
• The middle screen shows the screen after a user selected Joe Biden as preferred candidate.
• The screen on the right shows the screen after a user selected Donald Trump as their candidate.

The next two screenshots show the “ballot screen”.

Users can select the “ballot screen”, where candidates for all races are shown in a ballot-like format. The screenshot on the left shows the presidential ballot before any choice has been made, while on the right the ballot is shown after Robert Kennedy has been selected.
In summary, for any election, candidates are shown alphabetically (on first name) with their profile pictures, and after a selection has been made, the selected candidate is clearly visible.

2. Did the pollster publish the exact question wording and response options for every question mentioned in the poll release?
All ActiVote election polls are single race election polls. Any user who selects a candidate as their preference (within the field period of the poll) is included in our poll, independent of any other actions they may or may not take in our app.
Therefore, the exact question wording and response options are as described in the answer to the previous question: two possible types of screens (both available to each user) in which the user can visually select a candidate in the election race.

3. Did the pollster release both weighted and unweighted sample sizes for any demographic groups, or acknowledge the existence of a design effect in their data?
ActiVote includes in its crosstabs both the weighted and unweighted sample sizes (as percentage of the overall sample size) for all demographic groups that represent at least 5% of the overall unweighted sample and at least 5% of the overall population. For an example, see the table in the next answer.

4. Did the pollster publish crosstabs for every subgroup mentioned in the poll release?
ActiVote recognizes region, age, gender, affiliation, ethnicity, income and education as subgroups. We leave out any subgroup which has a weighted or unweighted representation of 5% or less. In practice this means that the following subgroups are left out in all national polls:
(1) “Other” party affiliations which nationally represent 2% of the electorate
(2) “Asian” and “Other” as ethnicities as they represent resp. 4% and 3% of the electorate.
For state-wide polls, included groups may differ, depending on their unweighted and weighted fraction of the electorate and the specific poll. However, we will only mention subgroups in a poll release, if they are also included in the crosstabs.

The following picture shows an example crosstab for a national presidential poll between Donald Trump and Joe Biden. It includes all subgroups for the mentioned characteristics, with the exception of affiliation-other, ethnicity-asian, and ethnicity-other, as they did not represent at least 5% of the overall population.

Each crosstab table contains the sample size (N=1000), the Margin-of-Error for the sample size (MoE=3.1%) and the scope the poll applies to (National). Vertically all characteristic types are listed (region, age, gender, affiliation, ethnicity, income, education), and for each characteristic the subgroups that represent 5%+ of the unweighted and weighted electorate. Next we show the unweighted fraction represented by each subgroup. In the example it shows that females were only 30% of the unweighted sample, while males were 70%. Next we show the weighted fractions, with females increased to 54% and males decreased to 46%. Finally, we show the weighted vote share for the various candidates. The candidate with the highest vote share in a subgroup has their vote share highlighted in their party’s color.

5. Did the pollster disclose the sponsor of the poll (if there was a sponsor)?
Every poll is conducted by ActiVote Inc., a non-partisan, registered Delaware Public Benefit Corporation and B Corp focused on helping people confidently vote in every election. Individual polls require no specific funding as all data is proactively provided by our users. The only costs involved, the operation of the ActiVote platform, are entirely funded by ActiVote.

6. Did the poll specify how the sample was selected (e.g., via a probability-based or non-probability method)? If the sample was probability, was the sampling frame disclosed? If non-probability, did the pollster disclose what marketplace or online panels were used to recruit responses or its model for respondent selection?
The survey participants are a non-probability sample: anyone who has downloaded our app and starts participating in our polls is included.
Users can identify themselves in any of three ways:
(1) By state, limiting their interaction to state-wide elections,
(2) Full address, allowing interaction on any elections,
(3) Name and full address, allowing us to link them to their voter registration as provided by the voter file from L2.
The level of identification determines how many characteristics we will have available of the user.
We do not use quotas for any characteristic. However, we do have decent representation for all characteristics, with the largest discrepancies being that our users skew a bit younger and more male than the general population.
Users are excluded from our polls if and only if we believe that they are not genuine in their participation in the app. We cannot be transparent about the various checks we perform, as that would render these safeguards less effective, but the goal is to ensure that we include all genuine users and exclude all pretenders.
There are no specific strategies to gain cooperation. However, the app selects for every user a daily action as a suggested action that they may wish to complete, which can be any type of survey question (policy question, voting on a bill, rating a representative, selecting a candidate in an election race). If a regular user has not yet selected a candidate for a particular upcoming election race, the daily action may thus prompt them to do so.
Our users are not compensated or incentivized in any way. They use the app for the user experience in participating in our democracy.

7. Did the pollster list at least three of the variables the poll is weighted on?
For most participants in our surveys, we have a link with their voter registration, which provides us information about the following characteristics: (1) age, (2) gender, (3) party affiliation, (4) ethnicity, (5) income, (6) education, and (7) voting chance based on data in the national voter file from L2.
We also weigh on region, approximately equally split in rural, suburban, and urban. We define this based on the population density per zip code: the 1/3 of the population living in zip codes with the lowest population density are deemed rural, those in the 1/3 living in zip codes with the highest population density are deemed urban. The final 1/3 are deemed suburban.
For each individual state, as well as for the nation, we use those same voter files to determine for each characteristic what percentage of the likely voters it represents. For the general presidential election on November 5th, 2024, the target percentages for likely voters are shown in the two tables below.

Finally, we also weigh on the survey participant’s Political Matrix position. This is an ActiVote-specific measure calculated for each survey participant based on the collection of all survey questions they have answered.

8. Did the pollster disclose the source of its weighting targets (e.g., “the 2022 American Community Survey”)?
As indicated in the previous answer, our weighting targets for six of eight characteristics (age, gender, affiliation, ethnicity, income and education) are based on the L2 voter files. Region is based on zip code. Likelihood to vote is based on the voting history of the user (based on the L2 voter registration) in combination with the voting history of voters with the same characteristics.
Finally, we weight on political matrix position, which is an ActiVote proprietary calculation which places each user in our 2-dimensional Political Matrix, representing their political leanings based on answers to any subset of over 500 policy questions. This weight ensures that we have the right balance of left, moderate left, centrist, moderate right and right leaning participants.

9. Did the poll report a margin of error or sample size for a “critical mass” of subgroups? We do not mandate this be a complete count, but if it looks like groups are intentionally missing (e.g., they are referenced in the press release but are missing in the crosstab documents), we withhold the point.

We report the unweighted and weighted sample size for all subgroups of all seven characteristics (region, age, gender, party affiliation, ethnicity, income and education) as percentage of the overall sample. We only exclude subgroups that represent 5% or less of the overall sample or the overall population, which means that typically “affiliation-other”, “ethnicity-asian” and “ethnicity-other” are excluded.
We do not discuss any subgroups in press releases that are not listed in the crosstabs.
See also the example crosstabs table in the answer to question 3.

10. Did the poll methodology or release include a general statement acknowledging a source of non-sampling error, such as question wording bias, coverage error, etc., in addition to the normal margin of sampling error inherent to surveying?
ActiVote’s polls are based on an opt-in app panel, meaning that anyone can participate by choice, which provides us with a non-probability sample. As a result, in each of our polls a particular subgroup of the overall electorate could be overrepresented.
Our polls aim to correct for such overrepresentation by weighing on various criteria, such as age, gender, party affiliation, ethnicity, income, education, voting chance, region and Political Matrix position.
Despite these steps to come to a representative poll, some subgroups may be underrepresented, potentially skewing the results in an unknown way.