ACS Data Assessment Sheds Light on Decision Not to Publish One-Year ACS Data for 2020
In July, the U.S. Census Bureau announced that it will not publish one-year American Community Survey (ACS) data for 2020. As a result, the U.S. Department of Housing and Urban Development will likely use the five-year ACS data to calculate income limits. Novogradac Income Limit Working Group research showed that changing from the one-year to five-year ACS would result in area median income limits that are, on average, 3.5% lower than they would be under the one-year ACS.
The Census Bureau provided some additional insight into its decision when it released a report Oct. 27 titled An Assessment of the COVID-19 Pandemic’s Impact on the 2020 ACS 1-Year Data and a blog post titled Pandemic Impact on 2020 American Community Survey 1-Year Data. The report discusses the issues found in the 2020 ACS one-year survey. Considering the significant effect these issues will have for low-income housing tax credit property income limits, it’s worth examining the Census Bureau’s findings.
Data Collection Issues
The data collection was severely hampered by the pandemic. The first issue arose when the National Processing Center (NPC), which handles all of the mailing of the ACS, had to shut down in March 2020. The ACS typically surveys about 3.5 million households per year, divided into 12 monthly increments called panels. The surveys are sent out monthly to gather survey responses throughout the year. Only about 25% of the surveys for March were mailed out and no mailings were done in April, May or June. Ultimately, the sample size was reduced by about 670,000 cases.
It wasn’t just the initial mailings that were impacted. Even when workers were able to return to the NPC, various factors such as social distancing limited the ability for workers to send out initial mailers as well as reminder letters that are typically send to surveyed households
The chart below shows the self-response rate by data collection panel and illustrates the low response rates for the panels from early to mid-2020.
The second issue was that the usual in-person follow-up was paused at various times throughout the year. The stay-at-home orders made it so that Census workers could not follow up in person with nonresponsive survey recipients. In addition, the stay-at-home orders varied by state and city, causing added sampling issues.
The chart below shows how mailing and in-person follow up was impacted in 2020.
During the stay-at-home orders, follow-up interviews were conducted by telephone. Once stay-at-home orders were lifted, in-person interviews were restarted.
However, the limited time frame that in-person visits were allowed affected the number of people who could be contacts and as will be discussed in more detail in the data-quality-issue section. Results were skewed toward those with stable housing, phone numbers connected to their housing and those with higher education levels. All of this resulted in an ACS that provided unreliable data.
Data Quality Issues
As mentioned above, the issues with the mail processing center led to a reduction in the sample size from the planned sample size of 3.54 million to 2.87 million (a reduction of 18.9%). In addition, the number of interviews conducted declined from 2.06 million in 2019 to 1.41 million in 2020. However, the reduction in sample size alone likely would have not have resulted in an unreliable one-year ACS.
The Census Bureau was concerned that critical sections of the population were not able to reply to the survey. From the report:
The coverage rates … confirm that the groups with usually high coverage rates in the survey, including White non-Hispanic and Asian non-Hispanic populations, continued to be well-represented in 2020. However, groups that tend to be underrepresented in the estimates, such as the Black non-Hispanic and the Hispanic populations, had lower coverage rates and were less represented in 2020. This provides some of our strongest direct evidence of shifts in the response universe for 2020, particularly for geographies in which these groups were concentrated.
The Census Bureau then benchmarked its findings against prior years and other source data to see if selected criteria indicated that the results of the survey looked out of line with expectations.
A large measurement that showed that the data was skewed toward homeowners was the building structure type. There was a significant increase in the number of responses indicating they lived in a single-family home opposed to multifamily or mobile homes. From the report to illustrate the issue:
This change from 2019 to 2020 would imply an increase of more than 3.2 million single-family homes during a period when the total number of housing units (of all types) increased by only 1.1 million units. Offsetting this increase, the 2020 ACS estimates also imply year-over-year losses of 570,000 mobile homes and 1.6 million units in buildings with 2 or more apartments between 2019 and 2020. As Section 3 on estimation modifications and Section 4 on quality measures demonstrated, the composition of the responding households was not representative of the housing units in typical years despite best efforts to adjust for the collection disruptions. These dramatic year-over-year changes from 2019 to 2020 raised concerns about the accuracy of the 2020 estimates and the nonresponse bias.
Another example to illustrate this point is the marital status of survey responders. “Between 2019 and 2020, the married population increased 1.4 percentage points and we had no reason to believe that this result reflected an actual change in the marital status of U.S. residents.”
The final example, to illustrate the concern that the population responding to the survey was biased to certain population is the level of education obtained. This data is very skewed. Again, from the report:
It would suggest that there was an increase of about 6 million people in the U.S. with a bachelor’s degree from 2019 to 2020. Comparing this result with the increase of around 2 million bachelor’s degrees in the ACS in recent years, we quickly saw that there was reason to doubt the accuracy of the 2020 ACS 1-year estimates. These estimates are especially suspicious given that other data sources, such as administrative records from the National Student Clearinghouse (Huie et al., 2021) do not show a large increase in the number of first-time graduates earning a bachelor’s degree in the 2019 to 2020 academic year.
There are additional examples in the report, but these really illustrate the data issues from the survey. After analyzing all of this data The Census Bureau concluded that the survey data was not accurate.
Median Income Issues Noted in the Report
The report contained a section on median income. As expected, since the information collected in the survey is skewed to those with a higher educational background and more stable living environments, the median income figures trended higher than anticipated. To help benchmark the reasonableness of the ACS estimates, the Census Bureau looked at the Current Population Survey Annual Social and Economic Supplements (CPS ASEC). The report stated, “While the ACS traditionally has shown lower median household incomes compared to the CPS ASEC for myriad reasons, estimates from the two surveys traditionally have not diverged to the degree seen comparing the 2020 ACS and the 2020-2021 CPS ASEC.
The median household income in the 2020 ACS was $1,454 higher than the 2020-2021 CPS ASEC, with all previous years showing the ACS with lower median household.”
2021 Outlook
The report did not address if the Census Bureau thought the data collection issue would be repeated with the 2021 survey. It is likely the Census Bureau will not be able to make this determination until the 2021 data is collected and analyzed. However, there are a few factors that will hopefully point to this being a one-year-only issue. The largest factor will likely be the effect of stay-at-home orders in 2021. Stay-at-home orders have not been as expansive in 2021 as they were in 2020. This will allow the NPC to process mailers for the entire year. In addition, this should allow Census workers in most jurisdiction to be able to visit nonresponding households throughout the year.