20. Appendix H: Survey and Suppression Rules
The table in this appendix provides information from a 2002 article with additional updates regarding rules for data suppression, statistical unreliability, and data quality in the display of data for Healthy People 2010. The main reasons for suppressing data in Healthy People 2010 included:
The number of events is too small to produce reliable estimates or may violate confidentiality requirements.
The sample design does not yield representative estimates for a particular group.
There is a high level of item nonresponse or a significant number of unknown entries.
Due to the varying criteria for data suppression adopted by the different data systems that monitor the health status of the U.S. population, Klein, et al, summarized these criteria and provided information on specific population groups for which data systems cannot reliably estimate health metrics. They outlined the criteria for the following surveys:
Behavioral Risk Factor Surveillance System (BRFSS)
Medical Expenditure Panel Survey (MEPS)
Monitoring the Future (MTF)
National Ambulatory Medical Care Survey (NAMCS) and National Hospital Ambulatory Medical Care Survey (NHAMCS)
National Crime Victimization Survey (NCVS)
National Health Interview Survey (NHIS)
National Health and Nutrition Examination Survey (NHANES) & Continuing Survey of Food Intake by Individuals (CSFII)
National Household Survey on Drug Abuse (NHSDA)
National Survey of Family Growth (NSFG)
School Health Policies and Programs Study (SHPPS)
Youth Risk Behavior Surveillance System (YRBSS)
National Hospital Discharge Survey (NHDS)
We have added the California Health Interview Survey (CHIS) and Consumer Assessment of Healthcare Providers & Systems (CAHPS) survey to the list above. The updated information for each survey is organized by a brief background, rules for statistical unreliability, and data de-identification/protection rules. Additionally, we have further updated guidance with information from a 2017 publication by the National Center for Health Statistics (NCHS).
Table 35: Surveys and Suppression Rules
California Health Interview Survey (CHIS)
The California Health Interview Survey (CHIS) began in 2001 as a biennial population-based, omnibus health survey of Californians. CHIS is a mixed-mode (web and telephone) survey that uses an address-based sampling (ABS) frame, making it representative of the state’s population.
CHIS data can be analyzed at the county level for the state's 41 most populated counties. The remaining 17 counties are combined into three different groups. Overall, the CHIS sample is designed to provide population-based estimates for most California counties and all major ethnic groups, including several ethnic subgroups.
Survey data (including the California Health Interview Survey) values are suppressed if the Relative Standard Error (RSE) is greater than 20% and, if statewide or non-stratified county-wide data, deemed likely to be misleading based on individual review of the data. The RSE is calculated using the standard methods noted above–this approach differs slightly from some other users of the California Health Interview Survey data, where the divisor for the RSE is 100 minus the percent, if the estimate is > 50%.
While 30% or 23% cut points are more standard, CHIS determined that a 20% cut point suppressed many potentially misleading values not suppressed with the standard values. CHIS did not use the “100 minus the percent” approach because they determined that it suppressed many values that were unlikely to be misleading. https://letsgethealthy.ca.gov/progress/understanding-this-data/
The CHIS Data Access Center generally does not allow the release of output containing the following information, unless special approval has been received:
• Frequencies that do not meet the cell suppression guidelines;
• Estimates run on sub-stratum geographical areas, i.e., by aggregated zip codes or for counties that comprise part of a stratum;
• Analyses that include the most highly sensitive or most highly identifiable variables with small cell size issues; or
• If most tables in the output require full suppression.
CHIS reviews all output for small (raw frequency) cell sizes and makes release decisions accordingly. CHIS does not generate frequency/counts or cross-tabulations at the sub-county level due to confidentiality and weighting concerns.
Small cell values are defined as less than 3 (unweighted) and less than 500 (weighted). In cross-tabulations, complementary cell values are also suppressed to avoid back-calculation.
Consumer Assessment of Healthcare Providers & Systems (CAHPS)
The CAHPS Survey is used to assess managed care beneficiaries’ satisfaction with their health care services. The goal of the CAHPS Health Plan Survey is to provide performance feedback that is actionable and will aid in improving overall beneficiary satisfaction.
https://www.cms.gov/data-research/research/consumer-assessment-healthcare-providers-systems
According to the National Committee for Quality Assurance (NCQA) Healthcare Effectiveness Data and Information Set (HEDIS) Specifications for Survey Measures, if a measure has fewer than 100 responses, the measure is not reportable.
There are 3 categories of suppression types: Item Suppression, Program Type Suppression, and Reporting Category Suppression.
1. Item Suppression: If there are fewer than 20 valid responses available for any item, the item’s results are suppressed.
2. Program Type Suppression: If there are fewer than 20 completed or partially completed surveys for a given program, the program is excluded from the Database.
3. Reporting Category Suppression: If there are fewer than 10 programs for a given characteristic (e.g., region), CAHPS does not show results for the characteristic. Given the limited number of programs in the 2024 Home and Community-Based Services CAHPS Database, no breakouts by program type or program characteristics are provided.
https://www.ahrq.gov/sites/default/files/wysiwyg/cahps/cahps-database/2024-hcbs-chartbook.pdf
Health Services Advisory Group (HSAG) suppressed MCPs results with fewer than 11 cases in the numerator to satisfy the Health Insurance Portability and Accountability Act of 1996 Privacy Rule’s de-identification standards.
Behavioral Risk Factor Surveillance System (BRFSS)
BRFSS is an ongoing, State-based system of health surveys conducted by telephone interview using random digit-dialed probability samples of adults ages 18 years and over.
Estimates are considered statistically unreliable and are suppressed if the denominator is based on fewer than 50 sample cases.
There are no published data-deidentification guidelines for the BRFSS. However, the CA BRFSS does adhere to guidelines set forth in the California Health Survey Data.
Medical Expenditure Panel Survey (MEPS)
MEPS is an annual, nationally representative subsample of respondents to the National Health Interview Survey (NHIS) and uses the stratified, multistage probability sample design of NHIS. The sample size is about 25,000 individuals.
Estimates are considered statistically unreliable and are suppressed if
1) the denominator is based on fewer than 70 sample cases, or
2) the relative standard error (RSE) of the estimate (expressed as a percentage) is greater than 30 percent.
Healthy People 2010 tracking data for American Indians or Alaska Natives and Asians or Pacific Islanders are suppressed because of their small numbers in the survey.
There is no general de-identification threshold.
Small numbers are generally recoded to mask identifiable information.
All person-level income amounts on the file, including both total income and the separate sources of income, are top-coded to preserve confidentiality. For each income source, top codes are applied to the top.
Medical Conditions Rules: To ensure confidentiality, age of diagnosis was top-coded to 85. For confidentiality reasons, AGEDIAG is set to Inapplicable (-1) for cancer conditions. In order to preserve confidentiality, all of the conditions provided on this file have been collapsed to 3-digit diagnosis code categories rather than the fully-specified ICD-10-CM code. For confidentiality purposes, approximately 7% of ICD-10-CM codes were recoded to -15 (Cannot be Computed) for conditions where the frequency was fewer than 40 for the total unweighted population in the file or less than 400,000 for the weighted population. Additional factors used to determine recoding include age and gender.
Clinical Classification Software Refined (CCSR) are used alongside ICD-10-CM diagnosis codes to group medical conditions into clinically meaningful categories.
For confidentiality purposes, less than 2% of the CCSR categories are collapsed into a broader code for the appropriate body system where the frequency is less than 40 for the total unweighted population in the file or less than 400,000 for the weighted population.
Consolidated Data Rules: percentile of all cases (including negative amounts that exceeded income thresholds in absolute value). In cases where less than one percent of all persons received a particular income source, all recipients are top-coded. Top-coded income amounts are masked using a regression-based approach.
When missing, values are imputed for certain persons’ hourly wages.
Hourly wages greater than or equal to $105.77 are top-coded to -10 and the number of employees variable is top-coded at 500.
Specific cancer diagnosis variables with a frequency count fewer than 20 and those considered clinically rare (i.e., appear on the National Institutes of Health’s list of rare diseases), are removed from the file, and the corresponding variable CAOTHER, indicating diagnosis of a cancer that is not counted individually, is recoded to “Yes” (1) as necessary. The age of diagnosis for arthritis (ARTHAGED) is included in this file and may be recoded in some cases to “Cannot be Computed” (-15). This variable is top-coded to 85 years of age.
The annual Disability Days variables, which represent the number of days a person missed work (DDNWRK21 and OTHNDD21), are top-coded to mask values that exceed the top one-half of one percent of the population.
NUMEMP indicates the number of employees at the location of the person’s current main job. This variable is top-coded at 500 or more employees.
Current main jobs are initially coded at the 4-digit level for both industry and occupation. Then, these codes are condensed into broader groups for release on the file. INDCAT31, INDCAT42, and INDCAT53 represent the condensed industry codes for a person’s current main job at the interview date. OCCCAT31, OCCCAT42, and OCCCAT53 represent the condensed occupation codes for a person’s current main job at the interview date. ... bottom coded to a value of ‘1951’ to preserve age confidentiality. https://meps.ahrq.gov/data_stats/download_data/pufs/h233/h233doc.pdf https://meps.ahrq.gov/data_stats/download_data/pufs/h231/h231doc.pdf
Monitoring the Future (MTF)
The MTF study uses self-administered questionnaires in annual national surveys of representative samples of 8th, 10th, and 12th graders in public and private schools in the continental United States. The sample size is about 45,000–50,000 students.
Data may be considered statistically unreliable when there are fewer than 100 cases in the denominator or if data values are less than 0.05 percent. For all objectives tracked by this survey, data for American Indians or Alaska Natives and Asians are statistically unreliable and are suppressed.
MTF provides threshold information for the restricted data available through NAHDAP (National Addiction & HIV Data Archive Program). All analyses, output, tables, and figures are examined for disclosure risk; any estimate to be reported by a researcher must be based on at least an N=30, especially for subgroups. MTF is in the process of revisiting this threshold of N=30 based on internal work and feedback it is receiving from users of the restricted MTF data.
There is no general threshold.
For MTF public use data, MTF routinely does the following to minimize disclosure risk:
· reports race/ethnicity only as Black, White, Hispanic, Other
· reports respondent's age in terms of number of months
· omits or truncates other sensitive questions. Internally, MTF uses its best judgment of the reliability and replicability of the estimates for any analyses, especially as related to questions new to a survey in a given year. MTF also considers cell size/sample size as it breaks the data into subgroups (sex, race/ethnicity, etc.)
National Ambulatory Medical Care Survey (NAMCS) and National Hospital Ambulatory Medical Care Survey (NHAMCS)
NAMCS and NHAMCS are the National Center for Health Statistics’ (NCHS) ambulatory healthcare surveys. The National Center for Health Statistics is part of the Centers for Disease Control and Prevention (CDC), Department of Health and Human Services. NCHS collects, analyzes, and disseminates timely, relevant, and accurate health data and statistics.
The annual NAMCS and NHAMCS use multistage probability sample designs. NAMCS collects data from over 3,000 non-federally employed office-based physicians and NHAMCS collects data on office visits from about 400 emergency departments and about 230 outpatient departments on samples of their patient visits during an assigned reporting period.
Per the Healthy People 2010 Criteria for Data Suppression, estimates are considered statistically unreliable if:
1) the numerator is less than 30, or
2) the RSE of the estimate is greater than 30 percent.
For all objectives tracked by these surveys, data for American Indians or Alaska Natives, Asians or Pacific Islanders, and Hispanics are suppressed. For American Indians or Alaska Natives and Asians or Pacific Islanders, the number of visits is too small. For the Hispanic origin variable, item nonresponse is too high.
Also, see the examples of the application of NCHS data presentation standards for proportions for the malignant neoplasms of colon and rectum visits.
NCHS follows standard statistical disclosure limitation (SDL) methods when they create public-use files. Their SDL threshold rule indicates that a cell in a table of frequencies is defined as sensitive if the number of respondents is less than some specified number. Some agencies require at least five (5) respondents in a cell, while others require three (3). Under certain circumstances, the number may be much larger. The choice of the minimum number is generally made in consideration of:
(a) the sensitivity of the information that the agency is considering to publish,
(b) the amount of protection the agency determines to be necessary given the degree of precision required to achieve disclosure.
A more recent publication from the National Institute of Standards and Technology (NIST) explains cells in contingency tables with counts lower than a predefined threshold can be suppressed to prevent the identification of attribute combinations with small numbers. NIST referenced the State of Washington's small numbers standards. Accordingly, department staff who are preparing confidential data for public presentation must:
1. Suppress all non-zero counts which are less than ten, unless they are in a category labeled “unknown.”
2. Suppress rates or proportions derived from those suppressed counts.
3. Use secondary suppression as needed to assure that suppressed cells cannot be recalculated through subtraction.
4. When possible, aggregate data to minimize the need for suppression.
5. Individuals at the high or low end of a distribution (e.g., people with extremely high incomes, very old individuals, or people with extremely high body mass indexes) might be more identifiable than those in the middle. If needed, analysts need to top- or bottom-code the highest and lowest categories within a distribution to protect confidentiality.
Note: Due to confidentiality reasons, NCHS cannot share specific information on their cell size suppression criteria as their survey data. For their vital statistics data, the cell suppression rule is to suppress cell sizes less than 10 at the subnational level. For more information, please follow the links below:
https://www.nber.org/research/data/national-hospital-ambulatory-medical-care-survey
https://www.cdc.gov/nchs/about/organization.html.
https://www.fcsm.gov/assets/files/docs/spwp22WithFrontNote.pdf
https://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-188.pdf
https://doh.wa.gov/sites/default/files/legacy/Documents/1500//SmallNumbers.pdf
National Crime Victimization Survey (NCVS)
NCVS is an annual, nationally representative survey of the civilian, noninstitutionalized population ages 12 years and over.
• A stratified, multistage cluster sampling strategy is used to select approximately 50,000 households for a series of telephone and in-person interviews.
•The survey obtains information on the frequency, nature, and consequences of criminal victimizations (including those not reported to police).
Estimates are considered statistically unreliable and are suppressed if they are based on 10 or fewer sample cases in the numerator.
The Bureau of Justice Statistics (BJS) sponsors the NCVS, and the U.S. Census Bureau collects the data and completes all data processing. Per the BJS:
1. The data confidentiality requirements are defined by the Census Bureau’s Disclosure Review Board. The detailed guidelines/thresholds for the NCVS are not publicly available.
2. But the Census Bureau cell size threshold often requires a minimum unweighted count to be at least three (3) for each cell.
3. Counts of zero are not considered disclosures, but very small counts are.
4. Cell suppression may be used for frequency count data. Utilizing Census Bureau software, sensitive cells (those with small unweighted counts) are recognized and then suppressed (the estimate is replaced with the letter “D”) from the published data. For more information, please follow the links below:
https://www2.census.gov/adrm/CED/Papers/CY19/2019-04-McKennaHaubach-Legacy%20Techniques.pdf
National Health Interview Survey (NHIS)
The National Health Interview Survey (NHIS) is the oldest ongoing national household health survey in the United States. The survey is conducted by the NCHS, which is part of the CDC.
NHIS uses a stratified, multistage probability sample design. It collects data annually on the civilian, non-institutionalized population by computer-assisted personal interview. The expected sample of 43,000 occupied respondent households yields a probability sample of about 111,000 persons.
According to the Health People 2010 Criteria for Data Suppression, estimates are considered statistically unreliable and are suppressed if:
1) the denominator is based on fewer than 50 sample cases, or
2) the RSE of the estimate is greater than 30 percent.
However, the examples of the application of NCHS data presentation standards for proportions use slightly different criteria.
The small cell size suppression policy will follow the NCHS guidelines as stated at the NAMCS and NHAMCS section above. For more information, please follow the link below:
https://www.cdc.gov/nchs/nhis-participants/what-to-expect/index.html
National Health and Nutrition Examination Survey (NHANES) & Continuing Survey of Food Intake by Individuals (CSFII)
NHANES is an annual, nationally representative examination survey of the U.S. civilian, noninstitutionalized population. A stratified, multistage probability sampling scheme is used to select approximately 5,000 persons for personal interview and examination.
CSFII is a nationally representative periodic survey, which uses a stratified multistage probability sample of the U.S. noninstitutionalized civilian population. It includes the collection of data on the kinds and amounts of foods consumed on each of two nonconsecutive days, sources of foods, time, and name of each eating occasion.
CSFII has been integrated with the National Health and Nutrition Examination Survey (NHANES); dietary data collection for the integrated survey began in 2002. For 1994–96, the sample size was about 16,000 individuals of all ages, and for 1998, an additional sample was collected from about 5,500 children aged 0–9 years.
In 2002, the Continuing Survey of Food Intakes by Individuals and the NHANES dietary component were merged, forming a consolidated dietary data collection. For more information please follow the link below:
Data from NHANES may be considered statistically unreliable for two reasons:
sampling design and/or small sample size.
-Estimates for Healthy People 2010 objectives that are based on fewer than 30 sample events in the denominator or that have a RSE greater than 30 percent are also considered statistically unreliable and are suppressed. Other sources may use different criteria to assess the statistical reliability of NHANES data.
-The NHANES sample size and design does not support producing estimates for certain racial and ethnic groups, including American Indians or Alaska Natives and Asians and Pacific Islanders. Healthy People 2010 tracking data for these populations are considered statistically unreliable and are suppressed. https://www.cdc.gov/nchs/data/statnt/statnt24.pdf
Per CSFII estimates between 25 percent and 75 percent are considered unreliable and are suppressed if the coefficient of variation of the mean (CV) is greater than 30 percent or the sample size is less than 30 times an average or generalized design effect. A variance inflation factor (VIF), equal to the ratio of the mean of the squared sampling weights to the square of the mean of the weights, is used in the generalized design. Estimates of 25 percent or less are considered unreliable and are suppressed if the sample size is less than eight times a generalized design effect divided by p, where p is the proportion expressed as a fraction.
- Estimates of 75 percent or greater are considered unreliable and are suppressed if the sample size is less than eight times a generalized design effect divided by (1–p).
-Data for American Indians or Alaska Natives and Asians or Pacific Islanders are suppressed because the size of the sampled populations for these groups is too small.
The National Health and Nutrition Examination Survey (NHANES), now combined with the Continuing Survey of Food Intake by Individuals (CSFII), is a major program of the NCHS. NCHS is part of the CDC and has the responsibility for producing vital and health statistics for the Nation. Therefore, the small cell size suppression policy will follow the NCHS guidelines as stated at the NAMCS and NHAMCS section above. For more information, please follow the links below and under the NAMCS and NHAMCS:
National Hospital Discharge Survey (NHDS)
NHDS is an annual survey that collects data from medical records to provide national estimates on hospital discharges from short-stay, noninstitutional hospitals and general and children’s general hospitals regardless of length of stay. Annually, the national estimate is based on a sample of about 300,000 records.
Population based rates are considered unreliable and are suppressed if the numerator is based on fewer than 30 records or if they have a RSE greater than 30 percent.
-Estimates based on 30–59 patient records are flagged to indicate they also have low reliability.
For all objectives tracked by this survey, data for American Indians or Alaska Natives and Asians or Pacific Islanders are statistically unreliable because of their small numbers in the survey. Data for Hispanics are suppressed because of high item nonresponse to the Hispanic origin variable.
The National Hospital Care Survey (NHCS) is a new survey that integrates inpatient data formerly collected by the NHDS with the emergency department (ED), outpatient department (OPD), and ambulatory surgery center (ASC) data collected by the National Hospital Ambulatory Medical Care Survey (NHAMCS). It is being collected by the CDC's NCHS.
The small cell size suppression policy will follow the NCHS guidelines as stated at the NAMCS and NHAMCS section above. For more information please follow the links below and under the NAMCS and NHAMCS section:
National Household Survey on Drug Abuse (NHSDA)
NHSDA is an annual, multistage national probability sample survey of the civilian, noninstitutionalized population (ages 12 years and over). In 1999, the sample size increased from about 25,000 to about 70,000 persons.
Estimated proportions are considered statistically unreliable and are suppressed if their RSE is greater than 17.5 percent. Estimated proportions are also considered statistically unreliable if p < 0.0005 or p ≥ 0.99995.
Data collection progress of the NHSDA was monitored during each quarterly survey by state. Small reserve samples were held back each quarter so that the assigned sample size could be adjusted if necessary during the course of data collection.
To protect the confidentiality of respondents, for example, in the 2019 National Survey on Drug Use and Health (NSDUH), the full analytic file of the individuals was treated using a statistical disclosure limitation method called MASSC, which consists of the following four major steps:
Micro Agglomeration, optimal probabilistic Substitution, optimal probabilistic Subsampling, and optimal sampling weight Calibration.
-All directly identifying information (such as name, phone number, and address) on the file was eliminated. In addition, census region, state, and other geographic identifiers were removed. Moreover, the household link between respondents from the same household was not included in the public use file.
Substance Abuse and Mental Health Services Administration (SAHMSA) reported that they are unable to provide detailed information on their disclosure avoidance techniques apart from that in their public-facing documentation. But according to their suppression criteria, for confidentiality protection, survey sample sizes greater than 100 were rounded to the nearest ten (10), and sample sizes less than 100 were not reported (i.e., are shown as “<100” in tables). For more information, please follow the links below:
https://www.datafiles.samhsa.gov/sites/default/files/field-uploads-protected/studies/NSDUH-2019/NSDUH-2019-datasets/NSDUH-2019-DS0001/NSDUH-2019-DS0001-info/NSDUH-2019-DS0001-info-codebook.pdf https://www.samhsa.gov/data/sites/default/files/reports/rpt41913/2021NSDUHmrbStatInference.pdf
National Survey of Family Growth (NSFG)
NSFG is a periodic survey based on a multistage probability design. It collects data on civilian, noninstitutionalized females 15–44 years of age by computer-assisted personal interview with a self-administered audio section for more sensitive topics. The sample size was 13,795 females in 1995. In 2002 civilian, noninstitutionalized males 15–44 years of age were added to the sample.
The Healthy People 2010 Criteria for Data Suppression describes data as statistically unreliable when RSE is greater than 30 percent or the denominator is based on fewer than 50 sample cases.
For all objectives tracked by this survey, data for American Indians or Alaska Natives and Asians or Pacific Islanders are statistically unreliable and are suppressed because the size of the sampled populations for these groups is too small.
The NCHS Data Presentation Standards for Proportions uses different application examples.
NSFG is conducted by the CDC NCHS with the support and assistance of a number of other programs and agencies within the U.S. Department of Health and Human Services (HHS).
The small cell size suppression policy will follow the NCHS guidelines as stated at the NAMCS and NHAMCS section above. For more information, please follow the links below and under the NAMCS and NHAMCS:
School Health Policies and Programs Study (SHPPS)
The periodic study conducted every six years consists of a census of all State education agencies; a national probability sample of public and private school districts; a national sample of public and private elementary, middle/junior high, and senior high schools; and a random sample of required health education and physical education classes.
Data based on fewer than 30 schools in the denominator are considered statistically unreliable and are suppressed.
The School Health Policies and Practices Study (SHPPS) is a national survey periodically conducted by the CDC to assess school health policies and practices at the state, district, school, and classroom levels every six years.
The small cell size suppression policy will follow the NCHS guidelines as stated at the NAMCS and NHAMCS section above. For more information, please follow the links below and under the NAMCS and NHAMCS:
SHPPS 2016 “Results from the School Health Policies and Practices Study” was originally retrieved December 2024 from the CDC website (removed in 2025), Copy is available through the National Coalition of STD Directors website: https://ncsddc.org/wp-content/uploads/2017/09/shpps-results_2016.pdf
Youth Risk Behavior Surveillance System (YRBSS)
The national Youth Risk Behavior Survey (YRBS) is a part of the YRBSS and is a biennial, school-based survey administered to samples of students in grades 9–12. In 1999 about 15,300 students participated.
Data based on a denominator of fewer than 100 students are considered statistically unreliable and are suppressed.
The national YRBS does not sample enough Asian or Pacific Islander or American Indian or Alaska Native adolescents from a single data year to present estimates for these race groups.
The CDC’s Division of Adolescent and School Health (DASH) routinely monitors youth health behaviors and experiences.
The small cell size suppression policy will follow the NCHS guidelines as stated at the NAMCS and NHAMCS section above. For more information, please follow the links below and under the NAMCS and NHAMCS:
Survey documentation was originally retrieved December 2024 from the CDC website: https://www.cdc.gov/healthyyouth/data/yrbs/pdf/trendsreport.pdf
Last updated
Was this helpful?