CalHHS programs are required to provide public reporting based on federal and California statute and regulations, court orders, and stipulated judgments, as well as by various funders. Although reporting may be mandated, unless the law expressly requires reporting of personal characteristics, publicly reported data must still be de-identified to protect against the release of identifying or personal information which may violate federal or state law.

Survey data, often collected for research purposes, are collected differently than administrative data and these differences should be considered in decisions about security, confidentiality and data release.

Administrative data sources (non-survey data) such as: vital statistics (e.g. births and deaths), healthcare administrative data (e.g. Medi-Cal utilization; hospital discharges), reportable disease surveillance data (e.g. measles cases) contain data for all persons in the population with the specific characteristic or other data elements of interest. Most of the discussions in this document pertain to these types of data.

On the other hand, surveys (e.g. the California Health Interview Study) are designed to take a sample of the population, and collect data on characteristics of persons in the sample, with the intent of generalizing to gain knowledge suggestive of the whole population.

The sampling methodology developed for any given survey is generally developed to maximize the sample size with the available resources while making the sample as un- biased (representative) as possible. These sampling procedures that are a fundamental part of surveys generally change the key considerations for protection of security and confidentiality. In particular, the main “population denominator” for strict confidentially considerations remains the whole target population, not the sampled population. But, if persons have special or external knowledge of the sampled populations (e.g. that a family member participated in the survey), further considerations may be required. Also, it is in the context of surveys that issues of statistical reliability often arise—which are distinct from confidentially issues, but often arise in related discussions.

Of particular note, small numbers (e.g. less than 11) of individuals reported in surveys do not generally lead to the same security/confidentiality concern as in population-wide data, and as such should be treated differently than is described within the Publication Scoring Criteria and elsewhere. In this case a level of de-identification occurs based on the sampling methodology itself.

About the Knowledge Base

The CalHHS Data Knowledge Base is composed of three main parts. These have historically been siloed documents, however, have now been published together to serve as a point of truth guide for all CalHHS data sharing needs.

Governance and Updates

As a product, the Knowledge Base is maintained by the Center for Data Insights and Innovation. It will be reviewed at least annually to ensure base functionality, such as no broken links, etc. For questions or contribution suggestions, please email [email protected]. Each section or component of the knowledge base will have it's own governance, update cycle, and contact information. Please see below.

The Numerator – Denominator Condition represents a combination of both the Numerator Condition and Denominator Condition and for which both conditions must be met or else a more detailed assessment is required. This may be considered as an initial screening of a data set.

The Numerator Condition sets a lower limit for the cell size of cells displayed in a table. The DDG has set this limit as any value representing aggregated or summarized records which are derived from less than 11 individuals (clients). Of note, values of zero (0) are typically shown since a non-event cannot be identified.

The Denominator Condition sets a minimum value for the denominator. The DDG has identified the lower limit for the denominator to be a minimum value of 20,000.

Since this is a Numerator – Denominator Condition, both the minimum cell size for the numerator and denominator must be met. If these conditions are met, the table can move to Step 5 for consideration for release to the public. If either the numerator of denominator condition is not met, then the review of the data must proceed to Step 3.

Necessity of criteria for this step will be determined by each department. This may vary depending on the purpose of the release and whether or not the department or program is a HIPAA covered entity or not. See for further discussion.

This step requires the use of a documented method to assess the risk that small numerators or small denominators may result in conditions that put individuals at risk of being re-identified.

Assessment of potential risk for a given data set must take into account a range of contributing considerations. This includes understanding particular characteristics of a given data set that is being released. For example, if the potential values for a specific personal characteristic, such as race, results in many small numbers in data set A but does not in data set B, then the risk may be low for data set B and high for data A if the groupings of the personal characteristics include the same categories. For this reason, each department or program may set different values for risk based on the underlying distribution of these variables in the data sets of interest.

There are many methods used to assess potential risk. Many of the methods that are in use throughout the country are described in the various references provided in Section 15. While each department will document the method(s) chosen for use, the following description of the Publication Scoring Criteria is provided as an example and may be adopted by departments as a method to assess potential risk.

CalHHS implemented an agency-wide governance structure in October, 2014. The governance structure acts both in a decision-making and advisory capacity to Agency leadership and its departments and offices. Implementation of the governance framework supports information technology (IT) initiatives that are more tightly aligned with meeting business objectives, enhanced project prioritization and improved strategic IT investment decisions. The Executive Sponsor is the Undersecretary of CalHHS. The Advisory Council consists of representatives of senior leadership from departments and offices in the Agency. There are five subcommittees that report to the Advisory Council, which include the Portfolio, Procurement, Infrastructure, Risk Management and Data Subcommittees. The Data De-identification Workgroup was convened by the Data Subcommittee with representation from all departments and offices in CalHHS.

CalHHS is engaged in improving transparency and public reporting through the Open Data Portal. As described in the CalHHS Open Data Portal Handbook, not all data is suitable for use on the open data portal. Data is Publishable State Data if it meets one of the following criteria:

1. Data that are public by law such as via the or

As described in and Figure 2, personal characteristics of individuals introduce the most significant risk with respect to identifying individuals in a data set. The following are examples of personal characteristics.

• Identifiers as defined in CA IPA

• Identifiers as defined in HIPAA

• Demographics typically reported in census and other reporting

The CalOHII is authorized by state statute to coordinate and monitor HIPAA compliance by all California State entities within the executive branch of government covered or impacted by HIPAA. To help ensure full compliance with HIPAA, CalOHII conducted a reassessment with all State Departments in January 2014 and updated as of July 27, 2015. The following are the self-reported results of this reassessment:

Many CalHHS programs oversee, license, accredit or certify various businesses, providers, facilities and service locations. As such, the programs report on various metrics, including characteristics of the entity and the services provided by the entity.

1. Characteristics of the entity are typically public information, such as location, type of service provided, type of license and the license status.

2. Services provided by the entity will typically need to be assessed to see if the reporting includes personal characteristics about the individuals receiving the services. Several examples are shown below.

   1. Reporting number of cases of mental illness treated by each facility – if the facility is a general acute care facility then the reporting of the number of cases does not tell you about the individuals receiving the services.

   2. Reporting number of cases of mental illness treated by each facility – if the facility is a children’s hospital then the reporting of the number of cases does tell you about the individuals receiving the services.

   3. Reporting number of psychotropic medications prescribed by a general psychiatrist does not tell you about the patients receiving the medications.

   4. Reporting number of psychotropic medications prescribed by a general psychiatrist to include the number of medications prescribed by the age group, sex or race/ethnicity of the patients receiving the medications does tell you about the patients receiving the medications.

In (a) and (c) above, assessment for de-identification is not necessary as there are no characteristics about the individuals receiving the services. However, in (b) and (d) above, the inclusion of personal characteristics which may be quasi-identifiers, especially when combined with the geographical information about the provider, does require an assessment for de-identification.

Taking the time to plan your project is essential. Whether you’re managing a team, analyzing or cleaning a portion of data, or drawing conclusions from your findings, completing any portion of the project requires a great deal of thought and planning. In the following section, we’ll provide a clear, step-by-step guide to the entire planning process, including everything you need to know about creating goals, determining a plan, and getting your data. It is our hope that you leave this section with a detailed and specific plan, and the confidence that you have the tools to carry out a successful project.

Set Clear and Measurable Outcomes

Be sure that you are measuring success and thinking critically about what your success metrics will be. You must have clear and actionable goals that you want to achieve with your data project.

Define Your Own Meaning of Data

Everyone is going to define data differently. Start by understanding what data means for your Department. It also will be important for you to prioritize your data. You must know which data are the highest value to your organization.

Start Small

You’re going to want to be sure to start small. Running a few pilots around data can’t hurt; this will help you get a better understanding of the lay of the land, what you can improve with data, and how you can identify the gaps.

Identify Staff to Manage Data

Every employee will need different kinds of accessibility, so make sure that your data systems maps to these needs and is not providing unauthorized access to information.

Focus on Education and Training

This will help staff clearly see the impact of the project and how data can improve effectiveness and efficiency. Including the Playbook in a new employee orientation could further enable staff and foster a culture of data within a Department.

Explore Shared Services Models

Don’t have access to the IT services you need? Maybe there is a shared service you can use with other Departments, or there are easier ways to get access to contemporary technologies. It’s also possible that you could iteratively re-engineer your existing IT infrastructure to gradually meet emerging needs.

After completion of the statistical de-identification process, each department will specify the additional review steps necessary for public release of various data products. Products may include but are not limited to reports, presentation, tables, PRA responses, media responses and legislative responses. See Section 7: Approval Process for further discussion.

CalOHII......... California Office of Health Information Integrity

CDC.............. Centers for Disease Control and Prevention

CDPH........... California Department of Public Health

CDSS............ Department of Social Services

CHHS........... California Health and Human Services Agency

CMS.............. Centers for Medicare and Medicaid Services

CPHS............ Committee for the Protection of Human Subjects

DDG.............. Data De-Identification Guidelines

DHCS........... Department of Health Care Services

HIPAA........... Health Insurance Portability and Accountability Act

IPA................. Information Practices Act

MHSOAC..... Mental Health Services Oversight and Accountability Commission

OSHPD......... Office of Statewide Health Planning and Development

PAR-DBR..... Public Aggregate Reporting - DHCS Business Reports

PHI................ Protected Health Information

PI................... Personal Information

PRA............... Public Records Act

PRT............... Peer Review Team

Governance for DDG will be provided by the Data Subcommittee with support from the Risk Management Subcommittee. The Subcommittees are part of the CHHS governance structure as described in the CalHHS Information Strategic Plan. Governance for the CalHHS DDG will provide the following support for departments and offices.

• Maintain the CalHHS DDG, which will include updates and revisions to the document as well as annual reviews for currency.

• Coordinate integration of the CalHHS DDG into the Statewide Health Information Policy Manual (SHIPM), Section 2.5.0 De-identification and the CalHHS Open Data Handbook.

• Convene a Peer Review Team (PRT).

• Provide for escalation of issues that cannot be resolved by the PRT.

The CalHHS PRT will include no more than two representatives from each department or office. Membership of the PRT is expected to include individuals with the following background and experience.

• Knowledge of and experience with generally accepted statistical and scientific principles and methods for rendering information not individually identifiable.

• Knowledge of and experience with legal principles associated with data de-identification in compliance with California IPA and HIPAA.

The PRT will have the following responsibilities:

• Provide review and consultation regarding a department’s DDG to ensure it is consistent with the CalHHS DDG. This may be particularly useful if a department incorporates methods for de-identification in the department’s DDG that have not already been documented in the CalHHS DDG.

• Provide for escalation and review of data de-identification questions or issues that a department is not comfortable resolving independently.

• Develop training tools to be used by departments when developing and implementing department specific DDGs based on the content of the CalHHS DDG.

The PRT will not review all disclosures or data released by each department.

Aggregate – formed or calculated by the combination of many separate units or items (Oxford Dictionary).

De-identified – generally defined under the HIPAA Privacy Rule (45 CFR section 164.514) as information (1) that does not identify the individual and (2) for which there is no reasonable basis to believe the individual can be identified from it.

Denominator – the portion of the overall population being referenced in a table or a figure representing the total population in terms of which statistical values are expressed (Oxford Dictionary).

Numerator – the number of specific cases as identified by the variable from a given population or the number above the line in a common fraction showing how many of the parts indicated by the denominator are taken (Oxford Dictionary).

Protected Health Information – information which relates to the individual’s past, present, or future physical or mental health or condition, the provision of health care to the individual, or the past, present, or future payment for the provision of health care to the individual, and that identifies the individual, or for which there is a reasonable basis to believe can be used to identify the individual (HIPAA, 45 CFR section 160.103).

Personal Information – includes information that is maintained by an agency which identifies or describes an individual, including his or her name, social security number, physical description, home address, home telephone number, education, financial matters, email address and medical or employment history. It includes statements made by, or attributed to, the individual (California Civil Code section 1798.3).

Publishable State Data – Data is Publishable State Data if it meets one of the following criteria: (1) data that are public by law such as via the PRA or (2) the data are not prohibited from being released by any laws, regulations, policies, rules, rights, court order, or any other restriction. Data shall not be released if it is highly restricted due to the Health Insurance Portability and Accountability Act (HIPAA), state or federal law (such data are defined as Level 3 later in this handbook.)

Re-Identified – matching de-identified, or anonymized, personal information back to the individual.

The following list of variables is important to consider when preparing data for release.

As stated previously, variables that are personal characteristics may be used to determine a person’s identity or attributes. When these characteristics are used to confirm the identity of an individual in a publicly released data set, then a disclosure of an individual’s information has occurred. Individual uniqueness in the released data and in the population is a quality that helps distinguish one person from another and is directly related to re-identification of individuals in aggregate data. Disclosure risk is a concern when released data reveal characteristics that are unique in both the released data and in the underlying population. The risk of re-identifying an individual or group of individuals increases when unique or rare characteristics are “highly visible”, or otherwise available without any special or privileged knowledge. Unique or rare personal characteristics (e.g., height above 7 feet) or information that isolate individuals to small demographic subgroups (e.g., American Indian Tribal membership) increase the likelihood that someone can correctly attribute information in the released data to an individual or group of individuals.

Variables that are event characteristics are often associated with publicly available information.

Therefore, increased risk occurs when personal characteristics are combined with enough granularity with event characteristics. One could argue that if no more than two personal characteristics are combined with event characteristics then the risk will be low independent of the granularity of the variables. This hypothesis will need to be tested using various population frequencies to quantify the uniqueness of the combination of variables both the in the potential data to be released as well as in the underlying population.

Data Sharing Guidelines

The Data Sharing Guidelines published here is a combination of the previously published CHHS Data Playbook, Data Playbook Beta, and the newly produced Data Sharing Guidelines from OTSI/CDII. The current version of the Data Sharing Guidebook is 1.0.

History

CalHHS Data Sharing Guidelines

CalHHS Data Playbook (Beta)

CalHHS Data Playbook

Introducing the CalHHS Data-Sharing Guidebook

Data is a core facilitator for the California Health and Human Services (CalHHS) vision of:

“… a connected network of state, local, and community providers, an ecosystem of coordinated services, and the secure, appropriate use of comprehensive demographic, socioeconomic, encounter, and outcomes data to generate insights that drive equitable policies, programs, and service delivery.” – CalHHS IT & Data Strategic Plan, March 2024

With over 33,000 employees, we at CHHS make up the largest agency under California’s executive branch. Our 13 departments collectively have access to an unprecedented amount of data -- it is our mission to use this data to improve the programs and services we deliver to our clients, amplify the impact of the data reports we create, and to create an organizational culture that is focused on data-driven decision making.

Realizing this vision requires data flow across the Agency’s departments and partner organizations. The CalHHS Data-Sharing Guidebook’s purpose is to be an asset that is broadly applicable to departments with various levels of data-sharing capabilities and experience. The diagram below provides the CalHHS Data-Sharing Guidebook’s purpose in California’s data- sharing ecosystem.

The Guide is composed of two sections:

• The Business Use Case Proposal Lessons Learned summarizes the learnings from data-sharing experts who participated in the discovery sessions used to create this Guide.

• The Data-Sharing Plays provide how-to guidance to implement capabilities that help establish data-sharing agreements and deliver meaningful data.

The Guide is a supplement to existing data-sharing assets, including:

• State of California

• The CalHHS

• Department-level data-sharing assets

The Guidebook is shared with you and your Department so you can use its content to supplement existing department data-sharing processes and training materials.

The Guidebook’s primary audience is data coordinators and data management teams that create data-sharing agreements, fulfill data requests, and consume shared data. The Guidebook’s Data-Sharing Plays also benefit your department’s data analyst, application development and security teams.

Play 1: Establish Data-Sharing Metrics and BUCP Tracking

If not already in place, we recommend creating a centralized department-level repository of BUCPs and mechanisms to track data-sharing metrics. Your previous BUCPs are valuable assets. Create a centralized department-level document repository for your BUCP artifacts to:

• Establish a library of reusable BUCP content to reduce content creation time.

• Maintain templates such as standard security controls and forms required by your department.

• Store reference material such as resolutions to disputes and solutions to statutory restrictions for data-sharing.

• Store quantitative and qualitative information on data-sharing outcomes to include in business cases to invest in data-sharing.

A departmental-level view of all BUCPs also helps prioritize the data-sharing improvement efforts by identifying frequently requested datasets.

Your BUCP document repository can also be a resource for lessons learned as BUCPs are approved and fulfilled. Collect team input to create “lessons learned” that you gleaned from working through the BUCP process. Retrospectives are an effective way to engage your team to collect and document lessons learned. Feel free to use the CalHHS Data-Sharing Guide: Lessons Learned section as a starting point for your department’s lessons learned.

Make sure your data-sharing team is aware of your department’s lessons learned as they develop. You can also review the lessons learned during BUCP kickoff efforts or as challenges arise during approval and data fulfillment. Other departments can benefit from your experience as well to apply or tweak them for their purposes. Please share them with other departments through CalHHS data-focused subcommittees and workgroups.

Identify BUCP-Related Metrics to Track

We recommend conducting a discovery session with your data-sharing team and relevant stakeholders to identify the BUCP-related metrics for tracking and reporting. Some examples of data points you may want to include in your BUCP tracking process are provided in the table below:

Once you have your metrics and data points identified, create a mechanism to collect them in your data-sharing tracking system.

Selecting BUCP Tracking Tools

You should check with your department’s technology and enterprise architecture teams to identify platforms for your BUCP tracking system. Also, consult with your department’s information security team to identify any required security controls for your BUCP repository. Your selected platform for BUCP tracking should include the following capabilities:

• Document repository to store BUCP’s and related documents.

• Capabilities to identify and retrieve BUCPs and other documents.

• Mechanism to track BUCP status and metrics.

• Task tracking to coordinate BUCP fulfillment activities.

Make sure your BUCP document repository is secured with access control to safeguard sensitive information including the contents of the Specialized Security and Specialized Privacy BUCP field content.

The State of California’s Office 365 (O365) provides tools to create a BUCP tracking mechanism using Excel and Microsoft Teams.

Describing Application Program Interfaces

Technology standards for metadata differ between Application Program Interfaces (API's) and data stored in databases or files. The techniques provided in the Guide to identify and describe data elements are pertinent to describing your APIs. If your data-sharing improvement effort is API- focused, reference this section during the execution of Play 5: Establish Your Metadata Repository to establish a repository compatible with API metadata standards.

API Description Standards

Industry standards and best practices will evolve with the field. Currently, there are two widely adopted API description standards:

• (OAS) is a broadly recognized metadata specification for API descriptions.

• (RAML) is an API modeling language for developing and publishing API descriptions.

Which one should you use? Each option has its considerations.

If your application development team has already adopted an API description standard, leverage their work and adopt the same standard. If your API descriptions are not current, execute Plays 2 through 6 to make them current.

If an API specification standard is not already in place, you need to work with your application development and enterprise architecture teams to jointly select a standard.

A primary consideration is which standards are supported by your API Gateway/Management platform. Review your platform’s documentation to identify supported standards. Converters are available to change formats between OAS and RAML; however, this adds an additional step to your deployment process.

Another critical factor is each standard's approach to capturing custom metadata that is beneficial for data sharing, such as security classifications and statute citations. OAS 3.x provides embedded in the description file. RAML extends its base metadata elements with Your development team should evaluate each option to determine which platform’s approach to custom metadata best aligns with their processes.

Publishing Your API Descriptors

For your API descriptors to be useful, they must be easily accessible by API consumers. Check if your API Gateway/Management platform provides an API catalog to publish your APIs. If this feature is available, it is the best option as it easily integrates publishing your API descriptors into your development and deployment processes.

If your platform does not provide an API catalog, you can make your API descriptors accessible using a web server or an object store (e.g., AWS S3, Azure Blob Storage). OAS and RAML publish descriptors as JSON files that are viewable using a web browser. Your team will need to create and maintain an index webpage that contains links to your API descriptions. Be sure to include steps to publish the API descriptor and update the index webpage in your development processes.

Many data catalog platforms maintain the JSON-based formats used by both OAS and RAML, creating an option for publishing your APIs for data consumers. Using a data catalog also allows custom metadata to be captured if using extensions/overlays is not an option. Using a data catalog also provides the benefit of establishing data lineage from APIs to backend databases.

Keeping Your API Descriptors Current

The approach described in Play 6.3: Keep Your Metadata Repository Current that integrates metadata maintenance into your development process applies to maintaining your API descriptions. RAML combines the creation of API descriptions with modeling and development. OAS generates descriptions from the underlying application code used to develop the API. Both options create a closed loop between development and descriptors to keep your API catalog current.

Play 7: Promote Data Awareness

Your department and its staff invested in data-sharing improvements by completing the previous Plays. It’s time to put your data into use internally within your department and externally with other organizations.

This Play provides ideas about how to communicate the available data for sharing so that your department yields benefits from data-sharing improvements. Promoting data awareness and consumption is important to justify future data-sharing investments and creates a sense of accomplishment for the team that helped the Plays.

Play 7.1: Internal Data Awareness

As previously noted, data-sharing improvements aren’t just for the benefit of data-requesting organizations. Your internal department teams benefit from the metadata you collected in . The Guidebook’s supplemental section, Benefits to Your Department from Executing the Plays, describes the benefits of executing the Plays. You can use this supplemental section to identify teams within your department and notify them of the enriched metadata and artifacts created in Play 6. Examples of stakeholders who may benefit from your team’s work include:

• Data Analytics

• Application and Data Management

• Information Security

• Departmental Programs

The first step is identifying internal stakeholders and teams benefitting from your data-sharing improvements. The supplement helps you identify internal analytics, technical, and security teams that benefit from your enhanced metadata repository. The notes on the department’s datasets collected in may identify programs that frequently share data within your department.

Start your data awareness campaign by emailing relevant stakeholders to notify them of the improved datasets and their benefits. Be sure to use terms that are relatable to your target audience. For example, using the term “data descriptions” is more easily understood than “metadata.” You can also schedule a Lunch and Learn or online meeting to review the benefits and how to use your metadata repository. This creates an opportunity for rich discussion about the data, how it is used, and ideas for future improvements.

Track the benefits and their impacts as your improved dataset is consumed by internal stakeholders. You can supplement your BUCP tracking repository with internal benefits to help justify investment in future data-sharing efforts.

Be sure to notify interested internal stakeholders as you incrementally improve data-sharing for new datasets.

Play 7.2: Agency-Level Data Awareness

This Play provides ideas for promoting awareness of the enhanced dataset with other CalHHS departments. A great way to promote data awareness is through the CalHHS Data Subcommittee and Data Coordinators workgroup. Participants in these groups can help spread the word about data to interested stakeholders within their departments.

The CalHHS Open Data Portal’s s a tool to identify datasets. To contribute to the open data catalog and promote awareness of your datasets, create a de-identified dataset for publication in the .

If you have datasets that other departments and organizations frequently request, publishing data-sharing information on your website is beneficial. The Department of Health Care Access and Information (HCAI) website is an example of a site that helps promote data awareness and data requestors’ understanding of data-sharing requirements.

Budget reporting may include both actuals and projected amounts. Projected amounts, although developed with models that are based on the historical actuals, reflect activities that have not yet occurred and, therefore, do not require an assessment for de-identification. Actual amounts do need to be assessed for de-identification. When the budgets reflect caseloads, but do not include personal characteristics of the individuals in the caseloads, then the budgets are reflecting data in the Providers and Health and Service Utilization Data circles of the Figure 2 Venn Diagram and do not need further assessment. However, if the actual amounts report caseloads based on personal characteristics, such as age, sex, race or ethnicity, then the budget reporting needs to be assessed for de-identification.

Facilities, Service Locations and Providers

Many CalHHS programs oversee, license, accredit or certify various businesses, providers, facilities and service locations. As such, the programs report on various metrics, including characteristics of the entity and the services provided by the entity.

1. Characteristics of the entity are typically public information, such as location, type of service provided, type of license and the license status.

2. Services provided by the entity will typically need to be assessed to see if the reporting includes personal characteristics about the individuals receiving the services. Several examples are shown below.

   1. Reporting number of cases of mental illness treated by each facility – if the facility is a general acute care facility then the reporting of the number of cases does not tell you about the individuals receiving the services.

   2. Reporting number of cases of mental illness treated by each facility – if the facility is a children’s hospital then the reporting of the number of cases does tell you about the individuals receiving the services.

In (a) and (c) above, assessment for de-identification is not necessary as there are no characteristics about the individuals receiving the services. However, in (b) and (d) above, the inclusion of personal characteristics which may be quasi-identifiers, especially when combined with the geographical information about the provider, does require an assessment for de-identification.

Mandated Reporting

CalHHS programs are required to provide public reporting based on federal and California statute and regulations, court orders, and stipulated judgments, as well as by various funders. Although reporting may be mandated, unless the law expressly requires reporting of personal characteristics, publicly reported data must still be de-identified to protect against the release of identifying or personal information which may violate federal or state law.

Current Version: 1.2 March 3, 2025

This is an important step as it will help us validate outcomes and determine successes. It also will help identify lessons learned, which will grow our toolbox and provide us with better intelligence. This ultimately will allow us to generate new content and additional best practices to help other Departments across the Agency.

Process Evaluations

Assess the processes involved in organizing and/or implementing the project. The focus here is on evaluating organizational and project capabilities rather than results.

Impact Evaluations

Assess short term objectives, which suggest that your larger goals are being achieved. Impact evaluations are much easier to measure because they consider benefits in terms of changes in beliefs and attitudes, skills, behavior and/or policies, structures and systems.

Outcome Evaluations

Assess how effective you have been in meeting big picture goals. The difficulties associated with outcome evaluations include: attributing change to any one particular project; long periods between the project and being able to see change; and finding reliable and valid ways of gathering this type of information.

Establishing an evaluation process will ensure that the benefits anticipated by the implementation of any particular program or policy change are realized and an assessment can be made of the project’s overall success.

What are the lessons learned? How will you iterate on the current solution? What are the next steps?

Purpose of the CalHHS Open Data Handbook

The California Health and Human Services (CalHHS) Open Data Handbook provides guidelines to identify, review, prioritize and prepare publishable CalHHS data for access by the public via the and – with a foundational emphasis on value, quality, data and metadata standards, and governance. This handbook is meant to serve as an internal resource and is also freely offered to any party that may be interested in improving the general public’s online access to data and to provide an understanding of the processes by which CalHHS makes its publishable data tables available. The handbook focuses on general guidelines and thoughtful processes but also provides linked tools/resources that operationalize those processes. The CalHHS Open Data Handbook is based on and builds upon the New York State Open Data Handbook, and we would like to acknowledge and thank the New York staff who created that document and made it available for public use.

The breadth of data and participation by departments and offices within CalHHS are continually being enhanced and expanded, making open data a dynamic, living initiative. This handbook, providing guidelines for broad publication of publishable state data in electronic, machine-readable form, is the first step in a major shift in the way CalHHS departments and offices share information publicly to promote efficiency, accessibility and transparency; and a significant improvement in the way CalHHS government engages citizens and fosters innovation and discovery in the scientific and business communities. It begins the process of standardizing the state’s data, which will make it easier to discover and use the data. Working in collaboration with others, this Handbook will be supplemented, as needed, with technical and working documents addressing specific formatting, data preparation, data refresh and data submission requirements. CalHHS and its departments and offices will use this handbook in their work as they consider various perspectives involved in governing business processes, data, and technology assets.

HIPAA covered entities in CHHS must de-identify data in compliance with the HIPAA standard. Under the HIPAA standard, either Safe Harbor or Expert Determination must be used. If Expert Determination is used then the documentation of the review is essential. The following may serve as a template for this documentation with the reference to the CHHS DDG to support the analysis documented.

Documentation of Expert Determination Template Name of Report:

Reason for Data Release:

Identify why the data release does not meet Safe Harbor. For example:

The request does not meet the Safe Harbor standard because it includes counts by county (geographic area smaller than the state) or counts by month (which does not meet the criteria for dates). Therefore, the steps in the CHHS DDG are being used to assess the tables.

After completion of the statistical de-identification process, each department will specify the additional review steps necessary for public release. This may vary depending on the purpose of the release and whether or not the department/program is a HIPAA covered entity.

Recognizing that some data analyses may be published as independent tables while other analyses will be part of larger reports, the final review of all data analyses must follow the department or office procedures for document review in addition to review procedures identified for the implementation of the DDG. The expectation is that the review of data for de-identification will fit into other routine review processes. Reviews outside the DDG portion may vary depending on whether data is being released for a PRA request, to the media, to the legislature, by the program as part of routine reporting, or for other reasons.

Departments and offices may consider the following components for reviews related to data that has been de-identified.

• Statistical Review to Assess De-identification (for HIPAA entities this may be an Expert Determination Review)

As noted in , the Publication Scoring Criteria is based on a framework that has been in use by the Illinois Department of Public Health, Illinois Center for Health Statistics. Various other methods have been used to assess risk and the presence of sensitive or small cells. Public health has a long history of public provision of data and many methods have been used. Some of those methods are highlighted here.

• Ohio Department of Health published a Data Methodology Standards for Public Health Practice. This method is framed around the concept that a Disclosure Limitation Standard for tabulations of confidential Ohio Department of Health data shall be suppressed when the table denominator value minus the table numerator value is less than 10.

The DDG describes a procedure, the Data Assessment for Public Release Procedure shown in Figure 5, to be used by departments in the CalHHS to assess data for public release. This section, section 4, describes specific actions that may be taken for each step in the procedure with additional supporting information being described in sections 5, 6 and 7. These steps are intended to assist departments in assuring that data is de- identified for purposes of public release that meet the requirements of the California IPA to prevent the disclosure of personal information.

The Data Assessment for Public Release Procedure includes the following steps:

1. Review the data to determine if it includes personal characteristics, directly or indirectly, that can be tied back to an individual;

2. If there is concern for personal characteristics, then assess the data for small numerators or denominators;

Metadata repository platforms save time in the maintenance of your data dictionary and improve access to metadata. Data catalog platforms also make it easier to manage custom metadata, including references to applicable statutes that govern data-sharing. Data catalog platforms also promote the use of your data-sharing artifacts by improving access through web access and search functions.

Standard data catalog features include:

• Automated collection of metadata from compatible data sources.

• Configurable and customizable metadata labels and tags to address the specific requirements.

Current Version: 1.0 September 23, 2016

Discovery sessions are an effective approach to effectively gather information from staff across your department. There is no set format for a discovery session. Discovery session questions will be context-specific to your department’s needs and the program(s) related to a data set.

Some key aspects of an effective discovery session include:

• Research Ahead of Time: Review any available data architecture and program documentation to identify initial questions and build context before the discovery session. Reviewing materials ahead of the meeting also ensures efficient use of participant time.

• Prepare Your Questions: Creating a set of discovery questions helps ensure you obtain the information you need on the data set. Be prepared for additional topics and questions to arise organically during the discovery session.

• Capture the Information: Ideally, have one person ask the discovery questions while another captures meeting notes. Often this is impossible. If acceptable to the participants, record the meeting for later review.

• Confirm Your Understanding: Communicate your understanding to participants for confirmation and clarity.

When scheduling discovery sessions, include context for the effort and the objectives for the discovery session. To establish a common understanding, the session should start with a definition of data-sharing; examples of data-sharing help provide additional context. The provides some short videos that review impactful data-sharing efforts.

The article from UX Everything provides useful tips for preparing and conducting discovery sessions.

Statistical masking provides an extensive set of tools that can be used to mitigate potential risk in a given data presentation. As discussed in Section 4.4, the data releaser will assess the need for statistical masking when the assessment in Step 3 identified potential risk. Each department will document statistical masking processes that are routinely used in data preparation for public release.

As discussed in section 4.4, initial methods to address sensitive or small cells, as well as complimentary cells include the following:

• Reduce Table Dimensions

• Reduce Granularity of Variable(s), aka Recoding or Aggregation

Establishing Minimum Numerator and Denominator

The DDG workgroup reviewed the published literature including information from other states and from the federal government. There was a great deal of variation in the numerical values chosen for the Numerator Condition. While the Centers for Disease Control and Prevention (CDC) WONDER database suppresses cells with numerators less than 10, the National Environmental Public Health Tracking Network suppresses cells that are greater than 0 but less than 6. Examples range from 3 to 40 with many being 10 to 15. The Centers for Medicare and Medicaid Services (CMS) uses a small cell policy of suppressing values derived from fewer than 11 individuals. As stated in a 2014 publication associated with a data release of Medicare Provider Data, “to protect the privacy of Medicare beneficiaries, any aggregated records which are derived from 10 or fewer beneficiaries are excluded from the Physician and Other Supplier PUF [public use file]

A critical step in reviewing data for public release is the consideration of what other data may be publicly available that could be used in combination with the newly released data to identify the individuals represented in the data. This section will highlight some specific data sets that are publicly available that may be used in combination with CHHS data that would contribute to potential increased risk.

Common kinds of data with personal information include: real estate records, individual licensing databases (MD, RN, contractors, lawyers, etc.), marriage records, news (and other) media reports, commercially available databases (data brokers, marketing), court documents, etc.

Vital Records Data

Another common data set for programs to be aware of are the publicly available electronic birth and death indices from Vital Records, as specified in Health and Safety Code section 102230(b).

CalHHS programs develop a wide range of information based on different types of data. This is reflected in the various categories shown on the entry page for the CalHHS Open Data Portal, which include:

• Diseases and Conditions

• Facilities and Services

• Healthcare

The CHHS Data Subcommittee requested the convening of the CHHS Data De- Identification Workgroup to develop the DDG.

The DDG Workgroup began with an orientation to the topic of data de-identification and presentations by the DHCS, OSHPD and California Department of Public Health (CDPH) regarding current practices and activities related to data de-identification. The DDG Workgroup used the Public Aggregate Reporting for DHCS Business Reports (PAR-DBR) as a starting point for initial drafts. The PAR-DBR had been developed between April and August, 2014 through a workgroup processes within DHCS with input and presentations from OSHPD, CDPH, and University of California, Los Angeles California Health Interview Survey. The PAR-DBR served as a basis for this document, including the literature review conducted as part of the development of the PAR-DBR.

The development process was designed to include an updated literature review, case examples and broad discussion among CHHS programs. Publishing data publicly is always a balance between the protection of confidentiality and the usability of the data.

The project timeline for the CHHS DDG Workgroup is below:

Disclosure Considerations

CalHHS collects, manages, and disseminates a wide range of data. As departments classify data tables and catalog their publishable state data, they should be mindful of legal and policy restrictions on publication of certain kinds of data. Following are general guidelines regarding disclosure to consider as departments begin to identify and review data tables.

The CalHHS Data Subcommittee commissioned the development of Agency-wide guidelines to assist departments in assessing data for public release. The are focused on de-identification of aggregate or summary data. Aggregate data means collective data that relates to a group or category of services or individuals. The aggregate data may be shown in table form as counts, percentages, rates, averages, or other statistical groupings. Refer to the for the specific procedures to be used by departments and offices.

This section will help Departments move from conversation to action. It will provide resources that will allow the Departments to succeed. This might include: roles and responsibilities; governance structures; and data standards. It also includes resources on project management and change management to help both staff implement and leadership enforce the importance of this work.

This content moves away from the theoretical and drives toward the tactical.

By detailing all critical steps before starting the project, the Department can anticipate factors they otherwise would not consider until encountered and identify potential problems and challenges on the front end.

The planning becomes proactive instead of reactive, which allows best practices to be used and ensures that energy and time are spent on implementing a high-quality, well‐thought‐out project rather than "putting out fires."

The planning and implementation processes will allow any person working on the project, regardless of his or her level of involvement, to fully understand the goal of the project and how it is to be accomplished. It ensures that everyone working on the project is on the same page and that any discrepancies are resolved before they become costly to the project or population served.

This play includes various resources designed to help with project management and data management.

The Data-Sharing Plays (Plays) improves data data-sharing by:

• Improving understanding of your data by external departments and internal data consumers.

• Mitigating late detection of security, privacy, and statutes during Business Use Case Proposal (BUCP) approvals.

• Providing a detailed description of your data to facilitate data requestors properly scoping datasets.

• Automating inventory of data fields to complete the BUCP Technical Fields.

• Implementing technical and security capabilities to receive shared data from other programs.

• Establishing BUCP metric tracking to optimize processes and data-sharing planning.

The Plays can be executed proactively before data-sharing requests or in parallel with a BUCP approval. Completing the Plays requires resource investment by your department but will provide benefits for data sharing with other organizations and direct benefits to your department. One of the participants in the Guidebook discovery sessions summarized the investment of time and future benefits as follows:

“We are willing to invest time upfront to save time in the future.”

– Discovery Session Participant from a CalHHS Department.

The Plays contain data architecture practices (the What) that support data-sharing and guidance to implement them (the How). A series of vignettes that depict a fictional department’s journey to improve its data-sharing capabilities are used to provide further context and implementation guidance. Not all Plays may be pertinent to your organization, depending on your existing data-sharing capabilities. We recommend reviewing the Guide in its entirety and leveraging concepts that benefit your department.

Technical Context

The examples and vignettes are based on extracting data from a database to address a common source of shared data. The Data-Sharing Plays are adaptable to data exchanges that use Application Program Interfaces (APIs) or other data integration architectures.

Data-Sharing Vignettes

The Plays include a series of vignettes featuring a fictional CalHHS department (California Department of Wellness) to provide context for each Play. The vignettes further elaborate each Play by telling a story that demonstrates how to its guidance into action. Reading the vignettes is optional but recommended as they offer valuable context and practical examples of the data- sharing plays in action.

The personas and their roles depicted in the vignettes are examples to illustrate the concepts described in the Plays. The roles and responsibilities of the vignette’s personas may be different from your department’s structure.

Vignette Context

The California Department of Wellness (CDW) was established in 1997 to improve the health and well-being of Californians. The CDW focuses on various aspects of wellness, like physical health, mental health, and social engagement. The CDW’s programs are interrelated with those managed by other departments under CalHHS. Although CDW is a small department, it manages over a dozen programs and supporting systems developed independently in response to various legislation over the past 20 years. Some primary CDW programs include:

• Healthy Habits is a CDW program that raises awareness about healthy lifestyle habits through various marketing campaigns, collaborations with Community-Based Organizations (CBO), and healthcare providers.

• Walk2Work (W2W) encourages people to engage in additional physical activity to improve their overall wellness and to facilitate social interaction between commuters. The Healthy Habits program collaborates with the W2W program to promote physical activity initiatives.

• Your Environment program aims to educate Californians about the significance and advantages of engaging with their communities and enjoying the beautiful spaces within the state. The program conducts outreach activities through marketing campaigns and works with community-based organizations. Your Environment also collaborates frequently with the Department of Volunteer Services and the Department of Outdoor Recreation to achieve its goals.

Other CalHHS departments utilize program data from the CDW to understand the impact of wellness on their program outcomes.

Meet the core team that is working on the CDW data-sharing improvement effort:

• Sally is the CDW Data Coordinator (DC). Prior to joining the CDW, she worked at a local government social services department that utilized data-driven practices. Sally leads the data-sharing improvement effort and coordinates with participating staff.

• Carlos recently joined the CDW 6 months ago as the Data Architect and Lead Database Administrator. He previously worked at another department outside of CalHHS as a database administrator and is learning about the CDW environment. Carlos is helping Sally with the technical aspects of data-sharing.

• Andrea is the manager of CDW Information Technology Services (ITS). Her team includes the application and data support teams that will provide information to enhance the CDW data architecture. The ITS team has minimal resource capacity for anything beyond regular system support.

Other CDW staff that will help improve data sharing are introduced as they appear in Play- specific vignettes.

Sally recently facilitated a team retrospective after providing data under a BUCP to share a portion of the Healthy Habits program data. Common themes from the team’s feedback include:

• We are sharing the same data; it would be helpful to have some reusable materials to reduce the time necessary to establish a BUCP and provide data.

• A statute governing data-sharing was identified late in the BUCP approval and created delays.

• Determining what data can be shared is labor intensive, as is identifying data elements, metadata, and their security classifications.

• After providing data, CDW staff are spending time answering data-related questions from departments receiving CDW data.

Based on these findings, Sally is launching an effort to improve CDW’s ability to share data using the Data-Sharing Plays.

Data-Sharing Plays Summary

This Guidebook comprises eight Data-Sharing Plays that aim to improve your ability to share and receive data. The roadmap below depicts the relationships of the Plays and their suggested order of execution:

The tables below summarize the Plays and their primary objectives:

Objective: Launch Your Data-Sharing Improvement Effort

Objective: Improve Your Ability to Provide Data

Objective: Improve Your Ability to Provide Data

Data de-identification practices will be implemented by each department and office (further referred to as department) in the agency. This DDG is the default policy for CalHHS departments. If a CalHHS department wants to create a department DDG, it must have appropriate references to departmental processes and the department must file a copy of their DDG with the Office of the Agency Information Officer (OAIO). For example, the Legal Review process and the Departmental Release Procedures for De- Identified Data require additional information to describe these steps within each department. Additionally, a department with programs not covered by HIPAA will not require specific HIPAA references. A department must request DDG consultation from the CalHHS peer review team (PRT), described in Section 8: DDG Governance prior to implementation. The PRT is available to review the department’s documentation to ensure it is consistent with the principles of the CalHHS DDG and meets requirements of the California IPA.

The CalHHS DDG is focused on the assessment of aggregate or summary data for purposes of de-identification and public release. Aggregate data means collective data that relates to a group or category of services or individuals. The aggregate data may be shown in table form as counts, percentages, rates, averages, or other statistical groupings.

Departments are sometimes asked to release record level data. Record level data refers to information that is specific to a person or entity. For example, a record for Jane Doe may include demographics and case information specific to Jane Doe.

However, summary data would include information from Jane Doe combined, or summarized, with data from other individuals. If record level data is to be publicly released, it must be assessed to ensure it is de-identified and does not include Personal Information (PI) or Protected Health Information (PHI). Although the DDG is focused on summarized data, it can be used to assist with review of individual or record level data. The record level data should be assessed both for uniqueness of the records and for the possibility that the data can be used in conjunction with other information available to the requester to identify individuals in the data. Record level data inherently has higher risk than summarized data, even after personal identifiers are removed.

Therefore, record level data for public release should be assessed on a case by case basis.

CalHHS collects, manages and disseminates a wide range of data. The focus for the DDG is on data that includes personal characteristics of individuals who have a legal right to privacy. Personal characteristics include but are not limited to age, race, sex, and residence and other identifiers specified in the IPA and HIPAA and listed in the tables below. These guidelines will focus on the assessment of personal characteristics that are included in various data sets or tables to assess risk for identification of the individuals to which they pertain.

Unique Identifiers

Assessing the risk of an unauthorized disclosure that violates an individual’s right to privacy and/or confidentiality, as provided by statute, may be achieved by associating personal characteristics with a person’s identity or attributes. When these characteristics can successfully confirm an individual’s identity in a publicly released data set, then release of this data results in disclosure of personal information.

Less obvious qualities in data sets and elements that may be used to identify individuals or groups can present uniqueness in data. Individual uniqueness in the released data and in the population is a quality that helps distinguish one person from another and is directly related to re-identification of individuals in aggregate data. Disclosure risk becomes a concern when released data reveal characteristics that are unique in both the released data and in the underlying population. The risk of re-identifying an individual or group of individuals increases when unique or rare characteristics are “highly visible”, or are readily accessible by the general public without any special or privileged knowledge. Unique or rare personal characteristics (e.g., height above 7 feet) or information that isolate individuals to small demographic subgroups (e.g., American Indian Tribal membership) increase the likelihood that someone can correctly attribute information in the released data to an individual or group of individuals.

Assessment of variables and their uniqueness

There are a number of variables that are unique to individuals that have been identified in various laws and are considered identifiers (PI/PHI). There are two primary laws that describe identifiers, shown in Figure 1, in California: the IPA and the federal HIPAA. Other variables that are commonly used to publish information to the public have been called quasi-identifiers because while they are not unique by themselves, they can become unique in the right combination. The variables shown in the Publication Scoring Criteria in Figure 6 can be considered quasi-identifiers and will be discussed further in Sections 4 and 6.

Assessment of risk in the context of maximizing the usefulness of the information presented

The removal of PI and PHI from datasets is often considered straight-forward, because as soon as data is aggregated or summarized the majority of the data fields defined as identifiers in the IPA and HIPAA are removed. However, various characteristics of individuals may remain that alone or in combination could contribute to identifying individuals. These characteristics have been described as quasi-identifiers. Figure 2 helps demonstrate the quasi-identifier concept. For instance, there is interest in reporting about providers, where providers may be individuals, clinics, group homes, or other entities. Each of these providers has a publicly available address and has publicly available characteristics. While patients may come to a provider from anywhere, they typically will visit providers within a certain distance of their residence. Thus, by publicly publishing details on providers, data miners with malicious intent would have a targeted geography that lists locality information, types of services offered and received, and demographic information about patients. To expand on this example, data that states a provider saw two patients with heart disease does not indicate who had the heart disease nor does it reveal the identity of the two patients amongst the thousands of patients that provider sees. However, datasets that display a provider within a given region with two Black or African American female patients under age 10 with heart disease may release enough personal characteristics about the patients to successfully reveal their identity. These compounding patient details released about providers that give geography information (address), health condition (heart disease), and person- based characteristics (quasi-identifiers) of the patients puts the dataset in the overlapping area of the diagram of Figure 2. This overlap, consequently, highlights potential risks associated with seemingly innocent summary data.

Step 1: Identify your Guiding Questions and Set your Goals

It is important to decide your vision and purpose behind your project, and identify what you hope your data project will accomplish. Be thoughtful — what impact do you hope to have? What changes are you trying to bring about? It is worth taking the time to write down your answers to the broader Guiding Questions, as they will be the foundation of your goals and strategic plan.

First, a bit about goals: for your project to execute smoothly, it is best to choose SMART Goals, of goals that are specific, measurable, achievable, realistic, and timely. Look at our summary of the SMART Goal checklist below:

Step 2: Develop a Strategic Plan

Now that you have identified your goals, you must develop a strategy for achieving your desired outcomes. A Strategic Plan is first and foremost a Roadmap to Success – the more care and thought you put into your plan, the more likely you are to produce a successful data project.

Utilize a strategic planning framework such as the use case diagram or a logic model. These frameworks will help you explicitly define each step necessary to achieve your goals as well as anticipate what challenges you may face throughout your project.

Strategy Tip:

Find the action words that best describe the work you’ll do: Action words are verbs that describe how you will approach each task in this project. They don’t describe your intended outcome (i.e. increase and reduce are not action words); rather, they describe roles you will take throughout your project to assure a successful outcome.

If you are creating a product:

• Update, Upgrade, Develop, Create, Implement, Evaluate, Produce If you are managing a project:

• Oversee, coordinate, supervise, manage, plan, support, transition If you are implementing the specifics of a project:

• Write, process, provide, maintain, reconcile, direct, administer

The Use Case Diagram

This framework is most helpful for projects where you intend to build some sort of system (e.g. website, smart phone app, etc.) that your users must interact with. You also must use a Business Use Case for any data you request using the Data Sharing Agreement form.

A Use Case Diagram will…

• Identify the goals of system-user interactions

• Define and organize functional requirements in your system

• Specify the context and requirements of a system

• Model the basic flow of events in a use case

Instructions for building a Use Case Diagram:

Step 1: Start by defining your actors, or the users that interact with your system. they can be anything from a person to an organization or outside system that interacts with your product.

Step 2: For each user, list all the ways they can interact with your system (these are the “use cases”) Note: Ensure you consider alternate/undesirable courses of events and use cases that aren’t obvious

Step 3: Draw lines between use cases to reflect commonalities or relationships among them.

Note: Identify the use case with the greatest number of relationships/associations – the most common use cases represent the functions in your project that should be essential.

Also check out this to build your own Use Case Diagram.

The Logic Model

The logic model framework focuses on visualizing the relationship between inputs, outcomes, and costs associated with your project. It is a graphical model where each component (or “phase”) of your project relates to a list of intended effects in an implicit, ‘if-then’ way.

The seven “components” you’ll consider are:

1. Inputs: The resources you need for your project

2. Activities: What the staff or the program does with those resources

3. Outputs: Tangible products, capacities, or deliverables that result from the activities

4. Outcomes: Changes that occur in other people or conditions because of the activities and outputs

To begin, simply create six headers as is shown in the diagram above — this can be done by hand, with sticky notes, or online.

Guiding Questions:

1. Identifying Impact: What measurable change are you seeking to achieve in the long-term?

2. Identifying Outcomes: What measurable changes are you seeking to achieve in the short-term?

3. Identifying Outputs: What tangible outcomes can you measure immediately following the implementation of your product/project?

4. Identifying Activities: What are some high-level steps you must take to complete your project?

List everything that comes to mind when you answer the those guiding questions above, drawing a box around each entry. Finally, draw arrows between boxes to signify the ‘if-then’ relationship.

Play 8: Prepare to Receive Data

The previous Plays provided guidance to improve data-sharing between programs within your department and other organizations. It is equally important to take measures to improve your department’s ability to request and receive data. This Play helps you receive shared data by:

• Establishing an environment or shared service to store and consume shared data.

• Demonstrate and document security controls when requesting data.

• Identify available options to transfer shared data.

This Play can be executed in parallel with other Plays to simultaneously improve your department’s ability to share and receive data. Completing the following Plays helps you develop a business case to secure resources or funding to improve your department’s ability to receive shared data:

• Play 1: Establish Data-Sharing Metrics and BUCP Tracking

• Play 3: Create a Business Case

This Play does not have a supporting vignette.

Play 8.1: Assess Your Shared Data Environment

A ready-to-use data-sharing environment is a critical technical component of receiving shared data. Identifying gaps in your shared data environment late in a BUCP approval or data fulfillment creates delays. Proactively identifying gaps in your shared data environment provides time to address them ahead of received data.

Some questions to address during your shared data environment assessment include:

• Does the environment have sufficient capacity to store shared data?

• Is the environment accessible to your department’s staff who will use shared data?

• Is the environment sufficiently secure to receive shared data?

• Are there any funding restrictions that would restrict the environment’s use?

Create a list of any gaps identified during your shared data environment assessment. Use the list of gaps to work with relevant teams, such as enterprise architecture and information security, to determine resolutions. Since resources to support data sharing are likely limited, create a resolution plan that implements mitigations in the short term (e.g., manual processes) with a plan for full resolution over time.

Use the guidance from Play 3: Create a Business Case to create a business case combined with shared data impacts from the BUCP repository created in Play 2: Identify Your Datasets to identify past data-sharing business results and secure needed resources.

The Center for Data Insights and Innovation (CDII) Agency Data Hub (Data Hub) is an agency- level data-sharing service provider. The Data Hub may be an option that leads to a faster path to receiving shared data than resolving gaps in your internal environment.

The Agency Data Hub is a secure cloud-based data-sharing ecosystem built on a modern platform for data science and analytics. The Agency Data Hub is a collaborative platform that is non-specific to any program or department, where staff across Agency departments and researchers can work together on focused efforts.

For more information on the Agency Data Hub, please contact the Center for Data Insights and Innovation (CDII) at

Play 8.2: Identify Data Transfer Platforms

Data transfer platforms are used to deliver data from providers to recipients. Delays due to data transfer capabilities were cited in several instances during the discovery sessions used to create this Guidebook. Some examples of data transfer technologies include:

• Secure File Transfer Protocol (SFTP) for file-based data sharing.

• Representational State Transfer (REST) Application Program Interfaces (API) for ongoing data interfaces

• Data streaming (e.g., Apache Kafka) for continuous data transfer.

Work with your department’s information technology team to identify data transfer platforms that are available for data-sharing. While creating your BUCP, evaluate your data transfer platforms to verify the following:

• Is the platform compatible across data provider(s) and recipient(s)? For example, does the data recipient have the technical capabilities to access shared data via an API?

• Does the data transfer platform have network connectivity between the data provider and recipient?

• Are there any restrictions (e.g., funding) that prohibit using the data platform technology?

• Does the data transfer platform meet information security requirements?

If a suitable data transfer platform is unavailable, consider using shared services including:

• The California Department of Technology (SAFE) service is an option. The SAFE service allows for the secure transfer of files over public and private networks using encrypted file transfer protocols (FTPS, SFTP/SSH, and HTTPS).

• Although not a data transfer service, the CalHHS Agency Data Hub (Agency Data Hub) is another option to provide data access.

• Another option for small-volume datasets with limited security controls is to leverage the State of California Office 365 to use Microsoft Teams for data transfer.

Play 8.3: Demonstrate Security Controls to Data Providers

A data provider’s Information Security team may require a review of the controls used to secure sensitive shared data. Creating an inventory of security controls is time-consuming and can create data-sharing approval delays. You can ready your department to receive shared data by proactively documenting the security controls of technical environments that contain shared data. In addition to improving data sharing, this exercise improves your system’s information security posture by:

• Establishing alignment with the State of California Statewide Information Management Manual (SIMM) 5300.5.

• Identify areas to improve security controls and funding needs.

• Providing input to a security roadmap or Plan of Action and Milestones (POAM).

One option is to document the security controls using the National Institute of (NIST SP 800-53) controls. Documenting your security controls using NIST 800-53 provides a commonly accepted set of controls for evaluation by the department providing data. This effort can also improve your data security outside data sharing by identifying needed security control improvements.

For security approvals of “Commercial off the Shelf” (COTS) platforms, such as Office 365 (O365), you can use the NIST Cyber Security Framework (CSFG) 2.0, which is a simplified, summarized control assessment model. The NIST CSF aligns with NIST SP 800-53 and the State of California SIMM.

If you want to provide additional demonstration of your security controls to data providers, your department can engage a third-party assessor to evaluate your security. The California Military Department provides independent .

Crafting your data narrative

Sharing your findings with the world is just like telling any good story — sometimes it’s more about the storyteller than the story itself.

All too often, truly meaningful and interesting data projects fall through the cracks because they lack a cohesive narrative or don’t convince the audience why they should care. Remember, it’s up to you to decide how to best leverage your data to tell your story in a way that is compelling, interesting, and true to you. Here are some guiding questions to get you started:

Who is your audience?

Your data story can and should change based on your intended audience. The contextualizing information you provide, anecdotes you share, or images you include in a professional journal would be completely different from those you’d choose to share to a group of high school science students. Consider the following questions:

• What is your relationship to your audience?

  • Are you their peer? Did you used to be in their shoes? Do you have anything in common?

• What can you do to understand your audience?

  • Create an audience profile for one of your readers/users

General Tips

• Use a word editing app like to improve the readability of your writing

  • Hemingway will highlight lengthy or run-on sentences, remove overly dense writing, offer alternatives for weak adverbs and phrases as well as poor formatting choices.

• Connect to your audience emotionally — how can you make this more personal?

• Visualize your story with a storyboard (see

Designing Effective Visualizations

Finding the ‘best’ way to visualize your data takes time and experience — if you’re a beginner, focus your efforts on learning from others and refining your methods to master the art of translating data to diagrams.

|If you just need a quick chart or table, check out these online tools — they are simpler to use than the advanced data visualization guides and may be more appropriate for your specific project:

• (interactive charts & simple data tools)

• (charts, tables, and maps)

• (beginner-friendly, collaborative, focuses on design thinking principles) |

For more complex data projects, choosing the right visualization is more than just deciding between a pie chart vs. a bar graph — it’s about understanding your audience’s learning style and design preferences, leaning in to your creative side, and asking for lots of feedback.

Here are some resources to help you understand all types of data visualization, how to create them, and which choices are most appropriate for your data:

• Beginner: summarizing general Data Visualization strategies and common methods used in different professions and sectors

• Beginner: : a Definition & Learning Guide with helpful examples

• Beginner: This Step-by-Step Guide to Data Visualization and Design written for beginners

• Beginner-Intermediate: teaches you how to implement some more basic, powerful data visualization techniques (line charts, scatter plots, and distributions) and how to choose the right one

Sharing with Others

Getting your message out there requires you to actively share and distribute what you discovered or created.

Building the Data Community at CHHS

Across the agency, there are a few existing groups and initiatives that exist to help you leverage your department’s resources to publicize your findings. Take advantage of the resources available to you, ask for help from those who’ve done this before, and be proud of yourself for completing your project!

• There are a number of “Data Showcase Teams” across the agency. They organize events to build a shared understanding of data, celebrate successes and failures, and learn from each other’s projects.

• Your department or program may have an established visual and brand style that provides credibility to your data analysis, thus increasing its chances of publication. These styles standardize color themes, fonts, and citation formats across agency publications.

• A repository of CHHS data assets is currently underway to streamline creation, maintenance, and sharing of each department’s resources.

Governance Structure for CalHHS Agency and its Departments and Offices

CalHHS departments and offices are required to engage in an internal review process and obtain approvals for data tables to be published on the CalHHS Open Data Portal. Departments and offices are responsible for driving toward increasing data content, quality, and accuracy, as well as ensuring compliance with all security, privacy, confidentiality laws, rules, regulations, and intellectual property rights requirements. The CalHHS Agency will also engage in strategic planning and approvals as needed.

The CalHHS Open Data Portal governance model takes a multi-level approach that provides oversight in the most efficient/streamlined way possible. Figure 2 provides a graphical representation of the model.

How the Governance Model Works - Overview

Within each CalHHS department or office a Data Coordinator will lead the data preparatory process. This individual will facilitate initial solicitation for data table suggestions for publication and subsequent preparation activities among the department's centers, divisions and offices. Recommendations for publication of data tables will be submitted from that level to the department or office's strategic level (i.e., the executive leadership team and/or policy committee) for decisions related to data table prioritization and publication. When warranted (based on determination by the department's executive leadership team), approvals will be moved up to the Agency level of governance. This plan constitutes the 'vertical' governance for open data.

The governance model, however, also includes a 'horizontal' governance structure meant to ensure uniform approaches to data publication and standards as a way to facilitate interoperability and sharing among CalHHS department and offices. This portion of the governance model originates at the CalHHS Agency level and centers around team members, representing CalHHS department and offices, serving on the Open Data Workgroup, led by the CalHHS Agency Information Officer.

Roles within the Governance Structure

Data Coordinator – Department/Office Level

Each office and department within CalHHS will designate a Data Coordinator. The Data Coordinator should be an individual who:

• Has authority equivalent to that of a Deputy Director or the head of a division or office within their department (e.g. Chief Information Officer, Informatics Deputy Director, etc.);

• Can identify appropriate persons with comprehensive knowledge of data and resources in use by their centers/divisions/offices/programs;

• Assumes responsibility for their department's compliance with this handbook, the , and future directives which may be needed to support the CalHHS Open Data program;

• Ensures that guidelines and tools, established by CalHHS governance, are applied to aggregate data for public reporting that: (1) adequately minimizes risk of re-identification, and (2) produce meaningful information (that is, they enable statistically reliable calculations).

The Data Coordinator acts as a liaison between Information Technology staff, departmental programs and leadership, and CalHHS Agency. As such, the Data Coordinator is best positioned to convey to the appropriate parties any specific needs of the Open Data Portal (such as formatting the data or defining a structure that is optimal for publication). Insofar as departments vary in terms of size, functions, and complexity, larger departments may also identify individuals within specific divisions, offices, and/or units to assist the department Data Coordinator.

Executive Leadership – Department/Office Level

The executive leadership team and their policy-making committees within CalHHS Departments and Offices will be responsible for ensuring alignment of relevant Department/Office strategic goals with data publication priorities on the CalHHS Open Data Portal. In addition, executive leadership will ensure their Department/Office compliance with this handbook and all approval processes set forth herein (see Department/Office Approvals section below).

CalHHS Governance – Open Data Workgroup

CalHHS has established a governance structure that includes an open data workgroup made up of representatives from each CalHHS department or office publishing to the Open Data Portal.

The Open Data Workgroup has the following responsibilities related to Open Data:

• Identify and standardize the use and governance of information in support of the missions and strategies of CalHHS Agency and its departments and offices

• Develop and maintain controls on data quality, interoperability and sources to effectively manage risk

• Identify new kinds, types and sources of data to drive innovation throughout the organization. Define processes for the effective, integrated introduction of new data

• Organize and lead the tactical open data governance activities at the CalHHS Agency level to apply four precepts: data principles, standards, policies and guidelines

CalHHS Governance – Agency Executive Leadership

The executive leadership team within CalHHS Agency is responsible for ensuring alignment of relevant CalHHS Agency strategic goals with data publication priorities on the CalHHS Open Data Portal. In addition, executive leadership will ensure compliance with this handbook among all participating Departments and Offices and provide oversight/approvals as set forth herein (see CalHHS Agency Approvals section below). A key position in the CalHHS Executive leadership team is the CalHHS Agency Information Officer who provides oversight and direction to the Open Data Workgroup and the CalHHS Open Data Portal.

Publication of Data

Each CalHHS department and office shall create a catalog of their publishable data and propose a schedule to CalHHS leadership, for making its data available on the CalHHS Open Data Portal. Each CalHHS department and office shall prioritize data publication in accordance with guidelines set forth in this handbook. Formatting of the catalog and publication schedules shall be determined by CalHHS leadership.

Department/Office Review and Approvals:

For each particular data table, at a minimum, departments and offices must obtain reviews and receive implicit or explicit approval (as applicable) from the individuals listed below. Standardized approval forms must be completed, signed, and a copy of the appropriate form filed with the Data Coordinator's office, prior to data table publication. Departments and offices may determine additional internal approvals and signatures are required, and should include appropriate additional persons in their review and sign-off process (e.g. Public Information Officer).

The department or office Data Coordinator is responsible for obtaining the following minimum approvals from within their department, including any individual internal approvals:

1. Data Steward: The Data Steward is the person who has the greatest familiarity with and knowledge of the data table, its contents, and the purpose for the collection of the data. The Data Steward should know the accuracy and currency of the data, and be best able to supply metadata elements describing the data. The data steward is responsible for ensuring the overall quality of the data and adherence to publication guidelines that include the creation of metadata, data dictionaries and small cell procedures.

2. Deputy Director: The Data Steward involves their center/division/office leadership (i.e. Deputy Director) to validate that the center/division/office wishes to proceed with publication of the data table and assumes responsibility for the global review of the data including an evaluation of sensitivities that may be associated with it.

3. Legal Counsel: Legal counsel will be in the best position to determine, when needed, whether the data table has internally been reviewed sufficiently to ensure compliance with privacy and security requirements, intellectual property rights, and Public Records Act (PRA) responsibilities. Legal counsel may recommend additional consultation with the department's chief privacy officer, chief security officer, and/or public affairs officer.

CalHHS Agency Approvals:

Data tables may fall into one of three categories as noted below:

• Level One: Unrestricted data tables that can be released to the public and published without restriction

• Level Two: Data tables that have some level of restriction or sensitivity (see detail below) but currently can be made available to interested parties with a signed data use agreement

  • Data use agreements between the participating Department and the end-user currently guide the sharing of data.

Data Publishing Decision Tree

The CalHHS Data Release Permission Decision Tree can help departments and offices determine a dataset's permission level. Based on review and comments provided by the department, Agency will determine whether the data table will be published on the Open Data Portal.

Armstrong, MP, G Rusthon, and DL Zimmerman, 1999, Geographically Masking Health Data to Preserve Confidentiality. Statistics in Medicine, 18, 497-525.

Bambauer, Jane R., Tragedy of the Data Commons (March 18, 2011). Harvard Journal of Law and Technology, Vol. 25, 2011. Available at SSRN: http://ssrn.com/abstract=1789749 or http://dx.doi.org/10.2139/ssrn.1789749

Benitez K1, Malin B., Evaluating re-identification risks with respect to the HIPAA privacy rule. J Am Med Inform Assoc. 2010 Mar-Apr;17(2):169-77. doi: 10.1136/jamia.2009.000026. http://www.ncbi.nlm.nih.gov/pubmed/20190059

CHHS Open Data Handbook - http://chhsopendata.github.io/ CHHS, Information Strategic Plan 2016.

Colorado Department of Public Health and Environment. “Guidelines for Working with Small Numbers.” Retrieved from http://www.cohid.dphe.state.co.us/smnumguidelines.html

Committee for the Protection of Human Subjects (CPHS), CPHS Bulletin & Update, January, 2005.

Federal Committee on Statistical Methodology, Interagency Confidentiality and Data Access Group. “Checklist on Disclosure Potential of Proposed Data Releases.” Washington: Statistical Policy Office, Office of Management and Budget, July 1999.

Federal Committee on Statistical Methodology, “Statistical Policy Working Paper 22 – Report on Statistical Disclosure Limitation Methodology.” Washington: Statistical Policy Office, Office of Management and Budget, 1994.

Golle, Philippe. “Revisiting the uniqueness of simple demographics in the US population. In Proceedings of the 5th ACM Workshop on Privacy in the Electronic Society. ACM Press, New York, NY. 2006: 77-80.

Howe, H. L., A. J. Lake, and T. Shen. "Method to Assess Identifiability in Electronic Data Files." American Journal of Epidemiology 165.5 (2006): 597-601. Print.

NAHDO-CDC Cooperative Agreement Project CDC Assessment Initiative. “Statistical Approaches for Small Numbers: Addressing Reliability and Disclosure Risk.” December 2004. Retrieved from *7jctjEJXwGDDMepE4_/ Statapproachesforsmallnumbers.pdf

NCHS Staff Manual on Confidentiality. Hyattsville, MD: National Center for Health Statistics, Department of Health and Human Services, “NCHS Staff Manual on Confidentiality.” 2004. Retrieved from .

NORC, “Case Study: The Disclosure Risk Implications of Small Cells Combined with Multiple Tables or External Data,” January 8, 2016.

NORC, “NORC Recommendations for California Department of Health Care Services (DHCS) Data De-Identification Guidelines (DDG),” January 8, 2016.

North American Association of Central Cancer Registries (NAACCR), “Using Geographic Information Systems Technology in the Collection, Analysis, and Presentation of Cancer Registry Data: A Handbook of Basic Practices,” October 2002.

Office of Civil Rights, U.S. Department of Health & Human Services. "Guidance Regarding Methods for De-identification of Protected Health Information in Accordance with the Health Insurance Portability and Accountability Act (HIPAA) Privacy Rule." November 26, 2012. Retrieved from .

Ohio Department of Public Health. “Data Methodology for Public Health Practice.” .

Panel on Disclosure Review Boards of Federal Agencies: Characteristics, Defining Qualities and Generalizability, 2000, Proceedings of the Joint Statistical Meetings, Indianapolis, Indiana.

Privacy Technical Assistance Center, U.S. Department of Education. “Data De- identification: An Overview of Basic Terms.” May 2013. Retrieved from

State of California, Department of Finance, Report P-1 (Race): State and County Population Projections by Race/Ethnicity, 2010-2060. Sacramento, California, January 2013. Retrieved from

State of California, Department of Health Care Services, Trend in Medi-Cal Program Enrollment by Managed Care Status - for Fiscal Year 2004-2012, 2004-07 - 2012-07, Report Date: July 2013. Retrieved from

Stoto, MA. Statistical Issues in Interactive Web-based Public Health Data Dissemination Systems. RAND Health. September 19, 2002.

Sweeney, L. “Information Explosion, Confidentiality, Disclosure, and Data Access: Theory and Practical Applications for Statistical Agencies,” L Zayatz, P Doyle, J Theeuwes and J Lane (eds), Urban Institute, Washington, DC, 2001.

Sweeney, L. “K-anonymity: a model for protecting privacy.” International Journal of Uncertainty, Fuzziness, and Knowledge-Based Systems. 2002; 10(5): 557-570.

Sweeney, L. Testimony before that National Center for Vital and Health Statistics Workgroup for Secondary Uses of Health information. August 23, 2007.

The Centers for Medicare and Medicaid Services, Office of Information Products and Data Analytics. “Medicare Fee-For Service Provider Utilization & Payment Data Physician and Other Supplier Public Use File: A Methodological Overview.” April 7, 2014.

Washington State Department of Health. "Guidelines for Working with Small Numbers." N.p., 15 October 2012. Retrieved from .

Public Use of the CalHHS Open Data Portal

The following section describes what the public will view and how the public will be able to use the CalHHS Open Data Portal.

Public Input

The CalHHS Open Data Portal encourages citizen engagement and participation through the website. The portal provides the capability by which the public can engage via the following mechanisms:

1. A survey tool asking for feedback about the website usability and data resources,

2. A dedicated email address by which the public can submit feedback or suggest specific datasets to publish,

3. An opportunity to join the Open Data listserv to receive updates about CalHHS Open Data, and

4. A showcase page where useful visualizations and applications created by users can be highlighted.

Data Tables

The CalHHS Open Data Portal supports two classifications of data tables: tabular and geospatial. A tabular data table is a flat file that conforms to a predefined schema. The schema defines the characteristics of a fixed number of columns, including the column name and data type. A geospatial data table contains information that can be readily rendered on an underlying map. Examples of geospatial features include points (buildings), polylines (bus routes), and polygons (school districts), along with attribute information that describes characteristics of each spatial feature.

Large Files

Public data often consist of historical archives, comprised of potentially millions of records collected over an extended period of time. The CalHHS Open Data Portal supports the loading, exporting and visualization of large data tables (> 1GB).

Application Program Interface (API)

The CalHHS Open Data Portal provides an open, standards-based application programming interface (API) to offer automatic access to the published data tables within the open data catalog. The portal's APIs allow the end user to get results back in JSON, XML, RSS, etc. This separation of data model and encoding allows the support of many different encoding standards, even ones that do not yet exist. This enables users to access data in a host of different file formats that are independent of the original format of the data.

Access to Data Tables

The CalHHS Open Data Portal supports the use of an API Strategy that allows the developer community to dynamically query a data table within the data catalog. Each hosted data table within the data catalog will:

• be readily and uniformly accessible

• be available for automated processing by applications and systems

• have a standard API endpoint

All communication with the API is done through an HTTPS protocol. The portal provides the following preferred response types which are made available by specifying the format as part of the URL, or by sending the appropriate HTTP "Accepts" header:

• JSON

• XML

• CSV

• RDF

Featured API Catalog

Additionally, the CalHHS Open Data Portal supports the creation of a featured API Catalog that provides custom endpoints to the developer community to dynamically query the data table based on "specified" data table elements. Just like published data tables, the featured API Catalog is categorized and tagged using the common domain and metadata schema. Additional information about the API can be found at .

Terms of Use

The Open Data Portal Terms of Use are subject to modification as conditions warrant. When the Terms of Use change, this will be indicated within the Terms themselves with notification of the "Last Modified Date." Therefore, users are required to review the Terms of Use each time they use the CalHHS Open Data Portal for any changes since their last visit. The Terms of Use will always be available on the landing page of the CalHHS Open Data Portal.

Use of Data to Develop Mobile Applications

Developers and the public are encouraged to develop applications that utilize the data on the CalHHS Open Data Portal. The state may post links to some of these on the CalHHS Open Data Portal, but will not generally be able to evaluate the content, accuracy, or functionality of these mobile applications.

The following table is provided for reference related to the race and ethnicity composition at the county level. It is State of California, Department of Finance, Report P-1 (Race): State and County Population Projections by Race/Ethnicity, 2010-2060. Sacramento, California, January 2013. The table is for year 2010.

Play 5: Establish Your Metadata Repository

For data to be valuable, it must be understandable by data consumers and your internal teams. Metadata or “data that provides information on other data1” provides the technical, business, and security information needed for consumers to work with your data. Metadata also provides input to complete the BUCP Technical Fields and support analysis for the Specialized Security and Specialized Privacy fields. Metadata contains the following types of information:

• Business Definitions

• Technical Information

• Security Classification

• Governing Statutes

The term “metadata” may not be familiar to your data consumers. When explaining the purpose of your metadata repository it may be more effective to use common language terms or descriptions such as “data definitions” in place of “metadata”.

This diagram depicts your metadata repository and its consumers across your department’s various teams:

Metadata Repository Consumers

Typically, metadata is stored in a repository and presented to data consumers in the form of a data dictionary. A data dictionary helps data recipients, such as departments receiving your data, understand the meaning of individual data elements.

Once your metadata repository is populated, you will have a data dictionary for data consumers to understand the meaning of individual data elements and support the following components of BUCPs:

• The content for the Technical Fields section.

• List of governing security statutes for analysis to complete the Specialized Security and Privacy Fields.

• List of governing statutes to understand data sharing requirements and limitations.

Your metadata repository also provides direct benefits to your department that are listed below and elaborated in the Guidebook’s supplemental section Benefits to Your Department from Executing the Plays:

• Internal data-sharing

• System changes

• Impact assessments

• Level of effort estimates

We recommend launching the effort to create your metadata repository in parallel with the data prioritization effort described in Play 4: Prioritize Your Data. Conducting these efforts in parallel provides time for the implementation of a data catalog product or for technology teams to implement an alternate solution using an existing database.

Describing Your Application Program Interfaces

The techniques to improve data-sharing provided by this Guide are relevant to data stored in databases and Application Program Interfaces (API). If your data-sharing improvement effort includes API, please review the Guidebook’s supplemental section Describing Application Program Interfaces for guidance on related metadata standards and practices.

Play 5.1: Identify Metadata Requirements

Your first step is to identify the type of information to be captured in your dataset’s metadata. Typically, metadata is captured at the field or attribute level and includes the following types of information:

• Field Name

• Field Label

• Data Type

• Field Definition

The attached metadata template provides an example of the types of metadata to collect. You can tailor the metadata attributes in this example spreadsheet to your department’s needs.

Collect sufficient data to convey the meaning of your data and analysis during BUCP approvals. Remove metadata attributes that are not relevant to your datasets to reduce metadata collection and maintenance efforts.

Additional examples of data dictionary templates are listed below:

The next step is to identify the technical platforms that store your datasets for use during your metadata repository selection process to identify technical attributes to include in your metadata repository. These are typically your department’s databases but can also include Application Program Interfaces (APIs) or datasets created by your analytics platforms. Use the dataset inventory you created in Play 2: Identify Your Data to identify technology platforms.

Play 5.2: Implement a Data Catalog and Metadata Repository

Depending on available resources, options to create a data catalog and metadata repository include:

• Commercially licensed software packages

• Open-source software packages

• Use a database to capture metadata

• Spreadsheets

Open-source and commercial metadata repository platforms include features for automated data element collection and web-based access. Metadata repository platforms save time in the maintenance of your data dictionary and improve access to metadata. Data catalog platforms also make it easier to manage custom metadata, including references to applicable statutes that govern data sharing. Data catalog platforms also promote the use of your data-sharing artifacts by improving access through web access and search functions.

The Guidebook supplemental section, Example Metadata Repository Tools, provides examples of data catalog tools, including those available (e.g., AWS, Azure, Google) via the State of California contract vehicles.

Use the requirements you created in Play 5.1: Identify Metadata Requirements to evaluate the identified options and make a platform selection. If you do not have budget or staff resources available to establish a data catalog platform, you can get started using existing tools such as your department’s databases, spreadsheets, and a collaboration platform (e.g., Microsoft Teams).

Spreadsheets avoid software license and infrastructure costs; however, they are time-intensive to maintain and use for large datasets. If funds are unavailable to support a commercial platform or host an open-source option, another strategy is to use one of your existing databases as a metadata repository. Most database platforms provide the ability to store descriptions and security classifications-based metadata. Using your database platform’s metadata features lets you:

• Integrate ongoing creation and updates of metadata into your development processes.

• Create reports for data consumers using SQL statements.

• Establish a source of metadata for Data Dictionary platforms.

Database platform metadata capabilities may only track a limited quantity of metadata fields. You will need to develop an approach to address such limitations. For example, your approach may need to combine security and legal statute references into a single field.

Later in the Play 5 vignette, we provide an example of using a database as a metadata repository.

Play 5.3: Create a Business Glossary

Data recipients who receive your metadata may have difficulty understanding your department’s use of specific terms and specialized meanings.

A business glossary extends context and understanding of your data definitions. A business glossary defines business concepts that are required to understand your data definitions. An example of a web-based business glossary is available . Your business glossary can be a worksheet in your metadata package transmitted to data recipients.

At a high level, a business glossary establishes a record of your department's or program's jargon-specific use of terms and acronyms. Some examples of items to include in your business glossary include:

• Terms specific to your department

• Specialized use of terms

• Acronyms specific to your department

You may already have the beginnings of a business glossary in your new employee onboarding library.

Spreadsheets are a simple mechanism to store your department’s business glossary if a data catalog platform is not available. A website improves accessibility if your department receives frequent data requests or actively publishes data on open data portals.

Play 5.3 Vignette: CDW Creates a Metadata Repository and Business Glossary

If your department has elected to implement a metadata repository tool or you are electing to use a spreadsheet, feel free to skip this vignette.

The CDW lacks the funds to purchase a data catalog for its metadata repository tool. When the data-sharing improvement effort is successful, Sally will try to secure funding for a data cataloging tool in the next fiscal year.

Carlos’ background as a database administrator helps solve the funding problem. He suggests using tools CDW already owns, including the department’s database and collaboration platform, Microsoft Teams. First, the CDW will use a combination of spreadsheets and description fields in their databases to collect and store their metadata. These tools provide automation that help scale the effort for the rest of the department’s datasets. Carlos’ metadata management process is depicted below:

The CDW Creates Its Metadata Repository Technology Platform

The high-level metadata collection and storage steps are as follows:

1. Create the Initial Data Element Inventory: Carlos runs a Structured Query Language (SQL) to extract the list of fields from the W2W database in a Comma Separated Value (CSV) format.

   1. Carlos places the file on the CDW’s collaboration platform, Microsoft Teams. The collaboration platform lets the CDW staff add metadata to a central file.

2. Collect the W2W Metadata: The CDW staff add supporting metadata, including:

Carlos documents the plan for review with the W2W development and database teams. Sally schedules a meeting with the W2W development lead and its DBA to review the plan. During the meeting, Sally and Carlos use the following agenda:

• Departmental Objectives

• Executive Support

• Benefits to the W2W Technical Teams

• Metadata Repository Approach

The W2W IT team lets Carlos know that they recently upgraded their database version. The new database version provides standard metadata fields for descriptions and security classifications.

Sally, Carlos, and the W2W development team agree on the approach and scope of a proof-of- concept to test its viability and gather lessons learned to improve the process. The group also agrees on how the metadata collection team will engage IT resources. The agreement helps ensure the IT staff can complete their daily job duties while supporting the metadata collection effort.

Sally decided to use a simple spreadsheet stored in Microsoft Teams for the CDW business glossary. The business glossary will be incrementally populated as the CDW data team populates the metadata repository.

Benefits to Your Department from Executing the Plays

Cataloging your department's datasets and capturing their metadata benefits external data recipients and your organization's analytics, application enhancement, and information security teams. You can use this section during the execution of the Plays to:

• Secure executive support by showing internal benefits and bolstering the business case for the data cataloging effort.

• Gain buy-in from the internal teams that are needed to support the effort by demonstrating direct benefit to their efforts.

• Support the business case for the acquisition of data management tools.

Benefits for Internal Cross-Program Data-Sharing

The Data-Sharing Plays provides benefits for internal data-sharing between your department’s programs. Your data catalog creates awareness across your department’s programs of available data and points of contact to coordinate data-sharing agreements.

Like external data-sharing, providing data between programs within your department may require agreements and security approvals. The detailed metadata created by executing the Plays provides detailed security classifications for compliance and verification of security controls. Participation by your department’s information security team in classifying data also builds familiarity and expedites security reviews.

Additionally, data recipients from other programs may not be familiar with your data. The metadata catalog and business glossary created by executing the Plays foster understanding and the ability to accurately use data from other programs.

Department Analytics Benefits

Your department's report and analytics creation efforts benefit from improved and comprehensive metadata by:

• Reduce Report Development Time: The inventory of data elements and their definitions allow your staff to create reports/analytics more quickly by:

  • Avoiding time spent on researching data element meaning.

  • Easy access to an inventory of your department's data.

• Providing support materials to explain report/analytics methodologies.

Creating your data catalog also addresses State Administrative Manual (SAM) Item 4, requirements for Agencies to maintain an enterprise data inventory.

Information Security Benefits

Having a detailed inventory of data elements and their data security classifications also benefits information security. The inventory of data elements and classifications provides the following:

• State Information Management Manual (SIMM) Requirements: SIMM 5305-A, through a reference to the Federal Information Processing Standards (FIPS) 199, requires that data maintained by the State of California maintain data classifications and other metadata.

• Requirements: SIMM 5305-A, through a reference to the , requires that data maintained by the State of California maintain data classifications and other metadata.

• Complete List of Applicable Security Requirements: The regulatory and department- level security requirements will vary by data element. Classifying each data element provides the information to establish the required data security controls. For example, some data elements may be subject to the / Health Information Technology for Economic and Clinical Health Act (HITECH). Other data elements may contain Personal Identifiable Information (PII) subject to .

Application and Database Development Benefits

The data catalog established by executing the Plays also benefits your department's application development teams. The inventory of data improves the efficiency of system enhancement efforts by:

• Reduced Design and Development Time: Your enriched metadata minimizes the time to create new application and database system changes through:

  • Data Architecture Design: The data catalog allows design staff to create database specifications more quickly through a consolidated view of existing data elements. The data dictionary format also provides a mechanism to communicate database specifications to developers.

  • Data Element Identification: The data catalog improves the ability to create mappings between your system's data tier with user interface (UI) and data interfaces (e.g., External Interfaces). The data catalog reduces the time to create thorough technical specifications.

System Modernization Benefits

Creating a data catalog also helps future modernization efforts by providing inputs to current state analysis and data conversion. A comprehensive inventory of data elements and their descriptions is one input to assist with planning modernization efforts by:

• Provide a source of requirements for new data platforms.

• Improve the accuracy of level of effort estimations for the data-related portions of a new system, including:

  • New system data architecture design and implementation.

  • Data migration/conversion efforts.

The data architecture artifacts created during the data cataloging effort support aspects of the including:

• Mid-Level Requirements

• Data Conversion Plan

• Reference Architectures (Data Architecture)

Success Stories

CalHHS Data-Driven Success Stories demonstrate how we leverage data and technology to improve services to Californians and become more client centric. The stories aim to create awareness of innovative ideas and improve interdepartmental coordination by providing a platform to collaborate, share ideas, and expand CalHHS’s data culture.

Linking Records to Improve Cross-Program Experiences

Former CalHHS Secretary Mike Wilkening and USC researcher Emily Putnam-Hornstein discuss the Record Reconciliation Project at the 2018 Data Expo. The goal of this project is to link and organize administrative, client-level records to improve statistical analysis of CalHHS clients. This CalHHS-USC collaboration is helping to break down program siloes and create a more holistic view of clients and their cross-program experiences.

Using Data Dashboards in Emergency Response

Marko Mijic, Former CalHHS Deputy Secretary, explains how bringing data together, streamlining data management processes, and mapping data with a dashboard helped to support timely decision-making and response to California wildfires.

The California Health and Human Services Agency (CalHHS) Data De-identification Guidelines (DDG) describes a procedure to be used by departments and offices in the CalHHS to assess data for public release. As part of the document, specific actions that may be taken for each step in the procedure are described. These steps are intended to assist departments in assuring that data is de-identified for purposes of public release that meet the requirements of the California Information Practices Act (IPA) of 1977 and the Health Insurance Portability and Accountability Act (HIPAA) to prevent the disclosure of personal information.

Additionally, the DDG support CalHHS governance goals to reduce inconsistency of practices across departments, align standards used across departments, facilitate the release of useful data to the public, promote transparency of state government, and support other CHHS initiatives, such as the CalHHS Open Data Portal.

Table of Contents

The CalHHS Data De-Identification Guidelines are divided up into multiple sections. The sections can be navigated using the list on the left side of this page or by clicking one of the section links below:

Creating Effective Data Element Definitions

This section explains the characteristics of effective data definitions and an approach to developing data definition skills across your team. Even if you have experience in this area, this guide helps develop data definition skills for the team members participating in the metadata collection effort.

The incentive to create effective definitions is that your staff spends less time fielding questions from data recipients. Additionally, an accurate understanding of data translates to more precise analytics that inform program, department, and agency-level decisions.

Some data element meanings are apparent and straightforward to define. Others have a very specific meaning in the context of your department's business and can be more difficult to define.

Creating effective data definitions may sound intimidating, but creating effective definitions just takes practice. Building any new skill takes a few rounds of

If Step 3 determined that the data set has a risk that small numerators or small denominators may result in conditions that put individuals at risk of being re-identified, then the data set must be assessed to determine the need for statistical masking of those small values and complimentary values. In performing the statistical masking, the data producer must consider what level of analysis may be sacrificed in order to produce a table with lower risk. Initial considerations for statistical masking are described below. For additional methods related to statistical masking, please see .

Reduce Table Dimensions

If there are more dimensions present in the table than necessary for the vast majority of analysis, the data producer should consider reducing the number of dimensions in a single table and produce multiple tables each with a subset of the dimensions in the table that resulted in small cells. For example, if there are six dimensions of interest for study, but a table that crosses all six dimensions produces a large number of small cells, the data producer could consider producing several tables each of which crosses four dimensions. This is especially effective if there are very few analytic questions requiring a cross section of all six variables.

Play 3: Create a Business Case

Improving your department’s ability to share data is a team effort. It requires support from department executives to allocate program, technical, and security resources. This Play provides general guidance on obtaining executive support by creating a business case for improving data-sharing. This Play focuses on creating a business case for data-sharing improvements that support department and program objectives by defining:

• Benefits to your department

• Required staff resources

• Required technology resources (e.g., Data Catalog, Metadata Repository)

• High-level implementation plan

We recommend positioning an incremental effort to minimize the department’s upfront investment and deliver benefits to sustain continued executive support. As the value of data- sharing is demonstrated, it becomes easier to gain executive support to improve sharing capabilities for additional datasets.