Data Tools

Supplemental: Example Metadata Repository Tools

Metadata repository platforms save time in the maintenance of your data dictionary and improve access to metadata. Data catalog platforms also make it easier to manage custom metadata, including references to applicable statutes that govern data-sharing. Data catalog platforms also promote the use of your data-sharing artifacts by improving access through web access and search functions.

Standard data catalog features include:

• Automated collection of metadata from compatible data sources.

• Configurable and customizable metadata labels and tags to address the specific requirements.

• Improved user access and consumption of metadata through web-based user interfaces and search functionality.

• Broad support for metadata types, including databases, APIs, and Commercial Off the Shelf (COTS) products.

• Compatibility with both cloud and on-premises database platforms.

• Store metadata across data sources (e.g., Databases, APIs)

Commercial and open-source platforms are available. Sparx EA (Enterprise Architect) is a commonly used tool that provides data dictionary functionality that may already be in use by your department's Enterprise Architecture team. Your metadata must be widely accessible to be useful. Your selected tool must be easy to use for non-technical staff.

Some example open-source offerings include:

• Dataedo Community

• DataHub

• OpenMetadata

If you elect to use an open-source platform, please have the product evaluated for your department's security and open-source use/license policies.

If you prefer a commercially supported product, there are many available options. Some open- source platforms offer Software as a Service (SaaS) options to access product support and avoid infrastructure maintenance.

A big advantage of general use data catalog products is the ability to store metadata beyond

just your data department’s databases. Many platforms support data cataloging for data sources including:

• APIs

• Data Extracts

• Data Created in Reports

These platforms typically provide useful functions including the ability to Entity Relationship Diagrams (ERD) diagrams and data lineage (traceability) across datasets.

Metadata repository Software as a Service (SaaS) offerings are available from the California Department of Technology (CDT) Off-Premises Cloud Services providers. These products have a low cost and are easily procured. You will need to verify that the cloud-based data catalog products are compatible with all data platforms that are the source of your metadata. Due to their low cost, adopting a cloud-based data catalog platform may be worth considering if it is compatible with your priority data sources. Most of these platforms allow spreadsheet uploads to address other data sources.

If a trial or SaaS option is available, it is a good practice to conduct a proof of concept (POC) before you make a final selection. Additionally, we recommend conducting a pilot with a single dataset before widespread adoption to verify your metadata schema and develop training materials.