Unifying the control layer across data, feature and model platforms在数据、特征、模型平台之间统一控制层

What Databricks, AWS, Google, Microsoft and Snowflake document about governing data, features and models together, what open-source catalogs offer, and what the evidence does and does not say about order and best practice.

Databricks、AWS、Google、微软、Snowflake 的文档如何描述数据、特征、模型的统一治理；开源目录能提供什么；以及关于顺序和“最佳实践”，证据能说什么、不能说什么。

Date · 日期
2026-10-04
Status · 状态
Draft — vendor documentation plus two engineering blogs; no independent comparison or outcome data草稿 — 厂商文档加两篇工程博客；没有独立对比或结果数据
Author · 作者
🌱
Tags · 标签
governance, catalog, lineage, feature store, model registry, control plane, OpenLineage
Question · 研究问题
When data-product, feature and model platforms were built separately, how far should the control layer be unified first, in what order, and what do vendors and open-source projects show?当数据产品、特征和模型平台是分别建设的，控制层应先统一到什么程度、按什么顺序？厂商和开源项目的做法说明了什么？
Addresses · 对应
Q7, overlaps Q2, Q5 · H4

Executive summary摘要

I found two distinct designs in the documents. In the first, one engine and one catalog govern everything: Databricks models tables, features, models, functions and AI services as securables in one namespace with inherited privileges [1][3], and Snowflake describes ML models under the same access model as tables [9]. In the second, a catalog sits on top of separate services and collects their metadata: AWS, Google and Microsoft each document an overlay catalog that ingests assets and lineage, partly automatically, partly through configuration or APIs [5][6][7][8][10]. An enterprise with separately built platforms is in the second situation. The documents show that overlay catalogs have limits (lineage depth, retention, automation coverage) [6][7][8][10], that OpenLineage is the common interchange format for lineage [5][8][11], and that LinkedIn's experience points to federated, event-based metadata over a single central store [14]. None of the sources prescribes an order of steps, and none is an independent evaluation. The order I suggest (shared asset IDs and catalog, then lineage, then identity and permissions, then approval) is my interpretation, not a documented best practice. Databricks' own SIGMOD paper argues for one catalog on grounds of consistent enforcement across access paths, yet also federates existing catalogs as a bridge [15]; Netflix, the closest example of separately built platforms I found, built a federated catalog first, then lineage, then a self-serve central catalog, without merging its platforms [20][21][23]; and the US, UK and EU regulatory texts I read ask for model inventories, data documentation and (EU, high-risk) logs, without naming lineage or catalogs [16][17][18][19].

我在文档中看到两种不同的设计。第一种是一个引擎、一个目录管理一切：Databricks 把表、特征、模型、函数和 AI 服务都建模为同一命名空间下的可授权对象，权限可继承 [1][3]；Snowflake 描述 ML 模型与表使用同一套访问模型 [9]。第二种是在各自独立的服务之上再放一个目录来汇集元数据：AWS、Google、微软各有一个“覆盖式”目录，部分自动、部分靠配置或 API 摄取资产和血缘 [5][6][7][8][10]。平台分别建设的企业属于第二种情形。文档显示覆盖式目录有局限（血缘深度、保留期、自动化覆盖）[6][7][8][10]；OpenLineage 是血缘的通用交换格式 [5][8][11]；LinkedIn 的经验指向“联邦的、基于事件的元数据”而非单一中央存储 [14]。没有任何来源规定步骤顺序，也没有一个是独立评估。我建议的顺序（共享资产 ID 与目录，其次血缘，再到身份与权限，最后审批）是我的解读，不是文档中的最佳实践。Databricks 自己的 SIGMOD 论文以“跨访问路径一致执行”为由主张用一个目录，但也把联邦现有目录当作过渡手段 [15]；Netflix 是我找到的、最接近“平台分别建设”的例子，它先建联邦式目录，再做血缘，再建自助式中央目录，并没有合并平台 [20][21][23]；我读到的美、英、欧监管文本要求模型清单、数据文档和（欧盟高风险系统的）日志，并没有点名血缘或数据目录 [16][17][18][19]。

Two designs, in plain terms两种设计，用大白话说

 A. One engine, one catalogA. 一个引擎、一个目录B. Overlay catalog on separate servicesB. 在独立服务之上的覆盖式目录
Examples例子Databricks Unity Catalog [1][3]; Snowflake Horizon [9]AWS SageMaker Catalog [5][6]; Google Knowledge Catalog [7][8]; Microsoft Purview [10]; open-source catalogs [11][12]
How governance works治理如何运作Assets live in the platform; the same privilege model applies to tables, features and models.资产存在平台里；表、特征、模型用同一套权限模型。Each service keeps its own assets and permissions; the catalog collects metadata and lineage about them.各服务保留自己的资产和权限；目录汇集关于它们的元数据和血缘。
Lineage血缘Captured automatically from queries run through the engine, with listed gaps [2].从经过引擎执行的查询中自动采集，文档列出了缺口 [2]。Automatic for some tools, configuration or API for others; OpenLineage events accepted [5][6][8].部分工具自动，其余需配置或调用 API；接受 OpenLineage 事件 [5][6][8]。
Fits an enterprise with separate in-house platforms?适合自研分散平台的企业吗？Only by moving onto that platform.只能通过迁移到该平台。Yes, this is the situation it is built for (my interpretation).是，这正是它设计面对的情形（我的解读）。

This two-way split is my reading of the documents; the vendors do not describe themselves this way.

这个二分法是我对文档的归纳，厂商并不这样描述自己。

Key findings主要发现

#Finding · 发现Cite
F1Unity Catalog lists tables, views, volumes, functions, models, services (model services, provider services, MCP, agents), secrets and features among its securable objects, in a catalog.schema.object namespace. Privileges granted on a catalog or schema apply to current and future child objects.Unity Catalog 的可授权对象包括表、视图、卷、函数、模型、服务（模型服务、提供方服务、MCP、智能体）、密钥和特征，使用 catalog.schema.object 命名空间。在 catalog 或 schema 上授予的权限适用于现有和未来的子对象。[1]
F2In Unity Catalog any Delta table with a primary key can be a feature table; it is available to all workspaces on the same metastore, and cross-metastore access uses a sharing mechanism. So a "feature platform" there is a convention on catalog tables, not a separate system.在 Unity Catalog 中，任何带主键的 Delta 表都可以作为特征表；同一 metastore 下的所有工作区都可访问，跨 metastore 使用共享机制。所以那里的“特征平台”是建立在目录表上的约定，不是另一个独立系统。[3]
F3Unity Catalog lineage is captured from queries run through Spark DataFrame or Databricks SQL, aggregated across workspaces on a metastore. Documented gaps: column lineage fails when a path rather than a table name is used, RDDs and global temp views are not captured, UDF lineage is table-level only, renamed objects lose lineage history. Catalog Explorer keeps lineage indefinitely (data after 2024-09-01); system tables keep one year.Unity Catalog 的血缘来自通过 Spark DataFrame 或 Databricks SQL 执行的查询，并在同一 metastore 的各工作区间聚合。文档列出的缺口：用路径而非表名时列级血缘失败；不采集 RDD 和全局临时视图；UDF 血缘仅到表级；重命名对象会丢失血缘历史。Catalog Explorer 无限期保留血缘（2024-09-01 之后的数据），系统表保留一年。[2]
F4The open-source Unity Catalog is Apache 2.0, a sandbox project at the LF AI & Data Foundation, supports tables, files, functions and AI models, is compatible with the Hive metastore and Iceberg REST catalog APIs, and its README says the APIs are evolving and should not be assumed stable.开源版 Unity Catalog 为 Apache 2.0，是 LF AI & Data 基金会的沙盒项目，支持表、文件、函数和 AI 模型，兼容 Hive metastore 与 Iceberg REST 目录 API；其 README 称 API 仍在演进，不应假定稳定。[4]
F5AWS: lineage in SageMaker Unified Studio is described as OpenLineage-compatible, capturing events from OpenLineage-enabled systems or APIs. Automated capture is listed for Glue crawlers and Redshift; Glue jobs and EMR are not automated by default and need configuration. The support matrix I read lists data tools and services, with no ML training, feature group or model entries (absence from that page is not proof of non-support).AWS：SageMaker Unified Studio 的血缘被描述为兼容 OpenLineage，可从启用 OpenLineage 的系统或 API 采集事件。自动采集列出的是 Glue 爬虫和 Redshift；Glue 作业和 EMR 默认不自动，需要配置。我读到的支持矩阵列的是数据工具和服务，没有 ML 训练、特征组或模型条目（该页面没有列出不等于不支持）。[5] [6]
F6Google: Dataplex Universal Catalog was renamed Knowledge Catalog (as of April 2026; API and IAM names unchanged). It ingests metadata from Vertex AI models, datasets, feature groups, feature views and online stores. Lineage is automatic for BigQuery and several services including Vertex AI Pipelines and Feature Store, and OpenLineage events can be imported. Limits: lineage retained 30 days, column lineage limited to top-level BigQuery columns, console graph depth 20.Google：Dataplex Universal Catalog 已更名为 Knowledge Catalog（截至 2026 年 4 月；API 和 IAM 名称不变）。它摄取 Vertex AI 的模型、数据集、特征组、特征视图和在线存储的元数据。BigQuery 及多个服务（含 Vertex AI Pipelines 和 Feature Store）的血缘自动采集，也可导入 OpenLineage 事件。限制：血缘仅保留 30 天，列级血缘限于 BigQuery 顶层列，控制台图深度 20 层。[7] [8]
F7Microsoft: a Purview scan of a Fabric tenant brings in item-level metadata and lineage for Fabric items including experiments and ML models. For non-Power BI items it does not support external upstream sources, cross-workspace lineage, or Notebook-to-Pipeline lineage, and sub-item lineage is not supported.微软：对 Fabric 租户做 Purview 扫描，可引入 Fabric 项目（含实验和 ML 模型）的项目级元数据和血缘。对非 Power BI 项目，不支持外部上游来源、跨工作区血缘、Notebook 到 Pipeline 的血缘，子项目级血缘也不支持。[10]
F8Snowflake's documentation says Horizon Catalog covers tables, ML models and data products, with models governed under the same role-based access model as tables, and column-level lineage across Snowflake, external databases, BI tools and OpenLineage feeds.Snowflake 文档称 Horizon Catalog 覆盖表、ML 模型和数据产品，模型与表使用同一套基于角色的访问模型，并提供跨 Snowflake、外部数据库、BI 工具和 OpenLineage 输入的列级血缘。[9]
F9OpenLineage is an LF AI & Data graduate project; its core model is datasets, jobs and runs, extended by facets. Listed integrations include Airflow, Spark, Flink, dbt, Trino and Feast. The page I read does not describe ML-specific facets.OpenLineage 是 LF AI & Data 的毕业项目；核心模型是数据集、作业和运行，用 facet 扩展。列出的集成包括 Airflow、Spark、Flink、dbt、Trino 和 Feast。我读到的页面没有描述 ML 专用的 facet。[11]
F10OpenMetadata's own site says it supports ML models and lineage with 130+ connectors; its trademarks are owned by Collate, which sells a managed version. This is a vendor-run page, so treat as a pointer.OpenMetadata 自己的网站称支持 ML 模型和血缘，有 130+ 连接器；商标归 Collate 所有，后者销售托管版本。这是厂商运营的页面，仅作线索。[12]
F11Uber's Palette feature store had metadata fragmented across services; a 2021 bad metadata change broke online serving, and most on-call time went to metadata changes. Consolidating into one schema-validated metadata repository cut onboarding deployment time by 95% and offline metadata updates from over an hour to minutes (Uber's figures). This is consolidation inside one feature store, not across platforms.Uber 的 Palette 特征库曾有元数据分散在多个服务中；2021 年一次错误的元数据变更导致在线服务中断，值班时间大多花在元数据变更上。整合为一个带模式校验的元数据仓库后，上线部署时间减少 95%，离线元数据更新从一小时以上降到几分钟（Uber 自己的数字）。这是单个特征库内部的整合，不是跨平台。[13]
F12LinkedIn describes three metadata architecture generations: crawler-based monolith, service API over a central store, and event-sourced federated. For the central-store generation it says the central team becomes a bottleneck and the coupling encourages separate catalogs for different use cases; the third lets teams add domain extensions without waiting on the central team.LinkedIn 描述了三代元数据架构：基于爬虫的单体、中央存储之上的服务 API、基于事件的联邦式。对中央存储这一代，它说中央团队会成为瓶颈，紧耦合还会促使不同用途各建各的目录；第三代让各团队无需等待中央团队就能增加领域扩展。[14]

Why Databricks chose one catalog, in its own wordsDatabricks 为什么选择“一个目录”：它自己的说法

The best primary source I found is Databricks' own SIGMOD 2025 industrial paper on Unity Catalog, written by 38 Databricks authors [15]. I read its full text (not through a summariser). It is a vendor's account of its own design, so it explains the reasoning but does not prove the design is better.

我找到的最好的一手来源，是 Databricks 自己发表在 SIGMOD 2025 的 Unity Catalog 工业论文，由 38 位 Databricks 作者撰写 [15]。我读了全文（不是通过摘要工具）。这是厂商对自己设计的说明，能解释它的思路，但不能证明这个设计更好。

#Finding · 发现Cite
D1Stated problem 1, inconsistent governance: with existing catalogs, access enforcement can differ depending on whether a table is reached by catalog name or by its cloud storage path. Stated problem 2, proprietary catalogs (the paper names AWS Glue) are not open to external workloads. Stated problem 3, operational catalogs lack discovery such as tag search and lineage. Stated problem 4, demand to govern models and unstructured data that table catalogs cannot represent.所述问题一，治理不一致：现有目录下，通过目录名访问还是通过云存储路径访问，权限执行可能不同。问题二，专有目录（论文点名了 AWS Glue）对外部工作负载不开放。问题三，运营目录缺少发现能力，如标签搜索和血缘。问题四，需要治理表目录无法表示的模型和非结构化数据。[15]
D2Mechanism for uniform access control: each cloud storage object maps to at most one catalog asset, and clients do not get direct storage access; the catalog vends temporary credentials after checking the asset's policy. Models use the same mechanism: Unity Catalog acts as an MLflow model registry by implementing the registry's storage interfaces for a registered-model asset type.统一访问控制的机制：每个云存储对象最多对应一个目录资产，客户端没有直接的存储访问权；目录在检查资产策略后签发临时凭据。模型使用同一机制：Unity Catalog 通过实现 MLflow 模型注册表的存储接口，增加“已注册模型”资产类型，充当模型注册表。[15]
D3Why not a separate discovery catalog: the paper says splitting operational and discovery catalogs creates three problems. Collecting lineage needs coordination with compute engines as well as the catalog; discovery catalogs poll for updates, trading freshness against load; enforcing the operational catalog's access policies inside discovery queries adds complexity. Unity Catalog instead exposes lineage APIs for engines to submit lineage, a change-event stream, and authorization APIs, and says the same interfaces serve third-party discovery tools such as Collibra and Alation.为什么不用独立的发现目录：论文称，把运营目录和发现目录分开会带来三个问题。采集血缘既要与目录协调，也要与计算引擎协调；发现目录靠轮询更新，要在新鲜度和负载之间取舍；在发现查询中执行运营目录定义的权限，会增加复杂度。Unity Catalog 的做法是提供血缘 API 让引擎提交血缘、变更事件流和授权 API，并称同样的接口也服务于 Collibra、Alation 等第三方发现工具。[15]
D4Unity Catalog also federates: an administrator can mount an external catalog (the paper names on-premise databases and the Hive metastore) into it, without copying data. Stated reasons: migration takes time, or the existing data is managed by a team separate from the one managing Unity Catalog. Metadata is mirrored on demand; the stated tradeoff is that simple clients such as the UI may see stale metadata until an engine mirrors it. About 16% of tables in the paper's usage data are foreign tables, across 26 foreign table types.Unity Catalog 也支持联邦：管理员可以把外部目录（论文提到本地数据库和 Hive metastore）挂载进来，不复制数据。所述原因：迁移需要时间，或现有数据由另一个团队管理。元数据按需镜像；论文写明的代价是，UI 这类简单客户端在引擎完成镜像前可能看到过期的元数据。论文使用数据中约 16% 的表是外部表，涉及 26 种外部表类型。[15]
D5Architecture: access control and audit logging sit in the core service (foreground); search, discovery and lineage are built as second-tier services (background) fed by metadata change events and tolerant of slight staleness. Databricks' scale figures: about 9,000 customers, 100 million tables, 550,000 volumes, 400,000 ML models (vendor figures, not independently checked).架构：权限控制和审计日志放在核心服务中（前台）；搜索、发现和血缘作为第二层服务（后台），由元数据变更事件驱动，可容忍轻微滞后。Databricks 给出的规模数字：约 9,000 个客户、1 亿张表、55 万个卷、40 万个 ML 模型（厂商数字，未经独立核实）。[15]

What this changes. My earlier answer, that it chose one catalog because it owns the engine, was only partly right. The paper's own argument is about enforcement: governance holds only if every access path (by name, by path, by engine) goes through one place that checks policy and vends credentials, and lineage is more reliable when engines report it to that same place. It also shows Databricks itself uses the overlay idea (federation) as a bridge for existing catalogs, with a named cost (stale metadata). So for an enterprise with separate platforms, the paper supports a staged approach, not a binary choice, but it describes Databricks' product, not a method for in-house platforms.

这改变了什么。我之前说它选择一个目录是因为拥有整个引擎，只对了一部分。论文自己的论证围绕强制执行：只有当每条访问路径（按名称、按路径、按引擎）都经过同一个检查策略并签发凭据的地方，治理才站得住；而血缘由引擎汇报到同一个地方时更可靠。论文也说明 Databricks 自己把覆盖式思路（联邦）当作接入现有目录的过渡手段，并写明了代价（元数据可能过期）。所以对于平台分散的企业，这篇论文支持“分阶段”而不是“二选一”，但它描述的是 Databricks 的产品，不是针对自研平台的方法。

What the regulators actually ask for监管方实际要求什么

#Finding · 发现Cite
R1EU AI Act Article 10 (high-risk AI systems): providers must document data collection processes and data origin, preparation operations (annotation, labelling, cleaning, enrichment, aggregation), assumptions about what the data measure, bias examination and mitigation, and identified data gaps. Article 12: high-risk systems must technically allow automatic recording of events (logs) over the lifetime of the system, for identifying risk situations, post-market monitoring and operational monitoring. Both apply to high-risk systems only; whether a given system is high-risk, and the application dates, I did not check.欧盟 AI 法案第 10 条（高风险 AI 系统）：提供方须记录数据收集过程和数据来源、数据准备操作（标注、标记、清洗、丰富、聚合）、关于数据所度量内容的假设、偏差检查与缓解，以及已识别的数据缺口。第 12 条：高风险系统须在技术上支持在系统生命周期内自动记录事件（日志），用于识别风险情形、上市后监测和运行监测。两条都只适用于高风险系统；某个系统是否属于高风险，以及适用日期，我没有核实。[16] [17]
R2US SR 26-2 (Fed, OCC, FDIC; 2026-04-17): says an effective model inventory holds enough information to understand model risk at individual and aggregate levels, calls for adequate documentation, and lists assessment of data quality, relevance and inputs as part of model testing. Generative and agentic AI models are out of scope (footnote 3); the principles apply to traditional statistical and quantitative models and non-generative, non-agentic AI models. The text I read does not contain the word "lineage" and does not mention feature stores or catalogs.美国 SR 26-2（美联储、OCC、FDIC；2026-04-17）：指出有效的模型清单应包含足够信息，以便在单个和整体层面理解模型风险；要求充分的文档；并把对数据质量、相关性和输入的评估列为模型测试的一部分。生成式与智能体 AI 模型不在范围内（脚注 3）；这些原则适用于传统统计和量化模型，以及非生成式、非智能体的 AI 模型。我读到的文本中没有“血缘”一词，也没有提到特征库或数据目录。[18]
R3UK PRA SS1/23: Principle 1.2 asks for a comprehensive model inventory, and a firm-wide inventory that helps identify direct and indirect model inter-dependencies; the inventory should record purpose and use, assumptions and limitations (including limits of the calibration data), validation findings and governance details. Principle 3.2 asks firms to show the development data are suitable for the intended use and representative. It also asks for data quality management procedures with rules and standards for quality, accuracy and relevance. This text also does not contain the word "lineage".英国 PRA SS1/23：原则 1.2 要求有全面的模型清单，以及能识别直接和间接模型相互依赖关系的全公司范围清单；清单应记录用途、假设与局限（包括校准数据的局限）、验证结论和治理信息。原则 3.2 要求证明开发数据适合预期用途且具有代表性。另要求建立数据质量管理程序，含质量、准确性和相关性的规则与标准。这份文本同样不含“血缘”一词。[19]

A search-result summary I saw earlier claimed SS1/23 covers "lineage of monitoring data". I read the full PDF and the word "lineage" does not appear in it, so I have not used that claim. I read the full text of SR 26-2 and SS1/23 from the regulators' own PDFs; the EU articles came through a summariser from a third-party reproduction (the EU text itself, S-0003, was not re-read for this note).

我之前看到的一条搜索结果摘要称 SS1/23 涵盖“监测数据的血缘”。我读了完整 PDF，其中并没有“lineage”一词，所以没有采用这一说法。SR 26-2 和 SS1/23 我读的是监管机构自己的 PDF 全文；欧盟条文是通过摘要工具读取的第三方转载版本（欧盟官方文本 S-0003 本次没有重新阅读）。

What this means for Q7. None of the three texts requires "data lineage" or a catalog by name. What they require is an inventory of models (with inter-dependencies in the UK), documentation of data origin, preparation and suitability, and, in the EU for high-risk systems, automatic logs. Lineage and a shared asset catalog are a practical way to produce that evidence across separate platforms, which is my inference, not a regulatory statement. So the regulatory pressure supports starting with an asset inventory and data-origin records (step 1 and 2 of my order) more than it supports any particular tool. SR 26-2 excludes generative and agentic AI, so it would not by itself drive this for LLM and agent assets (consistent with H5).

这对 Q7 意味着什么。三份文本都没有点名要求“数据血缘”或数据目录。它们要求的是：模型清单（英国还要求识别相互依赖关系）、对数据来源、准备和适用性的记录，以及（欧盟对高风险系统）自动日志。血缘和共享资产目录是在多个独立平台上产出这些证据的一种实用办法，这是我的推断，不是监管表述。所以监管压力更支持先从资产清单和数据来源记录做起（我顺序中的第 1、2 步），而不是支持某个特定工具。SR 26-2 把生成式和智能体 AI 排除在外，因此它本身不会推动 LLM 和智能体资产这样做（与 H5 一致）。

How Netflix does itNetflix 是怎么做的

Netflix is the closest match I found to an enterprise with separately built platforms: its own posts describe teams that choose their own tools, and a metadata layer built over them. I read the Netflix Tech Blog posts in a browser (the fetch tool was blocked). The catalog-history details come from a DataHub community post that retells a Netflix talk, so they are second-hand. All of it is Netflix's or its vendor's own account, not an independent evaluation, and the oldest posts are from 2018 and 2019.

Netflix 是我找到的、最接近“平台分别建设的企业”的例子：它自己的文章描述了各团队自选工具，以及在其上建立的元数据层。我在浏览器里阅读了 Netflix 技术博客的文章（抓取工具被拦截）。目录演进的细节来自 DataHub 社区转述 Netflix 演讲的文章，属于二手信息。所有内容都是 Netflix 或其供应商自己的说法，不是独立评估，最早的文章是 2018 和 2019 年的。

#Finding · 发现Cite
N1Metacat (2018) is described as a federated metadata access layer over Netflix's data stores (Hive on S3, Druid, Elasticsearch, Redshift, Snowflake, MySQL), serving Spark, Presto, Pig and Hive. The underlying stores remain the source of truth for schema metadata, so Metacat does not materialize it; it stores only business and user-defined metadata (for example connection information, TTL, metrics, owner-supplied audit information) and publishes everything to Elasticsearch for search. Because metadata changes go through it, it also emits change notifications (to SNS and Netflix's Keystone pipeline), for example so a janitor service can clean up S3 when a table is dropped.Metacat（2018）被描述为 Netflix 各数据存储（S3 上的 Hive、Druid、Elasticsearch、Redshift、Snowflake、MySQL）之上的联邦式元数据访问层，服务 Spark、Presto、Pig、Hive。底层存储仍是模式元数据的权威来源，所以 Metacat 不物化它；它只存业务和用户自定义元数据（例如连接信息、TTL、指标、负责人提供的审计信息），并把全部信息发布到 Elasticsearch 供搜索。因为元数据变更都经过它，它还会发出变更通知（发到 SNS 和 Netflix 的 Keystone 管道），例如表被删除时，清理服务可据此清理 S3。[20]
N2Its own "next steps" list schema and metadata versioning (to ask what metadata looked like in the past, for audit, debugging and rollback), and feeding contextual metadata such as table access frequency to a lineage service.它自己列出的“下一步”包括模式与元数据版本化（用于询问过去的元数据是什么样，服务于审计、调试和回滚），以及把表访问频率等上下文元数据交给血缘服务。[20]
N3Lineage (2019): Netflix says teams are free to choose tools within "paved paths", so no single consolidated, centralized source of truth exists to derive lineage from. The ingestion is mostly pull (scanning engine logs and metadata: Spark plans, Snowflake admin tables, scheduler APIs) plus some push (platform tools publish lineage events to Kafka topics). Data from the different sources is conformed into a generic graph model and enriched with Metacat and job-manager metadata, then served through a REST lineage service and a graph database. Its listed uses are search ranking, SLA alerting, and data-efficiency (cost attribution and cleanup of unused data).血缘（2019）：Netflix 说各团队在“铺好的路径”内可自由选择工具，因此不存在可用来推导血缘的、统一集中的权威来源。采集以拉取为主（扫描引擎日志和元数据：Spark 执行计划、Snowflake 管理表、调度器 API），辅以部分推送（平台工具把血缘事件发到 Kafka 主题）。来自不同来源的数据被整理成一个通用的图模型，并用 Metacat 和作业管理器的元数据做补充，再通过 REST 血缘服务和图数据库提供。列出的用途是搜索排序、SLA 告警，以及数据效率（成本归因和清理未使用数据）。[21]
N4Stated challenges: conforming diverse sources to one model; operational complexity of many ingestion patterns; every Spark upgrade changed the Spark plan, breaking ingestion unexpectedly. Responses: use S3 access logs to catch relationships the normal ingestion missed, move toward system-level instrumentation of engines, and add a REST API so anyone can publish lineage. The post lists ML and experimentation models among the artifacts to map, and ML infrastructure as a future onboarding target, so ML lineage was not yet shown there.所述挑战：把多样的来源整理成同一模型；多种采集方式带来的运维复杂度；每次 Spark 升级都会改变执行计划，使采集意外中断。应对：用 S3 访问日志补抓常规采集漏掉的关系，转向在引擎层面做系统级埋点，并提供 REST API 让任何人都能发布血缘。文章把 ML 和实验模型列为要映射的对象，把 ML 基础设施列为未来接入对象，所以当时还没有展示 ML 血缘。[21]
N5Catalog evolution (second-hand, from a DataHub community post about a January 2024 Netflix talk): Metacat worked as a metadata aggregator for the warehouse layer. Netflix then started a central catalog covering the warehouse, online stores and real-time layers, and found that building and managing connectors fell on the data platform team rather than the teams that own the data, and that it needed a policy engine. It changed to a self-serve model where source-system teams define their own entity types and ingest into the central catalog. Requirements named: new entity types, a custom ownership model, and custom properties defined by privacy and legal teams. Netflix chose DataHub after evaluating several tools, for its extensibility.目录演进（二手信息，来自 DataHub 社区文章，讲的是 Netflix 2024 年 1 月的演讲）：Metacat 在数据仓库层起到元数据汇集的作用。随后 Netflix 启动了覆盖数据仓库、在线存储和实时层的中央目录，发现构建和维护连接器的负担落在数据平台团队身上，而不是拥有数据的团队，并且需要策略引擎。于是转向自助模式：源系统团队自己定义实体类型，并接入中央目录。提到的需求包括：新的实体类型、自定义的责任人模型，以及由隐私和法务团队定义的自定义属性。Netflix 评估了几个工具后选择了 DataHub，看中它的可扩展性。[23]
N6DataHub's own customer-story page (dated 2026-03-20, vendor marketing) says Netflix uses DataHub as a catalog across data, ML models and software services, for ownership, PII classification, impact analysis and TTL and cost ownership, and quotes a Netflix manager calling it the "central nervous system" for discovery and governance. It gives no asset counts. Treat as a low-confidence pointer; I could not confirm the ML-model coverage from a Netflix source.DataHub 自己的客户案例页面（日期 2026-03-20，厂商营销）称，Netflix 把 DataHub 用作覆盖数据、ML 模型和软件服务的目录，用于责任人、PII 分类、影响分析以及 TTL 和成本归属，并引用 Netflix 一位经理称其为发现与治理的“中枢神经”。页面没有给出资产数量。仅作低可信度线索；我没能从 Netflix 自己的来源确认 ML 模型的覆盖范围。[24]
N7Netflix's ML feature layer, Axion (2022), is described as a separate fact store inside the ML platform, built on Keystone, Iceberg and EVCache, with its own data-quality monitoring (trend-based counts, consistent sampling, random sampling; Netflix says it caught over 95% of data issues early). The post does not mention a catalog, lineage or permissions, so I cannot say how Axion's assets are registered in the wider catalog.Netflix 的 ML 特征层 Axion（2022）被描述为 ML 平台内部一个独立的事实存储，建立在 Keystone、Iceberg 和 EVCache 之上，有自己的数据质量监控（基于趋势的计数、一致性抽样、随机抽样；Netflix 称及早发现了超过 95% 的数据问题）。文章没有提到目录、血缘或权限，所以我无法说明 Axion 的资产如何登记到更大的目录中。[22]

What Netflix suggests (my interpretation, one company). Netflix did not merge its platforms. It built a metadata layer on top of them in this order: (1) a federated catalog that leaves each store as the source of truth for its own schema (2018); (2) lineage gathered mostly by scanning what the engines already log, conformed into one graph (2019); (3) a broader central catalog where the owning teams register their own asset types, after finding that a central team writing every connector did not scale (by 2024). That is close to the order I proposed (catalog, then lineage), and it supports H4, but it is one company's account with strong engineering resources, the earliest steps are from 2018 and 2019, and the sources give no outcome measures. Two lessons are directly relevant to a company with separate platforms: lineage ingestion breaks when engine internals change, and the organisation question (who writes the connectors) decided the design more than the technology did.

Netflix 给出的启示（我的解读，仅一家公司）。Netflix 没有合并平台，而是在它们之上按这个顺序建了元数据层：（1）联邦式目录，让每个存储继续作为自己模式的权威来源（2018）；（2）血缘，主要靠扫描引擎已有的日志采集，再整理成一张图（2019）；（3）更大的中央目录，由数据所属团队自己登记资产类型，因为发现由中央团队来写所有连接器无法扩展（2024 年前）。这与我提出的顺序（先目录，后血缘）很接近，也支持 H4；但这是一家工程资源雄厚的公司的自述，最早的步骤来自 2018 和 2019 年，来源也没有给出结果度量。有两条教训对分散平台的公司直接适用：引擎内部变化会让血缘采集中断；以及“谁来写连接器”这个组织问题对设计的决定作用大于技术本身。

Evidence notes证据说明

Databricks documentation: services let you manage access to AI traffic with the same privileges used for data. [1]

Databricks 文档：服务让你用与数据相同的权限来管理 AI 流量的访问。[1]

All five vendors place ML models inside their catalog or governance scope. What differs is depth: single-engine designs give one permission model and engine-captured lineage, while overlay catalogs describe connector coverage, API-based lineage and stated limits. Whether that depth difference matters in practice is not shown by any source I read.

五家厂商都把 ML 模型纳入各自目录或治理范围。差别在深度：单引擎设计提供一套权限模型和由引擎采集的血缘；覆盖式目录则是连接器覆盖、基于 API 的血缘和各自写明的限制。这种深度差别在实践中是否重要，我读到的来源都没有给出。

Still not read or not found (no source yet): any independent comparison of these catalogs; independent case studies (Netflix's own account is now covered above, but it is the company's own description and I found no ML-specific lineage from Netflix sources); evidence on which step order works best; Snowflake's ML lineage and Feature Store lineage status (a search summary said preview, not verified on the page); Azure Machine Learning registry lineage; DataHub's ML entity coverage (the concepts page I read did not list ML entities, which is not conclusive); an AWS page titled "The importance of the sourceIdentifier attribute to lineage nodes" appeared in the docs index and I did not read it (a lead only); regulators outside the US, EU and UK (for example China's CAC and TC260, and EBA/ECB guidance) were not read. Resolved since the first draft: Databricks' own rationale (read in full, see above) and the US, UK and EU regulatory texts (see above).

仍未读到或未找到（暂无来源）：对这些目录的独立对比；独立案例研究（Netflix 自己的说法已在上文覆盖，但那是公司自己的描述，我也没有从 Netflix 来源找到 ML 专用的血缘）；哪种步骤顺序效果最好的证据；Snowflake 的 ML 血缘与 Feature Store 血缘状态（搜索摘要称为预览，未在页面上核实）；Azure 机器学习注册表的血缘；DataHub 对 ML 实体的覆盖（我读到的概念页没有列出 ML 实体，这并不能下结论）；AWS 文档索引中有一页题为“sourceIdentifier 属性对血缘节点的重要性”，我没有读（仅为线索）；美国、欧盟、英国以外的监管方（如中国的 CAC 和 TC260，以及 EBA/ECB 的指引）没有读。自初稿以来已解决：Databricks 自己的设计理由（已通读全文，见上）以及美国、英国、欧盟的监管文本（见上）。

Reading method and confidence: pages were read through a page summariser, not verbatim, so details carry lower confidence than the headline statements. Almost every source is a vendor's own documentation or a vendor-linked engineering blog describing its own product. The Snowflake page and OpenMetadata page are promotional in tone.

阅读方式与可信度：页面是通过摘要工具读取的，不是逐字阅读，所以细节的可信度低于总体陈述。几乎所有来源都是厂商自己的文档，或厂商相关工程博客对自己产品的描述。Snowflake 页面和 OpenMetadata 页面带有宣传性质。

Industry examples行业示例

CompanyProductWhat it showsSource
DatabricksUnity CatalogSingle-catalog design: data, features, models and AI services under one privilege model; lineage tied to its engines.单目录设计：数据、特征、模型和 AI 服务在同一权限模型下；血缘绑定其引擎。[1] [2] [3]
SnowflakeHorizon CatalogSame access model for tables and ML models; accepts external lineage feeds.表与 ML 模型同一访问模型；接受外部血缘输入。[9]
AWSSageMaker CatalogOverlay design with OpenLineage-compatible lineage; automation covers only some tools.覆盖式设计，血缘兼容 OpenLineage；自动化只覆盖部分工具。[5] [6]
Google CloudKnowledge CatalogCatalogs Vertex AI models, datasets and feature groups; imports OpenLineage; 30-day lineage retention.编目 Vertex AI 的模型、数据集和特征组；可导入 OpenLineage；血缘保留 30 天。[7] [8]
MicrosoftPurview + FabricScan-based, item-level lineage including ML models; no cross-workspace lineage for non-Power BI items.基于扫描的项目级血缘，含 ML 模型；非 Power BI 项目没有跨工作区血缘。[10]
LinkedInDataHub (design)Why a single central metadata store tends to bottleneck, and the federated alternative.为什么单一中央元数据存储容易成为瓶颈，以及联邦式的替代方案。[14]
NetflixMetacat, lineage service, DataHubFederated catalog, then lineage from scanning engines, then a self-serve central catalog; platforms not merged.先联邦式目录，再通过扫描引擎采集血缘，再建自助式中央目录；平台没有合并。[20] [21] [23]
UberPalette Meta StoreFragmented feature metadata caused an outage and on-call load; schema-validated consolidation fixed it.特征元数据分散导致一次故障和值班负担；带模式校验的整合解决了问题。[13]

Architecture implications架构含义

For an enterprise with separately built platforms, the documents describe the overlay pattern (design B) as the one that fits: leave each platform's build and serving in place and add shared services above them. The shared parts that recur across the vendors are a catalog with consistent asset identity, lineage collected as events (OpenLineage where possible), and one place to attach ownership and classification. Where a vendor offers a single permission model, it comes with the single engine; an overlay catalog does not by itself give one permission model across the platforms. Whether to adopt an open-source catalog or build in-house is not answered by these sources; the open-source Unity Catalog states its APIs are not stable [4], and OpenLineage is a specification and integrations, not a catalog [11].

对于平台分别建设的企业，文档描述的覆盖式模式（设计 B）是合适的：各平台的构建与服务保持不动，在其上增加共享服务。各厂商共有的共享部分是：具有一致资产标识的目录、以事件方式采集的血缘（尽可能用 OpenLineage），以及一个挂载责任人和分类的位置。厂商提供的单一权限模型是随单一引擎而来的；覆盖式目录本身并不能在各平台之间提供一套权限模型。采用开源目录还是自建，这些来源没有回答；开源 Unity Catalog 自己说 API 不稳定 [4]，OpenLineage 是规范和集成，不是目录 [11]。

MDLC implicationsMDLC 含义

If the thesis holds that evidence and approvals become the control points, then lineage is what lets a gate answer "which data and features produced this model". Vendors capture it at different depths and retention (30 days at Google [8], indefinite in Unity Catalog Explorer [2]), and that retention is a design input for any approval or audit step that needs history.

如果论点成立，即证据与审批成为控制点，那么血缘就是让审批环节回答“这个模型用了哪些数据和特征”的依据。各厂商采集的深度和保留期不同（Google 为 30 天 [8]，Unity Catalog Explorer 为无限期 [2]），对任何需要历史记录的审批或审计环节，保留期都是设计输入。

My interpretation我的解读

Order. The sources do not prescribe an order, so the following is my reasoning, not a documented best practice. (1) Shared asset identity and catalog first, because lineage events and permissions both need to refer to the same asset; (2) lineage next, because it gives the highest-value answers (impact of a change, what a model was built from) and can be collected without changing the platforms; (3) shared identity and groups; (4) approval and policy last, because they depend on the earlier steps. This ordering is consistent with the vendors' designs (catalog and lineage are universal, per-service permissions are not) but is not tested by any outcome data I found.

顺序。来源没有规定顺序，下面是我的推理，不是文档中的最佳实践。（1）先做共享资产标识和目录，因为血缘事件和权限都要指向同一个资产；（2）其次是血缘，因为它能回答价值最高的问题（变更的影响、模型由什么构建），且无需改动各平台即可采集；（3）共享身份和用户组；（4）审批和策略放最后，因为它们依赖前面的步骤。这个顺序与厂商的设计一致（目录和血缘是通用的，按服务划分的权限则不是），但我没有找到能检验它的结果数据。

Best practice. I could not find a source that establishes one. What the evidence supports is narrower: ML models are being pulled into the same governance scope as data by every vendor read here; lineage as events with an open format is the common interchange; and a single central metadata team has a documented bottleneck risk [14]. Anything stronger would need independent case studies.

最佳实践。我没有找到能确立某一种“最佳实践”的来源。证据能支持的范围更窄：本文读到的每家厂商都把 ML 模型纳入与数据相同的治理范围；以开放格式、事件方式传递血缘是通用的交换方式；单一中央元数据团队存在有文档记录的瓶颈风险 [14]。更强的结论需要独立案例研究。

Hypotheses假设

In enterprises with separately built platforms, unifying asset identity and lineage first delivers most of the audit and impact-analysis value, and unifying permissions can follow later or stay per platform.

在平台分别建设的企业中，先统一资产标识和血缘就能获得大部分审计与影响分析的价值；统一权限可以放到后面，或保持按平台分别管理。

Open questions未决问题

Do independent case studies exist of unifying governance across separately built data, feature and model platforms, and what order did they follow? How well do overlay catalogs capture ML-specific lineage (training runs, feature versions) in practice, given the support matrices list data tools? Does OpenLineage need ML-specific facets, and does anyone use them? What do regulators expect for lineage and record-keeping of ML assets (not read in this note)?

是否存在在分别建设的数据、特征、模型平台之间统一治理的独立案例研究，它们遵循什么顺序？考虑到支持矩阵列的是数据工具，覆盖式目录在实践中对 ML 专用血缘（训练运行、特征版本）的采集做得怎样？OpenLineage 是否需要 ML 专用 facet，有人在用吗？监管方对 ML 资产的血缘与记录保存有什么要求（本笔记没有读）？

What would change my view?什么会改变我的看法？

Link to hypotheses与假设的关系

H4 (tooling converges in the control plane while build and evaluation stay specialized): supported by this note. Every vendor read here governs models alongside data, while build and serving remain separate services in the overlay designs. Evidence is vendor documentation only, so this raises H4's support without testing it independently; the thesis page is not edited for a single note. Q7: partly answered on what vendors do, not on order or best practice. Q2 and Q5: adjacent only. H5: SR 26-2 excludes generative and agentic AI, consistent with H5 (already recorded in the convergence note). H1, H2, H3: no direct link.

H4（控制平面趋同，构建与评估保持专门化）：本笔记支持。本文读到的每家厂商都把模型与数据放在一起治理，而在覆盖式设计中构建与服务仍是独立的服务。证据仅来自厂商文档，所以这提高了 H4 的支持度，但并未独立检验；论点页不会因单篇笔记而修改。Q7：关于厂商怎么做已部分回答，关于顺序和最佳实践尚未回答。Q2、Q5：仅相邻。H5：SR 26-2 排除生成式和智能体 AI，与 H5 一致（趋同笔记中已记录）。H1、H2、H3：无直接关系。

References参考文献

  1. Databricks, “Unity Catalog securable objects reference,” read 2026-10-04 (via summariser). https://docs.databricks.com/aws/en/data-governance/unity-catalog/securable-objects vendor documentation · S-0024
  2. Databricks, “Data lineage in Unity Catalog,” read 2026-10-04 (via summariser). https://docs.databricks.com/aws/en/data-governance/unity-catalog/data-lineage vendor documentation · S-0025
  3. Databricks, “Feature tables in Unity Catalog,” read 2026-10-04 (via summariser). https://docs.databricks.com/aws/en/machine-learning/feature-store/uc/feature-tables-uc vendor documentation · S-0026
  4. Unity Catalog open source project, GitHub README, read 2026-10-04 (via summariser). https://github.com/unitycatalog/unitycatalog open-source project · S-0027
  5. AWS, “Data lineage in Amazon SageMaker Unified Studio,” read 2026-10-04 (via summariser). https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/datazone-data-lineage.html vendor documentation · S-0028
  6. AWS, “Data lineage support matrix,” read 2026-10-04 (via summariser). https://docs.aws.amazon.com/sagemaker-unified-studio/latest/userguide/datazone-support-matrix.html vendor documentation · S-0029
  7. Google Cloud, “Vertex AI assets in Knowledge Catalog” (model data catalog page), read 2026-10-04 (via summariser). https://docs.cloud.google.com/vertex-ai/docs/model-registry/model-data-catalog vendor documentation · S-0030
  8. Google Cloud, “About data lineage,” read 2026-10-04 (via summariser). https://docs.cloud.google.com/dataplex/docs/about-data-lineage vendor documentation · S-0031
  9. Snowflake, “Snowflake Horizon Catalog,” read 2026-10-04 (via summariser). https://docs.snowflake.com/en/user-guide/snowflake-horizon vendor documentation · S-0032
  10. Microsoft, “Metadata and Lineage from Fabric into Microsoft Purview,” page dated 2025-11, read 2026-10-04. https://learn.microsoft.com/en-us/purview/data-map-lineage-fabric vendor documentation · S-0033
  11. OpenLineage documentation, read 2026-10-04 (via summariser). https://openlineage.io/docs/ open-source specification · S-0034
  12. OpenMetadata website, read 2026-10-04 (via summariser; vendor-run page). https://open-metadata.org/ open-source project site · S-0035
  13. Uber Engineering, “Palette Meta Store journey,” read 2026-10-04 (via summariser). https://www.uber.com/blog/palette-meta-store-journey/ company engineering blog · S-0036
  14. LinkedIn Engineering, “DataHub: Popular metadata architectures explained,” 2020, read 2026-10-04 (via summariser). https://www.linkedin.com/blog/engineering/data-management/datahub-popular-metadata-architectures-explained company engineering blog · S-0037
  15. Chandra et al. (Databricks), “Unity Catalog: Open and Universal Governance for the Lakehouse and Beyond,” SIGMOD-Companion ’25, 2025, 13 pp. https://doi.org/10.1145/3722212.3724459 (PDF read in full 2026-10-04 from databricks.com). paper (vendor authors) · S-0038
  16. EU AI Act, Article 10 “Data and data governance,” read 2026-10-04 via a summariser from a third-party reproduction. https://artificialintelligenceact.eu/article/10/ regulation text (secondary copy) · S-0039
  17. EU AI Act, Article 12 “Record-keeping,” read 2026-10-04 via a summariser from a third-party reproduction. https://artificialintelligenceact.eu/article/12/ regulation text (secondary copy) · S-0040
  18. Federal Reserve, OCC, FDIC, SR 26-2 attachment “Revised Guidance on Model Risk Management,” 2026-04-17; PDF read in full 2026-10-04. https://www.federalreserve.gov/supervisionreg/srletters/SR2602a1.pdf supervisory guidance · S-0041 (same guidance as S-0006)
  19. Bank of England PRA, SS1/23 “Model risk management principles for banks,” 2023; PDF read in full 2026-10-04. https://www.bankofengland.co.uk/-/media/boe/files/prudential-regulation/supervisory-statement/2023/ss123.pdf supervisory statement · S-0042
  20. Majumdar and Li (Netflix), “Metacat: Making Big Data Discoverable and Meaningful at Netflix,” Netflix Technology Blog, 2018-06-14; read in a browser 2026-10-04. https://netflixtechblog.com/metacat-making-big-data-discoverable-and-meaningful-at-netflix-56fb36a53520 company engineering blog · S-0043
  21. Lin, Lingappa and Aswani (Netflix), “Building and Scaling Data Lineage at Netflix to Improve Data Infrastructure Reliability, and Efficiency,” Netflix Technology Blog, 2019-03-25; read in a browser 2026-10-04. https://netflixtechblog.com/building-and-scaling-data-lineage-at-netflix-to-improve-data-infrastructure-reliability-and-1a52526a7977 company engineering blog · S-0044
  22. Kaushal (Netflix), “Evolution of ML Fact Store,” Netflix Technology Blog, 2022-04-26; read in a browser 2026-10-04. https://netflixtechblog.com/evolution-of-ml-fact-store-5941d3231762 company engineering blog · S-0045
  23. DataHub Community, “How Netflix is Collaborating with DataHub to Enhance its Extensibility,” 2024-05-14 (retells a January 2024 Netflix talk); read in a browser 2026-10-04. https://medium.com/datahub-project/how-netflix-is-collaborating-with-datahub-to-enhance-its-extensibility-a34d33f45947 project blog, second-hand · S-0046
  24. DataHub, “Netflix Reimagines Discovery & Governance with DataHub,” customer story dated 2026-03-20; read through a summariser 2026-10-04 (vendor marketing, low confidence). https://datahub.com/customer-stories/netflix/ vendor marketing · S-0047

Last updated 2026-10-04.