Open-Source Ecosystem

Vendor profile — open-source ecosystem. Reviewed 2026-10-18 from project documentation only; no hands-on testing.

厂商档案 — 开源生态。于 2026-10-18 依据项目文档复核；没有实际操作测试。

Profile概况

Organization机构
Not a single organisation: a selection of open-source projects (MLflow, Langfuse, Promptfoo, OpenTelemetry GenAI conventions, KServe, plus Unity Catalog and OpenLineage already in the repo).不是单一机构：一组开源项目（MLflow、Langfuse、Promptfoo、OpenTelemetry GenAI 约定、KServe，以及仓库中已登记的 Unity Catalog 与 OpenLineage）。
Category类别
Open-source tooling, mostly vendor-neutral, covering evaluation, tracing, registries and serving. Several projects have a commercial company or hosted service around them.以厂商中立为主的开源工具，覆盖评测、追踪、注册表与服务。其中多个项目背后有商业公司或托管服务。
Primary buyer主要买家
Not applicable (no single seller). Users are engineering and ML platform teams that assemble the pieces themselves (interpretation).不适用（没有单一卖方）。使用者是自行拼装这些组件的工程与 ML 平台团队（解读）。
Relevant products相关产品
MLflow [1][2][3][4]; Langfuse [5][6]; Promptfoo [7]; OpenTelemetry GenAI semantic conventions [8][9]; KServe [10]; Unity Catalog OSS and OpenLineage [11][12].MLflow [1][2][3][4]；Langfuse [5][6]；Promptfoo [7]；OpenTelemetry GenAI 语义约定 [8][9]；KServe [10]；Unity Catalog 开源版与 OpenLineage [11][12]。
Last reviewed最后复核
2026-10-18

MDLC coverageMDLC 阶段覆盖

Coverage by stage按阶段的覆盖情况 — ● full完整 · ◐ partial部分 · ○ none无 · ? not yet researched尚未研究
Stage阶段Coverage覆盖Product / feature产品 / 功能Source来源
Require需求?No requirements tooling found in the projects read. Not researched in depth.所读项目中没有发现需求工具。未深入研究。—
Plan规划?Not researched.未研究。—
Build构建◐MLflow: experiment tracking, model packaging, model registry for classical ML; for LLM apps, a prompt registry with versions linked to the agent versions that use them, and automatic prompt optimisation [1][2][3]. Langfuse: prompt management with versions and labels [5]. Open-source coding agents and training frameworks were not researched, so only partial.MLflow：传统 ML 的实验跟踪、模型打包与模型注册表；对 LLM 应用，有提示词注册表（版本可关联到使用它们的智能体版本）与自动提示词优化 [1][2][3]。Langfuse：带版本与标签的提示词管理 [5]。开源编码智能体与训练框架未研究，因此只标为部分。[1][2][3][5]
Validate验证●Promptfoo: open-source CLI and library for evals and red teaming of LLM apps, runs locally and in CI/CD [7]. MLflow: LLM-as-judge metrics with pre-built and custom judges (README cites 50+ built-in metrics) [2][1]. Langfuse: datasets, experiments, LLM-as-judge, code evaluators and annotation queues [5]. Scope is LLM and agent evaluation; classical-ML validation tools (for example fairness libraries) were not researched.Promptfoo：面向 LLM 应用的评测与红队开源 CLI/库，可本地与在 CI/CD 中运行 [7]。MLflow：LLM 裁判指标，含预置与自定义裁判（README 称内置指标 50 多个）[2][1]。Langfuse：数据集、实验、LLM 裁判、代码评估器与标注队列 [5]。范围是 LLM 与智能体评测；传统 ML 的验证工具（如公平性库）未研究。[7][2][1][5]
Approve审批?MLflow Model Registry lets teams tag versions (for example validation status approved) but the page describes no formal approval workflow [3] (summariser output; absence not proven). Not marked.MLflow 模型注册表允许给版本打标签（如验证状态为已批准），但页面没有描述正式的审批流程 [3]（摘要器输出；不能证明不存在）。不打分。[3]
Deploy部署●MLflow: local inference server (mlflow models serve), Docker images, and targets including SageMaker, Azure ML and Databricks Model Serving; community plugins for others [4]. KServe (CNCF incubating): Kubernetes inference platform with autoscaling, canary rollouts (Knative install), LLM serving through vLLM and llm-d, InferenceGraph [10]. Scope is model serving; agent sandboxes and runtimes were not researched.MLflow：本地推理服务（mlflow models serve）、Docker 镜像，目标含 SageMaker、Azure ML、Databricks Model Serving，其他可用社区插件 [4]。KServe（CNCF 孵化项目）：基于 Kubernetes 的推理平台，支持自动扩缩、金丝雀发布（需 Knative 安装）、通过 vLLM 与 llm-d 的 LLM 服务、InferenceGraph [10]。范围是模型服务；智能体沙箱与运行时未研究。[4][10]
Monitor监控●Langfuse: traces, sessions, cost and latency dashboards, threshold alerts, built on OpenTelemetry [5]. MLflow tracing: OpenTelemetry-compatible capture of prompts, retrievals, tool calls and responses [2]. OpenTelemetry GenAI semantic conventions define spans, agent spans, metrics and events, now maintained in a separate repository whose pages do not state a maturity level [8][9]. KServe lists payload logging and drift and outlier detection [10].Langfuse：追踪、会话、成本与延迟仪表板、阈值告警，基于 OpenTelemetry [5]。MLflow 追踪：以兼容 OpenTelemetry 的方式记录提示词、检索、工具调用与响应 [2]。OpenTelemetry GenAI 语义约定定义了 span、智能体 span、指标与事件，现由单独的仓库维护，所读页面没有写明成熟度 [8][9]。KServe 列有请求负载日志与漂移、离群检测 [10]。[5][2][8][9][10]
Governance治理◐MLflow registry: version lineage to the producing run, aliases and tags; RBAC is described for Databricks or managed backends, not for open-source MLflow [3]. MLflow AI Gateway for access and cost control of model calls [2]. Unity Catalog OSS and OpenLineage address catalogue and lineage [11][12] (earlier summariser reads). No approval workflow, audit export or agent registry found across the set.MLflow 注册表：版本到产生它的运行的血缘、别名与标签；RBAC 的描述针对 Databricks 或托管后端，而非开源版 MLflow [3]。MLflow AI Gateway 用于模型调用的访问与成本控制 [2]。Unity Catalog 开源版与 OpenLineage 提供目录与血缘 [11][12]（此前通过摘要器读取）。在这组项目中没有发现审批流程、审计导出或智能体注册表。[3][2][11][12]

Observations观察

This column is a union of selected projects, not one product: no single project spans the stages, and each stage mark depends on which project is used. The selection (MLflow, Langfuse, Promptfoo, OpenTelemetry GenAI conventions, KServe, Unity Catalog and OpenLineage) is my choice, not the owner’s; others (for example Kubeflow, Ragas, Inspect, vLLM, open-source coding agents) were not researched.

这一列是所选项目的并集，而不是单一产品：没有哪个项目覆盖全部阶段，各阶段的标记取决于使用哪个项目。所选项目（MLflow、Langfuse、Promptfoo、OpenTelemetry GenAI 约定、KServe、Unity Catalog、OpenLineage）是我的选择，不是你的指定；其他项目（如 Kubeflow、Ragas、Inspect、vLLM、开源编码智能体）未研究。

The Langfuse README states that the repository is MIT licensed except for the ee folders [6]. The Promptfoo page calls the project open source, describes local runs, and also shows a hosted login and enterprise pricing; its footer names Promptfoo, Inc. [7]. The MLflow pages read here did not state a licence.

Langfuse 的 README 称仓库采用 MIT 许可，ee 文件夹除外 [6]。Promptfoo 页面称其为开源项目，描述本地运行，同时有托管登录与企业定价；页脚署名 Promptfoo, Inc. [7]。所读 MLflow 页面没有写明许可证。

OpenAI’s deprecations page points Evals users to a Promptfoo migration path [13] (summariser output; see the OpenAI profile).

OpenAI 的弃用页把 Evals 用户指向迁移到 Promptfoo 的路径 [13]（摘要器输出；见 OpenAI 页面）。

Method and limits: project documentation only, no hands-on testing, and every page was read through a page summariser, so details are lower confidence. Marks mean a feature is documented by the project, not that it is mature or widely used.

方法与局限：仅读项目文档，没有实际操作；所有页面都是通过页面摘要器读取的，因此细节可信度较低。标记只表示项目文档中有该功能，不代表成熟度或使用规模。

Interpretation解读

The open-source set is strongest where the work is specialised (LLM evaluation, tracing, serving) and weakest where an organisation-wide object is needed (approval, registry of agents, audit export). That fits the idea that evaluation and build tooling stays specialised while the control plane is where tooling converges (H4 on the thesis page), though it is one selection of projects read through a summariser, so it is weak support.

开源组合在专门化的工作上最强（LLM 评测、追踪、服务），在需要全组织统一对象的地方最弱（审批、智能体注册表、审计导出）。这与“评测与构建工具保持专门化，控制平面是工具汇聚之处”（论点页 H4）一致，但这只是一组经摘要器读取的项目，支持力度弱。

OpenTelemetry GenAI conventions may be what lets vendor and open-source traces interoperate: Microsoft Foundry, Google, AWS AgentCore, Langfuse and MLflow all describe OpenTelemetry-based tracing in their own pages. Whether they follow the same GenAI conventions was not checked.

OpenTelemetry GenAI 约定可能是厂商与开源追踪数据互通的基础：Microsoft Foundry、Google、AWS AgentCore、Langfuse 与 MLflow 在各自页面中都描述了基于 OpenTelemetry 的追踪。它们是否遵循同一套 GenAI 约定没有核对。

Strategic position战略位置

FACT. Each project documents the features attributed to it above; licences are as quoted in the observations.

FACT。各项目的文档载有上文归于它们的功能；许可证以观察部分所引为准。

INTERPRETATION. Strengths: vendor neutrality, local and self-hosted operation, and interoperability through OpenTelemetry. Gaps: assembly and operations fall on the user; no organisation-level approval or audit object; several projects sit beside a commercial company, which may shape their roadmaps.

INTERPRETATION。优势：厂商中立、可本地与自托管运行、通过 OpenTelemetry 互通。缺口：拼装与运维由用户承担；没有组织级的审批或审计对象；多个项目旁边有商业公司，可能影响其路线图。

HYPOTHESIS. Teams that need governance objects will pair these tools with a hyperscaler or data-platform control plane rather than extend them. Evidence that would test this: examples of MLflow or Langfuse traces feeding Purview, Agent Registry-style or Unity Catalog governance.

HYPOTHESIS。需要治理对象的团队会把这些工具与超大规模云或数据平台的控制平面搭配使用，而不是自行扩展它们。检验证据：MLflow 或 Langfuse 的追踪数据接入 Purview、类 Agent Registry 或 Unity Catalog 治理的实例。

Open questions待解问题

What maturity level do the OpenTelemetry GenAI conventions have today (the pages read do not say), and which vendors follow them?

OpenTelemetry GenAI 约定目前处于什么成熟度（所读页面没有写明），哪些厂商遵循它？

Is there an approval or audit capability in open-source MLflow beyond tags (the Model Registry page was read through a summariser)?

开源版 MLflow 除标签外是否有审批或审计能力（模型注册表页面是经摘要器读取的）？

Which other projects belong in this set (Kubeflow, Ragas, Inspect, vLLM, open-source coding agents)? The owner should say whether to extend or narrow it.

这个集合还应包含哪些项目（Kubeflow、Ragas、Inspect、vLLM、开源编码智能体）？需要你决定扩展还是收窄。

Require and Plan are “?”.

Require 与 Plan 仍为“?”。

References参考来源

  1. MLflow, project README (GitHub). https://github.com/mlflow/mlflow project documentation (via summariser) · S-0122
  2. MLflow documentation, “GenAI” section. https://mlflow.org/docs/latest/genai/index.html project documentation (via summariser) · S-0123
  3. MLflow documentation, “Model Registry.” https://mlflow.org/docs/latest/ml/model-registry/index.html project documentation (via summariser) · S-0124
  4. MLflow documentation, “Deployment.” https://mlflow.org/docs/latest/ml/deployment/index.html project documentation (via summariser) · S-0125
  5. Langfuse documentation, introduction. https://langfuse.com/docs project documentation (via summariser) · S-0126
  6. Langfuse, project README (GitHub). https://github.com/langfuse/langfuse project documentation (via summariser) · S-0127
  7. Promptfoo, “Introduction.” https://www.promptfoo.dev/docs/intro/ project documentation (via summariser) · S-0128
  8. OpenTelemetry, semantic-conventions-genai repository README. https://github.com/open-telemetry/semantic-conventions-genai project documentation (via summariser) · S-0129
  9. OpenTelemetry, semantic conventions for Generative AI (page now pointing to the new repository). https://opentelemetry.io/docs/specs/semconv/gen-ai/ standards page (via summariser) · S-0130
  10. KServe, project README (GitHub). https://github.com/kserve/kserve project documentation (via summariser) · S-0131
  11. Unity Catalog open source project, GitHub README (already registered; see the control-layer note). https://github.com/unitycatalog/unitycatalog open-source project · S-0027
  12. OpenLineage documentation (already registered; see the control-layer note). https://openlineage.io/docs/ open-source specification · S-0034
  13. OpenAI API docs, “Deprecations” (points to a Promptfoo migration path for Evals; see the OpenAI profile). https://developers.openai.com/api/docs/deprecations vendor documentation (via summariser) · S-0063