Open Questions
Running list of unresolved questions and what would resolve each.
Daily capture: each new question becomes the next Q. It is promoted to a research note only if it meets 2 of 3 — evidence is findable · it affects a hypothesis · the answer matters to me. · 每个新问题记为下一个 Q；满足 3 条中的 2 条才升级为研究笔记。
| # | Question | Area | What would resolve it | Status |
|---|---|---|---|---|
| Q1 | Where do classical ML and LLM/agent lifecycles diverge enough to need separate tooling? | Lifecycle | Practitioner interviews; vendor product boundaries. SR 26-2 excludes GenAI/agentic AI from US bank MRM guidance — see convergence note (H4, H5) | open |
| Q2 | Which systems of record will own agent action logs? | Architecture | Vendor roadmaps; enterprise deployments; concrete check: for vendor-hosted always-on agents (e.g. OpenAI dots), whether the customer can export a complete action log (the OpenAI pages read on 2026-10-04 show only an in-app Activity View) | researching |
| Q3 | Can evidence acceptance (G2) be delegated to an independent agent for low-risk models? | Governance | Regulatory guidance; MRM practice | open |
| Q4 | Does a structured workflow beat a model upgrade at equal cost? | Benchmarks | B-001 series | open |
| Q5 | In enterprise LLMOps platforms, which capabilities should be user self-service and which must be centrally gated — and does the boundary depend on risk tier? 企业 LLMOps 平台中，哪些能力应让用户自助、哪些必须集中审批？分界线是否取决于风险等级？ Why: drives platform design, ownership and buying decisions. Initial take (interpretation): self-service in the data plane (prompt iteration, sandboxes, eval runs, approved tools) inside guardrails; central control-plane gates (model/data access, production approval, budgets, new tools and agent permissions), tightening with risk. Related: H4. | Platforms / governance | Admin vs. self-service boundaries in 4–5 major platforms' docs; survey data on employees' use of unsanctioned AI tools; incident evidence by governance model | open |
| Q6 | For always-on agents that run without a release event, what is the minimum continuous evidence (sampled action review, permission-violation monitoring, scheduled eval re-runs) needed for a low-risk agent, and who accepts it: automated rule, independent agent, or a human reviewing exceptions only? 对没有“发版时刻”的常驻智能体，低风险场景下最少需要哪些持续证据（抽样复核动作、越权监控、定期重跑评估），由谁接受：自动规则、独立智能体，还是人只处理例外？ Why: release-time evidence (one eval report per version) stops describing an agent that keeps meeting new data and tools while nobody watches; it sets the cost and staffing of agent governance. Initial take (interpretation): for low-risk read-only agents, rule-based thresholds with automatic suspend and human review of exceptions; stricter tiers need human sign-off. Triggered by always-on agent launches (e.g. OpenAI Dots, 2026-09-29, checked against OpenAI's own pages on 2026-10-04) and state-level auditor regimes (unverified); the Dots facts are verified against vendor pages (see the dots note), the auditor regimes remain a lead. Related: H1; overlaps Q2 (action logs) and Q3 (delegated acceptance). Note: first-pass research note (2026-10-03). | Governance / AgentOps | Regulator or standards text on ongoing monitoring of AI systems; vendor docs for agent monitoring and auto-suspend; enterprise deployments of always-on agents; verified California auditor-regime bill text | researching |
| Q7 | For enterprises whose data-product, feature and model platforms were built separately, how far should the control layer (catalog and asset IDs, lineage, identity and permissions, approval and audit) be unified first, and in what order, without merging the platforms themselves? What do Databricks Unity Catalog, other cloud platforms and open-source catalogs show about what works? 当数据产品、特征和模型平台是分别建设的，控制层（目录与资产 ID、血缘、身份与权限、审批与审计）应先统一到什么程度、按什么顺序，而不合并平台本身？Databricks Unity Catalog、其他云平台和开源目录的做法和教训是什么？ Why: separate platforms break lineage and duplicate permissions and audit; unifying everything at once is costly and ignores real differences in build and serving needs. It decides sequencing, ownership and build-vs-adopt choices. Initial take (interpretation): unify metadata in this order: (1) global asset IDs and a shared catalog, (2) cross-platform lineage, (3) shared identity and groups, (4) approval and policy; keep build and serving specialized. Unity Catalog is reported by Databricks to govern tables, volumes, functions, models and agent-related services as securables under one permission model (vendor docs; not yet verified page by page). Related: H4 (control-plane convergence); overlaps Q2 (systems of record) and Q5 (self-service vs central gating). Note: first-pass research note (2026-10-04). | Platforms / governance | Vendor docs and architecture write-ups (Databricks Unity Catalog, AWS, Google Cloud, Microsoft, Snowflake); open-source catalog and lineage projects and their adoption reports; practitioner case studies or conference talks on unifying governance across data, feature and model platforms; regulator expectations on lineage | open |
| Q8 | Which US documents actually bind or guide how a bank (and other regulated firms) builds, validates and monitors ML models, data and generative or agentic AI in 2026, and what will the agencies' planned model-risk request for information add? 2026 年，哪些美国文件真正约束或指导银行（及其他受监管机构）如何构建、验证和监控 ML 模型、数据以及生成式或智能体 AI？监管机构计划发布的模型风险征求意见稿会补充什么？ Why: it decides which lifecycle evidence is required, which is expected and which is optional, and where the gap for LLMs and agents sits. Initial take (interpretation): SR 26-2 (2026-04-17) is the newest bank model-risk guidance, covers non-generative ML but not generative or agentic AI, and is soft law; the rest of the US picture is voluntary NIST standards, state transparency laws and older sector and data statutes, under a federal push to narrow state rules. Related: H5 (supported), H1 (weak); overlaps Q3 and Q7. Note: detailed reading note (2026-10-04). | Governance / regulation | Issued agency request for information and responses; final FTC statement; Commerce evaluation of state laws; primary FINRA, SEC, CFPB and state texts; enforcement actions that cite model governance | open |
| Q9 | When an agentic MDLC tool claims the models it builds beat those built by general AI tools, what is the minimum benchmark (offline performance, inference speed, build speed; same tasks and budget; repeated runs; replicable by users on their own data) that its users would accept? agentic MDLC 工具声称“建出的模型优于通用 AI 工具建出的模型”时，用户愿意接受的最小 benchmark 是什么（离线性能、推理速度、建模速度；同任务同预算、重复运行、用户可在自己数据上复现）？ Why: it decides how the tool's advantage can be claimed credibly and how much evaluation effort that costs. Initial take (interpretation): compare the tool plus its underlying LLM, not the model alone; report the three dimensions separately (no single score); give the opponent a tuned configuration and fix the LLM where possible; use several task types, include at least one where the tool may lose, and use users' real data for part of them; repeat runs and report ranges; ship scripts, splits and logs so users can rerun; state the limits. Grounded in recent papers (agent rankings reverse across harnesses; LLM-judge verdicts flip on reruns), read from abstracts only. Related: H3, H1; overlaps Q4 (B-001 series) and Q3. | Benchmarks | Harness-comparison papers (arXiv 2610.00917, 2610.00651); LLM-judge reliability (2610.12083); vendor benchmark methodology pages; the owner's own B-series runs | open |