mirror of
https://github.com/shareAI-lab/analysis_claude_code.git
synced 2026-09-22 05:13:48 +08:00
docs: make workflow and goal lessons harness-first
This commit is contained in:
Haoran
@@ -8,8 +8,6 @@ s01 → ... → s20 → s21 → `s22`
|
||||
>
|
||||
> **Harness 層**: Goal closure — turn 終端に program-controlled completion gate を追加します。
|
||||
|
||||
> **情報源の境界：** この章の製品詳細は Claude Code 2.1.177 の clean-room 行動再構成に基づく。後続リリースで名称や制限は変わり得る。`code.py` はオフライン教材モデルであり、製品ソースの複製ではない。
|
||||
|
||||
---
|
||||
|
||||
s01 から s21 まで、会話の 1 turn はどう終わったでしょうか。モデルが `tool_use` を出さなくなると、loop はそのまま `return` しました。one-shot task なら問題ありません。終わったら止まります。
|
||||
@@ -20,7 +18,7 @@ s01 から s21 まで、会話の 1 turn はどう終わったでしょうか。
|
||||
|
||||
## /goal: 各 turn の終端に gate を追加する
|
||||
|
||||
`/goal <condition>` を入力すると session-scoped stopping condition を設定します。program は active goal として保存し、各 turn の後に独立した lightweight model を evaluator として使い、transcript 内の trusted evidence が condition を満たすか確認します。不足なら gate が停止を拒み、次ラウンドへ「作業を続ける」prompt を queue します。十分なら goal を消して complete とします。
|
||||
`/goal <condition>` を入力すると session-scoped stopping condition を設定します。program は active goal として保存し、各 turn の後に evaluator が transcript 内の trusted evidence を condition と照合します。不足なら gate が停止を拒み、次ラウンドへ「作業を続ける」prompt を queue します。十分なら goal を消して complete とします。
|
||||
|
||||
|
||||
|
||||
@@ -40,8 +38,6 @@ if not has_tool_use(response):
|
||||
|
||||
この gate を制御するのは program です。モデルが自分を律しているのではありません。モデルは gate の存在すら知らず、次のラウンドの入力を受け取って作業を続けるだけです。
|
||||
|
||||
> 実際の Claude Code では `/goal` は session-scoped Stop hook で、workspace trust と hook restriction の管理下にあります。コードには `active_goal`、`goal_status`、`goal_met`、`tengu_goal_achieved` などの marker があります。
|
||||
|
||||
## Goal の設定: Evidence は command の後から数える
|
||||
|
||||
`set_goal` は active goal として、goal text、最大 turn budget、counter、そして evidence window の開始点 `start_index` を保存します。現在の transcript length を使うため、`/goal` command 自身は window の外です。これが最初の防御です。command が自分自身の完了を証明することはできません。
|
||||
@@ -55,8 +51,6 @@ def set_goal(self, objective, max_turns=20):
|
||||
}
|
||||
```
|
||||
|
||||
> 実際の Claude Code では `GoalRuntime.setGoal()` が active goal、開始位置、counter、budget を保存し、submit 後に `resetEvidenceStart()` で window を command 後へそろえます。
|
||||
|
||||
## Evaluator: 実在する evidence だけを信頼する
|
||||
|
||||
ここが仕組み全体の core です。evaluator は会話全体を見ず、evidence window 内で trusted source から来た message だけを見ます。3 層の filter が、「完了したと言ったから完了」という内容をすべて外へ止めます。
|
||||
@@ -79,9 +73,7 @@ def evidence_text(self):
|
||||
|
||||
効果は明確です。同じ `tests passed` でも、あなたが入力したものは数えず、background task notification が持ち帰ったものだけを数えます。モデルは「完了した」と自分で言うだけでは goal を complete にできません。これはコース全体に繰り返し現れた trust boundary の最後の登場です。s16 は protocol が理解ではなく field に依存すると言い、s19 は annotation が申告であり、申告は嘘をつけると言い、s22 は completion evidence を content ではなく origin で信頼します。
|
||||
|
||||
教材版の `goal_satisfied()` は決定的な keyword matching です。実際の版は evidence window を別の lightweight model へ渡して判定します。
|
||||
|
||||
> 実際の Claude Code の evaluator は作業モデルとは別の lightweight model で、`evaluatorModel`、`default small fast model` と記されています。任意の text を信じず、会話内の evidence を判断します。
|
||||
最小版の `goal_satisfied()` は決定的な keyword matching を使い、demo を offline かつ再現可能に保ちます。production harness では、この policy を独立した lightweight evaluator model に置き換えられますが、trusted evidence boundary はそのまま維持します。
|
||||
|
||||
## Gate の 3 状態: Completed / continuing / budget 超過
|
||||
|
||||
@@ -106,8 +98,6 @@ def evaluate_after_turn(self):
|
||||
|
||||
continuation prompt には、わざわざ自身を evidence にしないよう書き、filter でも除外します。これで false positive を防ぐ 3 層がそろいます。command text、reminder text、ordinary conversation のいずれも数えません。budget は s11 の古い規則に従います。automatic retry mechanism には必ず上限が必要です。そうでなければ、永遠に satisfied にならない goal が費用を燃やし続けます。
|
||||
|
||||
> 実際の Claude Code の `evaluateAfterTurn` は `goal_evaluated` event を出し、結果に応じて complete、continuation queue、gate の解除を行います。default budget は 20 turn です。
|
||||
|
||||
## Continuation prompt と外部 asynchronous message を分ける
|
||||
|
||||
continuation prompt は同じ `CommandQueue` に入りますが、task completion notification や monitor line といった外部 asynchronous event とは別の方法で消費します。`dequeue` には switch があり、外部 inbox を消費するときは goal continuation を既定で skip します。
|
||||
@@ -121,9 +111,7 @@ def dequeue(self, include_goal_continuations=True):
|
||||
return None
|
||||
```
|
||||
|
||||
なぜ分けるのでしょう。実際の model test では、モデルが continuation prompt を外部 notification と一緒に消費し、background evidence が到着する前に goal を complete と判定する bug が起きました。分離後は goal の進行が明示的な 1 step になり、asynchronous event に偶然運ばれません。
|
||||
|
||||
> 実際の Claude Code の `drainCommandQueue` は既定で `includeGoalContinuations=false` とし、goal continuation の消費を外部 asynchronous inbox から分けます。
|
||||
なぜ分けるのでしょう。同じ consumer が continuation prompt と外部 notification を一緒に取り出すと、background result が届く前に reminder text を新しい evidence と誤認する可能性があります。分離後は goal の進行が明示的な 1 step になり、asynchronous event に偶然運ばれません。
|
||||
|
||||
## 実際に動かす
|
||||
|
||||
|
||||
@@ -8,8 +8,6 @@ s01 → ... → s20 → s21 → `s22`
|
||||
>
|
||||
> **Harness layer**: Goal closure — a program-controlled completion gate at the end of each turn.
|
||||
|
||||
> **Source boundary:** Product details in this chapter are a clean-room behavioral reconstruction of Claude Code 2.1.177. Names and limits may change in later releases; `code.py` is an offline teaching model, not copied product source.
|
||||
|
||||
---
|
||||
|
||||
From s01 through s21, how does a conversation turn end? When the model stops emitting `tool_use`, the loop simply executes `return`. That is fine for one-shot work: finish and stop.
|
||||
@@ -20,7 +18,7 @@ This thread was present from the first chapter. s01 explained that exiting the l
|
||||
|
||||
## /goal: Add a Gate at the End of Every Turn
|
||||
|
||||
Entering `/goal <condition>` sets a session-scoped stopping condition. The program stores it as the active goal. After each turn, an independent lightweight model acts as evaluator and checks whether trusted evidence in the transcript satisfies the condition. If evidence is insufficient, the gate blocks the attempted stop and queues a "keep working" prompt for the next round. If it is sufficient, the goal is cleared and marked complete.
|
||||
Entering `/goal <condition>` sets a session-scoped stopping condition. The program stores it as the active goal. After each turn, an evaluator checks whether trusted evidence in the transcript satisfies the condition. If evidence is insufficient, the gate blocks the attempted stop and queues a "keep working" prompt for the next round. If it is sufficient, the goal is cleared and marked complete.
|
||||
|
||||
|
||||
|
||||
@@ -40,8 +38,6 @@ if not has_tool_use(response):
|
||||
|
||||
The program controls this gate. It is not the model restraining itself. The model does not even know the gate exists; it simply receives another round of input and continues working.
|
||||
|
||||
> In the real Claude Code, `/goal` is a session-scoped Stop hook governed by workspace trust and hook restrictions. The code contains markers such as `active_goal`, `goal_status`, `goal_met`, and `tengu_goal_achieved`.
|
||||
|
||||
## Setting a Goal: Evidence Starts after the Command
|
||||
|
||||
`set_goal` stores an active goal containing the objective text, a maximum-turn budget, counters, and `start_index`, the beginning of the evidence window. It uses the transcript's current length, placing the `/goal` command itself outside the window. This is the first defense: a command cannot prove its own completion.
|
||||
@@ -55,8 +51,6 @@ def set_goal(self, objective, max_turns=20):
|
||||
}
|
||||
```
|
||||
|
||||
> In the real Claude Code, `GoalRuntime.setGoal()` stores the active goal, start position, counters, and budget, then `resetEvidenceStart()` aligns the window to the position after command submission.
|
||||
|
||||
## The Evaluator: Trust Concrete Evidence Only
|
||||
|
||||
This is the core of the entire mechanism. The evaluator does not inspect the whole conversation. It sees only messages inside the evidence window that come from trusted sources. Three filters keep every form of "I said it was done, so it must be done" outside:
|
||||
@@ -79,9 +73,7 @@ def evidence_text(self):
|
||||
|
||||
The effect is clear. The same sentence, `tests passed`, does not count when typed by you, but does count when delivered by a background task notification. The model cannot bluff its way out by saying "I finished." This is the final appearance of the trust boundary repeated throughout the course. s16 said protocols rely on fields, not interpretation. s19 said annotations are claims and claims may be false. s22 says completion evidence is trusted by origin, not by content alone.
|
||||
|
||||
The teaching version's `goal_satisfied()` uses deterministic keyword matching. The real version asks a separate lightweight model to judge the evidence window.
|
||||
|
||||
> In the real Claude Code, the evaluator is a lightweight model separate from the working model, marked as `evaluatorModel` and the `default small fast model`. It judges evidence in the conversation rather than trusting arbitrary text.
|
||||
The minimal `goal_satisfied()` uses deterministic keyword matching so the demo stays offline and reproducible. A production harness can replace this policy with a separate lightweight evaluator model, while keeping the same trusted evidence boundary.
|
||||
|
||||
## Three Gate States: Completed, Continuing, or Over Budget
|
||||
|
||||
@@ -106,8 +98,6 @@ def evaluate_after_turn(self):
|
||||
|
||||
The continuation prompt explicitly says not to treat itself as evidence, and the evidence filter excludes it. That completes the three layers against false positives: the command does not count, the reminder does not count, and ordinary conversation does not count. The budget follows the old rule from s11: every automatic retry mechanism needs a limit. Otherwise, a goal that can never be satisfied becomes a perpetual money-burning machine.
|
||||
|
||||
> In the real Claude Code, `evaluateAfterTurn` emits a `goal_evaluated` event and either completes, queues a continuation, or stops blocking. The default budget is 20 turns.
|
||||
|
||||
## Keep Continuation Prompts Separate from External Asynchronous Messages
|
||||
|
||||
Continuation prompts enter the same `CommandQueue`, but they are not consumed in the same way as external asynchronous events such as task-completion notifications and monitor lines. `dequeue` has a switch, and consumption of the external inbox skips goal continuations by default.
|
||||
@@ -121,9 +111,7 @@ def dequeue(self, include_goal_continuations=True):
|
||||
return None
|
||||
```
|
||||
|
||||
Why separate them? A real model test exposed a bug where the model consumed the continuation prompt together with an external notification and marked the goal complete before background evidence arrived. With the paths separated, goal progression is an explicit step and cannot be carried along accidentally by asynchronous events.
|
||||
|
||||
> In the real Claude Code, `drainCommandQueue` defaults to `includeGoalContinuations=false`, separating goal-continuation consumption from the external asynchronous inbox.
|
||||
Why separate them? If one consumer drains continuation prompts together with external notifications, a reminder can be mistaken for new evidence before the background result arrives. With the paths separated, goal progression is an explicit step and cannot be carried along accidentally by asynchronous events.
|
||||
|
||||
## See It Run
|
||||
|
||||
|
||||
@@ -8,8 +8,6 @@ s01 → ... → s20 → s21 → `s22`
|
||||
>
|
||||
> **Harness 层**: 目标闭环 — 在轮次收尾处，加一道程序控制的完成闸门。
|
||||
|
||||
> **来源边界：** 本章产品细节来自对 Claude Code 2.1.177 的 clean-room 行为重建。后续版本可能更改名称与限制；`code.py` 是离线教学模型，不是产品源码复制。
|
||||
|
||||
---
|
||||
|
||||
从 s01 到 s21，一轮对话怎么结束？模型不再发 `tool_use`，循环就直接 `return` 了。一次性任务这么干没问题，做完就停。
|
||||
@@ -20,7 +18,7 @@ s01 → ... → s20 → s21 → `s22`
|
||||
|
||||
## /goal：每轮收尾加一道闸门
|
||||
|
||||
输入 `/goal <条件>` 就设了一个会话级的停止条件。程序把它存成当前活跃目标，每轮结束后，用一个独立的轻量小模型当判断器，看对话记录里的可信证据够不够满足条件。不够，闸门就把这次结束拦住，塞一条"继续干"的提示进下一轮；够了，就清除目标，标记完成。
|
||||
输入 `/goal <条件>` 就设了一个会话级的停止条件。程序把它存成当前活跃目标，每轮结束后，判断器检查对话记录里的可信证据够不够满足条件。不够，闸门就把这次结束拦住，塞一条"继续干"的提示进下一轮；够了，就清除目标，标记完成。
|
||||
|
||||
|
||||
|
||||
@@ -40,8 +38,6 @@ if not has_tool_use(response):
|
||||
|
||||
这道闸门是程序自己控制的。不是模型自己约束自己，模型甚至不知道有这么一道闸门，它只是收到了下一轮的输入，接着干就是了。
|
||||
|
||||
> 真实 Claude Code：`/goal` 是会话级的 Stop hook，受工作区信任和 hook 限制控制；代码里有 `active_goal`、`goal_status`、`goal_met`、`tengu_goal_achieved` 这些标记。
|
||||
|
||||
## 设目标：证据从命令之后开始算
|
||||
|
||||
`set_goal` 会存一个活跃目标：目标文本、最大轮数预算、计数器，还有 `start_index`——也就是证据窗口的起点。它取当前对话记录的长度，所以 `/goal` 这行命令本身在窗口外面。这是第一道防线：命令自己不能证明自己完成了。
|
||||
@@ -55,8 +51,6 @@ def set_goal(self, objective, max_turns=20):
|
||||
}
|
||||
```
|
||||
|
||||
> 真实 Claude Code：`GoalRuntime.setGoal()` 存活跃目标、起始位置、计数器和预算；提交后再 `resetEvidenceStart()` 把窗口对齐到命令之后。
|
||||
|
||||
## 判断器：只信实打实的证据
|
||||
|
||||
这是整个机制最核心的地方。判断器不看整段对话，只看证据窗口里来自可信来源的消息。三层过滤，把"嘴上说完成了但不算数"的内容全挡在外面：
|
||||
@@ -79,9 +73,7 @@ def evidence_text(self):
|
||||
|
||||
效果很明显：同样一句 `tests passed`，你打字说的不算，后台任务通知带回来的才算。模型糊弄不过去，它没法靠自己说一句"我做完了"就把目标判成完成。这是全课程反复出现的那条信任边界的最后一次登场：s16 说协议靠字段不靠理解，s19 说注解是申报、申报可以撒谎，s22 说完成证据只看来源不看内容。
|
||||
|
||||
教学版里 `goal_satisfied()` 是确定的关键词匹配；真实版会把证据窗口交给一个轻量小模型来判断。
|
||||
|
||||
> 真实 Claude Code：判断器是和干活的模型分开的轻量小模型（标记是 `evaluatorModel`、`default small fast model`），判断对话里的证据，不是随便什么文本都信。
|
||||
最小版的 `goal_satisfied()` 使用确定的关键词匹配，让演示保持离线和可复现。生产级 harness 可以把这条策略替换成独立的轻量判断模型，但仍然保留相同的可信证据边界。
|
||||
|
||||
## 闸门三态：完成/继续/超预算
|
||||
|
||||
@@ -106,8 +98,6 @@ def evaluate_after_turn(self):
|
||||
|
||||
那条"继续干"的提示里特意写了"别把这条提醒当成完成证据"，连提醒本身都被排除在证据之外。三层防误判就齐了：命令文本不算、提醒文本不算、普通聊天文本不算。预算则是 s11 教过的老规矩：任何自动重试的机制都得有上限，不然一个永远判不满足的目标就是个烧钱的永动机。
|
||||
|
||||
> 真实 Claude Code：`evaluateAfterTurn` 会发 `goal_evaluated` 事件，按结果完成/塞继续提示/拦截；默认预算是 20 轮。
|
||||
|
||||
## 继续提示和外部异步消息分开走
|
||||
|
||||
继续提示进的是同一个 `CommandQueue`，但它和外部异步事件（任务完成通知、监控行）不是同一种消费方式。`dequeue` 带个开关：消费外部收件箱的时候，默认跳过目标的继续提示。
|
||||
@@ -121,9 +111,7 @@ def dequeue(self, include_goal_continuations=True):
|
||||
return None
|
||||
```
|
||||
|
||||
为什么要分开？真实模型测试的时候出过一个 bug：模型把继续提示当成外部通知一起消费了，结果后台证据还没到，就提前把目标判成完成了。分开之后，目标的推进是显式的一步，不会被异步事件带着走。
|
||||
|
||||
> 真实 Claude Code：`drainCommandQueue` 默认 `includeGoalContinuations=false`，把目标继续提示和外部异步收件箱的消费分开。
|
||||
为什么要分开？如果同一个消费者把继续提示和外部通知一起取走，后台结果还没到，提醒文本就可能被误当成新证据。分开之后，目标的推进是显式的一步，不会被异步事件带着走。
|
||||
|
||||
## 跑起来看看
|
||||
|
||||
|
||||
@@ -1,9 +1,5 @@
|
||||
"""
|
||||
s22_goal_loop — /goal session goal loop (teaching version)
|
||||
|
||||
Clean-room behavioral reconstruction of Claude Code's `/goal` command. Grounded
|
||||
in @anthropic-ai/claude-code@2.1.177 observed behavior
|
||||
(reverse-research/cc_goal_loop), NOT leaked source.
|
||||
s22_goal_loop — minimal /goal session loop for a teaching harness
|
||||
|
||||
Idea:
|
||||
s01-s21 end a turn when the model emits no tool_use. `/goal` adds a
|
||||
@@ -27,12 +23,12 @@ Idea:
|
||||
Run:
|
||||
python code.py # /goal until tests pass + deploy green; watch the gate
|
||||
|
||||
Teaching simplifications (vs real /goal and runtime.mjs):
|
||||
Implementation choices:
|
||||
- The evaluator is a deterministic keyword check, not a small/fast model.
|
||||
- One mock task-notification produces the trusted evidence; the loop / monitor
|
||||
/ background-task plane (s13/s14) is out of scope — this chapter is just the
|
||||
goal gate.
|
||||
- The evidence trust boundary is the faithful part: only task-notification /
|
||||
- The evidence trust boundary is the important part: only task-notification /
|
||||
monitor-line origins count as evidence, so the `/goal` command text, the
|
||||
continuation reminder, and plain assistant prose can NOT satisfy the goal.
|
||||
Ordinary `submit()` calls cannot set those labels; only the host-event
|
||||
@@ -68,7 +64,7 @@ class Message:
|
||||
|
||||
|
||||
# ============================================================
|
||||
# CommandQueue — continuation prompts live here (mirrors CommandQueue)
|
||||
# CommandQueue — continuation prompts live here
|
||||
# ============================================================
|
||||
class CommandQueue:
|
||||
PRIORITY = {"now": 0, "next": 1, "later": 2}
|
||||
@@ -102,7 +98,7 @@ class CommandQueue:
|
||||
|
||||
|
||||
# ============================================================
|
||||
# GoalRuntime — the turn-completion gate (mirrors GoalRuntime)
|
||||
# GoalRuntime — the turn-completion gate
|
||||
# ============================================================
|
||||
class GoalRuntime:
|
||||
def __init__(self, transcript, queue):
|
||||
@@ -148,9 +144,8 @@ class GoalRuntime:
|
||||
return "\n".join(out)
|
||||
|
||||
def goal_satisfied(self):
|
||||
# Real Claude Code routes this to a small/fast evaluator model reading
|
||||
# the evidence window. The teaching version is a deterministic keyword
|
||||
# check so the lifecycle is reproducible.
|
||||
# A production harness can route this evidence window to a separate
|
||||
# evaluator model. The demo uses a deterministic keyword policy.
|
||||
objective = self.active["objective"].lower()
|
||||
evidence = self.evidence_text().lower()
|
||||
wants_tests = "test" in objective
|
||||
@@ -193,7 +188,7 @@ class GoalRuntime:
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Session — the main loop host with a Stop gate (mirrors submit / drain)
|
||||
# Session — the main loop host with a Stop gate
|
||||
# ============================================================
|
||||
class Session:
|
||||
def __init__(self):
|
||||
|
||||
Reference in New Issue
Block a user