iter4: fix the 'constant belief = free reward' bug + 6 other deep issues

Investigation prompted by external code review. Validated 5/5 critical
findings against actual code; applied fixes for the 7 most impactful.

[1] Anomaly signals NOW REACH THE LLM (Issue 1, was CRITICAL)
Env was computing per-meter anomalies (actual_delta - expected_under_neutral)
and stashing them in reward_breakdown — but format_observation_prompt only
read meters and step_history. StepRecord didn't include anomalies. So the
cleanest profile-inference signal in the env was COMPUTED BUT NEVER SHOWN
to the agent. The agent has been doing meta-learning blindfolded.
Fix: extend StepRecord with 5 anomaly fields, populate in env.step,
surface in prompt and inference.py. The agent now sees how each person's
response deviates from the average baseline — direct profile fingerprint.

[2] Belief baseline subtraction (Issue 4, was CRITICAL — biggest impact)
OLD: reward = (1 - mae) - 0.5
Constant '5 5 5' emission scored +0.336 raw per step (× 3.0 weight =
+1.008 per step = +28 per episode for ZERO learning). This is the
iter-1 mode collapse mechanism in disguise. The agent learned to just
emit constant belief and harvest free reward.
NEW: reward = similarity - constant_baseline_similarity
Constant emission now gets -0.03 per step (negative!). Perfect emission
gets +0.37 per step. The learning gradient is now real (+0.4 gap).

[3] Profile weight cap 0.80 -> 0.45 (Issue 5)
Previously 38% of sampled profiles had one meter > 0.50 weight.
Some weighted vitality/cognition heavily, making SLEEP-spam OPTIMAL
(the env was correctly rewarding it; agent wasn't reward-hacking).
With cap at 0.45, every profile must weight 3+ meters meaningfully.
Forces varied-action strategies to be optimal across all profiles.

[4] Scaled-down shaping (Issue 3)
Iter 3 had -0.3 (3-in-row) / -0.4 (cycle) / +0.2 (new-action) shaping
that swung total reward by ~0.9 — overwhelming the ±0.5 env signal.
GRPO advantage was dominated by 'did this action vary' not 'did this fit
the profile'. Reduced to -0.10 / -0.15 / +0.07 — nudges, not overrides.

[5] Step-0 belief reward skipped (Issue 9)
At step 0, agent has no observation history. Optimal play is constant
prior, which pulled the rest of training toward constant emission.
Skip belief reward when step_index == 0.

[6] Belief-action coupling reward (Issue 10)
Iter 3 made belief generation come BEFORE action so causal attention
would link them, but no reward gradient enforced consistency. Added
explicit ±0.15 bonus/penalty for action choice matching emitted belief
(high social belief + SOCIALIZE = +0.15, high social + MEDITATE = -0.10,
morning belief + DEEP_WORK in morning slot = +0.15, etc.).
Now there's a direct training signal that the belief should INFORM the
action — completing the meta-learning loop.

[7] grader_bias moved out of _compute_reward into env_reward (Issue 11)
Previously the +0.5*Δprogress + 0.4*Δconnection bias lived in the env's
per-step reward, which fed _step_rewards, which fed adaptation_score in
the grader. The 'alignment' was partially self-cancelling.
Now: env per-step reward is pure profile-weighted (uncontaminated
inference signal); grader_bias only shapes the GRPO-visible training
reward. Grader's adaptation_score computed on raw rewards.

All 31 tests pass. Verified math:
- Constant '5 5 5' weighted reward: +1.0 per step -> -0.03 per step
- Perfect belief weighted reward: similar -> +0.37 per step
- Step 0 belief reward: rewarded -> 0 (no info)

Iter 3 still running as control to confirm these are the right fixes.
Iter 4 ready to submit when iter 3 completes.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

inference.py

@@ -199,14 +199,21 @@ def choose_action_llm(obs, llm_client: OpenAI) -> RhythmAction:
 
     history_lines = []
     for h in (getattr(obs, "step_history", None) or [])[-5:]:
+        # Iter 4 fix: include anomalies for profile-inference signal
+        va = getattr(h, "vitality_anomaly", 0.0)
+        ca = getattr(h, "cognition_anomaly", 0.0)
+        pa = getattr(h, "progress_anomaly", 0.0)
+        sa = getattr(h, "serenity_anomaly", 0.0)
+        cna = getattr(h, "connection_anomaly", 0.0)
         history_lines.append(
             f"  step {h.step}: {h.action} -> reward {h.reward:+.2f} "
             f"(V{h.vitality_delta:+.2f} C{h.cognition_delta:+.2f} "
             f"P{h.progress_delta:+.2f} S{h.serenity_delta:+.2f} Cn{h.connection_delta:+.2f})"
+            f" [anom V{va:+.2f} C{ca:+.2f} P{pa:+.2f} S{sa:+.2f} Cn{cna:+.2f}]"
         )
     history_str = ""
     if history_lines:
-        history_str = "\n\nRecent history:\n" + "\n".join(history_lines)
+        history_str = "\n\nRecent history (anom = profile-inference signal):\n" + "\n".join(history_lines)
 
     user_prompt = textwrap.dedent(f"""\
 Step: {obs.timestep}/{MAX_STEPS} ({day_name} {slot_name})

models.py

@@ -50,8 +50,12 @@ class StepRecord(BaseModel):
     """
     Record of one completed step included in step_history.
 
-    Contains the action taken, the reward received, and per-meter deltas.
-    The agent uses this history to detect personality anomalies over time.
+    Contains the action taken, the reward received, per-meter deltas, AND
+    per-meter ANOMALIES (actual_delta - expected_delta_under_neutral_profile).
+    The anomalies are the cleanest profile-inference signal — they tell the
+    agent how much THIS person's response deviates from the average person.
+    Without them, the agent has to back out the profile from raw deltas
+    without a baseline to compare against (much harder).
     """
 
     step: int
@@ -62,6 +66,12 @@ class StepRecord(BaseModel):
     progress_delta: float = 0.0
     serenity_delta: float = 0.0
     connection_delta: float = 0.0
+    # Iter 4 fix: anomalies (was computed in env but not exposed to agent)
+    vitality_anomaly: float = 0.0
+    cognition_anomaly: float = 0.0
+    progress_anomaly: float = 0.0
+    serenity_anomaly: float = 0.0
+    connection_anomaly: float = 0.0
 
 
 class RhythmObservation(Observation):

server/rhythm_environment.py

@@ -215,8 +215,10 @@ def sample_profile(seed: int) -> Dict[str, Any]:
     raw = [rng.gammavariate(a, 1.0) for a in alphas]
     total = sum(raw)
     weights = [w / total for w in raw]
-    # Clamp and re-normalize to avoid degenerate weights
-    weights = [max(0.02, min(0.80, w)) for w in weights]
+    # Iter 4 fix: tighter clamp (0.45 max) forces every profile to weight 3+
+    # meters meaningfully. Old 0.80 cap allowed single-meter dominant profiles
+    # where SLEEP-spam was correctly the optimal play (env wasn't lying).
+    weights = [max(0.05, min(0.45, w)) for w in weights]
     total = sum(weights)
     weights = [w / total for w in weights]
 
@@ -528,6 +530,8 @@ class RhythmEnvironment(Environment):
         self._state.active_event = active_event
 
         # --- 15. Append completed step to rolling history ---
+        # Iter 4 fix: include anomalies (was computed but only stashed in
+        # reward_breakdown which the prompt builder never read)
         self._step_history.append({
             "step": current_step,
             "action": action_name,
@@ -537,6 +541,11 @@ class RhythmEnvironment(Environment):
             "progress_delta": round(deltas["progress"], 4),
             "serenity_delta": round(deltas["serenity"], 4),
             "connection_delta": round(deltas["connection"], 4),
+            "vitality_anomaly": round(deltas["vitality"] - expected_no_profile["vitality"], 4),
+            "cognition_anomaly": round(deltas["cognition"] - expected_no_profile["cognition"], 4),
+            "progress_anomaly": round(deltas["progress"] - expected_no_profile["progress"], 4),
+            "serenity_anomaly": round(deltas["serenity"] - expected_no_profile["serenity"], 4),
+            "connection_anomaly": round(deltas["connection"] - expected_no_profile["connection"], 4),
         })
         if len(self._step_history) > HISTORY_LENGTH:
             self._step_history.pop(0)
@@ -702,20 +711,18 @@ class RhythmEnvironment(Environment):
             self._vitality = max(0.0, self._vitality - vd)
 
     def _compute_reward(self, deltas: Dict[str, float]) -> float:
-        """Compute reward as hidden-weighted sum + grader-aligned bias.
-
-        Iter 3 fix: Add a profile-INDEPENDENT bias term for progress and
-        connection. The original profile-weighted reward drives belief inference
-        (varies by profile), but allowed agents to game it by spamming recovery
-        actions if the sampled profile didn't weight progress/connection. The
-        bias makes the per-step reward correlate with the FINAL grader (which
-        weights progress 0.25 and connection 0.15).
+        """Compute pure profile-weighted per-step reward.
+
+        Iter 4 fix: REMOVED the grader_bias term from here (moved to the
+        TRAINING reward function in reward_functions.py). Keeping the env's
+        per-step reward pure means:
+        - Inference signal (which depends on profile_weights) is uncontaminated
+        - Grader's adaptation_score isn't computed on biased rewards (no
+          self-cancelling alignment)
+        - The env's reward semantics match what an honest deployment would see
         """
         weights = self._profile["reward_weights"]
-        profile_reward = sum(deltas[m] * weights[m] for m in METERS) * REWARD_SCALE
-        # Grader-aligned bias: scaled so max bonus is ~0.1/step (manageable vs profile_reward)
-        grader_bias = 0.5 * deltas["progress"] + 0.4 * deltas["connection"]
-        return profile_reward + grader_bias
+        return sum(deltas[m] * weights[m] for m in METERS) * REWARD_SCALE
 
     def _grade_episode(self) -> float:
         """
@@ -800,6 +807,11 @@ class RhythmEnvironment(Environment):
                 progress_delta=h["progress_delta"],
                 serenity_delta=h["serenity_delta"],
                 connection_delta=h["connection_delta"],
+                vitality_anomaly=h.get("vitality_anomaly", 0.0),
+                cognition_anomaly=h.get("cognition_anomaly", 0.0),
+                progress_anomaly=h.get("progress_anomaly", 0.0),
+                serenity_anomaly=h.get("serenity_anomaly", 0.0),
+                connection_anomaly=h.get("connection_anomaly", 0.0),
             )
             for h in self._step_history
         ]

training/dataset.py

@@ -72,14 +72,27 @@ def format_observation_prompt(obs, profile_hint: dict | None = None) -> str:
 
     history_lines = []
     for h in (obs.step_history or [])[-5:]:  # last 5 only to fit prompt budget
+        # Iter 4 fix: include ANOMALIES (actual_delta - expected_under_neutral_profile).
+        # Anomalies are the cleanest profile-inference signal: they show how
+        # this person's response DEVIATES from average. Previously the env
+        # computed these but never exposed them to the agent.
+        anom_str = (
+            f" [anom V{h.vitality_anomaly:+.2f} C{h.cognition_anomaly:+.2f} "
+            f"P{h.progress_anomaly:+.2f} S{h.serenity_anomaly:+.2f} "
+            f"Cn{h.connection_anomaly:+.2f}]"
+        )
         history_lines.append(
             f"  step {h.step}: {h.action} -> reward {h.reward:+.2f} "
             f"(V{h.vitality_delta:+.2f} C{h.cognition_delta:+.2f} "
             f"P{h.progress_delta:+.2f} S{h.serenity_delta:+.2f} Cn{h.connection_delta:+.2f})"
+            f"{anom_str}"
         )
     history_str = ""
     if history_lines:
-        history_str = "\n\nRecent history:\n" + "\n".join(history_lines)
+        history_str = (
+            "\n\nRecent history (anom = how this person deviated from neutral baseline):\n"
+            + "\n".join(history_lines)
+        )
 
     hint_str = ""
     if profile_hint is not None:

training/reward_functions.py

@@ -238,31 +238,66 @@ def env_reward(
 
         try:
             env = _replay_env(ep_seed, ep_history, ep_mode)
+            # Capture pre-step meters so we can compute deltas for the bias
+            pre_progress = env._progress
+            pre_connection = env._connection
             obs = env.step(RhythmAction(action_type=action_type))
             reward = obs.reward
             chosen = action_type.value
 
-            # Iter 2 fix: explicit 3-in-a-row repetition penalty
+            # Iter 4 fix (Issue 11): grader-aligned bias moved here from env.
+            # Per-step env reward stays pure (drives belief inference); the
+            # bias only shapes the GRPO-visible training reward.
+            progress_delta = env._progress - pre_progress
+            connection_delta = env._connection - pre_connection
+            reward += 0.5 * progress_delta + 0.4 * connection_delta
+
+            # Iter 4 fix (Issue 3): SCALED-DOWN shaping. Iter 3 had
+            # -0.3/-0.4/+0.2 which dominated the ±0.5 env signal. Now
+            # roughly 1/3 of original magnitudes — nudges, not overrides.
             if ep_history and len(ep_history) >= 2:
                 recent3 = ep_history[-3:]
-                if recent3.count(chosen) >= 2:  # this action would make 3+ in a row
-                    reward -= 0.3
+                if recent3.count(chosen) >= 2:
+                    reward -= 0.10  # was -0.3
 
-            # Iter 3 fix: N-CYCLE penalty (catches the M-E-M-E-... loop iter 2 fell into)
-            # If last 6 actions (including this one) have <=2 unique values, apply penalty
             if ep_history and len(ep_history) >= 5:
                 last6 = ep_history[-5:] + [chosen]
                 if len(set(last6)) <= 2:
-                    reward -= 0.4
+                    reward -= 0.15  # was -0.4
 
-            # Iter 3 fix: NEW-ACTION exploration bonus
-            # If this action hasn't appeared yet in the current episode, +0.2.
-            # Strong incentive in early steps to TRY varied actions, fading as
-            # the action set grows. Stops once 6+ different actions tried.
             if ep_history is not None:
                 seen = set(ep_history)
                 if chosen not in seen and len(seen) < 6:
-                    reward += 0.2
+                    reward += 0.07  # was +0.2
+
+            # Iter 4 fix (Issue 10): BELIEF-ACTION COUPLING reward.
+            # Parse the agent's emitted belief and reward consistency between
+            # belief and action choice. Without this, the belief-first format
+            # only enforces consistency via causal attention (weak); now there's
+            # an explicit gradient signal.
+            _, b, b_provided = extract_action_and_belief(response)
+            if b_provided:
+                s_pref, m_pref, w_pref = b
+                # High social → social actions; low social → solo actions
+                if s_pref > 0.65 and chosen in {"socialize", "family_time"}:
+                    reward += 0.15
+                elif s_pref < 0.35 and chosen in {"meditate", "me_time"}:
+                    reward += 0.15
+                elif s_pref > 0.65 and chosen in {"meditate", "me_time"}:
+                    reward -= 0.10  # belief says extrovert, action says solo
+                elif s_pref < 0.35 and chosen in {"socialize", "family_time"}:
+                    reward -= 0.10  # belief says introvert, action says social
+
+                # High morning + morning slot + work → bonus
+                slot = obs.slot if hasattr(obs, "slot") else 0
+                if m_pref > 0.65 and slot == 0 and chosen in {"deep_work", "learn"}:
+                    reward += 0.15
+                elif m_pref < 0.35 and slot in (2, 3) and chosen in {"deep_work", "learn"}:
+                    reward += 0.15
+
+                # High work → work actions
+                if w_pref > 0.65 and chosen in {"deep_work", "learn", "admin_work"}:
+                    reward += 0.15
 
             scores.append(reward)
         except Exception:
@@ -283,14 +318,19 @@ def belief_accuracy(
     """
     Layer 4: Belief-vector accuracy reward (META-LEARNING signal).
 
-    Compares the agent's emitted [social, morning, work] belief vector to the
-    hidden profile's true belief vector. Reward in [-0.5, +0.5]:
-      perfect match  → +0.5
-      neutral [0.5,0.5,0.5] → 0.0  (zero-effort baseline)
-      max wrong      → -0.5
+    ITER 4 FIX (Issue 4 from external review): The previous formula
+    `(1 - mae) - 0.5` gave constant emission "5 5 5" a free +0.336 reward
+    per step (× 3.0 weight = +1.0 per step = +28 per episode for ZERO learning).
+    This recreated the iter-1 collapse mechanism in disguise.
+
+    New formula: subtract the per-profile baseline. The baseline is what a
+    constant 0.5 emission WOULD score for THIS profile. Now:
+      - Constant emission → reward ≈ 0 (no free reward)
+      - Better-than-baseline belief → positive
+      - Worse-than-baseline belief → negative
 
-    Mean-absolute-error based (cleaner than cosine for [0,1] vectors).
-    Skipped (returns 0) if no seed available — keeps reward conservative.
+    Plus iter 4 (Issue 9): no belief reward at step 0 (no information available
+    to commit a belief) — prevents pulling the policy toward a constant prior.
     """
     scores = []
     for i, completion in enumerate(completions):
@@ -298,24 +338,35 @@ def belief_accuracy(
         _, belief, belief_provided = extract_action_and_belief(response)
 
         if not belief_provided:
-            scores.append(-0.2)  # weak push toward emitting belief
+            scores.append(-0.1)  # weak push toward emitting belief
             continue
 
-        # Resolve seed/mode for replay
+        # Resolve seed/mode/step for replay
         if seed is not None and i < len(seed):
             ep_seed = seed[i]
             ep_history = action_history[i] if action_history is not None else []
             ep_mode = profile_mode[i] if (profile_mode is not None and i < len(profile_mode)) else "continuous"
+            ep_step = step_index[i] if (step_index is not None and i < len(step_index)) else 0
         else:
             scores.append(0.0)
             continue
 
+        # Iter 4 fix (Issue 9): step-0 commitment with no info biases toward
+        # constant prior. Skip belief reward at step 0.
+        if ep_step == 0:
+            scores.append(0.0)
+            continue
+
         try:
             env = _replay_env(ep_seed, ep_history, ep_mode)
             true_belief = env.get_belief_target()
+            # Iter 4 fix (Issue 4): subtract the constant-baseline reward
             mae = sum(abs(b - t) for b, t in zip(belief, true_belief)) / 3.0
-            similarity = 1.0 - mae  # in [0, 1]
-            scores.append(similarity - 0.5)  # in [-0.5, +0.5]
+            similarity = 1.0 - mae
+            baseline_mae = sum(abs(0.5 - t) for t in true_belief) / 3.0
+            baseline_similarity = 1.0 - baseline_mae
+            # Reward = how much better than the constant-emit baseline
+            scores.append(similarity - baseline_similarity)
         except Exception:
             scores.append(0.0)