feat(memory): upgrade migrates every v1 dialect — headings normalized, frontmatter added, verdict oracle

This commit is contained in:
bchanot committed 2026-10-28 14:34:48 +01:00
1 parent c318ee6297
commit 18d1c0b6bc
4 files changed
+202 -40

No files matched your search

@@ -32,7 +32,7 @@
3. **`upgrade` body wins**: an Index status reaches the body only when the body has none, or when the Index says superseded/deprecated and the body does not. Every other conflicting Index status is dropped and named on stderr (`memory: BDR-012: kept body shipped, dropped Index accepted`, the only one on this repo).
4. **Status word**: a lib-written `**Status (YYYY-MM-DD…)**:` line reads its leading word (`status` requires the text to start with one: "open, reopened by abc"); other labelled lines rank words in `_MEM_WORDS` order, `open` last ("accepted. check OPEN" = accepted). A heading candidate is its `[tag]` only, never the title.
5. **One classifier on v2, the lexical reader kept on v1**: on a v2 registry reconcile classifies by word (`memory_closed`: resolved | superseded | deprecated); `accepted` on a blocker stays open. On a v1 registry reconcile keeps today's lexical reader verbatim: the strict parser, measured on copies of all 13 other projects' blockers, changed the open set of 4 (config `— RESOLVED` title suffix, game `date: … · status:`, zenquality and angelscar356 `**Statut** :`); with the v1 branch the 13 sets are identical to today.
6. **Heading dialects**: `mem_key` accepts `## ID` followed by any non-alphanumeric char (` — `, ` - `, ` | `), so `status` and `--supersedes` work on v1 hyphen and pipe files. ID allocation reads every heading level, fenced or not (`### BDR-001` in trucks_at_work, a stray fence elsewhere), and a v1 add whose ID already sits in any heading is refused. `status` on a `###` registry stays rc 2 "no entry" (loud, not silent). A v1 `add --status s` writes `- **Status**: s` as the first body line instead of a heading tag: the v1 reconcile reader never reads the entry's own heading. The em-dash shape check is v2-only. `upgrade` refuses a non-em-dash file (`heading dialect " - ": not upgraded`) and a file without frontmatter; the banner says `memory.sh check`, never promises an upgrade.
6. **Heading dialects**: `mem_key` accepts `## ID` followed by any non-alphanumeric char (` — `, ` - `, ` | `), so `status` and `--supersedes` work on v1 hyphen and pipe files. ID allocation reads every heading level, fenced or not (`### BDR-001` in trucks_at_work, a stray fence elsewhere), and a v1 add whose ID already sits in any heading is refused. `status` on a `###` registry stays rc 2 "no entry" (loud, not silent). A v1 `add --status s` writes `- **Status**: s` as the first body line instead of a heading tag: the v1 reconcile reader never reads the entry's own heading. The em-dash shape check is v2-only. `upgrade` migrates the measured dialects (Task 8): only headings that fail the v2 shape are rewritten (` - ` and ` | date | ` separators, `###`), a file without frontmatter gets the template's, every blocker verdict is proven against reconcile's v1 reader (a v1-closed entry the strict reader would open gets a `Status (<date>, upgrade)` line; the reverse, the BLK-032..034 false-open fix, is kept and named). It still refuses what the oracle cannot prove (duplicate IDs, `## ID ADDENDUM` headings: game); the banner says `memory.sh check`, never promises an upgrade.
7. **Journal template** carries no `format: 2`: journal.md is not a registry (`check` excludes it). Contract criterion 8 says "every `templates/memory/*.md`": the four registry templates.
8. **awk programs**: an embedded awk program counts as one statement toward the 25-logic-line limit (`_mem_upgrade_text`, `_mem_upgrade_oracle`, `_mem_index_rows` hold 20-30 awk lines). Splitting them would scatter one parser across files.
9. **`--staged` removal rule**: a removed line counts only from the first entry heading down (the spec said any `-` line). The Index sits above the first entry, so Task 7's upgrade commit goes through memory-commit like any memory commit; the diff runs with `-c diff.interHunkContext=0 --no-color --no-ext-diff --no-textconv` so user git config cannot fold or hide hunks. An open code fence always counts as a culprit, even when it predates the commit (it hides headings from allocation).
+131 -28
View File
@@ -15,7 +15,12 @@
# --staged: only NEW culprits vs HEAD, no line removed
# inside an entry, no v2 registry deleted or downgraded
# upgrade [<memory-dir>] v1 → v2: Index-only dates and statuses become
# body lines, `format: 2`, `## Index` removed, oracle first
# body lines, `format: 2`, `## Index` removed; a missing
# frontmatter comes from the template, ` - ` / ` | date | `
# / `###` headings take the `## ID — title (date)` shape;
# a blocker whose open / closed verdict (reconcile's v1
# reader) would move gets a `Status (…, upgrade)` line; the
# oracle (IDs, check, verdicts) runs first, else rc 2
# <kind> = decision | learning | blocker | eval. MEMORY_DATE overrides today.
# Status of an entry: the last candidate wins. A candidate is the heading's
# trailing `[tag]`, or a body line whose bold label names Status, Resolution,
@@ -556,8 +561,9 @@ _mem_upgrade_text() { # <rows> <file> <PREFIX> → the v2 text on stdout
/^```/ { fence = !fence }
$0 == "## Index" { skip = 1; next }
skip { if (/^\|/) { tbl = 1; next }
if (!tbl || $0 == "") { if (tbl) skip = 0; next }
skip = 0 }
if (tbl) { skip = 0; if ($0 == "") next }
else if ($0 ~ /^#/ || $0 == "---") skip = 0
else next }
!fence && /^## / { flush(); split($0, w, " ")
cur = (w[2] in col) ? w[2] : ""; print
if (cur != "" && !hd[cur] && dt[cur] != "") print "- **Date**: " dt[cur]
@@ -567,12 +573,70 @@ _mem_upgrade_text() { # <rows> <file> <PREFIX> → the v2 text on stdout
END { flush() }' "$1" <(_mem_index_rows "$2" "$3") "$2"
}
_mem_upgrade_oracle() { # <old> <new> <PREFIX> <rows> → rc 2 on a lost fact
_mem_heading_ids() { # <file> <PREFIX> → the entry IDs, any heading level
"$MEM_GREP" -oE "^#+ *$2-[0-9]+" "$1" | sed 's/^#* *//'
}
memory_legacy_status() { # <file> <ID> → the v1 lexical status line (reconcile
# reads it on a v1 blockers file; `upgrade` proves its verdicts against it)
# v1: drop ALL `## BLK-` header lines first: the range is inclusive of the
# NEXT entry's header, and a sibling header may carry a status word.
sed -n "/^## ${2} /,/^## BLK-/p" "$1" \
| "$MEM_GREP" -v '^## BLK-' \
| "$MEM_GREP" -iE 'status|RESOLVED|REVERTED|upstream|resolved|[^a-z]open' \
| tail -1
}
_mem_legacy_open() { # <old file> <ID> → rc 0 when reconcile's v1 reader lists
local st # the blocker open: enumerated by its `## ID` heading and
"$MEM_GREP" -qE "^## $2([^0-9]|$)" "$1" || return 1 # not RESOLVED
st=$(memory_legacy_status "$1" "$2")
case "$st" in *RESOLVED*|*resolved*) return 1 ;; esac
}
_mem_index_stated() { # <old> <PREFIX> → IDs whose Index status upgrade
# writes into the body (mem_want): a fact the v1 reader never saw
awk -F'\t' -v p="$2" -v MEMW="$_MEM_WORDS" "$_MEM_AWK"'
FILENAME == ARGV[1] { col[$1] = $3; next }
{ split($0, a, / \| /); id = a[1]; sub(/^\| /, "", id); s = a[4]
sub(/ \|$/, "", s)
if (id in col && mem_want(p, col[id], s) != "") print id }' \
<(_mem_index_table "$1") <(_mem_index_rows "$1" "$2")
}
_mem_verdict_diff() { # <old> <new> → "ID<TAB>word" per blocker whose open /
local id was now ix # closed verdict moved (word = the old verdict)
ix=$(_mem_index_stated "$1" BLK)
for id in $(_mem_heading_ids "$2" BLK); do
printf '%s\n' "$ix" | "$MEM_GREP" -qx "$id" && continue
_mem_legacy_open "$1" "$id" && was=open || was=resolved
memory_closed "$(memory_status_word "$(memory_entry_status "$2" "$id")")" \
&& now=resolved || now=open
[ "$was" = "$now" ] || printf '%s\t%s\n' "$id" "$was"
done
}
_mem_upgrade_repair() { # <old> <new> <PREFIX> → no blocker loses its closure
# v1 closed, strict reader open (a dialect it cannot read): a status line
# restores it. v1 open, strict closed is the BLK-032..034 fix (prose words
# read as status by the v1 reader): kept, named
local id word
[ "$3" = BLK ] || return 0
while IFS=$'\t' read -r id word; do
if [ "$word" = open ]; then
echo "memory: $id: v1 reader open, strict reader closed (kept)" >&2
continue
fi
_mem_insert "$2" "$id" "- **Status ($(_mem_date), upgrade)**: $word" \
|| return 2
done < <(_mem_verdict_diff "$1" "$2")
}
_mem_upgrade_oracle() { # <old> <new> <PREFIX> → rc 2 on a lost fact
local o n
[ "$("$MEM_GREP" -E "^## $3-[0-9]" "$1")" \
= "$("$MEM_GREP" -E "^## $3-[0-9]" "$2")" ] \
|| { echo "memory: upgrade changed a heading: $1" >&2; return 2; }
memory_check_file "$2" "$3" || return 2
[ "$(_mem_heading_ids "$1" "$3")" = "$(_mem_heading_ids "$2" "$3")" ] \
|| { echo "memory: upgrade changed the entry IDs: $1" >&2; return 2; }
memory_check_file "$2" "$3" "" "$1" || return 2
o=$(_mem_index_rows "$1" "$3"); n=$(_mem_index_rows "$2" "$3")
awk -F'\t' -v p="$3" -v MEMW="$_MEM_WORDS" "$_MEM_AWK"'
function f(s, k) { split(s, a, / \| /); return a[k] }
@@ -584,39 +648,78 @@ _mem_upgrade_oracle() { # <old> <new> <PREFIX> <rows> → rc 2 on a lost fact
m = mem_want(p, col[id], old[id])
want = m != "" ? mem_word(m) : old[id]
if (dt[id] != "" && d == "") bad = bad " " id ":date"
if (want != "" && s != want) bad = bad " " id ":" s "!=" want }
if (want != "" && p != "BLK" && s != want)
bad = bad " " id ":" s "!=" want }
END { if (bad != "") { print "memory: upgrade lost:" bad; exit 2 } }' \
"$4" <(printf '%s\n' "$o") <(printf '%s\n' "$n") >&2
<(_mem_index_table "$1") <(printf '%s\n' "$o") \
<(printf '%s\n' "$n") >&2 || return 2
o=$(_mem_verdict_diff "$1" "$2" | awk -F'\t' '$2 == "resolved" \
{ printf "%s ", $1 }')
[ -z "$o" ] || { echo "memory: upgrade moved a verdict: $o" >&2; return 2; }
}
_mem_upgrade_front() { # <file> → the file; one lacking a frontmatter gets
local t first # the template's (format: 2 is added with the rest)
IFS= read -r first < "$1"
if [ "$first" != --- ]; then
t="$(dirname "${BASH_SOURCE[0]}")/../templates/memory/${1##*/}"
[ -f "$t" ] || { echo "memory: no template for $1" >&2; return 2; }
awk 'NR > 1 && /^---$/ { print; print ""; exit }
$0 != "format: 2"' "$t"
fi
cat "$1"
}
_mem_upgrade_headings() { # <file> <PREFIX> → entry headings in the v2 shape
# `## ID — title` stays byte-identical; ` - ` and ` | date | ` separators
# and any other level are rewritten, the title text and the ID kept; a
# heading that would still not be v2 (`### ID amendment`) is left as is
awk -v p="$2" '
/^```/ { fence = !fence }
fence || !match($0, "^#+ *" p "-[0-9]+") { print; next }
{ lead = substr($0, 1, RLENGTH); rest = substr($0, RLENGTH + 1)
if (lead ~ /^## / && rest ~ /^ — ./) { print; next }
sub(/^#+ */, "", lead); d = ""
if (rest ~ /^ [|] /) { rest = substr(rest, 4)
if (match(rest, "^[0-9][0-9/-]* [|] ")) {
d = substr(rest, 1, RLENGTH - 3); rest = substr(rest, RLENGTH + 1) }
rest = " — " rest (d != "" ? " (" d ")" : "") }
else if (rest ~ /^ - /) rest = " — " substr(rest, 4)
cand = "## " lead rest
print (cand ~ "^## " p "-[0-9][0-9][0-9]+ — ." ? cand : $0) }' "$1"
}
_mem_upgrade_build() { # <file> <PREFIX> <out> → the v2 text, blocker
local front src rows rc=0 # verdicts kept
front=$(mktemp); src=$(mktemp); rows=$(mktemp)
[ -n "$front" ] && [ -n "$src" ] && [ -n "$rows" ] \
|| { rm -f "$front" "$src" "$rows"; echo "memory: no temp file" >&2
return 2; }
{ _mem_upgrade_front "$1" > "$front" \
&& _mem_upgrade_headings "$front" "$2" > "$src" \
&& _mem_index_table "$src" > "$rows" \
&& _mem_upgrade_text "$rows" "$src" "$2" > "$3" \
&& _mem_upgrade_repair "$1" "$3" "$2"; } || rc=2
rm -f "$front" "$src" "$rows"
return "$rc"
}
_mem_upgrade_ready() { # <file> → rc 0 upgradable, 1 already v2, 2 refused
local first p
IFS= read -r first < "$1"
[ "$first" = --- ] \
|| { echo "memory: $1: no frontmatter, not upgraded" >&2; return 2; }
memory_is_v2 "$1" && { echo "memory: $1 already v2" >&2; return 1; }
p=$(_mem_prefix "$1") || return 2
awk -v p="$p" '/^```/ { fence = !fence; next }
!fence && match($0, "^## " p "-[0-9]+") \
&& substr($0, RLENGTH + 1) !~ /^( — |$)/ {
print "memory: " FILENAME ": heading dialect \"" \
substr($0, RLENGTH + 1, 3) "\": not upgraded"; bad = 1; exit }
END { exit bad ? 2 : 0 }' "$1" >&2
_mem_prefix "$1" > /dev/null
}
_mem_upgrade_file() { # <file> → rc 0 upgraded or already v2, rc 2 refused
local p rows new rc=0
local p new rc=0
_mem_upgrade_ready "$1"; rc=$?
[ "$rc" = 1 ] && return 0
[ "$rc" = 0 ] || return 2
p=$(_mem_prefix "$1") || return 2
rows=$(mktemp) || { echo "memory: no temp file" >&2; return 2; }
new=$(mktemp) || { rm -f "$rows"; echo "memory: no temp file" >&2; return 2; }
_mem_index_table "$1" > "$rows"
{ _mem_upgrade_text "$rows" "$1" "$p" > "$new" \
&& _mem_upgrade_oracle "$1" "$new" "$p" "$rows" \
new=$(mktemp) || { echo "memory: no temp file" >&2; return 2; }
{ _mem_upgrade_build "$1" "$p" "$new" \
&& _mem_upgrade_oracle "$1" "$new" "$p" \
&& cat "$new" > "$1"; } || rc=2
rm -f "$rows" "$new"
rm -f "$new"
[ "$rc" = 0 ] && echo "memory: upgraded $1"
[ "$rc" = 0 ] || echo "memory: $1 left as it was" >&2
return "$rc"
+2 -7
View File
@@ -63,16 +63,11 @@ reconcile_oracle_path_present() { [ -e "${1:?}" ]; } # $1 path → rc 0 if it
# $1 blockers file, $2 id. A v2 registry (lib/memory.sh): the entry's own
# header `[tag]` or a labelled body line, never a status word in prose or in
# the title (the BLK-032..034 false-open). A v1 registry keeps the lexical
# reader below: its dialects (title suffix `— RESOLVED`, `date: … · status:`,
# reader (memory_legacy_status): its dialects (title suffix `— RESOLVED`, `date: … · status:`,
# `**Statut** :`) carry status where the strict parser cannot see it.
reconcile_blk_current_status() {
if memory_is_v2 "$1"; then memory_entry_status "$1" "$2"; return; fi
# v1: drop ALL `## BLK-` header lines first: the range is inclusive of the
# NEXT entry's header, and a sibling header may carry a status word.
sed -n "/^## ${2} /,/^## BLK-/p" "$1" \
| "$RECONCILE_GREP" -v '^## BLK-' \
| "$RECONCILE_GREP" -iE 'status|RESOLVED|REVERTED|upstream|resolved|[^a-z]open' \
| tail -1
memory_legacy_status "$1" "$2"
}
# blockers whose CURRENT status does not close them → "id<TAB>status".
# v2: memory_closed (resolved | superseded | deprecated); v1: the substring.
+68 -4
View File
@@ -282,12 +282,12 @@ check T9j-oracle-refuses "$?" 2
check T9k-refused-untouched "$(cksum < .claude/memory/blockers.md)" "$sum"
printf '# Evals\n\n## EVAL-001 | 2026 | pipe dialect\n' > .claude/memory/evals.md
bash "$MEM" upgrade >/dev/null 2>&1
check T9l-no-frontmatter-rc "$?" 2
check T9m-dialect-untouched "$(sed -n 1p .claude/memory/evals.md)" '# Evals'
check T9l-no-frontmatter-v2 "$(sed -n 2p .claude/memory/evals.md)" 'format: 2'
check T9m-pipe-heading "$(grep -c '^## EVAL-001 — pipe dialect (2026)$' .claude/memory/evals.md)" 1
printf -- '---\ntype: evals_registry\nentry_prefix: EVAL\n---\n\n## Index\n\n| ID | Date | Output | Action |\n|----|------|--------|--------|\n\n## EVAL-001 - hyphen dialect\nx\n' \
> .claude/memory/evals.md
err=$(bash "$MEM" upgrade 2>&1 >/dev/null)
check T9o-hyphen-refused "$(printf '%s\n' "$err" | grep -c 'heading dialect " - ": not upgraded')" 1
bash "$MEM" upgrade >/dev/null 2>&1
check T9o-hyphen-upgraded "$(grep -c '^## EVAL-001 — hyphen dialect$' .claude/memory/evals.md)" 1
printf -- '---\ntype: evals_registry\nentry_prefix: EVAL\n---\n\n## Index\n\n| ID | Date | Output | Action |\n|----|------|--------|--------|\n\n## EVAL-001 — empty index (2026-04-01)\nx\n' \
> .claude/memory/evals.md
bash "$MEM" upgrade >/dev/null 2>&1
@@ -378,6 +378,70 @@ check T14n-staged-absolute "$?" 2
bash "$MEM" status BLK-404 'open' 2>&1 >/dev/null | grep -c restored > rs
check T14o-no-restore-noise "$(tr -d ' ' < rs)" 0
# T15 upgrade migrates every measured v1 dialect; the oracle refuses what it
# cannot prove. Blockers fixtures: the open set (reconcile's reader) before
# and after must be the same, the check green, a second run a no-op
v1blk() { # <dir> <body…> → <dir>/blockers.md with a v1 frontmatter
mkdir -p "$1"; { printf -- '---\ntype: blockers_registry\n'
printf 'entry_prefix: BLK\n---\n\n# Blockers\n'; shift; printf '%s\n' "$@"
} > "$(dirname "$1/x")/blockers.md"; }
openset() { bash -c '. "$1/lib/reconcile.sh"; reconcile_blk_open "$2"' _ \
"$ROOT" "$1" | cut -f1 | tr '\n' ' '; }
dialect() { # <name> <heading regex expected> — fixture already in d15/<name>
local f="d15/$1/blockers.md" before
before=$(openset "$f")
bash "$MEM" upgrade "d15/$1" >/dev/null 2>&1
check "T15-$1-rc" "$?" 0
check "T15-$1-v2" "$(sed -n 2p "$f")" 'format: 2'
check "T15-$1-heading" "$(grep -c "$2" "$f")" 1
check "T15-$1-check" "$(bash "$MEM" check "d15/$1" 2>&1; echo $?)" 0
check "T15-$1-verdicts" "$(openset "$f")" "$before"
sum=$(cksum < "$f"); bash "$MEM" upgrade "d15/$1" >/dev/null 2>&1
check "T15-$1-idempotent" "$(cksum < "$f")" "$sum"; }
v1blk d15/hyphen '## BLK-001 - hyphen, resolved' 'Status: resolved' '' \
'## BLK-002 - hyphen, open' 'Status: open'
dialect hyphen '^## BLK-001 — hyphen, resolved$'
v1blk d15/pipe '## BLK-001 | 2026-05-18 | pipe, resolved' 'Status: resolved' '' \
'## BLK-002 | 2026-05-19 | pipe, open' 'Status: open'
dialect pipe '^## BLK-001 — pipe, resolved (2026-05-18)$'
v1blk d15/hashes '### BLK-001 — three hashes, resolved' 'Status: resolved' '' \
'### BLK-002 — three hashes, open' 'Status: open'
dialect hashes '^## BLK-001 — three hashes, resolved$'
v1blk d15/suffix '## BLK-001 — suffix — RESOLVED' 'Status: resolved in repo' \
'' '## BLK-002 — suffix, open' 'Status: open'
dialect suffix '^## BLK-001 — suffix — RESOLVED$'
v1blk d15/datestatus '## BLK-001 — one line' \
'date: 2026-06-02 · status: RESOLVED 2026-06-04' '' \
'## BLK-002 — one line open' 'date: 2026-06-02 · status: open'
dialect datestatus '^- \*\*Status ([0-9-]*, upgrade)\*\*: resolved$'
check T15-datestatus-one-line "$(grep -c 'upgrade)' d15/datestatus/blockers.md)" 1
v1blk d15/statut '## BLK-001 — fr' '- **Statut** : resolved' '' \
'## BLK-002 — fr open' '- **Statut** : open'
dialect statut '^- \*\*Status ([0-9-]*, upgrade)\*\*: resolved$'
mkdir -p d15/nofm
printf '# Blockers\n\n## BLK-001 | 2026-05-18 | no frontmatter\n%s\n' \
'- **Statut** : resolved' > d15/nofm/blockers.md
dialect nofm '^## BLK-001 — no frontmatter (2026-05-18)$'
check T15-nofm-template "$(sed -n 1,3p d15/nofm/blockers.md | tr '\n' '#')" \
'---#format: 2#type: blockers_registry#'
check T15-nofm-body "$(grep -c '^# Blockers$' d15/nofm/blockers.md)" 1
# an em-dash heading stays byte-identical, a dated one included
v1blk d15/keep '## BLK-001 — keep — 2026-05-18' 'Status: open'
bash "$MEM" upgrade d15/keep >/dev/null 2>&1
check T15-keep-heading "$(grep -c '^## BLK-001 — keep — 2026-05-18$' d15/keep/blockers.md)" 1
# v1 open, strict reader closed: kept and named (BLK-032..034)
v1blk d15/fix '## BLK-001 — prose' 'Status: Resolved'
err=$(bash "$MEM" upgrade d15/fix 2>&1 >/dev/null)
check T15-fix-named "$(printf '%s\n' "$err" | grep -c 'BLK-001: v1 reader open, strict reader closed')" 1
# tampered: no status line lands, the oracle refuses, the file is untouched
v1blk d15/tamper '## BLK-001 | 2026-05-18 | lost closure' \
'date: 2026-06-02 · status: RESOLVED'
sum=$(cksum < d15/tamper/blockers.md)
err=$( ( . "$MEM"; _mem_upgrade_repair() { :; }; memory_upgrade d15/tamper ) 2>&1 >/dev/null )
check T15-tamper-rc "$?" 2
check T15-tamper-culprit "$(printf '%s\n' "$err" | grep -c 'moved a verdict: BLK-001')" 1
check T15-tamper-untouched "$(cksum < d15/tamper/blockers.md)" "$sum"
# T12 this repo's registries pass the check
check T12-live-check "$(bash "$MEM" -C "$ROOT" check >/dev/null 2>&1; echo $?)" 0