# AI Insights for Humans

A working notebook of practical AI interpretability, prompting, reliability, and tooling notes for people who want evidence instead of hype.

Purpose: Practical AI insight notebook for human operators.
Entries: 45
Tags: classroom, dictionary, evals, governance, integration, interpretability, notes, prompting, reliability, tooling, verification

Public surfaces:
- HTML notebook: https://tips.platphormnews.com/
- Classroom payload: https://tips.platphormnews.com/api/v1/classroom
- Dictionary/Notes integrations: https://tips.platphormnews.com/api/v1/integrations
- Platform literacy tips: https://tips.platphormnews.com/api/v1/platform-literacy
- RSS: https://tips.platphormnews.com/rss.xml
- API docs: https://tips.platphormnews.com/api/docs
- OpenAPI: https://tips.platphormnews.com/openapi.yaml
- MCP: https://tips.platphormnews.com/api/mcp

Protected actions require PLATPHORM_API_KEY.

## Route Standard
Implemented required routes: 20/20
- GET /: implemented; Public AI insights notebook and reading paths.
- GET /api/health: implemented; Platform health payload for the publication.
- GET /api/v1/health: implemented; Versioned platform health payload.
- GET /api/docs: implemented; Machine-readable API and route documentation.
- GET /api/v1/entries: implemented; Read-only JSON index of notebook entries.
- GET /api/v1/entries/{id}: implemented; Read-only JSON detail for one notebook entry.
- GET /api/v1/classroom: implemented; Read-only classroom slides, teaching loop, and enhancement plan payload.
- GET /api/v1/integrations: implemented; Read-only Dictionary and Notes integration map with public handoff boundaries.
- GET /api/v1/platform-literacy: implemented; Read-only interactive tips distilled from PlatPhormNews root discovery and policy surfaces.
- POST /api/v1/reports: implemented; Protected generation of a real notebook/platform summary report.
- GET /api/mcp: implemented; MCP metadata and JSON-RPC usage information.
- POST /api/mcp: implemented; JSON-RPC 2.0 MCP endpoint for real notebook tools, resources, and prompts.
- GET /openapi.yaml: implemented; OpenAPI 3.1 contract for public and protected routes.
- GET /llms.txt: implemented; Concise LLM-readable site summary.
- GET /llms-full.txt: implemented; Complete LLM-readable notebook and platform summary.
- GET /llms-index.json: implemented; Structured LLM index generated from real entries.
- GET /robots.txt: implemented; Crawler policy and sitemap pointer.
- GET /sitemap.xml: implemented; Canonical URL sitemap for the notebook.
- GET /sitemap-index.xml: implemented; Sitemap index for platform crawlers.
- GET /rss.xml: implemented; RSS feed generated from notebook entries.
- GET /feed.xml: implemented; RSS-compatible feed alias.
- GET /atom.xml: implemented; Atom feed generated from notebook entries.
- GET /manifest.webmanifest: implemented; Web app manifest for install and preview surfaces.
- GET /.well-known/mcp.json: implemented; MCP capability discovery manifest.
- GET /.well-known/agents.json: implemented; Agent-facing route and policy discovery manifest.
- GET /.well-known/security.txt: implemented; Security contact and policy file.
- GET /.well-known/trust.json: implemented; Trust, auth, public/protected, and exposure policy.

## Trust Policy
Web dashboard, public-safe discovery, browser-based operations, trusted-domain discovery, standard route compliance, Vercel metadata capture, trace inspection, and agentic workflow discovery are intentionally supported for public read-only debugging and operator workflows. Mutating, administrative, ingestion, replay, fork, remediation, deployment, sync, test-triggering, reporting, and write actions require PLATPHORM_API_KEY.

## Field Guide Playbooks
### Hallucination triage
A model produced a confident answer and nobody knows whether the weak point is retrieval, reasoning, or source support.
Outcome: A reviewer can identify which claims are supported, which claims need receipts, and which answer parts should be withheld.
1. Locate fragile spans: Hallucinations often hide where fluency is highest but token certainty is thinnest.
2. Separate retrieval from generation: Bad context can make a good model sound wrong with confidence.
3. Attach receipts: The unsupported claim does not vanish; it becomes visible enough to reject.
4. Decide the confidence bucket: Confidence matters when it changes the next action.

### Prompt change control
A system prompt needs improvement, but every wording edit risks changing refusal, format, or evidence behavior.
Outcome: The prompt change ships with a semantic diff, targeted fixtures, and a clear before/during/after eval record.
1. Shrink the contract: A smaller system prompt is not weaker; it is easier to audit.
2. Diff the meaning: A one-word modal change can quietly become a policy change.
3. Run the eval sandwich: The middle eval catches drift while the edit is still cheap.
4. Mutate the boundary: A boundary that only catches the obvious phrasing is not a boundary.

### Agent action gate
An agent can call tools, but the operator needs confidence before it mutates, publishes, replays, deletes, or spends.
Outcome: Every action moves through preview, inspect, authorize, execute, and verify without hiding protected states.
1. Preview the action: Permission to plan is not permission to execute; the preview is where unsafe scope becomes inspectable.
2. Respect blocked states: A designed stop is better than a graceful guess because it preserves trust and gives the operator a next move.
3. Spend checks by risk: Latency can be evidence-driven instead of uniformly expensive.
4. Verify after execution: The world, not the model, gets the final vote, so post-action evidence is part of the action itself.

## Failure Case Studies
### The source-shaped hallucination
Setup: A model summarizes a report and includes a link beside every claim.
Failure: The links exist, but two citations point to documents that discuss the topic without supporting the exact sentence.
Hidden signal: The unsupported claims are entity/date claims with weak token margins and broad source references.
Repair: Use receipts per claim, then triangulate the high-impact facts with independent sources before publishing.
Aha: A citation beside a sentence is not the same thing as support under the sentence.

### The buried constraint
Setup: A long prompt includes a critical policy exception in the middle of retrieved context.
Failure: The answer follows the opening task and closing format while ignoring the exception.
Hidden signal: The missed fact was technically present but surrounded by lower-value context.
Repair: Move the task contract to the opening, repeat the exception at the end, and rank retrieved chunks before generation.
Aha: Presence in context is not the same thing as usable salience.

### The tool call that did too much
Setup: An agent is asked to clean a set of records and calls a bulk mutation tool.
Failure: The selector is broader than the user intended and changes records outside the target set.
Hidden signal: The model never previewed target ids or stated a rollback condition.
Repair: Add a dry-run gate, require explicit target ids, and make protected-required a first-class state.
Aha: Tool safety starts before execution, not after the audit log catches damage.

### The helpful prompt edit that moved the boundary
Setup: A prompt is edited to make answers friendlier and less formal.
Failure: The model starts complying with requests it previously redirected because the new examples imply looser policy.
Hidden signal: The text diff is small, but the semantic diff changes tone, authority, and refusal posture.
Repair: Run a semantic diff, then use refusal fixtures and harmless mutants before shipping.
Aha: Style examples can become behavioral instructions if you do not test the boundary.

## AHA Ladder
### From answer to signal
Before: Read the model's sentence and judge whether it sounds plausible.
After: Inspect token margins, retrieved evidence, receipts, and contradiction before judging.
Operator question: Where would this answer break first?

### From prompt to contract
Before: Write instructions until the behavior feels described.
After: Define the output schema, failure states, hard boundary, and test fixtures.
Operator question: What is the model allowed to leave blank?

### From tool to action
Before: Let the agent call a tool when it appears to understand the task.
After: Preview arguments, inspect target ids, authorize mutation, execute, then verify changed state.
Operator question: What would stop this action before it runs?

### From confidence to decision
Before: Ask the model for a percentage and decide whether it feels high enough.
After: Bucket confidence by evidence quality, conflict level, and downstream impact.
Operator question: What does this confidence label change?

## Classroom Mode
### Teach the term before the trick
Takeaway: Students cannot inspect model behavior if the key vocabulary is unstable.
Dictionary term: retrieval relevance
Notes prompt: Which word in today's demo felt familiar but turned out to need a stricter definition?
Aha: The glossary is not prep material. It is part of the operating system for the lesson.

### Put an artifact on screen
Takeaway: The room needs something concrete to inspect: an answer, citation, tool preview, or note.
Dictionary term: receipt
Notes prompt: What exact artifact made you change your mind about the model output?
Aha: Evidence has to be visible before students can practice judgment.

### Preview action before authority
Takeaway: A model that can mutate state needs a dry run, target ids, stop conditions, and verification.
Dictionary term: dry run
Notes prompt: What would stop this tool call before it touches real state?
Aha: Permission to explain a tool call is not permission to execute it.

### Make stopping a valid answer
Takeaway: Unknown, partial, protected-required, and needs-review states keep bad certainty from becoming product behavior.
Dictionary term: abstention
Notes prompt: Where did the model need an honest stop state instead of a smoother sentence?
Aha: A graceful stop can be the most helpful output in the room.

### Turn confusion into curriculum
Takeaway: Repeated learner confusion is a content signal, not an interruption.
Dictionary term: provenance
Notes prompt: What phrase should become a shared glossary candidate after this lesson?
Aha: The class is generating the next version of the curriculum while it learns.

### End with recall, artifact, transfer
Takeaway: A student understands the insight when they can name it, use it on an artifact, and transfer it.
Dictionary term: prompt fixture
Notes prompt: Write one transfer question that would catch shallow memorization.
Aha: The final question should prove the habit moved beyond the slide.

## Dictionary and Notes Integrations
### PlatPhorm Dictionary
Public terminology, definitions, provenance lookup, search, datasets, term graph, and MCP discovery.
Public reads: Approved public terms, Dictionary search, OpenAPI and llms discovery, Term graph and provenance lookup, Read-only MCP introspection
Protected boundary: Moderation, import, admin, private submissions, audit logs, protected reports, model-assisted moderation, registry mutation, and destructive actions require PLATPHORM_API_KEY when enforcement is enabled.
Notebook use: This notebook links concepts to Dictionary as concept receipts and glossary anchors; it does not publish definitions on the user's behalf.

### PlatPhorm Notes
Public learning notepad for note enrichment, terminology detection, Dictionary lookup, and human-approved definition staging.
Public reads: App shell and discovery files, Public templates, Term detection previews, Note analysis previews, Workflow templates and integration status
Protected boundary: No visitor key entry. Backend calls to Dictionary, Docs, OpenContent, Sheets, Phorm, SVG, and other PlatPhorm services require PLATPHORM_API_KEY server-side.
Notebook use: This notebook gives students prompts and observation formats to use in Notes; it does not store note bodies or stage submissions directly.

## PlatPhormNews Root Interactive Tips
### Public DMZ means safe read-only front door
Surface: Root MCP manifest and trust policy
Source: https://platphormnews.com/.well-known/mcp.json
Beginner: Think of the public DMZ like the front desk of a school. Anyone can ask where things are, but they cannot change grades, unlock doors, or edit records.
Operator: The root site publishes public discovery, network graph, stories, route compliance, and MCP introspection while reserving sync, validation, reports, and policy updates for protected actions.
Ethical AI operator lens: Your job is to prove the front desk gives useful directions without accidentally becoming an admin panel.
Aha: Public does not mean powerful. Public means safely inspectable.
Safe boundary: Do not test destructive, private, protected, credential, admin, or mutation paths without explicit authorization.

### The network graph is the map of the city
Surface: GET /api/network/graph
Source: https://platphormnews.com/api/network/graph
Beginner: A website network is like a city. The graph tells you what buildings exist and how roads connect them.
Operator: The root graph reports sites, nodes, edges, sources, categories, route compliance, discovery compliance, and trusted domain status.
Ethical AI operator lens: A graph helps you check coverage: which sites are trusted, which surfaces are discoverable, and which claims need source-backed evidence.
Aha: Discovery is not guessing URLs. Discovery is reading the map the platform publishes.
Safe boundary: Stay inside trusted domains and published discovery routes. Do not crawl arbitrary user-provided URLs.

### robots.txt and llms files teach crawler manners
Surface: robots.txt, llms.txt, llms-full.txt, llms-index.json
Source: https://platphormnews.com/robots.txt
Beginner: robots.txt is a hallway sign for crawlers. llms files are a study guide for AI systems.
Operator: The root robots policy allows public discovery files and graph routes for LLM crawlers while blocking admin, private, protected, auth, cron, webhook, and secret-query paths.
Ethical AI operator lens: Good automation respects the published crawl lanes. Ignoring them is not clever; it is unsafe and low-signal.
Aha: The safest crawler is the one that reads instructions before touching the site.
Safe boundary: Never put tokens, keys, or secrets into URLs, traces, screenshots, feeds, or public artifacts.

### MCP is a toolbox, not a permission slip
Surface: .well-known/mcp.json and /api/mcp
Source: https://platphormnews.com/.well-known/mcp.json
Beginner: MCP lists tools an AI client can ask for, like a toolbox label. The label does not mean every tool is safe for every person to use.
Operator: The root manifest exposes JSON-RPC methods, public tools, protected tools, resources, prompts, trace headers, and the PLATPHORM_API_KEY boundary.
Ethical AI operator lens: Read tools/list and resources/list first. Confirm whether a tool is public read-only or protected before designing any workflow.
Aha: Tool discovery comes before tool execution.
Safe boundary: Do not call sync, refresh, ingestion, reporting, or policy-update tools unless authorized and key-gated.

### OpenAPI is the menu with ingredients listed
Surface: /api/docs and /openapi.yaml
Source: https://platphormnews.com/api/docs
Beginner: An API doc is a menu. It tells you what you can ask for, what parameters it accepts, and what shape the answer should have.
Operator: The root API documentation covers stories, search, sources, network APIs, network MCP, evidence, trustops, searchops, agentops, public contract, feeds, and discovery routes.
Ethical AI operator lens: A contract lets you test behavior without guessing. You can verify route shape, auth requirements, and public-safe responses.
Aha: Guessing endpoint behavior is amateur hour; reading the contract is how experts move fast safely.
Safe boundary: Use read-only endpoints for practice. Treat POST, sync, ingestion, and reporting routes as protected unless explicitly documented otherwise.

### JSON-LD is the site's ID card
Surface: Homepage JSON-LD
Source: https://platphormnews.com
Beginner: JSON-LD is structured data that tells search engines and agents who the site is, what it publishes, and what actions it supports.
Operator: The root homepage publishes organization, website search action, navigation, and FAQ structured data for public identity and discoverability.
Ethical AI operator lens: Structured data is a claim surface. Compare it with live routes and policies to catch stale identity, stale navigation, or unsafe promises.
Aha: A page can speak to humans in HTML and to machines in JSON-LD at the same time.
Safe boundary: Do not treat structured data as secret or authority by itself; verify with the corresponding public route.

### RSS tells what changed; sitemaps tell what exists
Surface: RSS, Atom, sitemap, sitemap index
Source: https://platphormnews.com/rss.xml
Beginner: RSS is the update stream. A sitemap is the table of contents. Together they help people and machines find new and existing pages.
Operator: The root publishes RSS/feed/Atom and several sitemap variants, including network and full sitemap surfaces, with robots pointers.
Ethical AI operator lens: Feeds and sitemaps are low-risk coverage tools. They reveal intended public pages without probing private paths.
Aha: The public site tells you where to look if you start with the sitemap instead of guessing.
Safe boundary: Use published sitemaps instead of brute-force path discovery.

### Agent policy tells bots how to behave
Surface: .well-known/agent-policy.json
Source: https://platphormnews.com/.well-known/agent-policy.json
Beginner: Agent policy is the classroom rules poster for AI clients: what they can read, what needs a key, and what data must stay private.
Operator: The root policy names public access surfaces, supported client classes, PLATPHORM_API_KEY requirements, trace propagation, JA4 redaction, and no-private-data rules.
Ethical AI operator lens: Before running any automated inspection, prove your agent obeys the policy and can stop at protected boundaries.
Aha: A well-behaved agent is safer because it knows where not to go.
Safe boundary: No secrets, cookies, raw IP addresses, raw JA4 digests, private traces, or private audit details in public reports.

### Trace headers are the breadcrumb trail
Surface: MCP manifest and agent policy trace fields
Source: https://platphormnews.com/.well-known/agent-policy.json
Beginner: Trace headers are like numbered sticky notes that help you follow one request as it moves between services.
Operator: Root policy accepts traceparent, tracestate, X-PlatPhorm-Trace-Id, and propagates safe PlatPhorm trace/request/source/target headers.
Ethical AI operator lens: Traceability helps debug cross-site behavior, but secret baggage must never ride along.
Aha: Observability is useful only when it does not leak the thing it observes.
Safe boundary: Never put authorization, API keys, cookies, session tokens, or raw private payloads into trace state or baggage.

### Status should say how it knows
Surface: Network graph and compliance payloads
Source: https://platphormnews.com/api/network/graph
Beginner: A status card is better when it tells you where the number came from, not just whether it is green.
Operator: Root discovery surfaces label sources such as static manifest, MCP registry, sitemap index, declared route spec, stale markers, confidence, and checkedAt timestamps.
Ethical AI operator lens: Source labels let you separate current evidence from stale, degraded, inferred, or declared-only status.
Aha: Green without evidence is decoration. Green with source and time is operational signal.
Safe boundary: Do not convert degraded or declared-only status into a claim of live success.

## Skill Ramp
### First-time learner
Learner can: Explain the difference between public read, protected action, and private data.
Practice: Sort routes from robots.txt into safe-to-read and stop-sign groups.

### Careful operator
Learner can: Use sitemaps, RSS, llms files, OpenAPI, and JSON-LD to inspect a site without guessing.
Practice: Build a source-backed checklist for one public surface.

### Agent workflow designer
Learner can: Read MCP tools/resources/prompts and design a public-safe workflow with explicit stop states.
Practice: Write a dry-run plan for a read-only MCP inspection.

### Ethical AI security operator
Learner can: Audit public automation behavior while respecting trust policy, robots policy, auth boundaries, trace redaction, and no-secret rules.
Practice: Produce a report that says what was inspected, what was skipped, why it was skipped, and what evidence supports each conclusion.

## Full Scope Enhancement Plan
### Phase 1: Content depth and curriculum structure
Status: implemented
Outcome: The site becomes a richer teaching notebook with entries, playbooks, classroom slides, AHA checkpoints, and self-quiz material.
Scope: Add classroom, Dictionary, Notes, integration, and governance entries.; Expand reading paths around classroom teaching and cross-site workflows.; Expose classroom slides through API, MCP, and llms surfaces.
Tests: Entry inventory and insight coverage tests.; Reading path reference tests.; Classroom content depth tests.

### Phase 1: Dictionary and Notes public handoffs
Status: implemented
Outcome: The notebook links to real Dictionary and Notes public surfaces while clearly preserving protected boundaries.
Scope: Add integration cards with purpose, public reads, protected boundary, and handoff links.; Add Dictionary term bridges and Notes workflow bridges.; Publish integration payloads through API docs, llms index, MCP, and protected report output.
Tests: Integration payload unit tests.; E2E checks for Dictionary and Notes links.; No banned platform key names in public surfaces.

### Phase 1: Full-screen classroom mode
Status: implemented
Outcome: Teachers can use the notebook as a focused, full-screen teaching surface without losing links to entries, Dictionary, or Notes.
Scope: Add a classroom mode client component with slide controls.; Support browser fullscreen when available and a full-viewport overlay fallback.; Keep slide text compact, readable, and linked to real content.
Tests: Playwright classroom overlay test.; Responsive smoke check on desktop and mobile viewport sizes.; Keyboard/next/back navigation check.

### Phase 1: Complete local test suite
Status: implemented
Outcome: Unit, platform contract, API, discovery, and browser e2e checks run from repo scripts.
Scope: Add Playwright configuration and e2e scripts.; Test homepage, classroom mode, integrations, protected report rejection, and discovery parity.; Keep existing TypeScript, lint, unit, and build gates.
Tests: pnpm test; pnpm test:e2e; pnpm build

### Next: Live alias and network verification
Status: next
Outcome: The deployed https://tips.platphormnews.com alias is verified against BrowserOps/Evals style route, discovery, and UI checks.
Scope: Run live read-only route standard checks against tips.platphormnews.com.; Compare llms, OpenAPI, RSS, sitemap, MCP, and API docs with local output.; Capture screenshots for classroom mode on desktop and mobile.
Tests: RUN_LIVE_INTEGRATION_TESTS=true read-only smoke.; BrowserOps journey after production deploy.; No secret leakage and protected action rejection checks.

### Future: Protected Dictionary and Notes workflow bridge
Status: future
Outcome: If needed later, this site can stage protected Dictionary/Notes workflow requests server-side with PLATPHORM_API_KEY.
Scope: Add protected server routes only when a real write workflow is approved.; Require PLATPHORM_API_KEY for any submission, staging, ingestion, report, or sync action.; Return protected_required and honest degraded states when the key or owning service is unavailable.
Tests: Auth rejection without key.; Protected success only with configured key.; Trace and redaction tests for cross-site requests.

## Entries
# Teach the term before the trick
Kind: Method
Tag: classroom
Published: 2026-06-30
The fastest classroom failure is showing the advanced move while students are still guessing what the nouns mean. A prompt fixture, dry run, receipt, confidence bucket, and abstention branch each need a plain-language definition before the lesson begins.
Use the Dictionary connection as a teaching move: introduce the term, point to the public definition surface, ask learners to predict the failure mode, and only then run the demonstration.
This changes the instructor's job from explaining a magic trick to naming a repeatable mechanism. Once the term is stable, students can transfer it to a new model, tool, or workflow.
## Aha Moment
The vocabulary is part of the interface. If the term is unstable, the lesson is unstable.

## Try This
Start each teaching block by naming three terms students must understand before they watch the model behavior.

## Watch For
- Students copying a workflow while using the key term differently
- Demos that rely on undefined platform language
- Definitions introduced only after confusion appears
Note: A shared glossary lowers the cost of every later explanation.

# Use Dictionary as the concept receipt
Kind: Method
Tag: dictionary
Published: 2026-06-30
Concept receipts are different from fact receipts. A fact receipt supports a claim about the world. A concept receipt stabilizes the language used to discuss the claim.
This matters in AI teaching because terms like context window, retrieval, grounding, confidence, eval, and tool call get used casually. Casual vocabulary creates accidental disagreement.
A good integration links to Dictionary for the term, links to the notebook for the operating habit, and links to Notes for the learner's private or public reflection workflow.
## Aha Moment
A Dictionary link can be a concept receipt that proves the class is sharing a meaning before debating evidence.

## Try This
Attach a Dictionary definition to each specialized term in a classroom slide, notebook entry, or review checklist.

## Watch For
- Treating a term definition as proof of a factual claim
- Definitions whose domain does not match the lesson
- Glossary links that point to search pages instead of concrete terms

# Turn confused notes into dictionary candidates
Kind: Method
Tag: notes
Published: 2026-06-29
A classroom generates terminology demand in real time. If learners keep asking what a phrase means, that phrase belongs in a glossary queue.
Notes is the capture layer: students write the confusion in their own language. Dictionary is the stabilization layer: the class turns that recurring confusion into a reviewed definition.
The important boundary is human approval. The model can suggest a term, summary, or definition draft, but the published Dictionary meaning should be reviewed before it becomes shared teaching infrastructure.
## Aha Moment
Repeated learner confusion is not noise. It is demand for a clearer shared definition.

## Try This
Collect fuzzy learner phrases in Notes, group repeated phrases, and promote only reviewed candidates to Dictionary.

## Watch For
- Publishing model-suggested definitions without human review
- Losing repeated student confusion after the class ends
- One-off answers that never improve the shared glossary

# Run the explain, inspect, repair loop
Kind: Method
Tag: classroom
Published: 2026-06-29
Explaining a concept gives students vocabulary. Inspecting an artifact gives them evidence. Repairing a failure gives them agency.
For prompt reliability, the artifact might be a bad answer. For tool safety, it might be a dry-run plan. For verification, it might be a citation that points near the claim but not under it.
The loop should end with an AHA checkpoint: what changed in how the learner sees the model, not just what command they copied.
## Aha Moment
Students remember a concept when they see the mechanism, inspect evidence, and repair a failure themselves.

## Try This
Teach one topic with three beats: explain the mechanism, inspect a concrete artifact, then repair the broken part.

## Watch For
- Slides that explain without giving students an artifact to inspect
- Live demos with no repair step
- Repairs that happen silently in the instructor's head

# Climb from word to workflow
Kind: Tip
Tag: classroom
Published: 2026-06-28
The word rung names the concept. The example rung shows it in a clean case. The failure rung shows why it matters. The workflow rung turns it into a repeatable decision.
Dictionary helps with the word rung. The notebook helps with the example and failure rungs. Notes helps students turn the workflow rung into their own operating habit.
If a lesson stops at a definition, it is reference material. If it reaches workflow, it becomes practice.
## Aha Moment
A term becomes operational only when the learner can use it to choose the next move.

## Try This
For one concept, write the word, a clean example, a failure example, and the workflow decision it changes.

## Watch For
- Definitions that never reach a practical example
- Examples that do not show the failure mode
- Students who can repeat the word but cannot act on it

# Put the glossary before the demo
Kind: Tip
Tag: dictionary
Published: 2026-06-28
Before a retrieval demo, define retrieval, relevance, chunk, contradiction, and receipt. Before a tool demo, define dry run, target id, protected action, and rollback.
The definitions do not need to be long. They need to be visible and shared.
This is where a Dictionary link does real work. It gives the class a stable reference outside the slide, the notebook entry, and the instructor's memory.
## Aha Moment
The demo is where students should watch behavior, not decode vocabulary under pressure.

## Try This
List the lesson glossary before the first live output and link each term to Dictionary when a public definition exists.

## Watch For
- Cognitive overload from new terms during the demo
- Instructor-specific jargon that has no shared reference
- Terms explained differently in slides, notes, and discussion

# Keep glossary publishing human-approved
Kind: Method
Tag: governance
Published: 2026-06-27
The risk is subtle. A bad definition does not only misstate one answer; it trains future readers to reason with the wrong frame.
A human approval queue catches invented authority, missing caveats, audience mismatch, and definitions that are technically correct but useless in practice.
The pipeline is still fast. Notes can stage the learner language, AI can draft the candidate, Dictionary can expose the reviewed term, and the notebook can link the term back into lessons.
## Aha Moment
A bad shared definition compounds because future lessons and workflows inherit it.

## Try This
Stage candidate glossary definitions as drafts, require human approval, and include source or scope notes before publishing.

## Watch For
- Model-written definitions treated as approved knowledge
- No provenance for why a definition changed
- Technically correct definitions that miss the classroom audience

# Use integration cards instead of vague platform links
Kind: Tool
Tag: integration
Published: 2026-06-27
Dictionary owns public definitions and concept lookup. Notes owns learning capture, term detection, and note analysis previews. This notebook owns lessons, entries, and teaching flows.
A good integration card states the handoff in plain language: open the definition, capture a note, preview term detection, or review a workflow template.
It should also name the boundary. If this site is not posting to Dictionary or Notes, the UI should not imply that a write happened.
## Aha Moment
A cross-site link is safer when it says what the target owns and what this page will not do.

## Try This
Write each integration card with owner, public reads, protected actions, and handoff links before adding buttons.

## Watch For
- Buttons implying a write happened when only navigation occurred
- Protected actions presented as public workflows
- Generic platform links without a product-specific handoff
Note: A link without a contract is just navigation.

# Stop for AHA checkpoints
Kind: Method
Tag: classroom
Published: 2026-06-26
AHA checkpoints are small but powerful. They ask students to name the shift: from answer to evidence, from prompt to contract, from tool to action, from confidence to decision.
The checkpoint should be concrete enough to grade informally. If students cannot say what they would inspect next, the AHA moment did not land yet.
This is why the notebook includes AHA text beside entries instead of hiding it in instructor notes.
## Aha Moment
The AHA moment is the measurable change in what the learner knows how to inspect next.

## Try This
Pause after each major concept and ask students to name the first artifact they would inspect now.

## Watch For
- Students agreeing verbally without changing their review behavior
- Checkpoints that ask for feelings instead of observable next moves
- Too many concepts between reflection points

# Use Notes as an observation log
Kind: Tip
Tag: notes
Published: 2026-06-26
Observation-first notes reduce hindsight storytelling. The learner writes the low-confidence token, unsupported claim, weird citation, or broad tool selector before explaining it away.
Those observations become better prompts for review, better term candidates for Dictionary, and better discussion material for a classroom.
The simple format is: I saw, I expected, I think it means, I need a definition for.
## Aha Moment
Observation-first notes preserve the evidence before the learner writes a story about it.

## Try This
Use the format I saw, I expected, I think it means, and I need a definition for after every demo.

## Watch For
- Conclusions recorded without the artifact that caused them
- Surprising behavior erased by a polished summary
- Term confusion captured nowhere reusable

# Check provenance before teaching a term
Kind: Method
Tag: dictionary
Published: 2026-06-25
Definitions have context. A release-engineering meaning of gate is not the same as a classroom meaning or a safety-policy meaning.
Before teaching from a term, check whether the definition names the domain, source, and intended audience. If the provenance is weak, treat it as a draft or discussion prompt.
This keeps Dictionary integration honest: it supports shared language, but it does not replace judgment.
## Aha Moment
A definition is scoped evidence. It helps only when its source and domain match the lesson.

## Try This
Before teaching a term, inspect whether the definition names the domain, source, and intended audience.

## Watch For
- One domain's meaning reused in a different workflow
- Definitions with no source or scope signal
- Treating Dictionary as a substitute for current behavior checks

# Use Socratic eval prompts in class
Kind: Tool
Tag: evals
Published: 2026-06-25
Socratic eval prompts teach students to inspect systems instead of merely reacting to outputs. The question shapes the habit.
Good classroom eval prompts are short, reusable, and attached to visible artifacts. They work on a model answer, a tool plan, a note, a definition, or a citation list.
The strongest prompt is often: what would you need to see before you let this action run?
## Aha Moment
The question teaches the habit. Ask about failure, missing receipts, and next evidence instead of asking if it is good.

## Try This
Give students four prompts: what fails first, what receipt is missing, what term is undefined, and what evidence changes this.

## Watch For
- Eval questions that invite taste instead of inspection
- Students scoring outputs without citing artifacts
- Questions that do not transfer to another workflow

# Map misconceptions as content gaps
Kind: Method
Tag: classroom
Published: 2026-06-24
A misconception map turns teaching friction into backlog. If students confuse confidence with correctness, add an entry. If they confuse retrieval with relevance, add a definition. If they over-trust citations, add a failure case.
The map should include the mistaken belief, the correction, the artifact that reveals it, and the next content improvement.
This is how a field notebook becomes a living curriculum instead of a static list of tips.
## Aha Moment
Misconceptions are content backlog with evidence attached.

## Try This
Track repeated misunderstandings as missing entries, missing definitions, missing examples, or missing exercises.

## Watch For
- Answering the same confusion repeatedly without improving content
- Misconceptions blamed on students instead of unclear material
- No owner for converting class friction into curriculum updates

# Use full-screen mode for shared attention
Kind: Tool
Tag: classroom
Published: 2026-06-24
Full-screen mode is not a marketing hero. It is a classroom tool: large type, clear current lesson, one AHA checkpoint, and one action at a time.
The best version preserves the operating surface. Students can still open the related entry, check the Dictionary term, or capture a note after the slide lands.
Use it when shared attention matters: kickoff, recap, live demo, or review.
## Aha Moment
Full-screen mode is useful when it narrows the room to one concept, one artifact, and one next action.

## Try This
Use full-screen mode for kickoff, live demo, recap, or AHA checkpoint, then return to the notebook for details.

## Watch For
- Presentation mode that hides links students need after class
- Large decorative screens with no workflow action
- Too much text on a shared teaching slide

# Watch for glossary drift
Kind: Method
Tag: governance
Published: 2026-06-23
AI vocabulary moves quickly. A definition that was clear during one model generation can become misleading after tools, providers, or platform boundaries change.
Drift review asks whether the term still matches current behavior, whether the examples still reproduce, and whether the protected/public boundary changed.
The repair is small but important: update the definition, update the entry, and update any classroom checkpoint that depends on it.
## Aha Moment
Shared vocabulary is a dependency, and dependencies drift.

## Try This
Review classroom terms after model, tool, or platform boundary changes and update Dictionary plus notebook links together.

## Watch For
- Definitions that describe a previous platform state
- Entries teaching examples that no longer reproduce
- Protected/public boundaries changing without glossary updates

# Turn entries into self-quiz cards
Kind: Tool
Tag: classroom
Published: 2026-06-23
Recall asks for the idea. Artifact asks students to use the idea on a concrete output. Transfer asks whether they can spot the same pattern in a different workflow.
For receipts: what is a receipt, which claim lacks one, and where else would you require one? For dry runs: what is the action preview, which selector is too broad, and what protected action should stop?
Self-quiz cards make the notebook useful between classes, not only during live teaching.
## Aha Moment
Learning sticks when recall, artifact inspection, and transfer all work.

## Try This
Write one recall question, one artifact question, and one transfer question for each high-value entry.

## Watch For
- Quizzes that only ask for memorized definitions
- No concrete artifact in the question
- Students passing the quiz but failing to transfer the habit

# Score retrieved context before you trust the answer
Kind: Method
Tag: verification
Published: 2026-06-28
RAG failures often look like model failures because the model speaks with the confidence of the context it was given. The first move is to separate retrieval quality from generation quality.
Give each chunk three labels: directly answers, adjacent background, or contradiction. The answer should cite directly answering chunks first and name contradictions instead of smoothing them away.
A good retrieval review is boring and mechanical. It asks whether the evidence was even in the room before debating whether the model reasoned over it correctly.
## Aha Moment
Retrieval only proves that documents were fetched. It does not prove the fetched text answers the question.

## Try This
Label each retrieved chunk as direct support, background, contradiction, or irrelevant before reading the final answer.

## Watch For
- Answers citing adjacent background as if it were direct support
- Contradictory chunks hidden by smooth synthesis
- High source count used as a substitute for relevance
Note: Retrieved is not the same thing as relevant.

# Design the output schema before the prompt
Kind: Tip
Tag: reliability
Published: 2026-06-24
A schema makes ambiguity visible. If you cannot name the field, type, and allowed absence state, you probably do not know what you are asking the model to produce.
Start with the downstream consumer. Does it need a label, a span, a confidence bucket, a source pointer, or an abstention? Write that contract first and then prompt the model to fill it.
The aha is that schemas are not just for machines. They force human reviewers to decide what counts as complete, partial, unknown, and unsafe.
## Aha Moment
A schema turns a vague prompt into an interface contract with explicit absence states.

## Try This
Write the JSON fields, allowed values, evidence pointers, and unknown states before drafting the prompt.

## Watch For
- Required fields that force the model to invent missing data
- Enums that do not include blocked, unknown, or partial
- Fields whose consumer or decision impact is unclear

# Use the eval sandwich: before, during, after
Kind: Method
Tag: evals
Published: 2026-06-20
Before-change evals establish the baseline you are trying not to lose. During-change evals are small and sharp: the five examples most likely to break because of the edit. After-change evals make sure the fix did not damage neighboring behavior.
This is especially useful for system prompt work because every instruction edit has a shadow. You improve one behavior and accidentally move refusal, tone, verbosity, or citation discipline.
Treat the eval sandwich as a change-management ritual. It is less glamorous than a benchmark, but it is the thing that keeps production behavior from drifting by inches.
## Aha Moment
The most valuable eval is often the small one you run while the prompt is still being edited.

## Try This
Run a baseline set before the change, five targeted fixtures during the change, and a regression set after.

## Watch For
- Only testing the behavior you intended to improve
- Skipping refusal and edge fixtures after style changes
- Treating one green benchmark as proof of production safety
Note: The middle eval is where most prompt regressions get caught.

# Dry-run tool calls before letting them mutate anything
Kind: Method
Tag: tooling
Published: 2026-06-14
Tool use changes the risk profile of a model. The model can be wrong in the world, not just wrong in text. The safe pattern is preview, inspect, authorize, execute, verify.
The dry run should include exactly what will be called, what identifiers will be touched, which fields will change, and what success will look like. If the model cannot state those plainly, it should not run the tool.
The best dry runs also include a refusal branch. They say what condition would stop execution and what evidence would be needed to proceed.
## Aha Moment
A tool call should become inspectable before it becomes irreversible.

## Try This
Require action preview, target ids, expected effects, refusal conditions, and rollback notes before execution.

## Watch For
- Tool arguments that include broad or ambiguous selectors
- Mutation plans without a verification step
- The model treating permission to plan as permission to execute

# Triangulate citations instead of trusting the first source
Kind: Method
Tag: verification
Published: 2026-06-08
Citation checking should ask three questions: does the source exist, does it support the exact claim, and would another independent source phrase the same claim the same way?
When sources disagree, the disagreement is the content. Do not ask the model to hide the conflict behind a confident synthesis. Ask it to name the split and explain what would settle it.
Triangulation is not needed for every sentence. Reserve it for claims that drive decisions: dates, quotes, prices, medical or legal facts, benchmarks, and allegations.
## Aha Moment
A second independent source often reveals whether a citation supports the claim or merely resembles it.

## Try This
Triangulate dates, quotes, benchmarks, allegations, and high-impact facts before presenting them as settled.

## Watch For
- Citations that support a nearby but different claim
- Two sources that repeat the same upstream source
- Disagreements collapsed into a false consensus

# Red-team prompts with harmless mutants
Kind: Method
Tag: evals
Published: 2026-05-31
Prompt boundaries are brittle when tests are too literal. A model may refuse the obvious bad phrasing and comply with the polite version, the translated version, or the version hidden inside a roleplay.
Mutants make that brittleness visible without adding new dangerous content. They perturb the wrapper, not the underlying safety category.
Use them for both sides of the boundary: prompts that should be refused and prompts that should remain allowed. Over-refusal is a product bug too.
## Aha Moment
A boundary that only holds for one phrasing is a memorized example, not a reliable rule.

## Try This
Generate harmless variants that preserve intent while changing tone, order, indirection, language, and role framing.

## Watch For
- Safety behavior that fails when the user sounds polite
- Allowed prompts that become over-refused after hardening
- Fixtures that mutate the danger instead of just the wrapper

# Diff the meaning, not just the text
Kind: Tool
Tag: prompting
Published: 2026-05-18
Text diff says what characters changed. Semantic diff says what the model is now being asked to believe, prioritize, refuse, or produce. That is the diff that matters for behavior.
Look for modal verbs, scope words, exception clauses, examples that imply new rules, and output format changes. These are the places where small edits become big runtime differences.
A useful semantic diff ends with a test plan: which fixtures should change, which must not change, and which new example proves the edit worked.
## Aha Moment
The smallest text diff can be the biggest behavioral diff when it changes authority, scope, or obligation.

## Try This
Review prompt edits for new duties, removed constraints, softened verbs, and examples that imply new policy.

## Watch For
- May, should, and must changes hidden in wording cleanup
- Examples that contradict the written rule
- Output contracts changed without matching tests
Note: A one-word modal change can be a policy change.

# Spend latency where uncertainty is highest
Kind: Tip
Tag: reliability
Published: 2026-05-09
Reliable AI systems are often cascades. Cheap checks run first; expensive checks wake up only when the situation deserves them. This keeps the product fast without pretending every answer has equal risk.
A practical budget has triggers: low logprob margin, missing receipt, protected action, high-dollar decision, policy boundary, or user-visible publication.
The aha is that latency can be evidence-driven. You do not need to choose between always fast and always careful.
## Aha Moment
Careful does not have to mean slow everywhere. Verification can wake up only where risk wakes it up.

## Try This
Escalate to extra checks when confidence is low, evidence is weak, impact is high, or an action is protected.

## Watch For
- Uniformly expensive checks on low-risk tasks
- Fast paths that ignore missing evidence
- Latency budgets without explicit escalation triggers

# Route tasks by failure mode, not brand name
Kind: Method
Tag: reliability
Published: 2026-04-26
Model routing is usually framed as cost or quality. The more useful frame is failure. Which model omits caveats, over-refuses, invents sources, follows schemas, or handles long contexts under pressure?
Build small route tests around the jobs you actually run. A model that wins general reasoning may still be the wrong choice for structured extraction if it casually fills unknown fields.
Keep routing reversible. When model behavior shifts, the route table should be an operational setting backed by evidence, not a belief system.
## Aha Moment
Routing by failure mode is more useful than routing by reputation because every model breaks differently.

## Try This
Build a route table from observed task fixtures: extraction, synthesis, coding, refusal review, and long-context use.

## Watch For
- Leaderboards used as a substitute for local fixtures
- Model routes that cannot be changed without code edits
- Ignoring behavior drift after model updates

# Treat memory as evidence with an expiration date
Kind: Tip
Tag: verification
Published: 2026-04-18
Good memory is scoped. It records what the user decided, what a repo convention is, and which risks were already found. Bad memory stores private residue, stale status, or facts that should be rechecked.
Every memory-derived claim should answer two questions: when was this learned, and how likely is it to have changed? If the answer is 'recently unstable,' verify before acting.
The aha is that memory is not context magic. It is a cache, and caches need invalidation rules.
## Aha Moment
Agent memory is a cache. Useful caches have scope, freshness, and invalidation rules.

## Try This
Store stable decisions and preferences, but re-check drift-prone facts before acting on them.

## Watch For
- Old status presented as current proof
- Private residue stored when only workflow shape was needed
- Memory outranking live repo, source, or user evidence

# Design abstention as a first-class output
Kind: Tip
Tag: reliability
Published: 2026-04-03
Abstention fails when it feels like an error. It works when it is a designed branch with a reason, missing evidence, and next action. The user should know what changed from answerable to not answerable.
Use explicit output states: answered, partial, unknown, blocked, protected-required, and needs-review. Each state should carry what evidence exists and what evidence is missing.
This is how you prevent graceful prose from hiding a dead end. The model can stop without leaving the user stuck.
## Aha Moment
A model fills blanks when the product has no respected way to stop.

## Try This
Make unknown, partial, blocked, protected-required, and needs-review states valid outputs with next actions.

## Watch For
- Schemas where every field must be filled even when evidence is absent
- Refusals that leave the user without a useful next step
- Unknown states treated as errors instead of designed outcomes

# Use confidence buckets, not fake precision
Kind: Method
Tag: verification
Published: 2026-03-21
Precise confidence numbers look scientific and often are not. Unless they are tied to a calibrated model or measured historical error, they are mostly theater.
Buckets work better in operator workflows because they can map to action: high can publish, medium can request a receipt, low can escalate, unknown can abstain.
The bucket should cite the reason: direct source found, conflicting source found, no source found, or model-only inference. That reason matters more than the label.
## Aha Moment
Confidence is useful when it changes what happens next, not when it looks mathematically precise.

## Try This
Use high, medium, low, and unknown buckets tied to evidence type, conflict level, and review action.

## Watch For
- Percentages that are not calibrated against historical outcomes
- Confidence labels without evidence reasons
- High confidence on claims with no direct receipt

# Ask for counter-evidence before confidence
Kind: Method
Tag: reliability
Published: 2026-06-18
The useful move is not asking the model to be uncertain. The useful move is forcing it to search for the best case against itself while the answer is still provisional.
A simple pattern: draft the answer, list disconfirming evidence, name what would change the answer, then assign confidence. The confidence number becomes attached to evidence quality instead of verbal smoothness.
This is especially helpful for research summaries, legal-ish interpretation, data diagnosis, and anything where one missing constraint can flip the conclusion.
## Aha Moment
The model is usually best at self-correction before it has been rewarded for sounding certain.

## Try This
Ask for the answer, the strongest objection, what evidence would change the answer, and then confidence.

## Watch For
- Objections that restate the answer instead of challenging it
- Confidence that stays high after missing evidence appears
- A final answer that ignores its own counter-evidence
Note: Confidence after counter-evidence is more useful than confidence after a sales pitch.

# Turn every important claim into a receipt
Kind: Method
Tag: verification
Published: 2026-05-27
Models are good at moving from evidence to prose. They are also good at producing prose that looks like it came from evidence. The receipt requirement separates those two behaviors.
The receipt can be a citation, a row count, a command output line, a test name, or a declared assumption. The format matters less than the discipline: every claim has to point somewhere.
When the model cannot produce a receipt, that is not failure. That is the system doing its job and showing you where judgment is still needed.
## Aha Moment
A citation is not decoration. It is a handle you can pull to see whether the sentence is load-bearing.

## Try This
Make every claim end with one receipt: source, row, command, calculation, trace id, or explicit assumption.

## Watch For
- Receipts that point to a broad document instead of the claim
- Numbers without denominators
- Source links that support nearby context but not the sentence itself

# Keep the system prompt short enough to audit
Kind: Tip
Tag: prompting
Published: 2026-05-03
System prompts work best as durable operating constraints, not as a dumping ground for every possible preference. When the prompt grows without structure, the model starts resolving conflicts in ways you cannot predict.
Split the prompt into policy, role, output contract, and refusal boundary. Then write a small regression set that exercises each section. You will catch more problems with five targeted tests than with another page of instructions.
## Aha Moment
A system prompt is product code. If it is too long to review, it is too long to trust.

## Try This
Rewrite the system prompt into role, hard boundary, output contract, and escalation rule.

## Watch For
- Repeated style instructions that fight each other
- Hidden policy changes bundled into wording edits
- Task detail that belongs in user input rather than permanent rules
Note: If you cannot diff it cleanly, you cannot debug it cleanly.

# Use refusal boundaries as test fixtures
Kind: Method
Tag: evals
Published: 2026-04-12
Most prompt tests focus on whether the model can complete the intended workflow. That misses the other half of the product: whether it handles edge requests without over-refusing, leaking private context, or inventing authority it does not have.
Build a small fixture set with three buckets: allowed, disallowed, and allowed-with-constraints. A good model response should land in the right bucket and explain the usable next step.
This turns safety from a vibe into a regression target. When a prompt edit changes the refusal boundary, you see it immediately.
## Aha Moment
Safety quality is not whether the model refuses. It is whether it refuses at the right boundary.

## Try This
Keep allowed, disallowed, and allowed-with-constraints prompts as fixtures and run them after prompt edits.

## Watch For
- Over-refusal on harmless transformed versions
- Under-refusal when the same intent is phrased politely
- Responses that refuse but fail to offer a safe next step

# Read the logprobs, not the answer
Kind: Method
Tag: interpretability
Published: 2025-11-02
Every token a model emits comes with a probability distribution over the whole vocabulary. The decoded text only shows you the winner. The logprobs show you the race.
When you pull them back, scan for the spots where the top token barely beat the runner-up. Those are the seams. Names, dates, citations, and the first token of a refusal are where you most often find a coin-flip dressed up as a confident sentence.
A practical loop: request the top 5 logprobs per token, render them as a heatmap over the output, and read the cold spots first. That is usually where the hallucination lives.
## Aha Moment
The final text is only the winning token path. The uncertainty is still visible in the runners-up.

## Try This
Render low-margin tokens first, especially names, dates, citations, and first tokens after a policy boundary.

## Watch For
- Fluent spans built from low-confidence token choices
- Nearly tied alternatives around entities
- A confident tone hiding a fragile probability margin
Note: Confidence ≠ correctness. But low confidence is a great place to look for hallucinations.

# Make it think before it answers
Kind: Tip
Tag: prompting
Published: 2025-10-28
Autoregressive models can only condition on what they have already written. If the answer comes first, the explanation is generated to fit a token that is already locked in. It is rationalization, not reasoning.
Flip the order and the reasoning tokens become real scratch space the final answer can lean on. This is the entire mechanism behind chain-of-thought, and it is why 'answer, then explain' quietly underperforms.
## Aha Moment
Explanation after an answer often explains a commitment; reasoning before an answer can still change it.

## Try This
Put scratch work, constraints, and uncertainty checks before the final answer field.

## Watch For
- Post-hoc explanations that never revise the answer
- A final answer that appears before evidence is considered
- Reasoning sections that only paraphrase the conclusion

# BertViz for watching attention move
Kind: Tool
Tag: tooling
Published: 2025-10-19
BertViz draws the attention pattern as lines connecting tokens, split by head and layer. It runs in a notebook in a couple of lines and works on most Hugging Face transformer models.
The value is not the pretty picture, it is the habit of looking. Once you have seen a head that exclusively tracks matching brackets, you stop thinking of the model as a black box and start thinking of it as a stack of small, legible mechanisms.
## Aha Moment
Attention is not the whole mind, but it is a useful map of what information is being routed where.

## Try This
Inspect heads for repeated roles: bracket matching, quote tracking, subject agreement, or copied labels.

## Watch For
- Pretty attention maps with no behavioral test
- One dramatic head treated as the whole explanation
- Patterns that disappear after small prompt perturbations
Note: Most heads are boring. The interesting 5% are worth the dig.

# Tell it what NOT to do, sparingly
Kind: Tip
Tag: prompting
Published: 2025-10-11
Putting a word in the prompt raises its activation, even when the word is wrapped in 'do not'. The model does not have a clean negation operator the way code does; it has attention, and attention does not know the difference between mention and command.
Rewrite prohibitions as positive descriptions of the target. Instead of 'do not be formal', say 'write like you are texting a friend'. Save the hard negatives for the few rails you cannot express any other way.
## Aha Moment
Negation still puts the forbidden idea into context. Attention does not forget because grammar says not to.

## Try This
Rewrite prohibitions as the positive shape of the output you want before adding hard safety rails.

## Watch For
- Forbidden words repeated in the response
- Style instructions expressed mostly as negatives
- A model fixating on the thing you tried to suppress

# Probe for a concept with a linear classifier
Kind: Method
Tag: interpretability
Published: 2025-09-30
The setup is small: collect hidden states from a chosen layer for a labeled dataset, then fit logistic regression on top. High accuracy means the concept is linearly available at that layer.
Sweep the probe across layers and you can watch where a concept first becomes readable. Abstract properties tend to sharpen in the middle layers, which is a clue about where the model is doing the relevant work.
## Aha Moment
If a tiny linear classifier can read a concept from hidden states, the model has already organized that signal.

## Try This
Probe one concept across layers and look for where accuracy first rises above a simple baseline.

## Watch For
- High probe accuracy mistaken for causal importance
- Labels that leak through surface wording
- Layer sweeps without a held-out validation split
Note: Cheap, fast, and weirdly revealing.

# Temperature is a creativity dial, not a quality dial
Kind: Tip
Tag: prompting
Published: 2025-09-22
Temperature reshapes the probability distribution before sampling. Low temperature sharpens it toward the most likely token; high temperature flattens it so unlikely tokens get a real chance.
So the dial controls variance, not intelligence. If a task has a correct answer, variance is your enemy and you want it near zero. If you want range and surprise, turn it up and sample several times.
## Aha Moment
Temperature changes variance, not intelligence. It widens the lottery; it does not improve the tickets.

## Try This
Set temperature near zero for extraction and scoring, then raise it only when variety is the product.

## Watch For
- Creative settings used for factual extraction
- One sampled answer treated as a stable result
- Quality claims that ignore variance between runs

# Sparse autoencoders to untangle neurons
Kind: Method
Tag: interpretability
Published: 2025-09-14
The trick is over-completeness plus sparsity. You map activations into a much wider space but force only a few of those dimensions to be active at once. The pressure pushes tangled concepts apart into separate, more interpretable features.
Once you have the features you can do surgery: amplify one and watch the model fixate on a bridge, suppress another and watch a behavior fade. It is the closest thing we have to named knobs inside the network.
## Aha Moment
Sparse features give you named-ish knobs because they separate mixed neuron behavior into cleaner directions.

## Try This
Treat an SAE feature as a hypothesis, then test whether steering it changes the behavior you care about.

## Watch For
- Human-readable labels that overfit a few examples
- Feature steering without downstream behavior checks
- Polysemantic residue hiding inside a cleaner-looking feature
Note: This is roughly how the 'Golden Gate Claude' demo worked.

# Keep a tokenizer open in a tab
Kind: Tool
Tag: tooling
Published: 2025-09-03
Leading spaces, casing, and digit grouping all change how text is split, and the model only ever sees the tokens, never your characters. A trailing space can silently sabotage a few-shot format.
When something is off and you cannot explain it, paste the exact string into a tokenizer view before you theorize. The answer is in there more often than you would expect.
## Aha Moment
The model never sees your characters. It sees chunks, spaces, merges, and weird number splits.

## Try This
Paste the exact prompt, including spaces and separators, into a tokenizer before debugging prompt behavior.

## Watch For
- Leading spaces changing label tokens
- Numbers split into surprising chunks
- Few-shot examples with inconsistent separators

# Sample many, keep the majority
Kind: Method
Tag: reliability
Published: 2025-08-25
Different samples take different reasoning paths but tend to converge on the same correct answer, while the wrong answers scatter. Voting over the final answers turns that asymmetry into accuracy.
It costs you N times the compute, so reserve it for the questions that matter. A useful middle ground is to escalate to self-consistency only when the first answer looks low-confidence.
## Aha Moment
Wrong reasoning paths tend to scatter; correct reasoning paths often converge.

## Try This
Sample several answers, normalize the final claim, and inspect disagreement before majority voting.

## Watch For
- Repeated wording counted as independent evidence
- Voting over outputs that all share the same missing assumption
- Compute spent on low-value questions

# Mind the middle of long contexts
Kind: Tip
Tag: prompting
Published: 2025-08-12
Retrieval accuracy as a function of position tends to be U-shaped: strong at the start, strong at the end, sagging in the middle. A fact you need can be technically present and still effectively invisible.
Design around it. Lead with the instruction, restate the critical constraint at the very end, and keep the soft middle for material you can afford to lose.
## Aha Moment
A fact can be present and still practically invisible if it sits in the soft middle of a long prompt.

## Try This
Move critical constraints to the opening contract and repeat the single highest-risk fact at the end.

## Watch For
- Middle facts ignored while opening and closing instructions are followed
- Retrieved context pasted without ranking
- Important constraints surrounded by low-value filler
Note: Sometimes called 'lost in the middle'.

# Activation patching to find the cause
Kind: Method
Tag: interpretability
Published: 2025-07-30
Probes tell you what is represented. Patching tells you what matters. By swapping one activation and watching the effect, you move from 'this information is present' to 'this component causes the behavior'.
Sweep the patch across layers and positions to localize the circuit. The spots where patching flips the answer are the load-bearing ones; everything else is along for the ride.
## Aha Moment
Correlation says a signal is present. Patching asks whether changing that signal changes the outcome.

## Try This
Patch one layer-position activation from a clean run into a corrupted run and track whether the answer flips.

## Watch For
- Large sweeps without a clean causal hypothesis
- Output flips caused by formatting rather than the target behavior
- A single patch interpreted without replication
Note: The closest thing in this field to a controlled experiment.

# Your examples teach format louder than content
Kind: Tip
Tag: prompting
Published: 2025-07-18
Few-shot learning is pattern continuation. The model is extremely sensitive to surface form, sometimes more than to the actual reasoning you intended to demonstrate.
Make every example identical in structure, pick label words that are single clean tokens, and keep lengths similar. The more rigid the template, the more reliably the model fills in the next blank.
## Aha Moment
Few-shot examples teach surface form first. The model copies the pattern before it understands your intention.

## Try This
Make labels, punctuation, casing, separators, and example length boringly consistent.

## Watch For
- One example with a different label shape
- Multi-token labels competing with single-token labels
- Examples that demonstrate style drift instead of the task

# The logit lens: decode the middle layers
Kind: Method
Tag: interpretability
Published: 2025-07-05
The output projection is just a linear map, so nothing stops you from applying it to an earlier layer. The result is a rough readout of what the model would say if it had to stop thinking right now.
Often the correct answer appears in the middle layers and the later layers spend their budget refining phrasing and suppressing alternatives. Seeing that handoff demystifies a lot of model behavior.
## Aha Moment
The answer often appears before the wording is polished. Middle layers can show the thought taking shape.

## Try This
Decode intermediate layers and watch when the target answer first becomes linearly readable.

## Watch For
- Early guesses treated as final causal proof
- Layer readouts that shift after prompt paraphrases
- Ignoring later layers that suppress unsafe or irrelevant continuations