Tools for studying how AI systems allocate attention and respond to context
Focal Prompt is an open-source research toolkit for the behavioural ecology of language models: decompose a prompt into foci, compare reported focus (model self-assessment of a completion) with perturbation-derived behavioural sensitivity (leave-one-focus-out ablation + permutation testing), and run the same experiment across models.
Experiment A — Reported focus
Ask the model how a single completion attended to each focus (scores summing toward 100). This is behavioural self-report, not transformer attention weights.
Experiment B — Perturbation sensitivity
Delete each verified focus span, resample, embed outputs, and test whether the embedding shift exceeds a permutation null. q-values use Benjamini–Hochberg FDR control.
Experiment C — Reported vs revealed
Compare the two lenses on the same foci. Agreement and disagreement are both scientifically useful — especially when embeddings miss structural instructions such as citation rules.
Live analysis on this site may be available under a capped demo budget. Inference is never billed to the project maintainers for open-source use — you bring your own keys.