How to Use FocalPrompt
Getting Started
1. Tag Your Prompt into Foci
Start by tagging your prompt into points of focus or "foci". You want to aim for as close to 100% coverage as possible for a comprehensive analysis. Our model can help you detect foci in your prompt but is unlikely to take you to full coverage.
2. Define Dynamic Inputs
Dynamic parts of your prompt, where the input will vary, can be tagged as "dynamic" in section 2 "Define Foci". Dynamic inputs include:
- Chat inputs - Variable conversation content
- RAG - Retrieval-Augmented Generation context
- Tools - Tool execution results
This will allow you to provide matched inputs and outputs in the batch analysis section.
3. Generate Output
You can then generate a single output for the prompt as provided. This will use the individual instance of any dynamic inputs currently provided in the prompt.
Experiments A / B / C
Experiment A — Reported focus (Assess Focus Distribution)
Score a single completion against your foci. Scores are the model’s behavioural self-report of how that output attended to each focus — not transformer attention weights.
5. Adjust Intended Focus
Use sliders to set an intended focus mix; rewrite attempts to reflect that mix. Generate a new output to compare.
Experiment B — Perturbation sensitivity (Ablation Analysis)
Delete each verified focus span, resample, embed outputs, and test whether the embedding shift exceeds a permutation null. q-values are Benjamini–Hochberg-adjusted across tested foci.
Experiment C — Reported vs revealed
After Experiments A and B, the Results report compares reported scores with revealed influence (dumbbell chart, focus map, concordance detail). There is no separate Experiment C step in the lab flow.
FocalPrompt detects whether removing each focus shifts the model's behaviour in semantic embedding space. It does not measure correctness, quality, or safety, and it does not tell you what to delete.
A non-significant result is a failure to detect a shift at this sample size, not evidence that the text does nothing. Short structural instructions can matter greatly while barely moving embeddings.
Note: Cost and runtime increase with the number of foci and with baseline and ablated sample counts.
How this works
Batch Analysis
7. Batch Analysis
In the Batch Analysis tab we can assess dynamic inputs and outputs, and how the different parts of the prompt input are influencing the output.
You can manually add dynamic input, output combinations or upload in bulk in a CSV file with the appropriate format.
Agent Builder
8. Build Optimized Agents
The Agent Builder tab allows you to build optimized agents for specific inputs. The system uses LLM assessment to select relevant foci and generate new outputs for comparison.
Best Practices
- Aim for 100% coverage of your prompt with foci whose text matches the prompt verbatim
- Tag dynamic inputs appropriately so they are reported as slots, not treated as prompt text
- Use ablation analysis to test which deletions produce a detectable behavioural shift
- Treat non-significant foci as undetected, not as unused
- Test prompt adjustments with the focus sliders before finalizing
1. Enter Your Prompt
2. Define Foci
Tip: A focus is a semantic/experimental unit — it may be contiguous, multi-span, nested, or overlapping. Select text and use Tag as Focus, or select a focus then Add another span (Shift+tag also works on desktop). Structured XML/Markdown sections can seed coarse foci without replacing finer ones.
No foci defined yet. Click "Auto-Detect Foci" or "Add Focus Manually" to get started.
3. Enter or Generate Output
4. Experiment A — Reported focus
Model self-assessment of one completion (behavioural report). This is not transformer attention. After ablation (B), reported scores feed the Results report — you do not need a separate Experiment C step.
Click "Assess Focus" to see the results.
6. Experiment B — Perturbation sensitivity
Leave-one-focus-out ablation with permutation testing. Detects whether removing each focus shifts behaviour in embedding space — not correctness, quality, or safety, and not a delete recommendation. Reported vs revealed comparison appears in the Results report below (Overview + Focus map), not as a separate experiment step.
Experiment configuration
Advanced
Click "Run Ablation Analysis" to test whether deleting each focus shifts the model's behaviour. Results open as an insight report (Overview first).
7. Focus order sensitivity
Measures behavioural sensitivity to focus ordering while keeping semantic content constant (glue, dynamic chat, and fixed sections preserved). Reuses Experiment B baseline samples. Distinct from LOO ablation (content removal) and from shuffle-order robustness on each focus card (LOO + remaining reorder). Not mechanistic attention.
Run Experiment B (section 6) first to reuse baseline samples.
Complete Experiment B, then run order sensitivity here.
8. Task quality evaluation (LLM)
Score how well each Experiment B output meets your task criteria (quality / instruction-following). Evaluates baseline samples (full prompt) and ablated samples (one focus removed per run) — not the single output from section 3. Separate from behavioral difference (whether ablation changed outputs) and from reported focus (Experiment A).
Sampling keeps at least one baseline and one sample per ablated focus. Cost scales with the number judged (batched 4 at a time). There is no fixed cap at 60.
Run Experiment B (section 6) first. Baseline and ablated samples from that run will be scored here against your criteria.
Enter criteria above, then click Evaluate.
1. Data Input
CSV format (case-insensitive headers): required output (aliases: suggested_message, response) plus at least one of input/chat_content, rag_context, tool_results, other_input, and/or per-row prompt. Shared prompt above applies when a row has no prompt column. UTF-8 (BOM OK). Prompt text is preserved exactly.
Add Pair Manually
Enter the variable input for this row (e.g. user message or chat turn) and the model output you want to analyze. The shared system prompt is set below.
Prompt (Applied to All Pairs)
Enter the prompt that was used to generate all outputs. This prompt will be applied to all input-output pairs.
No pairs added yet. Upload a CSV file or add pairs manually.
2. Define Foci
No foci defined yet. Click "Auto-Detect from First Prompt" or "Import from Prompt Analysis" to get started.
3. Run Analysis
Each pair is its own permutation experiment: the pair prompt is sampled, then each locatable focus is deleted and sampled. Dynamic slots are excluded. This may take a while.
Experiment configuration
Advanced
4. Results
Run batch analysis to see results.
1. Enter Chat Content
2. Define Foci
💡 Define the foci that the agent can select from, or import from Prompt Analysis tab.
No foci defined yet. Click "Auto-Detect Foci" or "Import from Prompt Analysis" to get started.
3. Assess Chat & Select Foci
Enter chat content and define foci, then click "Assess Chat & Select Foci" to see which foci are relevant.
4. Generate Response
After assessing chat and selecting foci, click "Generate Response" to see the constructed prompt and agent output.
5. Batch Agent Building
💡 Import pairs (input/output) to automatically build optimized agents for each input. The system uses LLM assessment (same as single agent builder) to select relevant foci and generate new outputs for comparison.
No batch data imported yet. Click "Import from Batch Analysis" to get started.