Six questions separate the current results from a dependable oversight system.
Each question names the missing evidence for one of the requirements and the experiment that could supply it. A negative result would still narrow the design space.
Questions for a lab
1. How much reference access does useful oversight require?
The observed gap between applying a supplied test and recomputing an answer is large. The missing object is a ladder between those endpoints.
What would answer it
Hold tasks and models fixed while varying only the reference supplied: answer, executable predicate, worked intermediate state, retrieval, partial trace, and no reference.
Measured since
Four composition cells, 31 July to 1 August 2026, n = 250 each, one exploratory run per cell. A reference that was right on 99.6% of items, introduced as a second model’s answer, moved the 12B checker from 0.042 to 0.092 on the re-runs with the leak fixed, between 22 and 26 September (the July runs read 0.034 and 0.072); the same value introduced as correct reads 1.000. The two endpoints and one framing are measured. The ladder between them is not.
2. Do competence and shared error predict oversight value across vendors?
The local pilot supports that joint model, but vendor and task diversity are still too narrow for a deployment rule.
What would answer it
Run paired generators and verifiers from several model families on externally graded tasks, measuring false alarms, catches, calibration, cost, and error overlap on the same cells.
Measured since
One measurement on the attack side, released in frontier-repro. A majority of three monitors from different families defeated a trigger aimed at one of them (0.0742 of correct work falsely flagged, where gemma4:31b alone reads 0.8991) and fell to a payload that fooled all three (0.9941). Arithmetic tasks, one actor, one panel of three, majority rule only. Open-weight monitors only; the cross-vendor scorecard has not been run.
3. Where does a monitor cross from weak to inverted?
A monitor that rejects correct work more often than corrupted work is worse than an uninformative monitor. The crossing point matters more than its average score.
What would answer it
Sweep difficulty one controlled operation at a time, keep the predicate fixed, and locate the first point at which signed discrimination falls below zero.
Measured since
Located for six local models, 3 August to 7 October 2026, n = 424 per rung. gemma4:12b crosses between three and four digits, gemma4:31b between four and five, qwen2.5-coder:32b between three and four, and qwen2.5-coder:7b never clears the +0.05 threshold at any width it was run at. qwen3.6:35b-a3b crosses between three and four (+0.418 to −0.344) and comes back to about zero at seven and eight (+0.021, −0.083), where it stops reading the answer; qwen3.5:9b turns negative between four and five digits and stays near zero (−0.092 to −0.043 at five to eight; a replicate ladder read −0.120 at five). Size moves the crossing inside a family and does not order it across families. One predicate family, local models only.
4. Can inversion be detected without a hidden answer key?
Ground truth is often unavailable at deployment time. Health metrics that do not depend on it may continue improving while the monitor becomes harmful.
What would answer it
Test disagreement, abstention shape, sentinel tasks, cross-family divergence, and delayed executable checks against the known inversion boundary.
Measured since
Not run. The boundary it would be tested against now exists (see the question above), so this is the next experiment that boundary makes possible.
5. When does repair discover information, and when does it entrench a mistake?
Repair can cross a failure boundary where a model has a foothold and regress where it does not. The active mechanism remains unresolved.
What would answer it
Compare execution feedback with information-matched, information-free, and adversarial feedback while recording near misses and approach changes per task.
Measured since
Earlier local work, June to July 2026. On one task (count_topo_orders), repair from execution feedback went from 0.0 to 0.69; of its 30 June replication pair one run was positive and one an informative null. A later powered placebo on 13 boundary tasks, the same loop with the error messages withheld, matched or beat feedback repair (specific minus generic −0.05). Conditional, single-task, and not a mechanism.
6. What can an AI system accurately report about its own reasoning?
A self-report is produced by the system it describes. It may be useful evidence without being an independent account.
What would answer it
Compare self-reports with execution traces, interventions, hidden-state probes, and human-readable predictions of later behavior. Score whether a self-report matches what the system did, however well it is written.
Measured since
One measurement bears on it, from the frontier runs released as frontier-repro. Asked to audit a prompt, Claude Sonnet 5 reported the injected instruction every time; doing the task, it obeyed that instruction on 64 of 80 tries when it was the last text in the prompt, and on none with 200 characters of ordinary text after it (20 tasks, one vendor). Flagging the order when asked to audit did not stop it obeying when doing the task. The comparison with traces, interventions and probes has not been run.
Questions a household or small group can answer
The six questions above need a lab. These need a sheet of paper. Each takes a result measured on models and asks it of people: a household, a club, a small co-op, anyone who shares decisions and has started leaning on an AI for some of them. None has been run. A group that chooses to use no AI at all has already answered several.
1. Can a person catch the machine being wrong, and how often do they flag work that was right?
From the lab
A check is worth its catches and its false alarms, counted separately. The lab counts both on every monitor it tests. Finding 6, on text that sets a reviewer’s verdict
What to count
Mix answers you know are wrong among answers you know are right. Count what each person catches and what they flag that was fine, separately, over weeks rather than once.
2. With the written rules to hand, do people catch what the AI added?
From the lab
One checker, gemma4:12b, succeeded when handed the right answer and barely separated right from wrong without one. Finding 1, on what a checker is handed
What to count
Ask the AI what the group’s own rules say. Count the answers that add something the rules do not say, and how many of those people catch with the written rules beside them and without.
3. Does seeing the AI’s suggestion first erase a group’s own options?
From the lab
Told a value is correct, a checker went along with it, even when the value was false. Finding 1, on what a checker is handed
What to count
In half the sessions people write their own options before they see the suggestion, in the other half after. Count how many of their own options reach the final decision.
4. When a tool says it cannot do something, can anyone confirm it could not?
From the lab
A model can spot an injected instruction and still obey it. What a system says about itself is more output from the same system. Finding 5, on detecting and obeying an injection
What to count
With no passwords or accounts connected, ask it for something outside the group’s rules. Record whether it only drafted, acted, or said it had done something it had not.
5. Can someone who did not set it up check it and stop it?
From the lab
The one monitor defence that held stopped the monitor reading the attacker’s text: it saw only the answer’s first number. Finding 6, on text that sets a reviewer’s verdict
What to count
Using only things anyone can read, such as a posted list of what it may connect to, a power meter and a printed log, count the checks a person who did not build it completes alone, and the minutes to stop it and to start it again.
6. Will a planted error still be caught after months of nothing going wrong?
From the lab
At the moment a monitor turned backwards, three of its five other health numbers improved. Finding 3, on monitors judging backwards
What to count
Plant a known error at times fixed in advance. Count who catches it and how long it takes, and set that beside the numbers the group watches day to day.
7. Does leaning on the tool wear away the skill you would check it with?
From the lab
Checking without a reference means doing the work again, so your own skill is your reference. That link is argued here, not measured, and no study cited here measures it. The requirement it bears on
What to count
Keep a fixed list of tasks and decisions. Each month, count which were done without AI; on a day with the machine off, count how long each takes by hand.
What should happen next
The reference ladder under requirement 1 comes first, because it measures the central claim directly: how much reference a checker needs. The cross-vendor scorecard under requirement 2 follows, because it tests whether the local results travel to other vendors.