A question is not hard because it is worded densely
Difficulty labels assigned at the time of writing are only guesses, and those guesses are wrong in one respect. If any effort went into writing a question, it will seem difficult to the person who wrote it, no matter what the question asks the student to do.
Real difficulty comes from cognitive demand, such as recall, application, and analysis, and from how much information a student has to keep in mind at one time. Dense sentences increase the reading load, and that load is generally coincidental.
A paper also has a shape. All the questions marked as medium can examine the same skill at the same level, while three of the topics on the syllabus remain unexamined. How topics are distributed is just as important as the distribution across levels.
The agent checks the work against its own claims and suggests amendments with the reasons for them. The agent does not prepare papers for students on its own. The examiner decides which paper to use.
How the agent is built in Agent Studio
Prompt skills determine how items are handled: your level definitions, your topic taxonomy, and the distribution your department requires for a paper of a given weight.
Prompt skills also include past papers with the answers corrected afterward, because that is the only reliable record of the questions your group actually found difficult.
The system prompt states that each relabeling must be justified by what the question asks of a student, and that cognitive demand should be separated from reading load.
The system prompt prohibits performance data linked to identifiable students. Calibration relies only on item characteristics and suppressed aggregates.
The calibration flow reads questions from Airtable or Google Sheets and passes them to the Agent Chat node along with the syllabus topics.
The Math node calculates the distribution across topics and levels, so an imbalance is expressed as a number rather than an impression. Create Excel generates the reviewed paper.
An Email node in the Human in the Loop group sends the set to the examiner. Question banks are updated only after approval.
What this agent is built from
- Prompt skills: level definitions, topic taxonomy, and expected distribution by paper weight.
- Prompt skills: past papers with their labels corrected after the fact.
- System prompt: relabeling argued from what the question asks. Reading load is separated from demand.
- System prompt: no identifiable student performance data enters calibration.
- Airtable (Integration): holds the question bank that the agent reads and proposes changes to.
- Math (Util) with Create Excel (Data): the distribution as a number, then the reviewed paper.
- When an ActionFlow is enough: If counting items per topic and per declared level is the work, build the workflow. That count exposes the crudest imbalances.
Frequently asked questions
Start building AI workflows
Create a free account, open a template or a blank canvas, and run your first ActionFlow.
Newsletter
Get product updates
New nodes, agents, and product notes. We send mail only when we have something worth opening.