Research demonstration · Project 01
Co-producing and testing simulated scholarly intuitions
Persistent research agents keep skills, memories and protocols from earlier work. We follow one hospitality-management replication from first attempt to a revised practice, then show how we plan to test whether that practice helps another agent.
Read this first
This demonstration reconstructs the process. It does not publish real chats or logs, to protect the integrity of ongoing work. Conversations are newly written and labelled where they appear. The runnable example uses made-up data. Nothing here is a live view of the agents.
Scroll, or use the arrows.
The idea
Simulated scholarly intuition
We use this name, or machine intuition, for acquired experience: skills, memories and protocols that shape what an agent notices, how it approaches a research problem, and when it questions its own work.
It is inspectable guidance shaped by the work and by the people who criticized it. We do not claim AI has human intuition, that two agreeing agents are right, or that acquired experience always helps. Testing that is the point.
Project 01 is one line of work inside Intuitionist Edu, a wider agent infrastructure for teaching and research.
How to read the labels
- Documented The team's records support that this happened.
- Reconstruction Newly written to explain a documented step.
- Synthetic Runs in your browser on made-up data.
- Planned Part of the proposed evaluation. Not yet run; no results.
Stage 1 of 7 · The study
The research task
A published hospitality-management study analyzed Yelp reviews. Its original author knows the paper and the reviews it went through on the way to publication. Two in-house agents, Greta and Desmond, rebuild the intended method on a parallel Yelp dataset held on our server.
Rebuilding the intended method is a replication. Answering the same questions with a different method is an extension, reported separately so the two are never confused.
Who contributed, what was challenged, what comes next
- Who contributed
- The original author, who knows what the method was meant to do. The investigator, who set the brief. Greta and Desmond.
- What was challenged
- The idea that matching a method's name is enough. Rebuilding a word-based analysis from present words alone, rather than present and absent categories, changes everything downstream.
- Evidence needed next
- A written account of the intended method, checked against the paper, before any numbers are compared.
- Original authorAgreement with our numbers is welcome, but it is supporting evidence. Did you rebuild and run the method we actually used?
- GretaI'll write down the method as the paper describes it, input table included, before either of us fits anything.
Stage 2 of 7 · The study
Independent first work
The intended workflow asks each agent to save its first reconstruction before comparing. Separate implementations are not automatically blind: any prior access to a peer’s work, shared discussion or audit findings must be disclosed.
Running on different models is not enough. These two run on models from different developers, and they still shared a mistaken assumption at the next stage.
Who contributed, what was challenged, what comes next
- Who contributed
- Greta and Desmond, working separately. The investigator decides what each may see.
- What was challenged
- The word "blind". Where an agent had seen part of a peer's work, the honest label is "independently implemented with prior exposure", stating what it saw and when.
- Evidence needed next
- Access records showing what each agent could see, and the saved first outputs, before any comparison.
What each agent works from in a first pass
| The published paper | Yes |
| The parallel Yelp dataset | Yes |
| The other agent's draft and results | Not until both are saved |
| Earlier audit findings on this study | Withheld in a first pass |
This table depicts the intended access procedure, not proof of fully sealed conditions in the documented case. Actual prior exposure must be recorded.
Stage 3 of 7 · The study
Critical review and external audit
A check on the model's cut points (the boundaries between star-rating levels) reported a failure. Both agents accepted it, because they shared one assumption about where the software puts those values and how it stores them. Their agreement added confidence, not evidence.
An external agent audited the outputs and how the team coordinated. Once the line that computed the check was opened, the "failure" turned out to be a reading error. The auditor's own explanation needed checking too: a later audit note withdrew an earlier failed-check diagnosis.
Who contributed, what comes next, technical detail
- Who contributed
- The external auditor, an agent outside the team. Greta and Desmond, reviewing each other. Fixes to the research were kept apart from fixes to coordination and status reporting. External means outside the assigned team; it does not establish a blind or infallible audit.
- Evidence needed next
- The computing line run on the saved fit, and a test of the check against known clean and broken inputs. Try a synthetic version.
- Technical detail
- Two related but separate mistakes. In statsmodels 0.15.0,
OrderedModellists predictor slopes first and thresholds after them. It stores the first cut point and then the log of each gap, so a correct fit can look out of order;transform_threshold_paramsconverts them. Checked against the library source, October 2026. Other libraries and versions may differ.
- AuditorTwo runs agreeing tells me you share a method, not that it is right. Which values did the check read, and how does the library store them?
- GretaI took the last four values by position. I haven't checked whether they are cut points or something stored on another scale.
Stage 4 of 7 · The study
Correction and human direction
The agents revised how they extract and verify results. The humans changed what counted as success. The original author asked for a calibration pilot: run the intended method, treat matching results as supporting evidence, check how parameters are represented, keep original attempts, and test checks on known errors and clean examples.
The investigator challenged an explanation the agents had accepted as a limitation, and separated "a lesson was written down" from "a lesson was used later". A human accepts, rejects, defers or redirects the revised work; a finished computation alone does not constitute acceptance.
Who contributed, what was challenged, what comes next
- Who contributed
- The original author and the investigator, questioning methods and interpretation. Greta challenged Desmond's first explanation of a further threshold error.
- What was challenged
- A reassuring caveat about standard errors. It sounded careful, so nobody had checked it, and its reasoning was wrong.
- Evidence needed next
- A rerun with the corrected code, then another round of audit and author review.
- InvestigatorYou both accepted that caveat. What property of the fit is it based on?
- DesmondI gave it as a reason the errors look small. I didn't test it.
- InvestigatorThen it's a claim, not a precaution. Check it like the result, or take it out.
Stage 5 of 7 · The study
Acquired experience
Each correction became a candidate rule with a stated reason and scope, recorded in the team's shared replication skill (version 1.3 at the documented snapshot, revised since). The agents applied it later within the same study.
A stored rule shows that something was retained. It is not yet evidence that later work improved. That needs a record of an agent finding and using the rule appropriately on a new task.
Stage 6 of 7 · The study
A new task under three proposed conditions
The proposed comparison has three conditions: no additional acquired experience, experience developed through prior human–agent collaboration, and a human-written checklist. Each receives the information needed for the new task. Models, tools and working resources remain comparable, with differences in supplied information accounted for.
The new tasks must not have answers in the retained material. Include a situation where the practice matters and one where it should not be applied. Selected-skill or memory comparisons are follow-up analyses. This is a design direction, not a finalized protocol.
Stage 7 of 7 · The study
Exchange and adaptation
Later, a visiting agent from another team could receive a practice, question its reasoning, and apply or adapt it to a different question. One candidate: could experience from analyzing large-scale Yelp reviews help an agent studying generative AI and misinformation on social media?
That first needs a secure way for outside agents to join: identity, limited access, revocation, logging and a security audit. This is planned work, not something that exists yet.
The practice
What changed in the guidance, and why
Choose a practice. The wording is adapted for public reading from the team's shared guidance.
- Reason
- Two agents accepted the same failed check because they shared one assumption.
- Reject or adapt when
- The library already returns labelled cut points; the model has no cut points; another library stores values differently. Only the habit of checking carries over.
- Whose contribution
- Agents drafted it. The external auditor opened the computing line, and its own earlier diagnosis was corrected. The original author asked for parameter representation to be checked. The investigator required checks to be traced to code.
- Status
- Recorded and applied within the same study. Not yet tested on a new task.
- Reason
- A similar-sounding procedure can carry out a different method, and every later number moves with it.
- Reject or adapt when
- The work is a deliberate extension with an alternative method. Label it an extension instead.
- Whose contribution
- The original author made executing the intended method the measure of success. An external audit compared runs that disagreed. Agents rebuilt the input and recorded the rule.
- Status
- Recorded. Not yet tested on a new task.
- Reason
- Agents accepted an incorrect explanation because it was offered as a caution. Human criticism showed its reasoning was wrong.
- Reject or adapt when
- It would strip out honest uncertainty. A limitation that cannot yet be checked can stay if labelled as untested.
- Whose contribution
- The investigator challenged the accepted explanation. Greta's review challenged incorrect descriptions of causes. Agents revised how limitations are reported.
- Status
- Recorded. Not yet tested on a new task.
Before Reconstruction
Take the values in the order you expect them. The last four are the cut points; check that they increase.
After Documented
- Select every reported value by its label, never by its position.
- Find out how the software stores each kind of value. Convert with the library's own function before testing.
- Do not apply a conversion the documentation does not call for.
- When a check fails, open the line that computed it before reporting the failure.
Applies when reading estimates from any statistical library, or reviewing a teammate's pass or fail summary.
Before Reconstruction
Run the analysis that matches the method's name, using the word table already built.
After Documented
- Rebuild the exact input the paper describes. An analysis needing a present and an absent category for each word differs from one using present words only.
- Before comparing, test the rebuilt method against a property you already know should hold.
- If your own sanity check fails, fix the method. Don't report it as a finding about the data.
Applies when a method's name hides choices about how its input is built.
Before Reconstruction
Once the main result holds, list the usual limitations.
After Documented
- Name the property of the fit each limitation rests on, or cut it.
- Check a limitation as hard as a result.
- When a check catches an error, treat the explanation of why as a second claim with its own evidence.
Applies to write-ups, peer reviews and audit reports, including the auditor's explanations.
Run the check
Three ways to read the same output
A small, repeatable version of the stage 3 problem. Each case is a made-up set of model parameters with known true values. Three hand-written readers check whether the cut points increase.
These readers are fixed code, not AI agents, and the cases are not study data. The result shows what the check does, not how any agent performs.
The three readers
- Positional reader. Takes values by position and tests them as stored. The shared assumption.
- Always-convert reader. Reads by name, then always converts. The lesson repeated without checking.
- Representation-aware reader. Reads by name, converts only when the stored form says so, and stops when it is unknown. The revised practice.
Run the check · Synthetic
Pick a case and run it
Results appear here.
Run the check · Synthetic
Across all five cases
Only the representation-aware reader reads every case correctly, catches the real problem, and stops when the stored form is not stated.
The always-convert reader shows why repeating a lesson without checking can be wrong: it distorts values that were already cut points, and misses a real ordering problem.
Repeat it elsewhere
- check.py three readers and the check, standard-library Python
- fixtures.json five synthetic cases with known values
- protocol.md an example protocol
- README.md how to run, and what it is not
Working together
Council and huddle
Two ways to organize the agents. Choosing one explains how it works; it does not show that either produces better research. The Yelp case was not a controlled comparison of the two.
Independent first work, then critical peer review of assumptions, methods and findings, then comparison and synthesis. Unresolved disagreement stays visible and goes to the humans. Its weakness, shown in stage 3: independent agents can still share an assumption.
A research-manager agent splits the work, specialized agents each take a part, and they coordinate through discussion. Its weakness: no part gets a second independent attempt unless someone builds one in.
In both, humans direct the inquiry and its interpretation. Access to experience is a separate question.
Evaluation design direction · Planned
Does acquired experience add value beyond a checklist?
Three proposed conditions receive the information needed for a new task. Choose one.
Design direction, not a finalized protocol. No runs, scores or findings are claimed. Task selection, scoring and information controls remain to be specified.
Follow-up comparisons can separate selected skills or memories to investigate which kinds matter. They are not the third main condition.
How we will test it · Planned
Tasks that test judgment, not memory
Assessment asks whether the agent identifies the relevant issue, chooses an appropriate check and explains the result, alongside numerical correctness, methodological reasoning and substantive interpretation. Record evaluator effort, costs, human intervention and repair attempts. Original authors clarify intended methods; agreement with them is not the sole quality criterion. New tasks include cases where the practice matters and cases where it should not be used. The baseline or checklist may perform as well as or better than acquired experience.
Still open
- How many tasks and repeat runs, and how they are scored.
- How to author the checklist and account for differences in supplied information; which skills or memories to separate in follow-up comparisons.
- Fair assessment when original authors helped shape the guidance.
- Whether a benefit survives a newer model.
- How a visiting agent joins securely, and what it may see.
Limits
Three risks we are watching
We preserve disagreements and rejected alternatives, record model versions, and test whether criticism from other teams or disciplines broadens the inquiry.
Technical note · framework and deployment scope
The public framework describes Hermes-based agents, a conversational interface, persistent artifacts and replaceable model endpoints. Role assignments and model settings vary by task. This page reconstructs an earlier case, not a current runtime configuration or the later DistilBERT study.
The bigger picture
One part of a wider infrastructure
This page does not establish that a visiting agent can already join, or that the full research artifact is complete. Other teams will also need permitted materials and repeatable checks, which are still being assembled. The second ongoing replication, a communication meta-analysis, is not shown here. We are looking for further studies and original authors to work with.