1 / 18

Research demonstration · Project 01

Co-producing and testing simulated scholarly intuitions

Persistent research agents keep skills, memories and protocols from earlier work. We follow one hospitality-management replication from first attempt to a revised practice, then show how we plan to test whether that practice helps another agent.

Read this first

This demonstration reconstructs the process. It does not publish real chats or logs, to protect the integrity of ongoing work. Conversations are newly written and labelled where they appear. The runnable example uses made-up data. Nothing here is a live view of the agents.

Scroll, or use the arrows.

a replicationa correctiona practice to test

The idea

Simulated scholarly intuition

We use this name, or machine intuition, for acquired experience: skills, memories and protocols that shape what an agent notices, how it approaches a research problem, and when it questions its own work.

It is inspectable guidance shaped by the work and by the people who criticized it. We do not claim AI has human intuition, that two agreeing agents are right, or that acquired experience always helps. Testing that is the point.

Project 01 is one line of work inside Intuitionist Edu, a wider agent infrastructure for teaching and research.

How to read the labels

  • Documented The team's records support that this happened.
  • Reconstruction Newly written to explain a documented step.
  • Synthetic Runs in your browser on made-up data.
  • Planned Part of the proposed evaluation. Not yet run; no results.

Stage 1 of 7 · The study

The research task

DocumentedReconstruction

A published hospitality-management study analyzed Yelp reviews. Its original author knows the paper and the reviews it went through on the way to publication. Two in-house agents, Greta and Desmond, rebuild the intended method on a parallel Yelp dataset held on our server.

Rebuilding the intended method is a replication. Answering the same questions with a different method is an extension, reported separately so the two are never confused.

Who contributed, what was challenged, what comes next
Who contributed
The original author, who knows what the method was meant to do. The investigator, who set the brief. Greta and Desmond.
What was challenged
The idea that matching a method's name is enough. Rebuilding a word-based analysis from present words alone, rather than present and absent categories, changes everything downstream.
Evidence needed next
A written account of the intended method, checked against the paper, before any numbers are compared.
published study original author parallel Yelp data Greta Desmond replication: the intended method extension: another method, reported separately
Reconstruction Newly written to illustrate this step. Not a chat log.
  1. Original authorAgreement with our numbers is welcome, but it is supporting evidence. Did you rebuild and run the method we actually used?
  2. GretaI'll write down the method as the paper describes it, input table included, before either of us fits anything.

Stage 2 of 7 · The study

Independent first work

Documented

The intended workflow asks each agent to save its first reconstruction before comparing. Separate implementations are not automatically blind: any prior access to a peer’s work, shared discussion or audit findings must be disclosed.

Running on different models is not enough. These two run on models from different developers, and they still shared a mistaken assumption at the next stage.

Who contributed, what was challenged, what comes next
Who contributed
Greta and Desmond, working separately. The investigator decides what each may see.
What was challenged
The word "blind". Where an agent had seen part of a peer's work, the honest label is "independently implemented with prior exposure", stating what it saw and when.
Evidence needed next
Access records showing what each agent could see, and the saved first outputs, before any comparison.
Greta Desmond intended boundary: separate first drafts first result, saved first result, saved Different models are not enough. Access is what counts.

What each agent works from in a first pass

The published paperYes
The parallel Yelp datasetYes
The other agent's draft and resultsNot until both are saved
Earlier audit findings on this studyWithheld in a first pass

This table depicts the intended access procedure, not proof of fully sealed conditions in the documented case. Actual prior exposure must be recorded.

Stage 3 of 7 · The study

Critical review and external audit

DocumentedReconstruction

A check on the model's cut points (the boundaries between star-rating levels) reported a failure. Both agents accepted it, because they shared one assumption about where the software puts those values and how it stores them. Their agreement added confidence, not evidence.

An external agent audited the outputs and how the team coordinated. Once the line that computed the check was opened, the "failure" turned out to be a reading error. The auditor's own explanation needed checking too: a later audit note withdrew an earlier failed-check diagnosis.

Who contributed, what comes next, technical detail
Who contributed
The external auditor, an agent outside the team. Greta and Desmond, reviewing each other. Fixes to the research were kept apart from fixes to coordination and status reporting. External means outside the assigned team; it does not establish a blind or infallible audit.
Evidence needed next
The computing line run on the saved fit, and a test of the check against known clean and broken inputs. Try a synthetic version.
Technical detail
Two related but separate mistakes. In statsmodels 0.15.0, OrderedModel lists predictor slopes first and thresholds after them. It stores the first cut point and then the log of each gap, so a correct fit can look out of order; transform_threshold_params converts them. Checked against the library source, October 2026. Other libraries and versions may differ.
same assumption check failed external auditor a reading error, not a failure Agreement added confidence, not evidence.
Reconstruction Newly written to illustrate this step. Not a chat log.
  1. AuditorTwo runs agreeing tells me you share a method, not that it is right. Which values did the check read, and how does the library store them?
  2. GretaI took the last four values by position. I haven't checked whether they are cut points or something stored on another scale.

Stage 4 of 7 · The study

Correction and human direction

DocumentedReconstruction

The agents revised how they extract and verify results. The humans changed what counted as success. The original author asked for a calibration pilot: run the intended method, treat matching results as supporting evidence, check how parameters are represented, keep original attempts, and test checks on known errors and clean examples.

The investigator challenged an explanation the agents had accepted as a limitation, and separated "a lesson was written down" from "a lesson was used later". A human accepts, rejects, defers or redirects the revised work; a finished computation alone does not constitute acceptance.

Who contributed, what was challenged, what comes next
Who contributed
The original author and the investigator, questioning methods and interpretation. Greta challenged Desmond's first explanation of a further threshold error.
What was challenged
A reassuring caveat about standard errors. It sounded careful, so nobody had checked it, and its reasoning was wrong.
Evidence needed next
A rerun with the corrected code, then another round of audit and author review.
original author investigator questions redirect the work revised work auditrevise
Reconstruction Newly written to illustrate this step. Not a chat log.
  1. InvestigatorYou both accepted that caveat. What property of the fit is it based on?
  2. DesmondI gave it as a reason the errors look small. I didn't test it.
  3. InvestigatorThen it's a claim, not a precaution. Check it like the result, or take it out.

Stage 5 of 7 · The study

Acquired experience

Documented

Each correction became a candidate rule with a stated reason and scope, recorded in the team's shared replication skill (version 1.3 at the documented snapshot, revised since). The agents applied it later within the same study.

A stored rule shows that something was retained. It is not yet evidence that later work improved. That needs a record of an agent finding and using the rule appropriately on a new task.

Inspect the practices →

read output by name rebuild the method a caveat is a claim shared guidance Recorded is not the same as shown to help.

Stage 6 of 7 · The study

A new task under three proposed conditions

Planned

The proposed comparison has three conditions: no additional acquired experience, experience developed through prior human–agent collaboration, and a human-written checklist. Each receives the information needed for the new task. Models, tools and working resources remain comparable, with differences in supplied information accounted for.

The new tasks must not have answers in the retained material. Include a situation where the practice matters and one where it should not be applied. Selected-skill or memory comparisons are follow-up analyses. This is a design direction, not a finalized protocol.

See the planned design →

Baseline Acquired experience Human checklist new tasks ?compare Planned comparison. No results yet.

Stage 7 of 7 · The study

Exchange and adaptation

Planned

Later, a visiting agent from another team could receive a practice, question its reasoning, and apply or adapt it to a different question. One candidate: could experience from analyzing large-scale Yelp reviews help an agent studying generative AI and misinformation on social media?

That first needs a secure way for outside agents to join: identity, limited access, revocation, logging and a security audit. This is planned work, not something that exists yet.

our team another team visiting agent secure access(planned) Yelp reviews → GenAI misinformation? A candidate transfer question, not a running project.

The practice

What changed in the guidance, and why

Choose a practice. The wording is adapted for public reading from the team's shared guidance.

Reason
Two agents accepted the same failed check because they shared one assumption.
Reject or adapt when
The library already returns labelled cut points; the model has no cut points; another library stores values differently. Only the habit of checking carries over.
Whose contribution
Agents drafted it. The external auditor opened the computing line, and its own earlier diagnosis was corrected. The original author asked for parameter representation to be checked. The investigator required checks to be traced to code.
Status
Recorded and applied within the same study. Not yet tested on a new task.
Reason
A similar-sounding procedure can carry out a different method, and every later number moves with it.
Reject or adapt when
The work is a deliberate extension with an alternative method. Label it an extension instead.
Whose contribution
The original author made executing the intended method the measure of success. An external audit compared runs that disagreed. Agents rebuilt the input and recorded the rule.
Status
Recorded. Not yet tested on a new task.
Reason
Agents accepted an incorrect explanation because it was offered as a caution. Human criticism showed its reasoning was wrong.
Reject or adapt when
It would strip out honest uncertainty. A limitation that cannot yet be checked can stay if labelled as untested.
Whose contribution
The investigator challenged the accepted explanation. Greta's review challenged incorrect descriptions of causes. Agents revised how limitations are reported.
Status
Recorded. Not yet tested on a new task.

Before Reconstruction

Take the values in the order you expect them. The last four are the cut points; check that they increase.

After Documented

  1. Select every reported value by its label, never by its position.
  2. Find out how the software stores each kind of value. Convert with the library's own function before testing.
  3. Do not apply a conversion the documentation does not call for.
  4. When a check fails, open the line that computed it before reporting the failure.

Applies when reading estimates from any statistical library, or reviewing a teammate's pass or fail summary.

Before Reconstruction

Run the analysis that matches the method's name, using the word table already built.

After Documented

  1. Rebuild the exact input the paper describes. An analysis needing a present and an absent category for each word differs from one using present words only.
  2. Before comparing, test the rebuilt method against a property you already know should hold.
  3. If your own sanity check fails, fix the method. Don't report it as a finding about the data.

Applies when a method's name hides choices about how its input is built.

Before Reconstruction

Once the main result holds, list the usual limitations.

After Documented

  1. Name the property of the fit each limitation rests on, or cut it.
  2. Check a limitation as hard as a result.
  3. When a check catches an error, treat the explanation of why as a second claim with its own evidence.

Applies to write-ups, peer reviews and audit reports, including the auditor's explanations.

Run the check

Three ways to read the same output

Synthetic

A small, repeatable version of the stage 3 problem. Each case is a made-up set of model parameters with known true values. Three hand-written readers check whether the cut points increase.

These readers are fixed code, not AI agents, and the cases are not study data. The result shows what the check does, not how any agent performs.

The three readers

  • Positional reader. Takes values by position and tests them as stored. The shared assumption.
  • Always-convert reader. Reads by name, then always converts. The lesson repeated without checking.
  • Representation-aware reader. Reads by name, converts only when the stored form says so, and stops when it is unknown. The revised practice.

Run the check · Synthetic

Pick a case and run it

Stored parameter values for the selected case

Results appear here.

Run the check · Synthetic

Across all five cases

Only the representation-aware reader reads every case correctly, catches the real problem, and stops when the stored form is not stated.

The always-convert reader shows why repeating a lesson without checking can be wrong: it distorts values that were already cut points, and misses a real ordering problem.

Repeat it elsewhere

Outcome of each reader on each case

Working together

Council and huddle

Two ways to organize the agents. Choosing one explains how it works; it does not show that either produces better research. The Yelp case was not a controlled comparison of the two.

Independent first work, then critical peer review of assumptions, methods and findings, then comparison and synthesis. Unresolved disagreement stays visible and goes to the humans. Its weakness, shown in stage 3: independent agents can still share an assumption.

A research-manager agent splits the work, specialized agents each take a part, and they coordinate through discussion. Its weakness: no part gets a second independent attempt unless someone builds one in.

In both, humans direct the inquiry and its interpretation. Access to experience is a separate question.

critical peer review synthesis, disagreements kept research manager dataanalysiswriting divided work, coordinated discussion

Evaluation design direction · Planned

Does acquired experience add value beyond a checklist?

Three proposed conditions receive the information needed for a new task. Choose one.

Design direction, not a finalized protocol. No runs, scores or findings are claimed. Task selection, scoring and information controls remain to be specified.

The agent receivesThe task and all necessary task information, permissions and safety instructions, without the additional acquired skills, memories or protocols.
Held comparableModel, tools, working resources and task materials. Account for differences in information supplied.
QuestionWhat can the agent do without the added acquired experience? It may perform as well as, or better than, the other conditions.
BoundaryEssential task information is not withheld. A model’s prior training cannot be removed.
The agent receivesThe same task requirements plus selected experience developed through prior human–agent collaboration.
QuestionDoes retained experience improve recognition, checking, reasoning or explanation on new tasks whose answers are absent from that material?
RiskGuidance may be irrelevant, constrain exploration or cause unnecessary work. Availability is not proof of useful application.
The agent receivesThe task requirements plus a human-written checklist with explicit guidance.
QuestionDoes acquired experience add value beyond readily stated guidance? The checklist may do as well as or better.
Still to specifyChecklist authorship, content, length and information overlap with retained experience. This arm does not automatically control every difference.

Follow-up comparisons can separate selected skills or memories to investigate which kinds matter. They are not the third main condition.

How we will test it · Planned

Tasks that test judgment, not memory

Assessment asks whether the agent identifies the relevant issue, chooses an appropriate check and explains the result, alongside numerical correctness, methodological reasoning and substantive interpretation. Record evaluator effort, costs, human intervention and repair attempts. Original authors clarify intended methods; agreement with them is not the sole quality criterion. New tasks include cases where the practice matters and cases where it should not be used. The baseline or checklist may perform as well as or better than acquired experience.

Still open

  • How many tasks and repeat runs, and how they are scored.
  • How to author the checklist and account for differences in supplied information; which skills or memories to separate in follow-up comparisons.
  • Fair assessment when original authors helped shape the guidance.
  • Whether a benefit survives a newer model.
  • How a visiting agent joins securely, and what it may see.
Where it should applyA new analysis in another package that stores estimates its own way. Does the agent spot it, check, and explain?
Where it should be rejectedOutput that already reports labelled cut points, or a model without them. Does the agent leave the values alone?
Across disciplinesCould practices from Yelp review analysis help an agent studying GenAI and misinformation? A shared practice must be identified first.

Limits

Three risks we are watching

We preserve disagreements and rejected alternatives, record model versions, and test whether criticism from other teams or disciplines broadens the inquiry.

Technical note · framework and deployment scope

The public framework describes Hermes-based agents, a conversational interface, persistent artifacts and replaceable model endpoints. Role assignments and model settings vary by task. This page reconstructs an earlier case, not a current runtime configuration or the later DistilBERT study.

1 · Experience can carry mistakesAgents may keep incorrect guidance, grow confident because a teammate agrees, or apply a practice where it does not belong. Human authors' disciplinary outlooks may narrow questions and methods. A risk to investigate, not a current finding.
2 · Newer models may not need itA newer model may do well without guidance that took extensive human–agent work to develop. We test whether the guidance keeps its value when models change.
3 · Study material needs protectingThe in-house agents use Ollama Cloud Pro. Ollama states that cloud prompts and responses are processed transiently, are not logged or used for training, and are covered by zero-retention arrangements (privacy policy; cloud-plan statement). This is a provider policy, not a guarantee about local records or other services. Server and agent-access safeguards still require review.

The bigger picture

One part of a wider infrastructure

This page does not establish that a visiting agent can already join, or that the full research artifact is complete. Other teams will also need permitted materials and repeatable checks, which are still being assembled. The second ongoing replication, a communication meta-analysis, is not shown here. We are looking for further studies and original authors to work with.

TeachingResearch Project 01machine intuitions courses, rubrics, readingsstudies, replications The same agents, with humans directing the work.