Assignment 3: Same Model, Different Harness

Module 7 Assignment | Agent Engineering Due: End of Module 10 Length: 4–6 pages of analysis, plus a data appendix Submission: report.pdf and an experiments/ folder (prompt, token logs, verification output, both produced programs) Model: Claude Haiku — pinned for you in the provided scripts. Both agents must run the identical model.


Premise

By now you have built a coding agent from scratch (Module 6, Assignment 2) and given it real context management — a sliding window, selective deletion, and compaction (Module 7, Lab 3). It works — but it is small. A few hundred lines, three or four tools, a simple loop. The agents people actually use in production — Aider, Claude Code, SWE-agent — are far more elaborate: repository maps, structured edit formats, planning steps, sophisticated context strategies.

It is tempting to assume the gap between your agent and a production agent is mostly the model. It is not. A coding agent is two things stacked together: the model (the LLM doing the reasoning) and the harness (everything wrapped around it — the prompt, the tools, how files are fed into context, how context is managed, how edits are applied). This assignment isolates the second.

You will run two agents on the same task — your own coding agent from Assignment 2 and Lab 3 and Aider, a mature open-source coding agent — while holding the model and the prompt identical across both. Because the model and the prompt are the same, every difference you measure is attributable to the harness alone. Your job is to measure that difference and explain it.

This is an analysis assignment, not a build assignment. You will not write a new agent. You will run two existing ones, measure carefully, and reason rigorously about why a wrapper around the same model produces a different result.


What Is Held Constant — and Why It Matters

The entire validity of this study rests on three controls. If any of them breaks, your measurements no longer mean what you claim.

  1. The model. Both agents call the same Claude Haiku model. The provided Aider launcher pins both Aider's main model and its "weak model" (Aider uses a separate cheaper model for some calls by default — if you do not pin it, your token comparison is silently contaminated). Your own agent must be configured to call the same model id.

  2. The prompt. Both agents receive the exact same prompt, byte for byte, from a single prompt.txt file. You write this prompt and feed the same file to both agents — don't coach one and not the other. The prompt.txt you submit is the one both runs used.

  3. The information. Neither agent ever sees the tests. Both build the program from the specification alone. The test suite is the grader's ground truth, run from outside each agent's workspace. This keeps the comparison about building the program, not about looping against a test command.

Why Haiku and not Sonnet? Because the model is arbitrary here — the point is the harness — so we pick the cheap, fast one. A smaller model also leans harder on good scaffolding, which makes the harness differences larger and easier to see. Running both agents costs a few dollars at most.


The Task: Build a Small Library From a Specification

You will be given a specification (spec.md) for a medium-sized, pure-Python library — roughly six modules with interlocking rules (no web server, no network, no database; everything is deterministic and unit-testable). The exact specification ships with the assignment; at a high level it is an expense-sharing library (think a minimal Splitwise): users and groups, expenses split in several ways, balance computation, and a debt-settlement step, with a layer of validation rules throughout.

It is deliberately large enough that a coding agent is genuinely exercised — many file reads and edits, real context accumulation across modules, real opportunities to lose track. A trivial one-file script would show nothing.

Each agent's job is to produce a working implementation that passes as many of the provided tests as possible, from the specification alone.


What You Will Measure

Exactly two things, for each agent.

1. Tests passed (accuracy)

The fraction of the provided pytest suite that passes against the program the agent produced. This is your objective success metric — it captures both correctness and how complete the build was. The provided verify.py runs the suite and reports X / N passed; it also confirms the agent did not tamper with the test files.

2. Tokens and cost (efficiency)

To produce that result, how much did each agent spend? Report, for each agent:

The provided collect.py reads both agents' logs and prints these in one comparison table. The headline of the whole assignment is the pair of numbers: tests passed, and tokens spent to get there.

We are intentionally not asking you to count tool calls or edit sizes separately. Those are just proxies for efficiency, which the token numbers already capture.


Procedure

The plumbing is provided so you can focus on running, measuring, and thinking. Everything ships in the assignment-03/ starter bundle — start by reading its README.md, which has the quickstart and a map of every file. The bundle includes: spec.md, the hidden tests/, requirements.txt (pins the exact Aider version), run_aider.sh, reset.sh, verify.py, collect.py, a one-page AIDER-PRIMER.md, AIDER-SETUP.md, a STUDENT-AGENT-CHECKLIST.md for configuring your own agent, and a ready-to-run reference agent in student-agent/.

  1. Install. pip install -r requirements.txt (this pins Aider and pytest).

  2. Write the shared prompt. In prompt.txt, write the single prompt both agents will receive. Treat this as the one knob you control. (See the prompt-iteration loop below.)

  3. Configure your agent per the provided checklist: set its model to the pinned Haiku id, point it at a clean workspace_mine/, make sure it cannot see tests/, and have your TokenTracker from Assignment 2 export a mine_tokens.csv, and choose which context-management strategy from Lab 3 to run (the one knob from Lab 3 you carry in) (a one-line addition — schema is provided).

  4. Run each agent once on the final shared prompt: your agent into workspace_mine/, Aider into workspace_aider/ via run_aider.sh. Use reset.sh to start each from a clean state.

  5. Measure. Run verify.py on each workspace (tests passed) and collect.py (token table).

  6. Write the report.

The prompt-iteration loop (this is the interesting part)

Your agent is weaker than Aider. Your first prompt will probably leave it far behind. Iterate the shared prompt — make the spec clearer, add structure, give better guidance — and re-run your agent until it performs as well as you can get it to. Then run both agents on that final prompt.

The point is to discover where prompting helps and where it cannot. A better prompt may take your agent from 8 passing tests to 20 — but it may still not reach Aider's 28, because some of Aider's advantage is structural to the harness and no prompt can close it. Document this journey; it is one of the richest parts of your analysis.

You only need one measured run per agent on the final prompt — because the prompt is fixed and identical, additional trials would only average over a variation you have deliberately removed. (If you observe meaningful run-to-run variance, note it; it is a bonus observation, not a requirement.)


What Your Report Must Contain

1. Setup and Controls

State the model id both agents used, confirm Aider's weak-model was pinned to the same model, and include the final prompt.txt in your appendix — the single file you fed to both agents. A reader must be able to confirm your three controls held.

2. The Results

The comparison table from collect.py (tokens and cost) alongside the verify.py results (tests passed), for both agents. State the headline plainly: how many tests each passed, and how many tokens each spent.

3. The Shared Prompt and Its Evolution

Show how your prompt evolved and what each change bought you. At minimum:

4. The Harness Explanation

This is the analytical core. The model and prompt were identical, so explain the differences you measured by naming specific harness mechanisms. Use the provided AIDER-PRIMER.md and your own logs. Strong explanations look like:

"Aider used 60% fewer peak input tokens because its repo-map feeds the model a compressed symbol map of the codebase instead of reading whole files into context, the way my agent does."

"My agent ran out of usable context by the fifth module and began re-implementing functions it had already written, because my sliding window dropped the earlier files; Aider's repo-map kept a persistent summary, so it never lost the structure."

Weak explanations name no mechanism ("Aider is just better") or attribute the difference to the model (it cannot be — the model was identical).

Cover both metrics: explain the token difference and the tests-passed difference, each tied to a concrete harness feature (Aider's or your own).

5. Synthesis

Tie it together in answer to the assignment's central question:


Asymmetric Results Are Valid

Your agent may lose badly. Aider may pass every test while yours passes half, on the same model and prompt. That is the lesson, not a failure — it is direct evidence of how much the harness contributes — and it earns full credit as long as you measure it cleanly and explain the harness reason. Likewise, a finding that prompting closed most of the gap is just as interesting as a finding that it closed almost none. The grader is looking for clean measurement and rigorous attribution, not for your agent to win.


Submission

assignment-03/
├── report.pdf
└── experiments/
    ├── prompt.txt              # the final shared prompt (identical for both agents)
    ├── mine_tokens.csv         # your agent's per-call token log
    ├── aider_llm.log           # Aider's raw call log (from run_aider.sh)
    ├── collect_output.txt       # the token comparison table
    ├── verify_mine.txt         # tests passed by your agent
    ├── verify_aider.txt        # tests passed by Aider
    ├── workspace_mine/         # the program your agent produced
    └── workspace_aider/        # the program Aider produced

Your report must be self-contained — a reader should understand your findings without opening the appendix. The experiments/ folder exists so a grader can verify your numbers.


Rubric (20 points)

Component Points Criteria
Setup & control integrity 3 Both agents ran on the identical model and the same submitted prompt.txt; tests were never exposed to the agents; data collected cleanly
Measurement & data 4 Both metrics reported correctly for both agents; token definitions correct; results clearly presented
Prompt evolution analysis 4 Shows what prompting improved on your agent and gives an honest, reasoned account of what it could not fix
Harness attribution 6 Explains both the token and tests-passed differences by naming specific harness mechanisms; never attributes a difference to the model; grounded in the logs and the primer
Synthesis 3 Answers the central question quantitatively; defends a single highest-value mechanism to adopt; surfaces any axis where the minimal harness wins

Total: 20 points.

A note on grading: the most common path to a low score is a broken control — different models (forgetting Aider's weak-model), or feeding the two agents different prompts — which invalidates the comparison. The most common path to a high score is naming the mechanism: tracing a measured difference to a specific, identifiable feature of the harness rather than waving at "Aider is more advanced."


What You Will Learn

By the end of this assignment, you should be able to:

These are the judgments you will rely on for the rest of the course, as you add RAG, memory, and skills to your agent — each of which is, fundamentally, another piece of harness wrapped around the same model.