§6.4 Research Agenda
The next step is to find out when these mechanisms matter, by how much, and at what cost. The central experimental difficulty is to distinguish improvements in the reasoning engine from improvements in the engineering environment through which it works. Three research directions organize that problem: representation engineering, governance and engineering capital, and the interaction between capacity and environment.
Table 6.4-1 collects four useful comparisons. Each holds the proposed move against an alternative that could plausibly work. Independent acceptance and the lifecycle accounting of §6.1 provide a common basis: include failed work and upkeep, report unresolved requests, and compare delivery under declared quality requirements and evaluation horizons.
| Question | Comparison | What to measure |
|---|---|---|
| Representation (P1) | Structured representation versus prose containing the same obligations | Correctness and reasoning cost across task families and agent capabilities |
| Assurance (P2) | Independent model-derived checks versus existing checks | Escape and false rejection, alongside correct delivery |
| Conversion (P3) | Corrections retained as instructions versus conversion into durable controls | Later capability, recurrence and cost per correct delivery, including construction and upkeep |
| Capacity × environment (P4) | Different model capabilities crossed with different engineering environments | Durable throughput, defect escape, and human attention |
6.4.1 Representation Engineering
To test representation leverage, compare agents given structured models with agents given prose containing the same obligations. Hold the task, foundation model, tools, and admission mechanism fixed. Measure correct delivery, reasoning cost, and the human effort required to interpret or repair the result. This isolates the effect of representation form on fulfillment. A separate elicitation experiment should test whether structured representations help engineers discover or express obligations they would otherwise omit, rather than assuming that every treatment begins with an equally complete specification.
The admission mechanism should also be varied independently. With a sound gate, covered violations cannot enter admitted work regardless of whether the producing agent receives the model, but the representation may reduce the number and cost of rejected attempts. With imperfect checks, compare escaped defects, false rejection, and correct delivery under each representation condition. Crossing representation and admission mechanism in a factorial design would distinguish their individual contributions and reveal whether they reinforce one another.
A second set of experiments should compare representations designed by engineers, recovered from existing implementations, and proposed by agents as new abstractions, using raw implementation as a common baseline. The six model classes in Chapter 2 were offered as useful, not proved necessary, sufficient, or minimal. Recovery should faithfully expose consequential structure already present in the system. Innovation should make useful relationships available that the existing representations did not.
For either route, discovery is only the beginning of evaluation. Use held-out engineering questions and failures to test usefulness; assess claimed correspondence independently; measure reconstruction cost, reasoning success, human comprehensibility, and maintenance. A representation that solves the demonstration question but misleads later work has not established leverage. The comparison should also ask whether any added structure is more economical to retain or to reconstruct when needed.
Representation studies should also examine whether the resulting models improve human engineering decisions, a question taken up in the review experiment below.
6.4.2 Governance and Engineering Capital
Governance research asks which interventions leave capacity that later work can actually inherit. One practical design follows a recurring failure class from first occurrence through diagnosis, local repair, any durable conversion, later reuse, and eventual maintenance or retirement. Record what the intervention enables as well as which failures recur: a model may make larger changes tractable, and a validator may make independent operation possible.
That history can show when later work began to inherit an intervention, but a decline in recurrence alone cannot establish its cause. The engineer may have become more experienced, or the work may have changed. Comparing similar work that received only local repair, or observing later work without the originating engineer, helps distinguish these explanations. Across organizations, the same design can examine how authority connects a local diagnosis to changes in the shared environment.
The economic comparison must include construction, carrying cost, and the work spent reconciling or retiring assets that lose fit. Several assets may contribute jointly to a result: a model, a typed operation, and an admission check can reinforce one another. A credible return estimate needs an explicit allocation or a comparison of the bundle, rather than assigning the entire gain to whichever asset was added last. The aim is to learn when inherited capacity pays and how organizations can preserve that value without merely accumulating controls.
6.4.3 Capacity and Environment
A factorial study can vary foundation-model capability and GEE quality independently over matched task families. GEE conditions should differ in specified representations, controls, or evidence pipelines, with their adequacy assessed independently of the measured outcomes. The study should measure correct admitted work, escaped defects, human intervention, and lifecycle cost over a common horizon. Interactions would show whether improvements to the GEE substitute for model capability, complement it, or contribute little under the tested conditions. Repeating the comparison as tasks and GEEs evolve would test whether the observed effects persist.
For example, two engineering environments could operate on the same codebase and task population, with the same foundation model and production tools. One could provide a validated dependency model and an admission check covering a declared set of architectural obligations; the other could rely on existing source files and ordinary review. The experiment would assess the model’s correspondence and the check’s coverage independently, before measuring delivery outcomes. Both conditions would use the same independent outcome assessment. This tests a specific engineering investment without assuming that GEE quality can be reduced to a universal score.
I expect the marginal value of a stronger reasoner eventually to be dominated by the value of a better engineering environment. Indeed, I suspect we have already reached that point for some classes of software-engineering work. A more capable agent may implement a change faster, but if the surrounding environment cannot preserve intent, supply adequate evidence, and govern acceptance, the additional capacity may produce little durable progress. The comparison is not between intelligence and engineering, but between investing in more intelligence and investing in the conditions under which intelligence becomes useful.
Human review deserves a direct comparison within this design. Give engineers the same review-time budget, but vary whether they begin with code or with changed models and verification evidence, retaining access to code when needed. Include cases where a passing check rests on an incorrect obligation or stale evidence; a reassuring display can make the wrong decision easier. As agent output increases, measure whether engineers still distinguish acceptable changes from faulty ones and how much correct work is admitted. Faster approval alone would not support P4 if more violations enter the system.
6.4.4 Optimizing the Engineering Environment
These comparisons could eventually inform engineering routing: choosing how a task should be engineered for realization, including its representation, foundation model, tools, iteration, and admission mechanisms. A routing decision may favor a stronger foundation model, a cheaper foundation model with better structure, or a deterministic operation for a sufficiently specified task. Across repeated work, the decision also concerns what to invest in the GEE. The engineer must allocate resources among reasoning, representation, implementation, and assurance while accounting for the environment that later work will inherit.
Let be an engineering policy that selects a production arrangement and any investment in the GEE for each task. Its choices depend on the inherited environment, the available evidence, and current conditions, including the capabilities and prices of available foundation models. Let be the set of policies available to the engineer. For a declared task sequence and evaluation horizon of engineering episodes, let denote the total cost incurred in episode : production, checking, repair, valued human attention, and investment or maintenance attributable to that episode. Charge initial construction to the first episode and allocate shared costs once across the horizon. One formulation of the investment problem is
Here counts independently assessed correct admitted delivery and is the minimum required delivery. is total human attention consumed over the horizon, measured in a declared unit such as engineer-hours, and is the available attention budget. The expectation is over the outcomes under policy . Human attention is both a valued cost and a limited resource; its constraint does not add a second charge. The condition represents the declared assurance obligations, which may combine invariants, probabilistic failure bounds, and required human authorization. These obligations must also specify which requests or task classes require delivery when an aggregate count would permit the policy to select only easy work. The positive delivery minimum rules out rejecting everything; coverage requirements address selective avoidance.
The final relation retains the schematic inheritance model: episode work , including any engineering investment, changes the GEE available to later work. The policy generalizes the earlier fixed-foundation-model comparison by allowing the reasoning engine and production arrangement to change between episodes. This is an illustrative problem formulation; particular settings will require their own assurance conditions, resource budgets, and valuations of useful delivery.
We do not know the optimal policy. Cost and transition functions remain uncalibrated, and the effects of representations and controls depend on the tasks. Foundation-model capabilities and prices keep changing. Better agents may make some maintained representations unnecessary by reconstructing their information cheaply. They may also make other representations economical to build and maintain. Requirements, dependencies, and organizational authority change as well. An engineering arrangement that is economical today may not remain so.
The formulation gives the research agenda a decision problem to address. The propositions identify mechanisms that can affect its objective and constraints; the proposed experiments can begin estimating those effects. Such evidence may eventually support policies that choose where to invest in durable structure, where to rely on probabilistic reasoning, and where human judgment remains necessary. The economically appropriate division of labor is an empirical question whose answer can change as the available technology and work change.
These experiments would establish effects only for the tasks, agents, and engineering environments studied. Replication across organizations and task families would show how far the relationships extend. Testing them outside software would require additional evidence that the relevant obligations can be represented and evaluated adequately, as the scope conditions above suggest.