Chapter 6 — The Theory  ::  §6.4

§6.4 Research Agenda

Testing MAGE requires measuring how representations, model capability, and engineering controls interact across tasks and over time. This section turns that requirement into a research program: what to measure, which comparisons could distinguish the proposed mechanisms, and what evidence would strengthen—or weaken—the account.

The equations in Chapter 6 state relationships to be measured, not fixed laws. The nine hypotheses collected in §6.2 identify quantities and relationships that empirical studies can test.

The hypotheses do not require nine unrelated studies. A small set of experimental designs covers much of the program.

Six questions carry the program, and two of them recur from §6.1: the vendor-factory question and the determinization frontier.

• How much representation can stronger reasoners economically replace?
• When does explicit representation beat repeated inference?
• Which properties can representation innovation move across the determinization frontier?
• How quickly does engineering capital depreciate?
• When does governance pressure produce adaptation rather than bureaucracy and control friction?
• Does increased capability improve outcomes more strongly in high-quality environments than in weak ones?

6.4.1 Parameterizing the Model

The theory's equations leave their parameters unspecified. A research program can ask what determines them.

Let $T$ denote an engineering task or task family, $M$ the reasoning model, $R$ the representation available to it, and $H$ the harness and process through which production is organized. In the vocabulary of Chapter 5, $H$ captures much of process design: context assembly, tools, decomposition, routing, permissions, coordination, and other machinery surrounding the fabricator. For a fixed engineering environment, write

$$p(T,M,R,H)$$

for the probability that one attempt produces an acceptable candidate, and

$$c(T,M,R,H)$$

for the expected cost of that attempt. Cost need not mean inference cost alone. Depending on the study, it can include tokens, compute, latency, tool use, realization, validation, and human intervention.

For sustained work, we reuse the dynamic formulation from §6.1. The environment itself becomes a state variable: writing ${\mathcal{G}}_i$ for the environment inherited at step $i$,

$$p_i=p(T_i,M,R_i,H,{\mathcal{G}}_i),{\mathcal{G}}_{i+1}=f({\mathcal{G}}_i,a_i)$$

Here the equations serve a different purpose. §6.1 used them to express path dependence; the research problem is to measure how much that dependence matters in practice and which features of the environment drive it.

Alignment introduces a related quantity. For an Alignment strategy $A$, let

$$q(T,R,A)$$

denote the probability that an unacceptable candidate escapes the mechanisms enforcing the relevant obligation. As §6.1 showed, it is zero for a sound deterministic gate over a fully covered decidable obligation, and weaker where validators are statistical, coverage is incomplete, mechanisms are defective, or the obligation is semantic.

What experiments would estimate

Agents do not have one fixed reliability or cost. The question is how these quantities vary across engineering tasks, model capability, representation, harness and process, and Alignment strategy, and how those factors interact.

Modeling and Alignment can produce similar observed improvements in final system quality through different mechanisms, so experiments must separate them. A stronger model may increase $p$. A better representation may increase $p$, decrease $c$, or both. A longer reasoning loop may increase the probability of eventually finding an acceptable candidate while increasing total cost. An independent gate may leave the producer's $p$ unchanged while decreasing $q$.

The decomposition refines the measurand. The Master Equation of §6.1 — encoding, interpretation, realization — refines what a study measures inside $p$. A factorial design can cross representation quality, reasoner capability, harness and process, realization task, and Alignment mechanism, then estimate $P(E\mid R)$, $P(I\mid E,R)$, $P(L\mid I,E,R)$, $q$, and $c$ separately rather than collapsing them into one aggregate success probability. This decomposition also supports a falsifiable prediction. Some representations should outperform natural language even with the same reasoner because engineers encode consequential intent into them more reliably. The human is one of the channels whose reliability MAGE is trying to improve.

The goal is to determine when capability, representation, harness and process, iteration, and enforcement substitute for or complement one another. Experiments can ask:

• Which factors systematically move $p$, $c$, and $q$?
• How do those effects vary by task family?
• When does representation substitute for model capability, and when does it complement it?
• When does dependence through the evolving engineering environment materially change retry cost, defect risk, or durable throughput?
• How much assurance can independent mechanisms provide without increasing producer capability?
• Does a structural representation improve a reasoner's interpretation of intent that was already correctly expressed — raising $P(I\mid E,R)$ independently of $P(E\mid R)$?
• Does structured elicitation raise the probability that humans express the intended obligation at all?
• At what repeated-use count does an explicit model become cheaper than repeated natural-language reconstruction?
• Where does a change of representation make a formerly semantic obligation mechanically decidable?
• Which process-design interventions primarily change interpretation, realization, cost, or several at once?
• When does representing a consequential property as an independently evaluable tolerance reduce dependence on producer reliability?

Crossed experiments can answer these questions: vary reasoning model and engineered environment over matched task families, then isolate representation and Alignment while holding the other fixed. Measurements of attempts, inference and tool cost, latency, intervention, admission, defect escape, and durable throughput can distinguish capability effects, representation leverage, retry economics, and independent assurance rather than collapsing them into one measure of agent performance.

Note that a detailed natural-language task description is itself a model of intended behavior. Adding a more formal architectural or behavioral model may provide little benefit when the agent can already reconstruct the relevant relationships from that description and the repository. Experiments should therefore compare representations against strong natural-language baselines, measuring their marginal effects on realization cost, assurance, and subsequent engineering work. Explicit engineering models are most likely to be valuable when consequential relationships are expensive to reconstruct, recur across tasks, or support independent checks.

These experiments measure how representation helps realization, not its full engineering value. As model capability improves, a representation may contribute less to the cost or probability of producing an acceptable change: the reasoner may reconstruct the relevant relationship cheaply and reliably without it. The same representation may still matter for retaining engineering control by giving engineers and independent mechanisms an inspectable account of consequential system knowledge. Chapter 7 takes up that second value directly.

6.4.2 Engineering Routing

Task difficulty need not be treated as fixed. Engineering can change the representation of a problem before selecting the model that will solve it. Conventional model routing assumes the opposite: it treats the task as given, estimates its difficulty, and selects a model whose capability and cost are appropriate to it.

Suppose a migration appears to require a frontier model to inspect hundreds of service configurations and reconstruct their dependencies. A trustworthy dependency model may turn the same work into a much smaller graph problem that a cheaper reasoner can handle reliably. Changing an authentication boundary may warrant a different strategy: the representation can still simplify the reasoning, but the consequence may justify the stronger model, additional evidence, and an independent admission mechanism as well. Engineering changes both the difficulty of the task presented to the reasoner and the assurance required around its result.

A difficult code-level reasoning task may become a simpler graph, state-machine, provenance, or constraint problem when an appropriate representation exposes the consequential relation directly. Better context or tooling may change the cost of reaching the answer. Iteration may purchase additional chances. Alignment may allow a less reliable producer to operate safely because an independent mechanism prevents covered failures from acquiring consequence. The model is therefore only one part of the realization strategy.

For an engineering task $T$, an automated system could in principle select the model $M$, representation $R$, harness $H$, Alignment strategy $A$, and retry budget $k$ jointly:

$$(M^{*},R^{*},H^{*},A^{*},k^{*})=\underset{M,R,H,A,k}{arg\hspace{0.17em}min}E[C_{\text{total}}\mid T,M,R,H,A,k]$$

subject to an engineering assurance requirement

$$P(\text{unacceptable outcome escapes}\mid T,M,R,H,A,k)\le {\epsilon }_T$$

The task-specific bound ${\epsilon }_T$ matters. Renaming a local variable, changing a user-visible layout, modifying a billing calculation, and altering an authentication boundary need not warrant the same assurance strategy. Routing should reflect the obligations and consequences of the work, not only its apparent implementation difficulty.

Call this engineering routing. A model router asks which reasoner should receive the task. An engineering router asks how the task should be engineered for realization: what the reasoner should reason over, which tools and context it should receive, how much iteration is economical, what evidence should be produced, and which independent mechanisms should govern admission. In the terms of the Master Equation (§6.1), it chooses where to spend: on representation, process design, capability, or evidence and enforcement.

A suitable representation can lower the capability a task requires; conversely, a strong model may make additional Modeling uneconomical for a simple or infrequent task. The best choice depends on the task, available capability, environment, assurance requirement, and recurrence.

The research implication is concrete. Measuring $p$, $c$, and $q$ across task families and engineered environments could show when each strategy is preferable. The question becomes not merely how difficult is this task for a model? but how cheaply can we engineer this task to be safe to delegate?

6.4.3 Representation Selection, Induction, and Innovation

Choosing the right representation is itself an engineering problem. Different representations expose different properties, support different analyses, and impose different costs. The six model classes of Chapter 2 were offered as useful, not proven necessary, sufficient, or minimal.

A representation is useful only relative to the questions and consumers it serves. A sufficient representation family preserves enough information to answer the declared engineering questions and govern the declared obligations to the required standard; a minimal useful family removes representations whose added reasoning, assurance, or independence benefit does not justify their carrying cost. Mathematical minimality is not the target, since a strictly minimal set may be difficult for engineers to reason about. We can test redundancy in principle: two representations that encode overlapping information can both be valuable when one serves a different question, reasoner, or independent assurance path, while a representation that changes none of the relevant answers, governable obligations, independent evidence, or reconstruction cost may be carrying cost without leverage.

A dependency graph makes the test concrete. Suppose engineers repeatedly reconstruct permitted service dependencies from handlers, configuration, and code. Adding an architectural graph costs something to create, reconcile, and maintain—but if later engineers and agents can answer the dependency question directly, and a validator can use the same representation to reject forbidden edges, the reconstruction and residual-assurance cost falls. A second representation that answers no additional question and adds no independent evidence adds cost without leverage.

Chapter 4 already treated representation induction as a practical engineering move. An existing implementation can contain stable structure that no explicit model names. Static analysis, clustering, repeated failures, architectural context, and GenAI-assisted interpretation can help recover that latent structure and turn it into an explicit representation. This is not representation innovation in the stronger sense: the induced model makes explicit consequential structure that the existing system already contains.

A stronger research question is whether commodity intelligence can innovate representations. Given an engineering system and a set of recurring questions, failures, and decisions, can a reasoner propose a representation that makes important relationships tractable even when that representation was neither supplied in advance nor plainly latent in the implementation? Such a system might propose a new decomposition, intermediate state, relation, abstraction boundary, or combination of existing model forms because that representation better supports the engineering questions being asked. Together, induction and innovation form the ordered claim of H4: recover what is latent, then propose representations the existing set does not supply.

The problem can also be framed as learning. Given a distribution of engineering questions, failures, and decisions, one could ask which representation preserves the distinctions needed to answer them while suppressing irrelevant state—the question representation learning asks in machine learning. MAGE brings a strong engineering prior: software and systems engineering have spent decades developing abstractions for architecture, state, ownership, interfaces, requirements, quantities, provenance, and assurance, and the experience in this book suggests many are also unusually usable by foundation models. A complementary direction is to train models specifically for engineering abstractions,¹1. Safayat Bin Hakim et al., “Neuro-Symbolic Agentic AI: Architectures, Integration Patterns, Applications, Open Challenges and Future Research Directions,” Computer Science Review 60 (2026): 100902, https://doi.org/10.1016/j.cosrev.2026.100902.²2. Miltiadis Allamanis et al., “Learning to Represent Programs with Graphs,” in International Conference on Learning Representations (2018).³3. Linhao Luo et al., “G-Reasoner: Foundation Models for Unified Reasoning over Graph-Structured Knowledge,” in International Conference on Learning Representations (2026). which might let smaller models perform reasoning that currently requires expensive general-purpose intelligence. Where only machine reasoners and trustworthy mechanisms use a representation, it might trade human readability for compactness or stronger mechanical analysis, provided the results can be shown to humans where human judgment resumes. The open question is not whether learned representations should replace engineering models, but which representations best divide the reasoning among humans, agents, and mechanisms for the question at hand.

An experiment can distinguish induction from innovation — the test of H4. Compare expert-authored, machine-induced, and machine-innovated representations against both the raw implementation and one another. For induction, measure whether the proposed model faithfully recovers consequential structure already present in the system. For innovation, use held-out engineering questions and failures to test whether the proposed representation exposes useful structure that the existing representations did not. In both cases, measure reasoning success, reconstruction cost, correspondence failures, carrying cost, human comprehensibility, and benefit to later humans and agents.

6.4.4 Experimental Designs

Separating Modeling from Alignment

A factorial design can cleanly separate Modeling from Alignment. Hold the task, reasoning engine, and admission mechanism fixed while varying whether the producer receives the structured representation used by the environment. If Alignment provides the claimed independence, covered defect escape should remain unchanged; if Modeling provides the claimed leverage, attempts, inference cost, latency, or intervention required to reach admission should fall when the representation is available. This design tests H1 and H5 directly.

Multi-project longitudinal design

This design should vary or observe engineered-environment quality while tracking durable throughput, defect escape, and human-attention burden over weeks or months. The strongest version is a crossed experiment: several reasoning engines from weaker to frontier, several otherwise comparable environments from thinly to richly engineered, and the same task families across every cell. The results would distinguish three possibilities. If richer environments primarily substitute for raw model capability, weaker models should close part of the gap in stronger environments. If the two are complementary, the strongest models should gain most from strong environments. If environment quality contributes little, the cells should move mainly with model capability. This design directly tests H1, H8, and H9.

The crossed experiment should also measure where human work occurs and how much productive capacity resides outside the agent. Hold the task family and reasoning engine fixed, vary environment quality, and record how much human attention remains across coding, debugging, test construction, review, environment setup, integration, recovery, and other activity classes, alongside task scale, reconstruction cost, intervention burden, defect escape, and durable throughput. A richer environment that reduces only code-production time would support the narrower productivity account; one that moves several classes of realization work out of repeated human execution while maintaining quality would support MAGE's broader account. Kumar et al.’s sixteen activity classes provide an existing measurement vocabulary, avoiding the need for another MAGE-specific taxonomy.⁴4. Sukrit Kumar et al., “Time Warp: The Gap between Developers' Ideal Vs Actual Workweeks in an AI-Driven Era,” in Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) (2025), https://doi.org/10.1109/ICSE-SEIP66354.2025.00007.

Subsystem-level design

Compare engineering surfaces that differ in representation and governance while holding as much surrounding repository context constant as possible. Track model correspondence, reconstruction effort, intervention, churn, and defect escape. Stepped introduction of a model or mechanism is particularly useful because the same subsystem supplies its own pre-intervention baseline. This tests whether representation leverage (H1), determinization (H3), and mechanized assurance (H5) appear where the theory predicts, rather than merely accompanying well-run projects; the correspondence tracking bears directly on H2.

Event-centered longitudinal design

Follow one failure class through its whole life: first occurrence, repeat exposure, structural recognition, local repair, any durable conversion, propagation to later work, later recurrence, and eventual maintenance or retirement of the resulting asset. This is the strongest way to separate "the engineer got better" from "the environment learned," and it tests H6 directly. Applied across organizations, it also tests H7: whether authority pathways determine whether a local diagnosis becomes shared engineering capital.

What the designs still need

All of these designs require credible measures of durable throughput, human-attention burden, environment quality, and engineering-capital return. Until those are available, support ratio, mechanism count, and repository size should remain descriptive rather than proxies for maturity or value.^** Candidate measures such as redundant-encoding density and agent-legibility remain exploratory and are not part of MAGE's canonical vocabulary.

Commodity intelligence may make some of these experiments unusually tractable: the reasoning engine, repository state, task family, and engineered environment can be held approximately fixed while one representation or mechanism varies.

Finally, whether the theory transfers beyond software engineering is an open empirical question. §6.3 named what a domain must supply; this book does not test that generalization. Chapter 7 gives several concrete examples, and testing MAGE beyond software engineering remains part of the research agenda.

Works Cited

1. Hakim, Safayat Bin, Muhammad Adil, Alvaro Velasquez, and Houbing Herbert Song. “Neuro-Symbolic Agentic AI: Architectures, Integration Patterns, Applications, Open Challenges and Future Research Directions.” Computer Science Review 60 (2026): 100902. https://doi.org/10.1016/j.cosrev.2026.100902.
2. Allamanis, Miltiadis, Marc Brockschmidt, and Mahmoud Khademi. “Learning to Represent Programs with Graphs.” In International Conference on Learning Representations. 2018.
3. Luo, Linhao, Zicheng Zhao, Junnan Liu, et al. “G-Reasoner: Foundation Models for Unified Reasoning over Graph-Structured Knowledge.” In International Conference on Learning Representations. 2026.
4. Kumar, Sukrit, Drishti Goel, Thomas Zimmermann, Brian Houck, Balasubramanyan Ashok, and Chetan Bansal. “Time Warp: The Gap between Developers' Ideal Vs Actual Workweeks in an AI-Driven Era.” In Proceedings of the 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). 2025. https://doi.org/10.1109/ICSE-SEIP66354.2025.00007.

© James C. Davis, 2026–present