§5.3 Evaluating the Product and Factory
The engineering machinery described in the preceding section made substantial delegation possible. Its value, however, depends on the software it produces and the cost of producing and maintaining it. DocAble must improve document accessibility while preserving the properties its users depend on; the factory must accomplish that work economically. Purdue’s undergraduate remediation effort provides a practical reference for evaluating the resulting documents.
5.3.1 Human Remediation as a Reference
Across approximately six months, 42 trained undergraduates billed 1,303.4 hours, and the recovered corpus contains 382 identifiable before/after pairs from 20 courses. That is 3.41 billed hours per pair. At an assumed typical wage of $13, estimated direct labor is $44.36 per pair. These are corpus-level quantities; individual document timings and subsequent review effort were not measured 1.1. James C. Davis, DocAble: Automated Document-Accessibility Remediation, Good Fast and Cheap, SEIP manuscript, October 28, 2026, 2026.
The student outputs document an institutional workflow rather than expert-verified accessibility. I developed DocAble against standards using independently obtained documents and did not examine the students’ remediated files until near the end of development. Eligibility screening yielded 315 pairs, randomly split by document and stratified by format into 252 development documents and 63 validation documents. One validation spreadsheet had prior development exposure. DocAble produced outputs for 62 attempts and failed on one PDF.
The evaluation in Figure 5.3-1 compared the original, human-remediated, and DocAble-remediated versions of each document along two dimensions. Syntactic accessibility concerns selected mechanically testable properties, measured with veraPDF’s PDF/UA-1 checks and Microsoft Office’s October 2026 Accessibility Checker. For each document, the study calculated the signed reduction in findings relative to the original, then summarized those reductions by format. A reduction does not establish that every accessibility obligation holds.
Semantic accessibility asks what information the document conveys. A GPT-5.2 rubric assessment examined text transcripts, with the original identified and the two outputs presented under randomized labels concealing their provenance. The model judged individual criteria and supplied transcript evidence. A deterministic scorer credited improvements and penalized regressions relative to the original; an output-score difference within one point defined the parity band. The rubric was refined on the development partition before the validation run.
DocAble achieved larger median checker reductions for PDF and DOCX; the human workflow achieved larger reductions for PPTX. Both workflows had the same frequency of increased finding counts. The spreadsheet had no original findings, so its percentage reduction was undefined. The failed PDF’s partial artifact was excluded from completed-output comparisons.
Across the 62 scored outputs, the semantic rubric rated DocAble better on 20, within the parity band on 41, and worse on 1. Gains in complex-image descriptions accounted for much of its advantage. These are weighted document scores; the gain/regression panel separately counts document–criterion judgments, not unique defects.
One presentation makes the distinction concrete. It contained an image of a thermochemical table. The human-authored alternative text repeated the table’s title, which satisfied a checker’s requirement for alternative text but did not communicate the entries. DocAble supplied a description and a transcription next in the reading order. In the same presentation, the human version transcribed mathematical content that DocAble missed. These author-inspected examples explain disagreements between the measures; they are not systematic expert adjudication.
The semantic assessment covered only part of the intended rubric. Of 3,392 criterion-level rows, 2,467 were inconclusive for DocAble, including 1,617 whose automatic adapters were pending. Inconclusive judgments contributed no points; they were not evidence of satisfactory accessibility. Two scoring passes agreed on 58 of 62 document classifications, but repeatability does not establish agreement with accessibility experts.
The evaluation did not include vision-tier or screen-reader assessments, nor systematically measure complete content preservation or visual fidelity. The Office checker had also changed since the students’ summer work, although all versions were evaluated with the same current tools. The findings show improvements on measured properties and identify remaining weaknesses. They do not establish full WCAG conformance or equivalent accessibility for outputs within the parity band.
5.3.2 Runtime and Cost
Median elapsed time across all 63 attempts, including the failed attempt, was 1.36 minutes. The 62 completed outputs had a median inference expenditure of $0.066, with a measured range of approximately $0.001–$0.440. Full cloud execution costs were not measured for this cohort. The paper separately reports coarse operating-cost estimates; those should not be mistaken for the measured distribution 1.
These quantities show that the specialized system can produce remediation outputs quickly and with little inference expenditure. The human reference required 3.41 billed hours per document pair, compared with 1.36 minutes of unattended DocAble runtime. These are different measures of production effort. Estimating labor savings would additionally require measuring the competent review and correction needed after automated remediation and human remediation. To the best of my knowledge, the faculty whose courses were manually remediated have not been surveyed regarding quality of the result and the need for subsequent changes. As to DocAble's outputs, so far, faculty describe a few up to tens of minutes of follow-up per output.
5.3.3 General-Purpose Agents and a Specialized Service
Could a general-purpose agent perform the remediation directly? A separate comparison in the DocAble study used community document tools, unlike the engineered DocAble operations reused in Substituting the Reasoning Engine. Opus and Fable each used one configuration on ten validation documents: five PDFs, two Word documents, two presentations, and one spreadsheet. Opus and DocAble completed all ten; Fable timed out on one PDF.
Quality varied by format. On the four PDFs completed by every configuration, DocAble had the largest median reduction in checker findings. Both agents exceeded DocAble’s reductions on the two Word documents, while human remediation led on the two presentations. Median runtime was 8.22 minutes for Opus, 7.77 for Fable, and 1.31 for DocAble; median agent inference costs were $2.466 and $3.977, compared with less than $0.50 for DocAble. These costs concern this ten-document cohort, not the larger cohort underlying the $0.066 median above.
The result concerns particular configurations, community tools, and automatic findings. It does not establish comparable semantic quality or a lasting capability boundary between a specialized service and a general-purpose agent. The engineering problem includes acquiring a useful rubric, providing suitable tools, and delivering a workflow users can operate. A specialized service packages that accumulated work and can reduce the cost of applying it repeatedly; its advantage depends on the quality of that work, operating economics, and usability.
5.3.4 What the Economics Establish
A factory’s startup cost, its marginal change cost, and its carrying cost are different numbers — a separation §6.1 develops formally. The DocAble study estimates approximately $110,000 to develop DocAble over six months: $90,000 in engineering labor and $20,000 in cloud expenditure. These are author-supplied estimates, not an independently audited ledger 1. Much of that expenditure purchased reusable capability. Dividing it by the documents processed during development would confuse the cost of building the factory with the cost of using it.
The alternatives incur costs in different places. Manual remediation purchases labor for each document. General-purpose agents need a rubric, tools, and supervision. A specialized service incurs development and maintenance costs to make the recurring workflow easier to run. Cheap output production is therefore only part of the return on the factory. Establishing that the outputs serve their readers, integrating the service into institutional work, and maintaining it remain engineering and economic obligations.
5.3.5 Limitations and Maintenance Costs
The engineering machinery that made delegation possible could also mislead the agents it was meant to support. One model named a code symbol that was later moved and renamed. Nothing invalidated the reference; the model continued to look authoritative, agents continued to reason from it, and the failure surfaced days later during deployment. A subsequent audit showed that the stale pointer was not an isolated defect: re-running closed work against current code found genuine model-to-code drift that a green definition of done had missed. This incident motivated correspondence checking between models and implementation. The factory needed machinery not only to preserve engineering knowledge, but also to establish that its representations remained connected to the system they purported to describe.
The apparatus requires maintenance of its own. Models drift, controls conflict, and evidence becomes stale as workloads change. A gate can continue enforcing an obsolete assumption long after the incident that justified it. Correspondence checks help detect some of these failures, but must themselves be operated and repaired. At the final snapshot, supporting source was about 3.8 times the production code. Its size is a liability as well as an investment; some of it will cease to justify its upkeep.
The evidence has clear limits. This is one longitudinal production case led by one primary engineer directing an agent fleet, using one contemporary model ecosystem, without a controlled comparison against another engineering process. It cannot establish that another organization will encounter the same failures, build the same mechanisms, or achieve the same economics — and checker scores do not establish complete end-user accessibility. What the case can establish is mechanism and sequence: these pressures occurred, these responses followed, and selected structures persisted long enough to be exercised under continuing change.
Appendix B offers a practical starting point for applying these techniques, including brownfield migration, operation of a governed engineering environment, and measurement.
DocAble provides evidence about one factory’s development, operation, outputs, and costs. It shows how an engineer can construct machinery that permits substantial delegation while preserving selected engineering obligations, but also how that machinery can fail or become expensive to maintain. These observations do not establish that other organizations need the same arrangements. They do, however, raise a question worth examining: when agents increase implementation capacity in other software organizations, what changes in the surrounding production system?
Works Cited
- Davis, James C. DocAble: Automated Document-Accessibility Remediation, Good Fast and Cheap. SEIP manuscript, October 28, 2026, 2026.