Migrating a data warehouse can take years and cost millions. Building a new platform can demand a similar investment. For the organisation funding the work, that means a long wait for capabilities it needs, while people and budget remain committed to delivery.
I have spent more than 25 years building, transforming and migrating data platforms and warehouses. The work has always been difficult and costly. Many enterprises have lived through enough of these programmes to know how much effort sits behind a new platform going live.
AI has given this familiar problem a new significance. Businesses want to put agents to work, and those agents need dependable data they can find, understand and use appropriately. At the same time, AI coworkers give us new ways to undertake the engineering and knowledge work needed to provide it.
Evorthon Data Harness grew progressively from my attempts to use AI to solve problems I have known and grappled with for decades. I have released it as a software factory for data platform delivery, combining a specialist AI coworker with tools for repeatable engineering and verification.
The aim is to reduce the effort, cost and time involved in modernising an existing estate or building a new platform. The opportunity is to apply accumulated engineering knowledge through a delivery system that teams can keep using as their platform develops.
AI makes the data foundations more valuable
I have long been passionate about proper data engineering and management. Clear definitions, discoverability, governance and dependable processing affect what an organisation can do with its data. They determine how easily people can find an answer and how much work it takes to establish that the answer is sound.
Those disciplines become practical requirements when agents use enterprise data to carry out work. An agent investigating service performance needs to find the relevant datasets and understand what their measures mean. It also needs to work within the permitted access and use data appropriate to the period being investigated.
A measure of response time, for example, might use working hours or elapsed hours. A status might describe a case today or at the end of the reporting period. Those choices change the answer. The definitions need to be available alongside the data so that a person or an agent can use it correctly.
For a data team, the ambition to deploy AI therefore creates a concrete demand for the work it already knows matters. Source history, ownership, quality checks and documented meaning all contribute to making data usable by agents. Discoverability gives them a way to locate the right material, while governance defines how it may be used.
The organisation gains more from its AI investment when these foundations are in place. It also needs a practical way to build and improve them across an estate that may already have decades of accumulated processing.
AI also changes how we can do the work
The opportunity extends into the delivery process itself. AI can assist with work that involves reading existing material, investigating a problem, drafting a design and implementing changes. With a suitable working environment, it can contribute across several stages of an engineering task.
The growing work on agent harnesses gives this a practical form. A harness supplies the context, tools and working process around the model, helping an agent make progress through a task and check its results. Specialist coworkers apply that approach to a particular kind of work.
In my own work, AI has changed how quickly I can investigate a problem and develop a working solution. I have been applying it to data problems I know, which gives me a concrete basis for judging what is useful. That includes understanding an existing process and establishing why a replacement produces a different result.
Evorthon developed through that work. As I used AI to address those problems, the methods and supporting software progressively came together into a more complete delivery capability. The project gives that capability a reusable form, from understanding the business need to presenting a working result for acceptance.
For a team facing its own platform programme, the value is access to a specialist coworker with an explicit engineering method, connected to software that carries the work forward.
Turn knowledge of the estate into work the factory can deliver
A migration begins with understanding the behaviour that consumers rely on. Schemas, transformation code, reports and schedules contain much of that knowledge. The coworker helps extract the relevant facts, records where they came from, and brings unresolved questions to the people who can answer them.
A filter in a query might exclude a particular case status. The current code establishes the behaviour; the business owner decides whether the replacement should preserve it. The confirmed answer is then available when the new processing is designed, implemented and checked.
This makes the discovery work useful throughout delivery. Engineers can follow an agreed definition back to its source. A reviewer can assess a proposed change against the intended outcome, and a new team member can see how a decision was reached.
For a new platform, discovery starts with the capabilities and operating needs the business requires. The coworker helps turn those requirements into a proposed architecture, source and output contracts, delivery increments and acceptance criteria. People review the proposals and approve the decisions.
Each use case keeps a connected account of what it serves and what is needed to deliver it. That gives the factory a basis for organising the work and gives the team something concrete to review at each stage.
Reuse the engineering across the programme
Once the team has agreed what to build, Evorthon Data Harness connects approved work to tools for tracking, pipeline generation and automated builds. Engineers and agents share a record of the work and its dependencies. Eligible items can proceed through build and review cycles, with findings returned for resolution.
Suitable source descriptions and rules can also drive repeatable pipeline generation. The generator is Ergasterion, which I described in AI Is Building Faster. Reliable Data Still Takes Engineering. Evorthon Data Harness provides the wider delivery method around that factory, including the work needed to reach an agreed description and accept the result.
The economic benefit comes from applying the same engineering approach across successive deliveries. A source description continues to be useful when another field is added. Confirmed business definitions inform the implementation and its checks. Once a dataset has been accepted, another use case can build on it with its scope and evidence already recorded.
The platform team keeps its choice of target technology and connects its environment through defined adapters. The investment in the delivery method can then be applied to further work on that platform.
Make progress while the rest of the estate becomes clear
Large programmes have uneven readiness. Some sources are well understood, while others need further discovery or access approval. The team may already know enough to deliver an initial consumer dataset while broader requirements are still developing.
Evorthon Data Harness assesses readiness for individual spans of processing within a use case. Where the output is sufficiently defined, engineers can progress that part with missing inputs represented by labelled synthetic data or tracked follow-up work. The outstanding dependencies remain visible.
A transformation can therefore be developed and exercised while the source owner resolves an input. When the required evidence becomes available, the team can verify the result against it.
Acceptance can cover part of a use case. Each accepted version records the outputs and scenarios it covers, the evidence used, and the work still outside its scope. An accepted intermediate dataset can then support another delivery.
For the programme sponsor, this creates a clearer connection between the work being funded and the capabilities becoming available. In a migration, each increment can also carry the conditions for retiring the corresponding existing process.
Establish that the new platform does the right work
Verification begins when the team agrees what success means. Evorthon Data Harness records the inputs, reference data, expected outputs and comparison rules for a verification case, together with available processing checkpoints.
Engineers can change the candidate implementation and compare it against a fixed version of the expected result. Captured evidence lets routine verification continue independently of the legacy system’s operating schedule. When results differ, the engine reports the earliest difference confirmed by the available checkpoints and identifies where more evidence is needed.
The included customer service example shows this within one delivery increment. Service leaders need a daily view of demand, response and resolution by 09:00. The replacement preserves the agreed cutoff and metric definitions, with checks covering the case population and published measures.
The example uses synthetic data and exercises successful publications alongside deliberate defects, including missing keys, changed totals and rounding differences. It follows the work through two daily publications and named acceptance, including the conditions for retiring the previous manual route.
A second synthetic example covers a new platform for renewable asset monitoring. It follows the required capabilities through to measures, alerts and a consumer view, using agreed scenarios to check the result.
The team can inspect both examples from the original outcome to the acceptance evidence. They make the delivery method concrete and provide a starting point for applying it to another use case.
Put the accumulated experience to work
Evorthon Data Harness builds on established practices, including metadata driven engineering, explicit requirements, incremental delivery, independent review and reproducible verification. It brings them together with a specialist AI coworker to address work that has occupied a substantial part of my career.
The released project includes the coworker pack, a Python package, delivery templates and worked examples. The adopting team supplies its platform connections, authorises model access and provides the people who own the business and technical decisions.
For an organisation investing in AI, improving its data foundations and improving how it delivers them can support the same objective. The platform provides data that people and agents can use. The factory gives the team a repeatable way to build and change it, retaining the knowledge and engineering developed along the way.
A useful starting point is one bounded use case from a real programme. Carry it through discovery, design, build and acceptance, then assess where the method can reduce effort across the wider estate.
The source and worked examples are available on GitHub. The adoption guide explains how to start, and the Python package is available on PyPI.