AI has changed how quickly we can produce software. I use it every day, and the difference is material. A small team can build and test systems at a pace that would have required much more time and more people a few years ago.
The data those systems depend on has not become easier to trust. Many enterprise AI programmes can reach a model quickly and still spend months turning source feeds into data that people can use. Before the difficult questions of identity and meaning, there is a large amount of repeated engineering: inspect the columns, cast the types, preserve changes, add keys, write tests and document the output.
A new field often means editing several models by hand. A new source brings another set of staging code, history tables, mappings and checks. When a source or target schema changes, each copy of that plumbing has to be found and updated. The work is familiar, but it consumes time and creates many opportunities for pipelines that should follow the same pattern to drift apart.
Ergasterion was created to remove that repeated work. You describe a source once in configuration: its tables, columns, keys and the business entities they refer to. The factory generates the source-facing dbt pipeline from that description. Add a field to the configuration and run the factory again; the generated pipeline changes with it. Add a source by creating another declaration, mapping it to the domain and configuring where it will run. The same factory builds the new pipeline without a copied flow to maintain.
The harder data problems sit on top of that basic capability. Modern data platforms do not decide whether two records describe the same person, which source should win when values disagree, or how an earlier value should be reconstructed after a correction. People still have to define those things. Ergasterion lets them write those decisions as readable rules and applies the rules through the same generated pipeline.
I have spent more than 20 years designing, building and advocating for configuration and metadata driven data pipelines. I first built this factory approach using Microsoft Data Transformation Services (DTS), then built later versions with SSIS, Informatica, Ab Initio and Spark. Across those technologies and several financial services domains, one division of responsibility has held up well: people describe the sources and own the rules, while a factory generates the repeatable engineering.
I have released that factory method as an open source project called Ergasterion.
Ergasterion is the ancient Greek word for a workshop. This workshop generates the repeatable parts of a dbt estate: typed staging, business keys, append-only source history and the models that prepare each entity for use. On top of that plumbing, it can resolve overlapping identities and apply rules for choosing values. It also emits contracts, product descriptors and a typed map of the domain. The same engine builds worked e-commerce and investment examples without carrying either domain’s vocabulary in its code.
What the demonstration does
The public repository includes invented source data and a complete local demonstration. No warehouse account is required.
- Configuration describes the tables and columns sent by three retail systems, together with the business entities they represent.
- Ergasterion generates typed staging, business keys and append-only history models from those descriptions.
- DuckDB runs the complete project locally, including the generated plumbing and the tables designed for use by people and tools.
- Where several records describe the same customer, declared rules resolve the identity and choose the value to use for each field.
- The factory also produces open data contracts and a data product descriptor. Generated graph files carry the domain relationships into common graph forms.
The investment example exercises the same method with funds, management firms, cash movements, valuations and a deal pipeline. It uses different nouns and different business rules. The generator remains the same.
Change the description, rebuild the pipeline
Suppose the storefront adds a preferred language field to its customer feed. The engineer adds that column to the source declaration and runs the factory. Ergasterion regenerates the typed staging and history models; only the output affected by the declaration changes. The engineer reviews that change and runs the tests. There is no generated SQL to find and patch by hand.
A new source follows the same path. The engineer creates a declaration for its tables and columns, maps them to the domain, and configures the target. The factory generates the standard plumbing for that feed. If it describes an entity that no other source supplies, the record passes through without an identity argument. If it overlaps an existing source, the domain’s matching and value-selection rules apply.
Supported warehouse targets use the same declarations. The target configuration and shared adapter rules own the SQL differences, so changing a target schema or moving between supported targets does not require a manual rewrite of every generated model.
Follow one customer through the factory
The e-commerce example follows an invented customer called Ava Thompson. A storefront, a marketplace and a customer relationship system each hold a record for her. The records use different identifiers and slightly different contact details.
People responsible for the domain define the matching rules in readable configuration. In this example, a shared loyalty identifier is the strongest signal. A normalised email comparison follows when that identifier is absent. The rules are ordered and cautious, so a weak resemblance does not silently merge two people.
Ergasterion turns each incoming table into a typed model, then stores every source’s version of Ava’s details with its source and effective time. A later correction adds another version. It does not erase the earlier record. This append-only history uses established Data Vault patterns.
The three source records then resolve to one customer identity. Each match records the rule that produced it. The process is deterministic: the same records and rules produce the same identity.
The sources disagree about Ava’s city. The domain configuration says that the customer relationship system owns contact attributes, while the storefront owns marketing preferences and consent. Ergasterion applies those rules field by field. The final customer record says Manchester and records the source that supplied it.
History also matters after identity resolution. Ava moves from silver to gold loyalty status in May. An order placed in March remains attributed to silver, while a June order uses gold. A question about March continues to receive the March answer after the later change.
The final tables are shaped for use by people, applications and analytical tools. Their data contracts describe the columns and types in the Open Data Contract Standard. A product descriptor groups those outputs at domain level, and generated graph files describe how customer, order and product relate.
People own the meaning
Configuration drives the repeatable mechanics. People still decide what the important entities are, which keys identify them, how cautious matching should be and which source owns each attribute.
Those choices are visible and changeable. They sit in the source and domain descriptions, where a domain owner and an engineer can review them together. The factory applies the accepted choices across every generated model.
The clean tables and measures that consumers use are also designed by people. Ergasterion generates the source-facing plumbing beneath them. It does not infer the right commercial definition of revenue, performance or customer value.
This boundary matters for enterprise AI. Faster code production does not remove the need for accountable decisions about meaning. It makes a clear division between judgement and repeatable engineering more valuable, because the implementation can now move much faster than the organisation’s ability to inspect it.
How the output is verified
Identical descriptions produce identical generated files. A check fails if a generated file is edited by hand, directing the change back to its source description. The repository uses named tests over invented data to verify the business results, including Ava’s identity, her winning city, dated loyalty status and order totals.
The matching rules favour precision. A record that does not meet a declared rule remains separate. In the investment example, uncertain matches go to a human review queue and the recorded decision is reused on later runs.
The local demonstration runs the complete seeded project in DuckDB. Snowflake has a worked live path for an account owner who wants to deploy it. BigQuery support covers generation, SQL portability checks and static parsing.
Every row in the public examples is synthetic. The engine is released under the MIT licence, and the source can be inspected alongside the declarations, generated output, contracts and tests.
Foundations
Ergasterion builds on established work. Metadata driven engineering has a long history. dbt provides the transformation framework, Data Vault provides the append-only history pattern, and the Bitol standards provide open contracts and product descriptors.
Each of those platforms required different generated artefacts and runtime conventions. The architecture remained the same: people described sources and rules, while the factory generated the repeatable pipelines.
dbt is the execution framework for the open source implementation because its adapter model and built-in checks provide a practical implementation base. Ergasterion’s central design is independent of that choice. Source and domain declarations enter the system; an engine-specific emitter produces pipeline artefacts and verification. An emitter could produce native Databricks pipelines and jobs, Spark SQL or PySpark workloads, Azure Data Factory pipelines, or artefacts for another suitable engine. Each emitter would have to preserve the same rules for history, identity, deterministic generation and testing.
My contribution is the assembly and the operating boundary. The source descriptions, identity rules, survivorship rules, generated pipeline and open boundary artefacts are connected in one working project. The same mechanism is exercised across two unrelated domains and can run locally on a laptop.
Platform AI assistants
Most cloud data platforms now include AI assistance that can create and modify data pipelines from a natural language prompt. These tools can be useful when building or changing an individual pipeline.
My concern is the unit of control. If each pipeline begins as a separate prompt, the assistant can choose a different key pattern, history model, naming scheme or test structure each time. Persistent instructions can reduce that variation, and a capable assistant can implement entity resolution code when it has enough context. A collection of individually plausible pipelines still needs one source of truth for the estate.
Entity resolution makes the boundary visible. Identity keys, matching rules, survivorship and human decisions span several sources and must remain stable as the estate changes. Those choices should not belong to whichever prompt happens to create the next pipeline.
The factory provides that coherence. Versioned source descriptions and domain rules generate the pipeline, and identical inputs produce identical files. AI assistants can help draft declarations, propose mappings, explain failures and build new emitters. The declarations and deterministic generator remain the system of record for how the estate is built.
Scope and limits
Ergasterion begins once a source table is available to dbt. It does not extract data from operational applications or replace the transport that delivers it. Import commands can draft the mechanical parts of a source description from DDL or an existing contract, and they leave the judgement as explicit work for a person.
The factory will not discover business meaning from raw columns or decide which source an organisation should trust. It generates the pipeline from decisions that people have made and written down.
It is also not a universal answer to enterprise AI delivery. Governance, ownership, operating models and access to source data remain organisational work. Ergasterion addresses a narrower problem: producing dependable, inspectable data engineering from descriptions and rules that people own.
Run it locally
The engine is available from PyPI:
python -m venv .venv
source .venv/bin/activate
pip install "ergasterion-factory[duckdb]"
ergasterion init my-estate
For the complete worked demonstration, clone the GitHub repository and follow the local DuckDB demonstration. It builds both example domains, then prints the e-commerce result, customer resolution and investment result from the same local database.
I would value reports from practitioners who apply it to a different source shape. Please tell me where installation is unclear, a source cannot be described, the generated boundary is wrong, or a real domain needs a pattern the factory does not yet express.