Project formats

​

Why you would write your own

​

The three tiers

​

Reading the semantic layer twice, for two different questions

@runtime_checkable
class SemanticCatalogProject(Protocol):
    def semantic_catalog(self) -> SemanticCatalogView: ...

​

Where an edit lands, and what you own

from exmergo_dex_core.adapters.project import ProjectView
from exmergo_dex_core.transform.plans import EditKind


class MyProject:  # ... tier 3 methods as above
    def load(self) -> ProjectView:
        return MyView(root=self.root, files=self.declarations())

    def edit_path(self, kind: EditKind, model: str) -> str | None:
        if kind is EditKind.SCHEMA_YML:
            return f"declarations/{model}.yml"
        return None  # no authored staging model in this format

    def editing_surface(self) -> list[str]:
        return ["declarations"]

​

What reconcile writes into a declaration

• On ClickHouse, nullability is the type constructor, so a column that starts or stops accepting nulls is reported as column_retyped and never as nullability_changed. It therefore arrives as advice where every other connector produces an edit.
• On Snowflake, a precision or scale change produces no schema finding at all, because both spellings render identically.

​

The one rule that is not visible in the signatures

​

Writing one

from exmergo_dex_core.dbt_project import ProjectDefinitions


class MyProject:
    name = "my_format"

    def __init__(self, graph):
        self._graph = graph

    def definitions(self) -> ProjectDefinitions:
        defs = ProjectDefinitions(present=True)
        defs.built_relation_names = sorted(self._graph.models)
        # ...declared keys, joins, metric models
        return defs

​

Proving it works

pip install "exmergo-dex-core[project-conformance]"

from exmergo_dex_core.adapters.conformance import ExploreProjectContract


class TestMyProject(ExploreProjectContract):
    def make_project(self):
        return MyProject(nothing_declared())

• make_unreadable_project() returns a project whose source your format cannot parse. It defaults to None, which skips the never-raises assertions with a message saying so. That is a compromise, because a format reduced from an in-memory object may genuinely have no unparseable state. If yours does have one, override it: those are the assertions worth the most.
• DeclaringProjectContract, mixed in beside your tier contract, checks that a declared key and a declared join actually arrive. It is separate because the two contracts answer different questions: whether dex can safely call your format, and whether reading it was worth doing. It carries two further hooks that default to skipping, and both are worth supplying if your format can reach the state:
  • a_project_declaring_a_composite_key() returns (project, model, columns). declared_composite_keys is a separate field from declared_keys and nothing else in the suite reaches it. Declare more than two columns: a format that handles a composite grain by special-casing the pair passes a two-column fixture and fails a four-column one, so a pair cannot tell you what you came to find out. A truncated composite key is the expensive failure here, because it does not read as a missing declaration. It reads as a narrower grain that is simply wrong.
  • a_project_declaring_a_join_with_differently_named_sides() returns the same shape as a_project_declaring_a_join, with column != to_column (the contract checks that and refuses a mirrored fixture). If both ends of your join fixture are spelled the same, an implementation that reads the source column and copies it onto the target satisfies the plain join assertion exactly, and the defect ships. A key whose two ends are named differently is the ordinary case, and a format that mirrors joins on a column the target may not have, surfacing as a wrong answer rather than as a failure.
• SemanticProjectContract, mixed in beside MaintainProjectContract if your format declares semantics at all. Its one hook, a_project_declaring_a_semantic_model(), returns (project, name, dimensions, measures) where the two mappings are field name -> the warehouse column behind it. It asserts tier 2 first, against the project the hook returns. semantic_layer is a tier-2 member, so a format mixing this in beside ExploreProjectContract would otherwise fail with AttributeError: no attribute 'semantic_layer', which names the missing attribute rather than the tier it belongs to. This is the gap most worth closing, because the tier contract cannot see it. MaintainProjectContract checks that an empty project yields an empty semantic layer and never looks at a populated one, so a format that reads every field name and drops the column behind each passes the tier suite completely. SemanticModelDef keys every field to a column and the drift detector skips any field whose column is None, correctly, since it cannot resolve what it was not given. A layer mapped entirely to None therefore validates, serializes, and compares clean forever: the check does not fail, it never runs, and a dropped warehouse column that should raise dangling_reference at high severity raises nothing. Map a field to None when you genuinely have no bare column for it, and include one such field if you can produce one, because categorical_dimensions takes str rather than str | None and a field that is categorical and unresolved has to be left out of that mapping rather than given an invented column.

• A conflict refuses the apply, not the conflicting edit within it. The edit set is the unit. A writer that lands the clean edits and holds back the conflicting one leaves the project matching neither the proposal nor what the human had, while the apply reports itself refused, so nothing records which half arrived. A single-edit set cannot see this, which is why the assertion stages two.
• A create pinned to no prior content is refused when the target now exists. old_content_hash=None is a claim that the file was absent at plan time. If one is there now, the claim is false and honoring it costs the human who created it during review the whole file rather than the lines that diverged. A writer reading None as “nothing to compare, go ahead” passes every other assertion, because in the staged conflict the pinned hash is a real one.

​

Constructing one

engine = DexEngine.from_repo(repo_root, project_format=MyProject(graph))

project:
  format: mypkg.projects:my_project
  options:
    graph: orders

# in your own pyproject.toml
[project.entry-points."exmergo_dex_core.projects"]
acme = "dex_acme_format:acme_format"

​

What a factory is

• repo_root, the directory dex was pointed at, or None when there is no repository in the picture.
• project_dir, where within that repository the project was pinned, relative to repo_root. None when nothing pinned one.
• connector, the name of the warehouse dex resolved for this run, or None when nothing named one. A name, never a live adapter and never a credential, so reading it opens no connection and costs nothing. It is here because a format may have to read an authored expression or relation name the way the active warehouse would, and identifier arity, quoting, and unquoted-case folding are the connector’s rules: a format that guesses them links a declaration to the wrong column or to none. dbt reads its own target and ignores this.
• options, your format’s own coordinates, passed through verbatim. dex does not interpret them, so the keys are yours to define and yours to validate.

​

Two alternatives that were rejected