[duckdb] is the warehouse and [ossie] is the document validator. They are
independent: reading, validating, fingerprinting, and authoring a native layer
all happen without a warehouse, and a warehouse install carries no Ossie
support unless you ask for it.
1. A warehouse to point at
dex demo writes dex_demo.duckdb (7 tables, 29,512 rows) and a
.dex/config.yml beside it, refusing rather than overwriting anything that is
already there. The data is generated from a pinned seed and is deliberately
flawed, which is what makes the rest of this page interesting.
Run explore inventory before writing a line of the document. On DuckDB the
catalog name comes from the filename, so the identifiers are
dex_demo.main.orders and not demo.main.orders, and a document written
against a guess will parse cleanly and link to nothing. The identifiers it
reports are:
2. The document
Save this assemantics/commerce.ossie.yaml. It is written to exercise every
branch of the read path at once, so it is denser than a real first document
would be.
3. Configuring it
Ossie is selected on the semantic axis. It is never the transformation project, so there is noproject: block here at all:
semantic.ossie.files is the whole of the layer’s coordinates. There is no
discovery step and no glob, because a document dex was not told to read is a
document nobody reviewed. Paths are relative to the repository root and confined
to it: an absolute path, a .. that walks out, and a symlink resolving out are
each refused.
4. Reading the layer
dimension_scope is declarations, meaning one row per dimension the document
declares rather than one row per token a query could group by. The layer comes
back as six semantic models (one per dataset, namespaced
<semantic model>.<dataset>), eighteen dimensions, two metrics, and:
unavailable block says so rather than leaving you to infer it:
commerce.revenue carries
semantic_models: ["commerce.orders"], because its expression names
orders.order_total_eur and that reference resolves. commerce.order_count is
SUM-free and names nothing, so its lineage is empty rather than every dataset
in the model:
metric ‘order_count’ names no dataset field dex could resolve, so its lineage is empty. Ossie states no metric-to-dataset reference, and naming every dataset in the semantic model would be a claim the document does not make.
5. What did not link, and why
Every dimension carries the physicalcolumn behind it, and four of them carry
none. That is a decision, not a failure, and each one says which decision:
field ‘orders.order_total_eur’ is computed rather than a bare column (ANSI_SQL: order_total * 0.92), so it carries no physical column. A column guessed out of an expression makes the PII gate screen the wrong one and report it as evidence
field ‘orders.quoted_status’ is a quoted identifier (“status”), so it carries no physical column. An unquoted identifier is folded the way the warehouse folds it while a quoted one is exact, so the two need not name the same column and dex cannot tell which was meant
dataset ‘commerce.recent_orders’ has a source dex cannot read as one relation on duckdb, so it carries no physical link. Ossie documents a source as a table reference or a query without a portable way to tell them apart, and a query read as a relation would reach the PII gate as false evidenceThe fourth is the one to notice, because it is a check dex ran and reported rather than a check it passed:
MAQL, MDX expressions are preserved as written and were not checked: they are not SQL, so dex neither parses them nor reads a physical column out of oneBoth non-SQL expressions are still in the payload, verbatim, under the element’s
vendor_params.dialects. Nothing the document says is dropped. What changes is
that dex will not read a column out of them, and says so instead of staying
quiet.
The document also warns, and a warning is not a partial refusal. The layer is
read in full and the warning rides along:
relationship ‘returns_to_order_items’ joins to_columns [‘order_id’, ‘order_item_id’] on dataset ‘order_items’, which do not cover its primary key or any of its unique keys. The join may fan out
6. The governed route when query and values refuse
Ossie is catalog-first. list answers; query, values, and
--for-dimension refuse, and --api refuses too:
Apache Ossie specifies interchange metadata, not a portable query runtime: it defines no filter grammar, no join planning, and no execution semantics, so dex has nothing to render a governed statement from.That is a property of the format rather than a gap in this implementation, and the refusal names the alternative rather than leaving you stuck. A dimension carries its
semantic_model, that model carries its relation, and the
relation is an ordinary warehouse object:
customers.email at confidence 0.95, so:
query refused: the projection would carry values from PII-flagged column(s): customers.email (email). Use a measuring aggregate over them (COUNT, COUNTIF, APPROX_COUNT_DISTINCT, AVG(LENGTH(…))), or drop them from the output.The flag reached that column because the field
customers.email is a bare
identifier on a relation dex can address. A field that carries no column is
screened by the name heuristic instead, and the payload says which happened.
7. What the declarations reach in exploration
dex_demo.main.products and
dex_demo.main.warehouse_locations come back with semantic_models: [],
which is what separates a load-bearing table from a merely large one.
The declared grain. order_items has a composite primary_key in the
document, and it supplies a grain measurement could not prove, saying where the
grain came from rather than leaving that silent:
grain order_id, product_id comes from the project’s declared composite key (measurement found no key of its own)That is the case the declaration earns its keep in. Nothing keys this table:
order_item_id is unique on 13,000 of its 14,000 rows because a batch was
loaded twice, and the column pairs that are unique here are unique only because
of that shortfall, so profiling reports none of them and says so in
key_evidence. A declaration cannot fix duplicates, and it does not claim to.
It states what the grain is meant to be, maintain grain is where that gets
verified against the data, and the measurement keeps the finding.
The declared joins, at confidence 1.0, with the composite kept whole:
explore diagram, which labels the edge
order_id = order_id, return_id = order_item_id. No first-column proxy is ever
emitted or measured, because a composite measured one column at a time can
report a healthy join three times over while no complete tuple matches.
--verify is where the document meets the data. orders_to_customers verifies
with no orphans. web_events_to_customers does not:
dex_demo.main.web_events.customer_id -> dex_demo.main.customers.customer_id is declared as a foreign key but 100% of values have no match in the parent; the project and the warehouse disagreeIts confidence stays at the 1.0 the document asserts. A measurement never revises a declaration; it reports the disagreement and leaves the decision to you. The composite edge is measured as a complete tuple and comes back with no orphan fraction at all, because
returns is the demo’s half-failed load and has
no rows: an honest absence of evidence rather than a clean bill of health.
8. A baseline, and drift against it
transform_layer is null because there is no transformation project here and
Ossie declares no build step, so a transform baseline over it would be a
baseline of nothing. The two layers are fingerprinted independently, which is
exactly why a repository with a semantic layer and nothing else still gets one.
The semantic side records each dataset and metric with a content hash, the
relation behind it, the column each field resolves to, the declared keys in the
arity they were written, and every relationship with its ordered column pairs.
It also records that it captured them, so a baseline written before that was
possible reports the relationship axis as unchecked rather than as clean.
Now break something. Point web_events.page_path at a column that does not
exist, and change one side of the composite relationship:
[high] broken_relationship: relationship ‘returns_to_order_items’ from ‘commerce.returns’ to ‘commerce.order_items’ names column pair(s) that no longer resolve: [[‘return_id’, ‘line_item_id’]]
[high] dangling_reference: dimension ‘page_path’ on semantic model ‘commerce.web_events’ references column ‘page_url’, which is gone from dex_demo.main.web_eventsTwo more findings at
low record that the definitions themselves changed. All
of it is free: this axis reads the documents and the cache and opens no
warehouse connection. The command’s paid half, dimension cardinality, needs a
semantic model that names a transformation model, and Ossie names none, so
data.offer is null and there is nothing to confirm.
maintain grain is where the declaration is measured:
[high] (order_id, product_id) on dex_demo.main.order_items is no longer unique: 13952 distinct combinations over 14000 rows (~48 duplicate rows); joins on it will fan outThe declared composite reached the grain channel, was measured as a composite, and the warehouse disagreed with it. It is a finding, never an automatic rewrite. On DuckDB this is free; on a metered warehouse it goes through the same
--confirm --budget handshake every scanning command does.
maintain reconcile has nothing to author here. A native semantic layer is not
an editable transformation project, so its proposals are advisory and no plan is
stored. Authoring is the next section, and it is a different command.
9. Authoring into the layer
kind may be omitted because
this command implies semantic_document:
cache validation skipped 1 query-backed, quoted, or otherwise opaque dataset source(s); dex does not guess a physical relation
cache validation skipped 2 computed or non-SQL field expression(s); dex validates only a direct physical column reference
cache validation checked 5 direct relation(s) and 29 physical column reference(s) using stored evidence only; no warehouse connection was openedThose are notes. A reference the cache can positively contradict is a refusal, and no plan is stored:
the prospective Ossie semantic layer has references contradicted by the exploration cache, so no plan was stored: dataset[5] ‘suppliers’ names source ‘dex_demo.main.suppliers’, but the cached inventory for namespace ‘dex_demo.main’ does not contain itThe namespace guards are the difference between the three modes.
define
refuses a semantic-model name the layer already has, update refuses one it
does not, and plan accepts both and reports each under defined or updated:
semantic ossie define found names already defined: commerce; useNothing is written until you apply:semantic ossie updateorsemantic ossie plan
these files changed after the plan was made (human edits are authoritative);
re-plan against current state, or re-run with --confirm to overwrite
deliberately
On a clean apply the accepted bytes are written exactly as authored. dex does
not parse and re-serialize the document, so comments, key order, quoting, and
whitespace all survive, and configured documents the payload did not mention are
untouched. The comment above the new field is still there afterwards, and
explore semantic list now reports customers__lifetime_value beside the
others.
10. What this page did not prove
Catalog-first is not a roadmap item. Ossie specifies interchange metadata and no portable query runtime. The refusals in section 6 are the format’s shape. Upstream has an active working group on a query language and a reference engine; when that lands it becomes a declared runtime with its own governed contract, not a loosening of these commands. Validation is not an execution guarantee. There is no external validator here the waydbt parse is for dbt: there is no published Ossie package and no
runtime to load a document into. A document that passes is schema-valid and
internally consistent. Whether any engine can execute it is a separate question
dex has no basis to answer, and no dex surface says otherwise.
dbt was never involved. No dbt project, no MetricFlow, no manifest, at any
step above. The two axes are independent in both directions: dbt does not depend
on Ossie, Ossie does not depend on MetricFlow, and a repository may have either,
both, or one of each.
dex reads Ossie; it does not convert it. There is no exporter to or from
another semantic format, and dex says nothing about what another tool would make
of a document it read.
The schema dex validates against is pinned by content rather than by version
string, and upgrading it is a reviewed procedure rather than a version bump. The
pin, the upstream commit, the known deltas, and the upgrade steps are in
Apache Ossie compatibility.
Source: references/ossie-walkthrough.md (edit on main). This page is generated; change the source, not the site.