MMarquitoOpenLineage Visualizer

OpenLineage Visualizer

Choose a sample dataset to explore, or upload your own OpenLineage JSONL file.

Sample Datasets
Apache Spark
Spark ETL Pipeline

A custom Spark Scala ETL pipeline ingesting 4 CSV sources through Delta Lake tables with column-level lineage.

50 events · openlineage-from-spark-demo-customer.json
dbt
dbt Jaffle Shop

The classic dbt jaffle_shop tutorial project — customers, orders, and payments models with staging and mart layers.

90 events · openlineage-from-spark-dbt-jaffle-shop.json
dbt
dbt AdventureWorks

A larger dbt project modeling the AdventureWorks sample database with sales, products, and customer dimensions.

232 events · openlineage-from-spark-dbt-adventureworks.json
Or bring your own
Upload Custom JSONLUpload your own OpenLineage JSONL file — parsed entirely in your browser.

OpenLineage Visualized

How lineage flows from query to OpenLineage event — hover any block for sample data.

Unresolved planOptimized planExecution eventsSparkListenerEventQueryExecutionDataset listFacetsRunEvent objectSerialized JSONDeltaTableV2 node1User QuerySQL / DataFrame API
Entry point — user submits SQL or chains DataFrame transformations via SparkSession
2Catalyst EngineParse → Analyze → Optimize
Parses SQL into an unresolved plan, resolves references via catalog, then applies 70+ optimization rules
3Logical PlanOptimized plan tree
A tree of relational operators — Project, Join, Filter, Aggregate — with resolved columns & types
4Listener BusEvent dispatch to all listeners
Spark fires SparkListenerEvents on job start/end, stage completion, SQL execution — broadcast to registered listeners
5Delta LakeACID transactions & log
DeltaTableV2 plugs into Spark's DataSource V2, providing ACID commits via OptimisticTransaction and a JSON-based _delta_log
6OL Spark ListenerHooks into Spark events
Extends SparkListenerInterface — captures onJobStart, onJobEnd, onOtherEvent and routes them to the OpenLineage context
7Plan VisitorsWalk the LogicalPlan tree
A chain of QueryPlanVisitor subclasses that pattern-match on LogicalPlan nodes to extract input/output datasets
8Column LineageField-level dependency edges
ColumnLevelLineageBuilder traces each output column back through expressions to source columns, building a dependency graph
{"event":"facets":}9RunEvent OutputOpenLineage JSON emitted
OpenLineageRunEventBuilder assembles the final RunEvent with job, run, input/output datasets, and all facets including column lineage
10Event EmitterRoutes to OpenLineageClient
EventEmitter.emit(event) delegates to OpenLineageClient which serializes the RunEvent and passes it to the configured Transport
11File TransportWrites JSONL to disk
FileTransport appends each RunEvent as a single JSON line to the output file — this is the lineage JSONL file you load into this visualizer
LEGENDApache SparkOpenLineageDelta Lake
  1. 1.User Query — A user submits SQL or DataFrame transformations via SparkSession. 📄 SparkSession.scala
  2. 2.Catalyst Engine — Spark parses the SQL, resolves table/column references through the catalog, and applies 70+ optimization rules. 📄 QueryExecution.scala
  3. 3.Logical Plan — The result is an optimized tree of relational operators (Project, Join, Aggregate) with resolved types. 📄 LogicalPlan.scala
  4. 4.Listener Bus — As execution proceeds, Spark fires SparkListenerEvents (job start/end, stage completion) to all registered listeners. 📄 SparkListener.scala
  5. 5.Delta Lake — Delta Lake tables appear in the plan as DeltaTableV2 nodes, backed by ACID transactions and a JSON commit log. 📄 OptimisticTransaction.scala
  6. 6.OL Spark Listener — OpenLineage's listener intercepts these Spark events and triggers lineage extraction. 📄 OpenLineageSparkListener.java
  7. 7.Plan Visitors — A chain of QueryPlanVisitor classes pattern-match on plan nodes to identify input/output datasets. 📄 QueryPlanVisitor.java
  8. 8.Column Lineage — ColumnLevelLineageBuilder traces each output column back through expressions to its source columns. 📄 ColumnLevelLineageBuilder.java
  9. 9.RunEvent Output — The final OpenLineage RunEvent JSON is assembled with job, run, datasets, and column lineage facets. 📄 RunEventBuilder.java
  10. 10.Event Emitter — EventEmitter receives the built RunEvent and delegates to the OpenLineageClient, which routes it to the configured Transport. 📄 EventEmitter.java
  11. 11.Transport (File) — FileTransport appends the JSON event as a single line to the configured output file. Other transports include HttpTransport (POST to API) and ConsoleTransport. 📄 FileTransport.java