Skip to content

[FLINK-XXXX] [table] Support lineage extraction for DataStreamscan/sink providers#27727

Open
jsingh-yelp wants to merge 1 commit intoapache:masterfrom
jsingh-yelp:support-lineage-for-datastream-providers
Open

[FLINK-XXXX] [table] Support lineage extraction for DataStreamscan/sink providers#27727
jsingh-yelp wants to merge 1 commit intoapache:masterfrom
jsingh-yelp:support-lineage-for-datastream-providers

Conversation

@jsingh-yelp
Copy link

@jsingh-yelp jsingh-yelp commented Mar 3, 2026

What is the purpose of the change

This pull request adds lineage vertex extraction support for DataStreamScanProvider and DataStreamSinkProvider.

Brief change log

  • This pull request adds lineage vertex extraction support for DataStreamScanProvider and DataStreamSinkProvider in the table planner.
  • Previously, only Source, SinkFunction, and SinkV2 providers had their LineageVertex extracted for FLIP-314 lineage events.
  • Connectors that use DataStream providers (e.g. Apache Paimon) were silently missing lineage dataset facets. This ensures all scan/sink provider types are treated consistently for lineage reporting.
  • Specifically was trying to solve this issue for reference: [Feature] Support native Flink 2.0 job lineage (FLIP-314) for Paimon tables paimon#7306

Verifying this change

This change added tests and can be verified as follows:

  • Added unit tests for the changes done in flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/lineage/TableLineageUtilsTest.java.
  • Verified manually by running a flink cluster that the changes made are running as expected.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): ( no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)

@flinkbot
Copy link
Collaborator

flinkbot commented Mar 3, 2026

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

@jsingh-yelp jsingh-yelp force-pushed the support-lineage-for-datastream-providers branch from 9fef5c9 to 16bda1f Compare March 3, 2026 04:55
@jsingh-yelp
Copy link
Author

I don't have access to the JIRA board I have requested for access in the mean time can anyone please help creating a ticket for this PR?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants