Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[prism] Triggers and Panes and Windowing Strategies #31438

Open
Tracked by #29650
lostluck opened this issue May 29, 2024 · 0 comments
Open
Tracked by #29650

[prism] Triggers and Panes and Windowing Strategies #31438

lostluck opened this issue May 29, 2024 · 0 comments

Comments

@lostluck
Copy link
Contributor

lostluck commented May 29, 2024

Triggers are the "last" complex bit of modeling necessary for Prism's Table stakes for being incorporated for use by other SDKs.

Triggers are a way to refine how and when aggregations occur, but require sophisticated accounting Runner side to implement. https://beam.apache.org/documentation/programming-guide/#triggers has the user side details.

Triggers (and WindowingStrategy in general) contribute to the "fun" bit of the batch and stream unification that the Beam model provides. To help manage this, Panes are metadata passed along with the elements, and indicate the upstream aggregation conditions that produced the values.

https://docs.google.com/document/d/12r7frmxNickxB5tbpuEh_n35_IJeVZn1peOrBrhhP6Y/edit describes how Lateness and late data is handled WRT panes.

Ultimately, in prism, this will probably be handled via the state and timer system instead of an additional layer of separate logic, which has the cost of less efficient behavior for "only watermark" triggered aggregations. Robust State and Timers allow for all these.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant