1 Release published by 1 person

• v2.59.0-RC1 Beam 2.59.0 release

  published Aug 24, 2024

39 Pull requests merged by 14 people

• Add doc string and signature to generated Python wrappers

  #32337 merged Aug 27, 2024

• Preserve numeric string literals when reading from json.

  #32303 merged Aug 27, 2024

• Refactor PubsubIO Lineage metrics to work with all runners

  #32319 merged Aug 27, 2024

• fix failing ProtoSchemaTranslator on proto3 optional fields

  #32216 merged Aug 27, 2024

• Move beam summit banner to top of swiper slide

  #32323 merged Aug 27, 2024

• Add schema to SpannerIO read

  #32008 merged Aug 27, 2024

• Add beam summit banner

  #32320 merged Aug 27, 2024

• Fixed invalid links on programming-guide.md

  #32317 merged Aug 26, 2024

• Add patch release docs

  #32318 merged Aug 26, 2024

• Cherry Pick #32178: Loose BigQuery GCP project ID regex restrictions

  #32314 merged Aug 26, 2024

• Make autosharding test more robust

  #32293 merged Aug 25, 2024

• Cherry Pick #32294 (Java PrismRunner and TestPrismRunner) to the release.

  #32304 merged Aug 23, 2024

• Implement Java TestPrismRunner and PrismRunner

  #32294 merged Aug 23, 2024

• Fix TextIO.read() split with non-default delimiter

  #32298 merged Aug 23, 2024

• Rename delimiter to sep to pass to pandas.

  #32301 merged Aug 23, 2024

• Parallelize building wheels per language version

  #32297 merged Aug 23, 2024

• Remove some setup steps we don't need anymore

  #32300 merged Aug 23, 2024

• Loose BigQuery GCP project ID regex restrictions

  #32178 merged Aug 23, 2024

• Disable sickbay scheduled runs

  #32299 merged Aug 23, 2024

• [CHANGES.md] Add 2.60.0 section.

  #32287 merged Aug 22, 2024

• Cherry pick: Check experiment for hotkey logging (#32285)

  #32290 merged Aug 22, 2024

• Download BQMS catalog jar at build time

  #32282 merged Aug 22, 2024

• Check experiment for hotkey logging

  #32285 merged Aug 22, 2024

• Move RunInference GC to finalize_bundle

  #32281 merged Aug 22, 2024

• Cherrypick #32276 to 2.59.0 release branch.

  #32280 merged Aug 22, 2024

• [prism] Add an idle shutdown timout to prism binary.

  #32276 merged Aug 22, 2024

• Add SetState support for SparkRunner

  #32226 merged Aug 22, 2024

• Better error message when trying to get Beam Schema from proto schema with Struct

  #32260 merged Aug 21, 2024

• Do not create new Executor everytime createRunner

  #32272 merged Aug 21, 2024

• Better error when Python xlang download jar not available

  #32269 merged Aug 21, 2024

• Update requests package to 2.32.0

  #32270 merged Aug 21, 2024

• Revert "Portable runner fixes"

  #32271 merged Aug 21, 2024

• Fixes a breakage related to Kafka upgrade

  #32262 merged Aug 21, 2024

• Bump github.com/testcontainers/testcontainers-go from 0.32.0 to 0.33.0 in /sdks

  #32267 merged Aug 21, 2024

• Add timeout to runinference

  #32237 merged Aug 21, 2024

• Allow setting BigQuery endpoint

  #32153 merged Aug 21, 2024

• [Managed Iceberg] Support BigQuery Metastore catalog

  #32242 merged Aug 21, 2024

• Add ErrorHandler DLQ API to Python

  #31856 merged Aug 21, 2024

• Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager from 1.17.11 to 1.17.12 in /sdks

  #32266 merged Aug 21, 2024

27 Pull requests opened by 16 people

• [WIP] Support Protobuf 5.x

  #32273 opened Aug 21, 2024

• [WIP] Remove Python 3.8 Support

  #32279 opened Aug 22, 2024

• [WIP] Remove Python 3.8 targets from Actions

  #32283 opened Aug 22, 2024

• [DO NOT MERGE] Run all PostCommit and PreCommit Tests against Release Branch

  #32284 opened Aug 22, 2024

• [yaml] Add enrichment transform to Beam YAML

  #32286 opened Aug 22, 2024

• [yaml] Add examples for Spanner IO in YAML

  #32288 opened Aug 22, 2024

• [yaml] Add use cases for Enrichment transform in YAML

  #32289 opened Aug 22, 2024

• Bump github.com/aws/aws-sdk-go-v2/service/s3 from 1.60.0 to 1.60.1 in /sdks

  #32295 opened Aug 23, 2024

• Bump google.golang.org/api from 0.192.0 to 0.194.0 in /sdks

  #32296 opened Aug 23, 2024

• Add WindowedValueParam option to DoFns.

  #32305 opened Aug 23, 2024

• 2.59.0 Beam site updates.

  #32308 opened Aug 24, 2024

• Update Python Dependencies

  #32310 opened Aug 25, 2024

• Add predicate to control which columns are propagated when propagating successful rows

  #32312 opened Aug 25, 2024

• Bump cloud.google.com/go/datastore from 1.17.1 to 1.19.0 in /sdks

  #32313 opened Aug 26, 2024

• Update direct_runner.py

  #32325 opened Aug 27, 2024

• WIP: Ordered list state python

  #32326 opened Aug 27, 2024

• Revert "Download BQMS catalog jar at build time"

  #32327 opened Aug 27, 2024

• Fix remote execution test flakiness in tearDown

  #32328 opened Aug 27, 2024

• [DO NOT MERGE] Python 3.8 deprecation workflow validation

  #32329 opened Aug 27, 2024

• Use proper coders in interactive cache.

  #32330 opened Aug 27, 2024

• Remove beam summit banner

  #32331 opened Aug 27, 2024

• Colab Notebook for Unit Tests in Beam

  #32336 opened Aug 27, 2024

• Document dynamic destination key should be unique over the pipeline

  #32338 opened Aug 27, 2024

• Fix BigtableIO.write() client sharing

  #32340 opened Aug 27, 2024

• [YAML] Better errors for unconsumed error outputs.

  #32341 opened Aug 27, 2024

• Fix writing raw messages to pubsub

  #32342 opened Aug 27, 2024

• Fix Kafka with Redistribute

  #32344 opened Aug 28, 2024

19 Issues closed by 6 people

• The Playground CI Nightly job is flaky

  #30612 closed Aug 27, 2024

• The PostCommit TransformService Direct job is flaky

  #30960 closed Aug 27, 2024

• Performance Regression or Improvement: cogbk_python_batch_load_test_2GB_of_100B_records_with_a_multiple_key:runtime

  #32275 closed Aug 27, 2024

• Performance Regression or Improvement: cogbk_python_batch_load_test_reiterate_4times_10KB_values:runtime

  #32292 closed Aug 27, 2024

• Performance Regression or Improvement: cogbk_python_batch_load_test_2GB_of_100B_records_with_a_single_key:runtime

  #32274 closed Aug 27, 2024

• [Bug]: ProtoMessageSchema.getSchema fails on proto3 messages with optional fields

  #32199 closed Aug 27, 2024

• [Task]: Fix invalid links in `Beam Programming Guide`

  #32316 closed Aug 26, 2024

• [Failing Test]: apache_beam.io.gcp.bigquery_test.PipelineBasedStreamingInsertTest is flaky

  #32069 closed Aug 25, 2024

• [Task][Prism]: Create a PipelineResult implementation

  #31935 closed Aug 23, 2024

• [Task][Prism]: Create a PrismRunnerRegistrar

  #31967 closed Aug 23, 2024

• [Bug]: TextIO.read() with non-default delimiter doesn't split at the right position

  #32251 closed Aug 23, 2024

• [Task]: build_wheels workflow takes too long for release candidates

  #32217 closed Aug 23, 2024

• [Feature Request]: Add timeout to RunInference operations

  #32137 closed Aug 22, 2024

• Sorry, only users with push access can use that command.

  #32268 closed Aug 22, 2024

• [Feature Request]: Support `SetState` in SparkRunner

  #32225 closed Aug 22, 2024

• [Feature Request]: Add to ClickHouseIO dedicated useragnet

  #32252 closed Aug 22, 2024

• The Clean Up GCP Resources job is flaky

  #31846 closed Aug 22, 2024

• [Bug]: [Go SDK]: State API writes may become corrupted with Go 1.23

  #32245 closed Aug 22, 2024

• [Feature Request]: Allow setting BigQuery endpoint, for example to use bigquery emulator

  #28149 closed Aug 21, 2024

15 Issues opened by 11 people

• Performance Regression or Improvement: cogbk_python_batch_load_test_reiterate_4times_2MB_values:runtime

  #32343 opened Aug 27, 2024

• [Bug]: Add check for self-referencing input in YAML transform

  #32339 opened Aug 27, 2024

• [Bug]: BigQueryIO storage write api streaming dynamic destination conflicts if multiple transforms set same destination key

  #32335 opened Aug 27, 2024

• [Feature Request]: Emit warnings when pcollection elements approach or exceed 2GB limits.

  #32334 opened Aug 27, 2024

• [Bug]: Better error message for specifying inputs on chain pipeline in Beam YAML

  #32333 opened Aug 27, 2024

• [Bug]: Support error handling in PyTransform

  #32332 opened Aug 27, 2024

• The PreCommit GoPortable job is flaky

  #32324 opened Aug 27, 2024

• [Feature Request]: Auto-detect Schema for Parquet Sink

  #32322 opened Aug 27, 2024

• [Failing Test]:

  #32315 opened Aug 26, 2024

• [Feature Request]: Add A Freature to get bigquery schema dynamically from a POJO class.

  #32311 opened Aug 25, 2024

• [Bug]: Beam Interactive Tour Mismatch

  #32309 opened Aug 24, 2024

• [Feature Request]: Running Word-Count with Gradle for PowerShell

  #32307 opened Aug 24, 2024

• [Bug]: Issue with the script for adding execute task in "Convert from Maven to Gradle" section

  #32306 opened Aug 24, 2024

• [Feature Request]: Protobuf 5+ support

  #32302 opened Aug 23, 2024

• [Bug][Prism]: Java Prism Runner tests always pass despite Job failure

  #32263 opened Aug 21, 2024

56 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• Add correct shutdown mechanics for direct path

  #31902 commented on Aug 27, 2024 • 20 new comments

• [yaml] Adding Spanner IO Providers for Beam YAML

  #31987 commented on Aug 27, 2024 • 14 new comments

• Improve BatchElements documentation

  #32082 commented on Aug 22, 2024 • 9 new comments

• Rag opensearch usecase with Beam's MLTransform

  #32018 commented on Aug 23, 2024 • 8 new comments

• Add JobServerOption for `--jar_cache_dir`

  #32033 commented on Aug 28, 2024 • 5 new comments

• Remove expensive shuffle of read data in KafkaIO when using sdf and commit offsets

  #31682 commented on Aug 21, 2024 • 2 new comments

• add ExternalTransformProvider example

  #30666 commented on Aug 27, 2024 • 2 new comments

• added custom watermark for kinesis reader

  #28763 commented on Aug 24, 2024 • 0 new comments

• [Feature Request]: Support writing to a Solace message broker

  #31905 commented on Aug 27, 2024 • 0 new comments

• The PerformanceTests Kafka IO job is flaky

  #31929 commented on Aug 27, 2024 • 0 new comments

• The PostCommit Java Examples Dataflow ARM job is flaky

  #31901 commented on Aug 27, 2024 • 0 new comments

• [Python] Add a couple quality-of-life improvemenets to `testing.util.assert_that`

  #30771 commented on Aug 27, 2024 • 0 new comments

• Managed BigQueryIO

  #31486 commented on Aug 26, 2024 • 0 new comments

• [Python] Managed Transforms API

  #31495 commented on Aug 26, 2024 • 0 new comments

• The LoadTests Python Combine Flink Streaming job is flaky

  #30520 commented on Aug 26, 2024 • 0 new comments

• Support writing to Pubsub with ordering key; Add PubsubMessage SchemaCoder

  #31608 commented on Aug 26, 2024 • 0 new comments

• [Feature Request]: Add examples for using metrics with Python to Beam Website

  #30419 commented on Aug 25, 2024 • 0 new comments

• Add jobLabelsMap parameter to BigQueryOptions

  #31698 commented on Aug 24, 2024 • 0 new comments

• Optimize Datastream for batch

  #31950 commented on Aug 23, 2024 • 0 new comments

• Add logging for DataChangeRecord and ChangeStreamQuery

  #31956 commented on Aug 24, 2024 • 0 new comments

• [yaml] Add tests for provider schema validation

  #31975 commented on Aug 22, 2024 • 0 new comments

• SolaceIO write connector

  #32060 commented on Aug 27, 2024 • 0 new comments

• make FieldValueTypeInformation creators take a TypeDescriptor parameter

  #32081 commented on Aug 26, 2024 • 0 new comments

• Kafka poll interval

  #32162 commented on Aug 28, 2024 • 0 new comments

• added methods for support ValueProvider in `MqttIO`

  #32181 commented on Aug 26, 2024 • 0 new comments

• Bump commons-cli:commons-cli from 1.8.0 to 1.9.0

  #32191 commented on Aug 26, 2024 • 0 new comments

• Bump cloud.google.com/go/pubsub from 1.41.0 to 1.42.0 in /sdks

  #32250 commented on Aug 21, 2024 • 0 new comments

• WIP to improve custom delimiter to support overlapping and spanning buffers without exception.

  #32258 commented on Aug 21, 2024 • 0 new comments

• The TypeScript Tests job is flaky

  #30506 commented on Aug 22, 2024 • 0 new comments

• The PostCommit Java PVR Spark Batch job is flaky

  #30512 commented on Aug 22, 2024 • 0 new comments

• The PostCommit XVR Direct job is flaky

  #30517 commented on Aug 22, 2024 • 0 new comments

• The PostCommit XVR GoUsingJava Dataflow job is flaky

  #30519 commented on Aug 22, 2024 • 0 new comments

• The PostCommit Python ValidatesContainer Dataflow With RC job is flaky

  #30525 commented on Aug 22, 2024 • 0 new comments

• The PostCommit Java IO Performance Tests job is flaky

  #30527 commented on Aug 22, 2024 • 0 new comments

• The PostCommit Java Sickbay job is flaky

  #30529 commented on Aug 22, 2024 • 0 new comments

• The PostCommit Java Nexmark Dataflow job is flaky

  #30606 commented on Aug 22, 2024 • 0 new comments

• [Bug]: ReadAllFiles does not fully read gzipped files from GCS

  #31040 commented on Aug 22, 2024 • 0 new comments

• MQTT IO connector for Python

  #21060 commented on Aug 22, 2024 • 0 new comments

• [Task]: Sunset Python 3.8 support

  #31192 commented on Aug 22, 2024 • 0 new comments

• [Task]: Migrate to build with Java11

  #31677 commented on Aug 22, 2024 • 0 new comments

• Portable runners should be able to issue checkpoints to Splittable DoFn

  #20979 commented on Aug 23, 2024 • 0 new comments

• The Dask Runner Tests job is flaky

  #30528 commented on Aug 23, 2024 • 0 new comments

• The PostCommit Sickbay Python job is flaky

  #30882 commented on Aug 23, 2024 • 0 new comments

• The LoadTests Go SideInput Flink Batch job is flaky

  #30515 commented on Aug 23, 2024 • 0 new comments

• The LoadTests Go GBK Flink Batch job is flaky

  #30507 commented on Aug 23, 2024 • 0 new comments

• The LoadTests Go CoGBK Flink Batch job is flaky

  #30502 commented on Aug 23, 2024 • 0 new comments

• The LoadTests Go Combine Flink Batch job is flaky

  #30521 commented on Aug 23, 2024 • 0 new comments

• The PerformanceTests xlang KafkaIO Python job is flaky

  #30526 commented on Aug 23, 2024 • 0 new comments

• [Feature Request]: Improve performance of generating successful writes in Java's BigqueryIO connector

  #31918 commented on Aug 23, 2024 • 0 new comments

• [Bug]: Java SDK BigQueryIO's RowMutationInformation class is not backward compatible with previous releases

  #31993 commented on Aug 23, 2024 • 0 new comments

• [Task]: Update the minor version of protobuf library in the upper bound prior to Beam release.

  #25590 commented on Aug 23, 2024 • 0 new comments

• [Task][Prism]: Create a PrismRunner

  #31793 commented on Aug 23, 2024 • 0 new comments

• [Tracking Umbrella] Prism Runner areas for contribution.

  #29650 commented on Aug 23, 2024 • 0 new comments

• [Task][prism]: Be able to execute non-Go SDKs on Prism.

  #28187 commented on Aug 23, 2024 • 0 new comments

• The Clean Up Prebuilt SDK Images job is flaky

  #30514 commented on Aug 24, 2024 • 0 new comments

• Add support for dynamic destinations when writing to MQTT

  #19376 commented on Aug 25, 2024 • 0 new comments