Release Notes - Beam - Version 2.36.0 - HTML format

Sub-task

  • [BEAM-10212] - Add support for Java SDK harness state caching
  • [BEAM-13507] - Python PostCommit Failures -
  • [BEAM-13508] - Java PostCommits (Examples, PerformanceTests, XVR] Failing with ClassNotFound
  • [BEAM-13645] - Vendor gRPC 1.43.2
  • [BEAM-13646] - Vendor calcite with protobuf 3.19.2

Bug

  • [BEAM-12587] - Python "object" row type rejects null values.
  • [BEAM-12869] - Upgrade tensorflow to 2.6.x
  • [BEAM-13009] - DynamoDBIO misses writing items if `withDeduplicateKeys` is not set
  • [BEAM-13209] - DynamoDBIO silently drops unprocessed items
  • [BEAM-13213] - OnWindowExpiration does not work without other state
  • [BEAM-13286] - RowJson fails to serialize DATETIMEs produced by ZetaSQL
  • [BEAM-13294] - innerBroadcastJoin fails when batch column Nullable, streaming column not
  • [BEAM-13313] - Use snake case for field name for WindowingStrategy proto
  • [BEAM-13341] - Serialization of AvroCoder for dataflow workers discard reflect option
  • [BEAM-13354] - Java SDK Harness MultimapUserState not using structural value when comparing keys
  • [BEAM-13374] - ToJson() of a Row with a DateTime field fails with an exception
  • [BEAM-13402] - Pub/Sub Lite commits can take 10s of hours without completing
  • [BEAM-13421] - Python DeferredDataFrame.xs differs from Pandas
  • [BEAM-13443] - Poor handling of aggregated records in KinesisIO.read
  • [BEAM-13445] - Fix chunked encoding for S3WritableButeChannel
  • [BEAM-13454] - Dataframe read_fwf fails reading incrementally.
  • [BEAM-13456] - beam_PostCommit_Java consistently timing out.
  • [BEAM-13467] - Schema translation to proto doesn't handle null arguements
  • [BEAM-13526] - Unable to use the KafkaIO.Read.withValueDeserializer(DeserializerProvider) method
  • [BEAM-13539] - Gradle warning: processResources overrides
  • [BEAM-13599] - Overflow in Python Datastore RampupThrottlingFn
  • [BEAM-13603] - Shared object does not read from cache when using tag
  • [BEAM-13604] - NPE while getting null from BigDecimal column
  • [BEAM-13611] - CrossLanguageJdbcIOTest.test_xlang_jdbc_write failing in Python PostCommits
  • [BEAM-13616] - Update protobuf-java to 3.19.2 and other vendored dependencies that use protobuf
  • [BEAM-13628] - [Go SDK] Make Side input cache fit resolved semantics.
  • [BEAM-13665] - Spanner IO request metrics requires projectId within the config when it didn't in the past
  • [BEAM-13668] - Java Spanner IO Request Count metrics broke backwards compatibility
  • [BEAM-13716] - Clear before creating a new virtual environment in setupVirtualenv
  • [BEAM-13830] - XVR Direct/Spark/Flink tests are timing out
  • [BEAM-14004] - [CosmosDB] com.mongodb.MongoCommandException: Command failed with error 115: Command is not supported

New Feature

  • [BEAM-12865] - Allow customising batch duration when streaming with WriteToBigQuery
  • [BEAM-13052] - Pub/Sub Lite support for Python SDK
  • [BEAM-13159] - Support for Redis 5.0 streams (XADD/XREAD/XRANGE) in RedisIO
  • [BEAM-13351] - Add a connector for FhirIO's GetPatientEverything functionality
  • [BEAM-13389] - Enable JdbcIO expansion service to more easily receive JDBC Driver as a dependency
  • [BEAM-13617] - StorageApi sink might try to reuse closed stream in batch mode and in STORAGE_API_AT_LEAST_ONCE mode
  • [BEAM-13697] - Add SchemaFieldNumber annotation

Improvement

  • [BEAM-3304] - Go triggering support
  • [BEAM-7559] - Add an Integration Test for DynamoDBIO
  • [BEAM-9706] - Add ignoreUnknownValues flag to beam.io.WriteToBigQuery
  • [BEAM-10584] - Replace getErrorInfo by quotaExceeded method in InsertAll for BigQuery
  • [BEAM-11703] - Support apache-beam python install on ARM64
  • [BEAM-12561] - Implement truncate for DataFrame and Series
  • [BEAM-12565] - Implement compare for DataFrame and Series
  • [BEAM-13171] - Support for stopReadTime on KafkaIO SDF
  • [BEAM-13233] - Replace AWS API used to list Kinesis shards from DescribeStream to ListShards (SDK v2)
  • [BEAM-13244] - Add STS Assume role credentials provider to AwsModule (AWS SDK v2)
  • [BEAM-13276] - Beam Dependency Update Request: com.fasterxml.jackson.core
  • [BEAM-13335] - DataFrame sources produce excessively large index
  • [BEAM-13397] - Update numpy required version to >=1.21.0 for M1 chip
  • [BEAM-13410] - Add integration test for S3FileSystem
  • [BEAM-13430] - Upgrade Gradle version to 7.3
  • [BEAM-13441] - Use quiet mode for batch deletes in AWS S3FileSystem
  • [BEAM-13459] - Dataflow python runner should cache uploaded artifacts across job runs
  • [BEAM-13481] - Upgrade shadow plugin (log4j)
  • [BEAM-13488] - ClassGraph scanner is nondeterministic/incomplete when using parallel scans
  • [BEAM-13494] - KinesisIO.read requires default rater limiter for low throughput shards
  • [BEAM-13541] - Use runtime information to improve CoGroupByKey caching
  • [BEAM-13591] - Bump log4j2 version to 2.17.1 due to the vulnerability

Task

  • [BEAM-12733] - [beam_PostCommit_Java] org.apache.beam.sdk.extensions.ml.RecommendationAIPredictIT.predict failing
  • [BEAM-13236] - Properly close kinesis producer on teardown
  • [BEAM-13781] - grpc-netty-shaded version conflict

Edit/Copy Release Notes

The text area below allows the project release notes to be edited and copied to another document.