Release Notes - Beam - Version 2.27.0 - HTML format

Sub-task

  • [BEAM-7937] - Support Hadoop 3.x on Hadoop File System
  • [BEAM-8615] - Support Hadoop 3.x on Hadoop Format
  • [BEAM-8639] - Beam Dependency Update Request: org.apache.kafka:kafka-clients
  • [BEAM-8774] - Beam Dependency Update Request: org.apache.kafka:kafka_2.11
  • [BEAM-9782] - README on how to keep go katas up-to-date

Bug

  • [BEAM-7794] - DynamoDBIOTest is blocking forever
  • [BEAM-10706] - DynamoDBIO fail to write to the same key in short consecution
  • [BEAM-11016] - Serialize AwsCredentialsProvider for AWS SDK v2 IO connectors
  • [BEAM-11208] - BigQuery storage streams fail with QUOTA_EXCEEDED errors in split
  • [BEAM-11216] - StreamingDataflowWorker ReaderCache usage can be incorrect in presence of retries
  • [BEAM-11230] - ReadFromBigQuery fails when the table has repeated records
  • [BEAM-11266] - Cannot use Python MongoDB connector with Atlas MongoDB
  • [BEAM-11331] - AwsOptions from sdk.io.aws2 is incompatible with DataflowPipelineOptions
  • [BEAM-11375] - Java & Spark postcommits failing.
  • [BEAM-11384] - Improve StreamingDataflow worker to avoid escaping the processing key and including with generated json logs
  • [BEAM-11386] - BigQuery streaming inserts should fail if SchemaUpdateOptions specified
  • [BEAM-11400] - StreamingDataflowWorker stuck commits logic triggers exceptions if commits eventually complete
  • [BEAM-11401] - StreamingDataflowWorker ReaderCache performs blocking close of readers on possibly important threads
  • [BEAM-11412] - beam_PostCommit_Python_VR_Spark failing
  • [BEAM-11413] - spark_runner_test default job server is incorrect.
  • [BEAM-11417] - StreamingDataflowWorker can leak UnboundedSource finalization callbacks
  • [BEAM-11433] - Dataflow Multi-language pipelines broken for Java 11 users
  • [BEAM-11435] - Reuse already set timers instead of recreating them in Flink Streaming Runner
  • [BEAM-11472] - taxi_dataframe example broken
  • [BEAM-11481] - FlinkRunner might not advance output watermark on change of watermark hold
  • [BEAM-11524] - (Deprecated) WriteStringsToPubSub broken on Dataflow
  • [BEAM-11529] - Issue with trying to run Kafka to Pub/Sub example with unencrypted communication between Kafka and the pipeline
  • [BEAM-12265] - FlatMapElement Kotlin Beam non Serializable lambda

New Feature

  • [BEAM-10438] - Turn on functions supported by BeamZetaSqlCalcRel
  • [BEAM-10896] - Support UNNEST an array of structs
  • [BEAM-10962] - Multiple PubSub Source Reader
  • [BEAM-11173] - Create Bigtable table provider
  • [BEAM-11338] - Beam schema for thrift data
  • [BEAM-11373] - Add write support to Bigtable Table for BeamSQL

Improvement

  • [BEAM-7003] - Test KafkaIO against different versions of Kafka
  • [BEAM-8569] - Support Hadoop 3 on Beam
  • [BEAM-9278] - Make HBase client a provided dependency in HBaseIO
  • [BEAM-10095] - Add hyperlinks to the beam-overview page.
  • [BEAM-10278] - Remove Python-dependent tasks from runner modules
  • [BEAM-10992] - Add ability to select case format for schema fields inferred from Java types
  • [BEAM-11065] - Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub
  • [BEAM-11226] - Support larger tables in BigQuery storage API bounded source
  • [BEAM-11312] - When prebuilding workflow on CloudBuild fails, we should tell the user where to look for build logs, ideally a link, or a pointer where the build logs are.
  • [BEAM-11337] - Make the ThriftCoder class visible from outside its package
  • [BEAM-11343] - Make ExpansionServiceClient accesible outside of core-construction-java
  • [BEAM-11360] - Enable portable job submission for Dataflow multi-language pipelines (Python SDK)
  • [BEAM-11365] - Running Direct Runner Test on Windows
  • [BEAM-11374] - Add filters to Bigtable Table for BeamSQL
  • [BEAM-11419] - Add flink 1.12 build target
  • [BEAM-11426] - FhirIO Search for Beam connector
  • [BEAM-11438] - Dockerfile for dev container updates (macOS)
  • [BEAM-11444] - MongoDB Python connector shows password in display_data
  • [BEAM-11474] - Populate transform id field in Java SDK harness logs.
  • [BEAM-11522] - PR/13220 introduced a regression in CombineFn.from_callable

Test

  • [BEAM-8876] - Tests to confirm Hadoop version compatibility
  • [BEAM-11172] - Set up Java Kafka performance test with Dataflow runner v2
  • [BEAM-11407] - Add IT test to Bigtable for BeamSQL

Edit/Copy Release Notes

The text area below allows the project release notes to be edited and copied to another document.