Sub-task
- [BEAM-7937] - Support Hadoop 3.x on Hadoop File System
- [BEAM-8615] - Support Hadoop 3.x on Hadoop Format
- [BEAM-8639] - Beam Dependency Update Request: org.apache.kafka:kafka-clients
- [BEAM-8774] - Beam Dependency Update Request: org.apache.kafka:kafka_2.11
- [BEAM-9782] - README on how to keep go katas up-to-date
Bug
- [BEAM-7794] - DynamoDBIOTest is blocking forever
- [BEAM-10706] - DynamoDBIO fail to write to the same key in short consecution
- [BEAM-11016] - Serialize AwsCredentialsProvider for AWS SDK v2 IO connectors
- [BEAM-11208] - BigQuery storage streams fail with QUOTA_EXCEEDED errors in split
- [BEAM-11216] - StreamingDataflowWorker ReaderCache usage can be incorrect in presence of retries
- [BEAM-11230] - ReadFromBigQuery fails when the table has repeated records
- [BEAM-11266] - Cannot use Python MongoDB connector with Atlas MongoDB
- [BEAM-11331] - AwsOptions from sdk.io.aws2 is incompatible with DataflowPipelineOptions
- [BEAM-11375] - Java & Spark postcommits failing.
- [BEAM-11384] - Improve StreamingDataflow worker to avoid escaping the processing key and including with generated json logs
- [BEAM-11386] - BigQuery streaming inserts should fail if SchemaUpdateOptions specified
- [BEAM-11400] - StreamingDataflowWorker stuck commits logic triggers exceptions if commits eventually complete
- [BEAM-11401] - StreamingDataflowWorker ReaderCache performs blocking close of readers on possibly important threads
- [BEAM-11412] - beam_PostCommit_Python_VR_Spark failing
- [BEAM-11413] - spark_runner_test default job server is incorrect.
- [BEAM-11417] - StreamingDataflowWorker can leak UnboundedSource finalization callbacks
- [BEAM-11433] - Dataflow Multi-language pipelines broken for Java 11 users
- [BEAM-11435] - Reuse already set timers instead of recreating them in Flink Streaming Runner
- [BEAM-11472] - taxi_dataframe example broken
- [BEAM-11481] - FlinkRunner might not advance output watermark on change of watermark hold
- [BEAM-11524] - (Deprecated) WriteStringsToPubSub broken on Dataflow
- [BEAM-11529] - Issue with trying to run Kafka to Pub/Sub example with unencrypted communication between Kafka and the pipeline
- [BEAM-12265] - FlatMapElement Kotlin Beam non Serializable lambda
New Feature
- [BEAM-10438] - Turn on functions supported by BeamZetaSqlCalcRel
- [BEAM-10896] - Support UNNEST an array of structs
- [BEAM-10962] - Multiple PubSub Source Reader
- [BEAM-11173] - Create Bigtable table provider
- [BEAM-11338] - Beam schema for thrift data
- [BEAM-11373] - Add write support to Bigtable Table for BeamSQL
Improvement
- [BEAM-7003] - Test KafkaIO against different versions of Kafka
- [BEAM-8569] - Support Hadoop 3 on Beam
- [BEAM-9278] - Make HBase client a provided dependency in HBaseIO
- [BEAM-10095] - Add hyperlinks to the beam-overview page.
- [BEAM-10278] - Remove Python-dependent tasks from runner modules
- [BEAM-10992] - Add ability to select case format for schema fields inferred from Java types
- [BEAM-11065] - Apache Beam Template to ingest from Apache Kafka to Google Pub/Sub
- [BEAM-11226] - Support larger tables in BigQuery storage API bounded source
- [BEAM-11312] - When prebuilding workflow on CloudBuild fails, we should tell the user where to look for build logs, ideally a link, or a pointer where the build logs are.
- [BEAM-11337] - Make the ThriftCoder class visible from outside its package
- [BEAM-11343] - Make ExpansionServiceClient accesible outside of core-construction-java
- [BEAM-11360] - Enable portable job submission for Dataflow multi-language pipelines (Python SDK)
- [BEAM-11365] - Running Direct Runner Test on Windows
- [BEAM-11374] - Add filters to Bigtable Table for BeamSQL
- [BEAM-11419] - Add flink 1.12 build target
- [BEAM-11426] - FhirIO Search for Beam connector
- [BEAM-11438] - Dockerfile for dev container updates (macOS)
- [BEAM-11444] - MongoDB Python connector shows password in display_data
- [BEAM-11474] - Populate transform id field in Java SDK harness logs.
- [BEAM-11522] - PR/13220 introduced a regression in CombineFn.from_callable
Test
- [BEAM-8876] - Tests to confirm Hadoop version compatibility
- [BEAM-11172] - Set up Java Kafka performance test with Dataflow runner v2
- [BEAM-11407] - Add IT test to Bigtable for BeamSQL
Edit/Copy Release Notes
The text area below allows the project release notes to be edited and copied to another document.