Release Notes - ASF JIRA

Release Notes - Beam - Version 2.20.0 - HTML format

Configure Release Notes

Sub-task

[BEAM-4461] - Create a library of useful transforms that use schemas
[BEAM-7274] - Protobuf Beam Schema support
[BEAM-7333] - Select need the ability to rename fields
[BEAM-8616] - ParquetIO should have Hadoop dependencies as provided
[BEAM-8625] - Implement servlet in Dataflow runner for sdk status query endpoint
[BEAM-8626] - Implement status api handler in python sdk harness
[BEAM-8676] - Beam Dependency Update Request: com.google.api:gax-grpc
[BEAM-8685] - Beam Dependency Update Request: com.google.auth:google-auth-library-oauth2-http
[BEAM-8691] - Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
[BEAM-8695] - Beam Dependency Update Request: com.google.http-client:google-http-client
[BEAM-9146] - [Python] PTransform that integrates Video Intelligence functionality
[BEAM-9205] - Regression in validates runner tests configuration in spark module
[BEAM-9229] - Adding dependency information to Environment proto
[BEAM-9247] - [Python] PTransform that integrates Cloud Vision functionality
[BEAM-9248] - [Python] PTransform that integrates Cloud Natural Language functionality
[BEAM-9258] - [Python] PTransform that connects to Cloud DLP deidentification service
[BEAM-9262] - Update ApiServiceDescriptor to have open ended authentication method
[BEAM-9353] - ByteBuddy Schema code does not properly handle null values
[BEAM-9442] - Schema Select does not properly handle nested nullable fields

Bug

[BEAM-4409] - NoSuchMethodException reading from JmsIO
[BEAM-5086] - Beam Dependency Update Request: org.apache.kudu
[BEAM-6566] - SqlTransform does not work for beam version above 2.6.0 if RowCoder explicitly chosen
[BEAM-7427] - JmsCheckpointMark can not be correctly encoded
[BEAM-8374] - PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata
[BEAM-8490] - Python typehints: properly resolve empty dict type
[BEAM-8492] - Python typehints: don't try to strip_iterable from None
[BEAM-8525] - trivial_inference: slice of Const[List[T]] returns T instead of List[T]
[BEAM-8532] - Beam Python trigger driver sets incorrect timestamp for output windows.
[BEAM-8590] - Python typehints: native types: consider bare container types as containing Any
[BEAM-8629] - WithTypeHints._get_or_create_type_hints may return a mutable copy of the class type hints.
[BEAM-8739] - Consistently use with Pipeline(...) syntax
[BEAM-8965] - WriteToBigQuery failed in BundleBasedDirectRunner
[BEAM-9003] - test_reshuffle_preserves_timestamps (apache_beam.transforms.util_test.ReshuffleTest) does not work in Streaming VR suite on Dataflow
[BEAM-9057] - Make sure restriction_tracker.deferred_remainder is never called more than once for one <element, restriction>
[BEAM-9113] - Protobuf NanosType<T> serialization issues
[BEAM-9116] - Limit the number of past invocations stored in the job service
[BEAM-9124] - Upgrade linkage checker 1.1.2 to avoid Maven HTTPS problem
[BEAM-9126] - TestStreamTest.testDiscarding mode fails for Dataflow since Dataflow TestStream only updates watermarks at 1s resolution.
[BEAM-9132] - State request handler is removed prematurely when closing ActiveBundle
[BEAM-9143] - Add withOutputParallelization to RedisIO.Read/ReadAll
[BEAM-9161] - Inconsistent view of current ActiveBundle from main and bundle timer thread
[BEAM-9204] - HBase SDF @SplitRestriction does not take the range input into account to restrict splits
[BEAM-9215] - FileBesedSink may suppress exceptions during close
[BEAM-9218] - Template staging broken on Beam 2.18.0
[BEAM-9225] - Flink uber jar job server hangs
[BEAM-9227] - Perform bounded source computations on the worker.
[BEAM-9228] - _SDFBoundedSourceWrapper doesn't distribute data to multiple workers
[BEAM-9240] - Check for Nullability in typesEqual() method of FieldType class
[BEAM-9241] - Fix inconsistent nullability mapping for Protobuf to Schema
[BEAM-9242] - Processing Stuck messages are reported by Dataflow as errors
[BEAM-9252] - Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
[BEAM-9253] - SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
[BEAM-9265] - @RequiresTimeSortedInput does not respect allowedLateness
[BEAM-9277] - Beam 2.19 raises exception in IPython notebook
[BEAM-9288] - Conscrypt shaded dependency
[BEAM-9290] - runner_harness_container_image experiment is not honored in python released sdks.
[BEAM-9304] - beam-sdks-java-io-google-cloud-platform imports conflicting versions for BigTable and Spanner
[BEAM-9311] - ZetaSQL Named Parameters are lower case, don't treat as case-sensitive
[BEAM-9313] - beam_PostRelease_NightlySnapshot failure due to ClassNotFoundException: org.apache.beam.model.pipeline.v1.StandardWindowFns$SessionsPayload$Enum
[BEAM-9317] - PostCommit PVR failures
[BEAM-9333] - DataCatalogPipelineOptions is not registered
[BEAM-9345] - "Multiple environments cannot be created in detached mode"
[BEAM-9357] - Bump upper end of Google Bigquery dependencies for python
[BEAM-9394] - DynamicMessage handling of empty map violates schema nullability
[BEAM-9400] - Upgrade Linkage Checker 1.1.4
[BEAM-9405] - Python PostCommit is flaky: 'PortableRunner' object has no attribute 'create_job_service'
[BEAM-9413] - [beam_PostCommit_Py_ValCont] build failed
[BEAM-9417] - Unable to Read form BigQuery and File system in same pipeline
[BEAM-9423] - Re-Add the stop button to the Flink web interface for pipelines
[BEAM-9452] - PipelineResources algorithm is not working on Windows
[BEAM-9465] - Reshuffle should trigger repeatedly
[BEAM-9475] - New style metrics in portability throw error
[BEAM-9478] - Update samza runner page to reflect new changes
[BEAM-9485] - Dataflow Silently drops Non implemented transform in fnapi mode.
[BEAM-9503] - SyntaxError in process worker startup
[BEAM-9548] - Bad error handling with errors from TestStreamService when using Interactive Beam
[BEAM-9557] - Error setting processing time timers near end-of-window
[BEAM-9566] - Performance regression of FlinkRunner stream mode due to watermark holds update
[BEAM-9573] - Watermark hold for timer output timestamp is not computed correctly
[BEAM-9601] - Interactive test_streaming_wordcount failing
[BEAM-12500] - Dataflow SocketException (SSLException) error while trying to send message from Cloud Pub/Sub to BigQuery

New Feature

[BEAM-3453] - Allow usage of public Google PubSub topics in Python DirectRunner
[BEAM-6857] - Support dynamic timers
[BEAM-7810] - Allow ValueProvider arguments to ReadFromDatastore
[BEAM-8550] - @RequiresTimeSortedInput DoFn annotation
[BEAM-8561] - Add ThriftIO to Support IO for Thrift Files
[BEAM-8564] - Add LZO compression and decompression support
[BEAM-8614] - Expose SDK harness status to Runner through FnApi
[BEAM-9072] - [SQL] Add support for Datastore source
[BEAM-9149] - Support ZetaSQL positional parameters
[BEAM-9184] - Add ToSet() combiner, similar to ToList() and ToDict()
[BEAM-9220] - Add use_runner_v2 argument for dataflow
[BEAM-9305] - Support ValueProvider for BigQuerySource query string
[BEAM-9340] - Properly populate pipeline proto requirements.
[BEAM-9344] - Enable bundle finalization in Java SDK

Improvement

[BEAM-1833] - Restructure Python pipeline construction to better follow the Runner API
[BEAM-6120] - Support retrieval of large gbk iterables over the state API.
[BEAM-7310] - Confluent Schema Registry support in KafkaIO
[BEAM-8042] - Parsing of aggregate query fails
[BEAM-8271] - StateGetRequest/Response continuation_token should be string
[BEAM-8298] - Implement state caching for side inputs for Python
[BEAM-8335] - Add streaming support to Interactive Beam
[BEAM-8399] - Python HDFS implementation should support filenames of the format "hdfs://namenodehost/parent/child"
[BEAM-8537] - Provide WatermarkEstimatorProvider for different types of WatermarkEstimator
[BEAM-9022] - Publish spark job server container images in release process
[BEAM-9030] - Bump grpc to 1.26.0
[BEAM-9059] - Migrate PTransformTranslation to use string constants
[BEAM-9140] - Update to ZetaSQL 2020.01.1
[BEAM-9160] - Update AWS SDK to support Kubernetes Pod Level Identity
[BEAM-9162] - Upgrade Jackson to version 2.10.2
[BEAM-9169] - Extra character introduced during Calcite unparsing
[BEAM-9175] - Introduce an autoformatting tool to Python SDK
[BEAM-9176] - Update dataflow Java container
[BEAM-9203] - Programmatically determine if SQL exception is user error, unsupported, or bug
[BEAM-9230] - Enable CrossLanguageValidateRunner test for Spark runner
[BEAM-9231] - Annotate as Experimental/Internal missing classes in beam-sdks-java-core
[BEAM-9236] - Mark missing Schema based classes and methods as Experimental
[BEAM-9264] - Upgrade Spark to version 2.4.5
[BEAM-9268] - SpannerIO: Better documentation and warning about creating tables in the pipeline
[BEAM-9273] - Explicitly fail pipeline with @RequiresTimeSortedInput with unsupported runner
[BEAM-9276] - python: create a class to encapsulate the work required to submit a pipeline to a job service
[BEAM-9280] - Update commons-compress to version 1.20
[BEAM-9281] - Update commons-csv to version 1.8
[BEAM-9292] - Provide an ability to specify additional maven repositories for published POMs
[BEAM-9296] - Add typing annotation to python SDF
[BEAM-9315] - HadoopFileSystemOptions unable to interpret HADOOP_CONF_DIR with multiple paths
[BEAM-9326] - JsonToRow transform should not use bounded Wildcards for its input
[BEAM-9329] - Support request of schemas by version on KafkaIO + Confluent Schema Registry
[BEAM-9343] - Upgrade ZetaSQL to 2020.02.1
[BEAM-9349] - Upgrade to joda time 2.10.5 to get updated TZDB
[BEAM-9352] - Ensure consistent usage of jackson version brought in by transitive dependencies
[BEAM-9359] - Use DataCatalog client libraries rather than gRPC stubs
[BEAM-9364] - Refactor KafkaIO to use DeserializerProviders
[BEAM-9436] - Improve performance of GBK

Test

[BEAM-8141] - Add an integration test suite for cross-language transforms for Spark runner
[BEAM-9347] - Remove default image for Unified Worker

Task

[BEAM-4150] - Standardize use of PCollection coder proto attribute
[BEAM-6628] - Update GCP dependencies to a recent version
[BEAM-7518] - Protobuf Schema: Introduce logical type for Timestamp, Duration and other
[BEAM-8437] - Consider using native doubles in standard_coders_test.yaml
[BEAM-9037] - Instant and duration as logical type
[BEAM-9063] - Migrate docker images to apache namespace.
[BEAM-9084] - Cleaning up SDK docker image tagging
[BEAM-9121] - Bump vendored calcite to 1.21.0
[BEAM-9263] - Bump python sdk fnapi version to enable status reporting
[BEAM-9301] - Add a script to check Java linkage issues (beam-linkage-check.sh)
[BEAM-9310] - Make SpannerAccessor in Java package-private to reduce API surface
[BEAM-9472] - Remove excessive logging in python fn_api_runner
[BEAM-9560] - Beam Release Process breaks by "Project '1.10' not found in project ':runners:flink'"

Edit/Copy Release Notes

The text area below allows the project release notes to be edited and copied to another document.

Release Notes - Beam - Version 2.20.0
    
<h2>        Sub-task
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-4461'>BEAM-4461</a>] -         Create a library of useful transforms that use schemas
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-7274'>BEAM-7274</a>] -         Protobuf Beam Schema support
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-7333'>BEAM-7333</a>] -         Select need the ability to rename fields
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8616'>BEAM-8616</a>] -         ParquetIO should have Hadoop dependencies as provided
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8625'>BEAM-8625</a>] -         Implement servlet in Dataflow runner for sdk status query endpoint
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8626'>BEAM-8626</a>] -         Implement status api handler in python sdk harness
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8676'>BEAM-8676</a>] -         Beam Dependency Update Request: com.google.api:gax-grpc
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8685'>BEAM-8685</a>] -         Beam Dependency Update Request: com.google.auth:google-auth-library-oauth2-http
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8691'>BEAM-8691</a>] -         Beam Dependency Update Request: com.google.cloud.bigtable:bigtable-client-core
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8695'>BEAM-8695</a>] -         Beam Dependency Update Request: com.google.http-client:google-http-client
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9146'>BEAM-9146</a>] -         [Python] PTransform that integrates Video Intelligence functionality
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9205'>BEAM-9205</a>] -         Regression in validates runner tests configuration in spark module
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9229'>BEAM-9229</a>] -         Adding dependency information to Environment proto
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9247'>BEAM-9247</a>] -         [Python] PTransform that integrates Cloud Vision functionality
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9248'>BEAM-9248</a>] -         [Python] PTransform that integrates Cloud Natural Language functionality
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9258'>BEAM-9258</a>] -         [Python] PTransform that connects to Cloud DLP deidentification service
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9262'>BEAM-9262</a>] -         Update ApiServiceDescriptor to have open ended authentication method
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9353'>BEAM-9353</a>] -         ByteBuddy Schema code does not properly handle null values
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9442'>BEAM-9442</a>] -         Schema Select does not properly handle nested nullable fields
</li>
</ul>
            
<h2>        Bug
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-4409'>BEAM-4409</a>] -         NoSuchMethodException reading from JmsIO
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-5086'>BEAM-5086</a>] -         Beam Dependency Update Request: org.apache.kudu
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-6566'>BEAM-6566</a>] -         SqlTransform does not work for beam version above 2.6.0 if RowCoder explicitly chosen
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-7427'>BEAM-7427</a>] -         JmsCheckpointMark can not be correctly encoded
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8374'>BEAM-8374</a>] -         PublishResult returned by SnsIO is missing sdkResponseMetadata and sdkHttpMetadata
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8490'>BEAM-8490</a>] -         Python typehints: properly resolve empty dict type
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8492'>BEAM-8492</a>] -         Python typehints: don&#39;t try to strip_iterable from None
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8525'>BEAM-8525</a>] -         trivial_inference: slice of Const[List[T]] returns T instead of List[T]
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8532'>BEAM-8532</a>] -         Beam Python trigger driver sets incorrect timestamp for output windows.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8590'>BEAM-8590</a>] -         Python typehints: native types: consider bare container types as containing Any
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8629'>BEAM-8629</a>] -         WithTypeHints._get_or_create_type_hints may return a mutable copy of the class type hints.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8739'>BEAM-8739</a>] -         Consistently use with Pipeline(...) syntax
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8965'>BEAM-8965</a>] -         WriteToBigQuery failed in BundleBasedDirectRunner
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9003'>BEAM-9003</a>] -         test_reshuffle_preserves_timestamps (apache_beam.transforms.util_test.ReshuffleTest) does not work in Streaming VR suite on Dataflow
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9057'>BEAM-9057</a>] -         Make sure restriction_tracker.deferred_remainder is never called more than once for one &lt;element, restriction&gt;
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9113'>BEAM-9113</a>] -         Protobuf NanosType&lt;T&gt; serialization issues
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9116'>BEAM-9116</a>] -         Limit the number of past invocations stored in the job service
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9124'>BEAM-9124</a>] -         Upgrade linkage checker 1.1.2 to avoid Maven HTTPS problem
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9126'>BEAM-9126</a>] -         TestStreamTest.testDiscarding mode fails for Dataflow since Dataflow TestStream only updates watermarks at 1s resolution.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9132'>BEAM-9132</a>] -         State request handler is removed prematurely when closing ActiveBundle
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9143'>BEAM-9143</a>] -         Add withOutputParallelization to RedisIO.Read/ReadAll
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9161'>BEAM-9161</a>] -         Inconsistent view of current ActiveBundle from main and bundle timer thread
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9204'>BEAM-9204</a>] -         HBase SDF @SplitRestriction does not take the range input into account to restrict splits
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9215'>BEAM-9215</a>] -         FileBesedSink may suppress exceptions during close
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9218'>BEAM-9218</a>] -         Template staging broken on Beam 2.18.0
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9225'>BEAM-9225</a>] -         Flink uber jar job server hangs
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9227'>BEAM-9227</a>] -         Perform bounded source computations on the worker.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9228'>BEAM-9228</a>] -         _SDFBoundedSourceWrapper doesn&#39;t distribute data to multiple workers
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9240'>BEAM-9240</a>] -         Check for Nullability in typesEqual() method of FieldType class
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9241'>BEAM-9241</a>] -         Fix inconsistent nullability mapping for Protobuf to Schema
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9242'>BEAM-9242</a>] -         Processing Stuck messages are reported by Dataflow as errors
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9252'>BEAM-9252</a>] -         Problem shading Beam pipeline with Beam 2.20.0-SNAPSHOT
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9253'>BEAM-9253</a>] -         SQL PostCommit failure: ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9265'>BEAM-9265</a>] -         @RequiresTimeSortedInput does not respect allowedLateness
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9277'>BEAM-9277</a>] -         Beam 2.19 raises exception in IPython notebook
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9288'>BEAM-9288</a>] -         Conscrypt shaded dependency
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9290'>BEAM-9290</a>] -         runner_harness_container_image experiment is not honored in python released sdks.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9304'>BEAM-9304</a>] -         beam-sdks-java-io-google-cloud-platform imports conflicting versions for BigTable and Spanner
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9311'>BEAM-9311</a>] -         ZetaSQL Named Parameters are lower case, don&#39;t treat as case-sensitive
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9313'>BEAM-9313</a>] -         beam_PostRelease_NightlySnapshot failure due to ClassNotFoundException: org.apache.beam.model.pipeline.v1.StandardWindowFns$SessionsPayload$Enum
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9317'>BEAM-9317</a>] -         PostCommit PVR failures
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9333'>BEAM-9333</a>] -         DataCatalogPipelineOptions is not registered
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9345'>BEAM-9345</a>] -         &quot;Multiple environments cannot be created in detached mode&quot;
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9357'>BEAM-9357</a>] -         Bump upper end of Google Bigquery dependencies for python
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9394'>BEAM-9394</a>] -         DynamicMessage handling of empty map violates schema nullability
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9400'>BEAM-9400</a>] -         Upgrade Linkage Checker 1.1.4
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9405'>BEAM-9405</a>] -         Python PostCommit is flaky: &#39;PortableRunner&#39; object has no attribute &#39;create_job_service&#39;
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9413'>BEAM-9413</a>] -         [beam_PostCommit_Py_ValCont] build failed
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9417'>BEAM-9417</a>] -         Unable to Read form BigQuery and File system in same pipeline
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9423'>BEAM-9423</a>] -         Re-Add the stop button to the Flink web interface for pipelines
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9452'>BEAM-9452</a>] -         PipelineResources algorithm is not working on Windows
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9465'>BEAM-9465</a>] -         Reshuffle should trigger repeatedly
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9475'>BEAM-9475</a>] -         New style metrics in portability throw error
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9478'>BEAM-9478</a>] -         Update samza runner page to reflect new changes
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9485'>BEAM-9485</a>] -         Dataflow Silently drops Non implemented transform in fnapi mode.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9503'>BEAM-9503</a>] -         SyntaxError in process worker startup
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9548'>BEAM-9548</a>] -         Bad error handling with errors from TestStreamService when using Interactive Beam
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9557'>BEAM-9557</a>] -         Error setting processing time timers near end-of-window
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9566'>BEAM-9566</a>] -         Performance regression of FlinkRunner stream mode due to watermark holds update
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9573'>BEAM-9573</a>] -         Watermark hold for timer output timestamp is not computed correctly
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9601'>BEAM-9601</a>] -         Interactive test_streaming_wordcount failing
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-12500'>BEAM-12500</a>] -         Dataflow SocketException (SSLException) error while trying to send message from Cloud Pub/Sub to BigQuery
</li>
</ul>
            
<h2>        New Feature
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-3453'>BEAM-3453</a>] -         Allow usage of public Google PubSub topics in Python DirectRunner
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-6857'>BEAM-6857</a>] -         Support dynamic timers
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-7810'>BEAM-7810</a>] -         Allow ValueProvider arguments to ReadFromDatastore
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8550'>BEAM-8550</a>] -         @RequiresTimeSortedInput DoFn annotation
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8561'>BEAM-8561</a>] -         Add ThriftIO to Support IO for Thrift Files
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8564'>BEAM-8564</a>] -         Add LZO compression and decompression support
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8614'>BEAM-8614</a>] -         Expose SDK harness status to Runner through FnApi
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9072'>BEAM-9072</a>] -         [SQL] Add support for Datastore source
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9149'>BEAM-9149</a>] -         Support ZetaSQL positional parameters
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9184'>BEAM-9184</a>] -         Add ToSet() combiner, similar to ToList() and ToDict()
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9220'>BEAM-9220</a>] -         Add use_runner_v2 argument for dataflow
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9305'>BEAM-9305</a>] -         Support ValueProvider for BigQuerySource query string
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9340'>BEAM-9340</a>] -         Properly populate pipeline proto requirements.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9344'>BEAM-9344</a>] -         Enable bundle finalization in Java SDK
</li>
</ul>
    
<h2>        Improvement
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-1833'>BEAM-1833</a>] -         Restructure Python pipeline construction to better follow the Runner API
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-6120'>BEAM-6120</a>] -         Support retrieval of large gbk iterables over the state API.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-7310'>BEAM-7310</a>] -         Confluent Schema Registry support in KafkaIO
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8042'>BEAM-8042</a>] -         Parsing of aggregate query fails
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8271'>BEAM-8271</a>] -         StateGetRequest/Response continuation_token should be string
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8298'>BEAM-8298</a>] -         Implement state caching for side inputs for Python
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8335'>BEAM-8335</a>] -         Add streaming support to Interactive Beam
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8399'>BEAM-8399</a>] -         Python HDFS implementation should support filenames of the format &quot;hdfs://namenodehost/parent/child&quot;
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8537'>BEAM-8537</a>] -         Provide WatermarkEstimatorProvider for different types of WatermarkEstimator
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9022'>BEAM-9022</a>] -         Publish spark job server container images in release process
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9030'>BEAM-9030</a>] -         Bump grpc to 1.26.0
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9059'>BEAM-9059</a>] -         Migrate PTransformTranslation to use string constants
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9140'>BEAM-9140</a>] -         Update to ZetaSQL 2020.01.1
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9160'>BEAM-9160</a>] -         Update AWS SDK to support Kubernetes Pod Level Identity
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9162'>BEAM-9162</a>] -         Upgrade Jackson to version 2.10.2
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9169'>BEAM-9169</a>] -         Extra character introduced during Calcite unparsing
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9175'>BEAM-9175</a>] -         Introduce an autoformatting tool to Python SDK
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9176'>BEAM-9176</a>] -         Update dataflow Java container
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9203'>BEAM-9203</a>] -         Programmatically determine if SQL exception is user error, unsupported, or bug
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9230'>BEAM-9230</a>] -         Enable CrossLanguageValidateRunner test for Spark runner
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9231'>BEAM-9231</a>] -         Annotate as Experimental/Internal missing classes in beam-sdks-java-core
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9236'>BEAM-9236</a>] -         Mark missing Schema based classes and methods as Experimental
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9264'>BEAM-9264</a>] -         Upgrade Spark to version 2.4.5
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9268'>BEAM-9268</a>] -         SpannerIO: Better documentation and warning about creating tables in the pipeline
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9273'>BEAM-9273</a>] -         Explicitly fail pipeline with @RequiresTimeSortedInput with unsupported runner
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9276'>BEAM-9276</a>] -         python: create a class to encapsulate the work required to submit a pipeline to a job service
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9280'>BEAM-9280</a>] -         Update commons-compress to version 1.20
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9281'>BEAM-9281</a>] -         Update commons-csv to version 1.8
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9292'>BEAM-9292</a>] -         Provide an ability to specify additional maven repositories for published POMs
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9296'>BEAM-9296</a>] -         Add typing annotation to python SDF
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9315'>BEAM-9315</a>] -         HadoopFileSystemOptions unable to interpret HADOOP_CONF_DIR with multiple paths
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9326'>BEAM-9326</a>] -         JsonToRow transform should not use bounded Wildcards for its input
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9329'>BEAM-9329</a>] -         Support request of schemas by version on KafkaIO + Confluent Schema Registry
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9343'>BEAM-9343</a>] -         Upgrade ZetaSQL to 2020.02.1
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9349'>BEAM-9349</a>] -         Upgrade to joda time 2.10.5 to get updated TZDB
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9352'>BEAM-9352</a>] -         Ensure consistent usage of jackson version brought in by transitive dependencies
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9359'>BEAM-9359</a>] -         Use DataCatalog client libraries rather than gRPC stubs
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9364'>BEAM-9364</a>] -         Refactor KafkaIO to use DeserializerProviders
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9436'>BEAM-9436</a>] -         Improve performance of GBK
</li>
</ul>
    
<h2>        Test
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8141'>BEAM-8141</a>] -         Add an integration test suite for cross-language transforms for Spark runner
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9347'>BEAM-9347</a>] -         Remove default image for Unified Worker
</li>
</ul>
        
<h2>        Task
</h2>
<ul>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-4150'>BEAM-4150</a>] -         Standardize use of PCollection coder proto attribute
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-6628'>BEAM-6628</a>] -         Update GCP dependencies to a recent version
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-7518'>BEAM-7518</a>] -         Protobuf Schema: Introduce logical type for Timestamp, Duration and other
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-8437'>BEAM-8437</a>] -         Consider using native doubles in standard_coders_test.yaml
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9037'>BEAM-9037</a>] -         Instant and duration as logical type 
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9063'>BEAM-9063</a>] -         Migrate docker images to apache namespace.
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9084'>BEAM-9084</a>] -         Cleaning up SDK docker image tagging
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9121'>BEAM-9121</a>] -         Bump vendored calcite to 1.21.0
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9263'>BEAM-9263</a>] -         Bump python sdk fnapi version to enable status reporting
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9301'>BEAM-9301</a>] -         Add a script to check Java linkage issues (beam-linkage-check.sh)
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9310'>BEAM-9310</a>] -         Make SpannerAccessor in Java package-private to reduce API surface
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9472'>BEAM-9472</a>] -         Remove excessive logging in python fn_api_runner
</li>
<li>[<a href='https://issues.apache.org/jira/browse/BEAM-9560'>BEAM-9560</a>] -         Beam Release Process breaks by &quot;Project &#39;1.10&#39; not found in project &#39;:runners:flink&#39;&quot;
</li>
</ul>