Sub-task
- [BEAM-2528] - BeamSql: support create table
- [BEAM-2600] - Python SDK harness container
- [BEAM-2876] - Add provision api proto
- [BEAM-2881] - GCS artifact server proxy
- [BEAM-2904] - Python SDK support for portable progress reporting
- [BEAM-2906] - Dataflow supports portable progress reporting
- [BEAM-2949] - Initial splitting
- [BEAM-2963] - Use portable ParDoPayload in DataflowRunner
- [BEAM-3125] - Portable flattens in Java SDK Harness
- [BEAM-3143] - Fix type inference in Python 3 for generators
- [BEAM-3179] - [Nexmark][SQL] Refactor Generator
- [BEAM-3181] - [Nexmark][SQL] Implement a basic pass-through SQL query
- [BEAM-3422] - The jar files uploaded to maven do not work with Java 9
- [BEAM-3427] - Upgrade Maven and Gradle to build using Java 8
- [BEAM-3428] - Merge java and java8 examples all together
- [BEAM-3430] - Update website documentation about Apache Beam 2.3.0+ being Java 8
- [BEAM-3432] - Merge sdks/java/io/hadoop/jdk1.8-tests into sdks/java/io/hadoop/input-format
- [BEAM-3461] - Drop redundant "beam-" from project names in gradle
- [BEAM-3466] - Remove Java 7 and any related task from Jenkins
- [BEAM-3467] - Remove Java 7 from the docker development (reproducible) build images / website
- [BEAM-3534] - Add a spark validates runner test for metrics sink in streaming mode
- [BEAM-3563] - Revise Fn API metrics protos
Bug
- [BEAM-409] - Incorrect use of Math.ceil in ApproximateQuantiles
- [BEAM-682] - Invoker Class should be created in Thread Context Classloader
- [BEAM-793] - JdbcIO can create a deadlock when parallelism is greater than 1
- [BEAM-1187] - GCP Transport not performing timed backoff after connection failure
- [BEAM-1487] - BufferingStreamObserverTest, BeamFnLoggingClientTest repeatedly times out in precommit
- [BEAM-1868] - CreateStreamTest testMultiOutputParDo is flaky on the Spark runner
- [BEAM-2101] - Reading public GCS files requires authentication
- [BEAM-2102] - deprecated use of pip --download command
- [BEAM-2257] - KafkaIO write without key requires a producer fn
- [BEAM-2270] - Examples archetype bundles Hadoop 2.6 in its jar for ApexRunner; cannot run on Hadoop 2.7?
- [BEAM-2271] - Release guide or pom.xml needs update to avoid releasing Python binary artifacts
- [BEAM-2273] - mvn clean doesn't fully clean up archetypes.
- [BEAM-2304] - State declared with one class cannot be accessed as a superclass (applies to BagState|CombiningState <: GroupingState)
- [BEAM-2320] - Update DataflowRunner dependency check for Google / Apache Beam Distribution
- [BEAM-2498] - Dataflow runner should shade Runner/Fn API protos
- [BEAM-2566] - Java SDK harness should not depend on any runner
- [BEAM-2576] - Split up portability model artifacts
- [BEAM-2607] - Enforce that SDF must return stop() after a failed tryClaim() call
- [BEAM-2704] - KafkaIO: NPE without key serializer set
- [BEAM-2779] - PipelineOptionsFactory should prevent non PipelineOptions interfaces from being constructed.
- [BEAM-2870] - BQ Partitioned Table Write Fails When Destination has Partition Decorator
- [BEAM-2872] - AvroIO.TypedWrite#to() method produces a compilation error under JDK 1.7
- [BEAM-2957] - Fix flaky ElasticsearchIOTest.testSplit in beam-sdks-java-io-elasticsearch-tests-5
- [BEAM-2996] - Metric names should not be null or empty
- [BEAM-3013] - The Python worker should report lulls
- [BEAM-3016] - BeamFnLoggingClient shutdown stability
- [BEAM-3030] - watchForNewFiles() can emit a file multiple times if it's growing
- [BEAM-3048] - RAND_RANGE in WindowedWordCount is not used
- [BEAM-3049] - Java SDK Harness bundles non-relocated code, including Dataflow runner
- [BEAM-3051] - no beta on gcloud
- [BEAM-3052] - ReduceFnRunner sets end-of-window hold even when no data is buffered
- [BEAM-3054] - org.apache.beam.sdk.io.elasticsearch.ElasticsearchIOTest is flaky
- [BEAM-3088] - BigQuery source should consider streaming buffer when determining estimated sizes of tables
- [BEAM-3107] - Python Fnapi based workloads failing
- [BEAM-3110] - The transform Read(UnboundedKafkaSource) is currently not supported
- [BEAM-3118] - Fix thread leaks in ElasticsearchIO 5 write tests
- [BEAM-3120] - Jenkins postcommit test suite triggered for a release branch uses code from master branch.
- [BEAM-3121] - Dockerized jekyll server fails
- [BEAM-3130] - View.asMap() causes a ClassCastException in Apex runner
- [BEAM-3137] - BigQueryIO.write() should better verify user schemas
- [BEAM-3139] - Update dataflow.version in beam root pom.xml
- [BEAM-3155] - Python SDKHarness does not use separate thread for progress reporting
- [BEAM-3160] - Type based coder inference incorrectly assumes that a coder for one type is equivalent to every other coder for that type.
- [BEAM-3161] - Cannot output with timestamp XXXX
- [BEAM-3174] - Master python sdk seems broken with test_harness_override_present_in_dataflow_distributions on Py 2.7.6
- [BEAM-3186] - In-flight data loss when restoring from savepoint
- [BEAM-3187] - Spark runner does not respect ParDo's lifecycle on case of exceptions
- [BEAM-3196] - Python postcommit test_wordcount_fnapi_it failing on Dataflow
- [BEAM-3202] - Multiple deserializations of PipelineOptions leaks memory
- [BEAM-3206] - Spark runner does not shut down after pipeline completion
- [BEAM-3219] - DataflowRunner: @Setup not called for batch stateful DoFn
- [BEAM-3240] - Quickstart examples archetype dependencies conflict for Apex on YARN
- [BEAM-3244] - Flink runner does not respect ParDo's lifecycle on case of exceptions
- [BEAM-3282] - MqttIO reader should use receive with timeout
- [BEAM-3284] - Python SDK, dataflow runner, the method modify_job_status is calling the wrong API endpoint
- [BEAM-3336] - MqttIO read tests are flaky
- [BEAM-3348] - Beam SQL DSL not support non-ascii characters in sql
- [BEAM-3354] - Timer setRelative() does not reset previously set timer.
- [BEAM-3357] - Python SDK head fails to run tests due to Requirement.parse('protobuf<=3.4.0,>=3.2.0')
- [BEAM-3362] - Create an example pipeline that uses State.
- [BEAM-3366] - FnApiDoFnRunner should be registered for the ParDo URN
- [BEAM-3369] - Python HEAD fails tests due to a ValueError
- [BEAM-3382] - Validate count for trigger AfterCount
- [BEAM-3387] - Bigtable integration tests failed due to client version dependency change
- [BEAM-3389] - Clean receive queue in dataplane
- [BEAM-3391] - Upgrade apitools to >= 0.5.18
- [BEAM-3397] - beam_PreCommit_Java_MavenInstall failing on Dataflow integration test because of too long commandline
- [BEAM-3411] - Test apache_beam.examples.wordcount_it_test.WordCountIT times out
- [BEAM-3416] - File is not properly closed in VcfSource when exception is thrown
- [BEAM-3436] - RetryHttpRequestInitializerTest takes 4min to complete
- [BEAM-3438] - KinesisReaderIT fails due to a missing PipelineOptions property
- [BEAM-3450] - RemoteGrpcPorts should contain the wire format
- [BEAM-3452] - NullPointerException when executing BigQuerySourceBase.split()
- [BEAM-3458] - Go SDK beam.Create & beam.CreateList should support complex types
- [BEAM-3478] - Flink checkstyle broken
- [BEAM-3486] - Progress reporting for python sdk fix
- [BEAM-3492] - Spark Integration Tests fail with a Closed Connection
- [BEAM-3499] - Watch can make no progress if a single poll takes more than checkpoint interval
- [BEAM-3510] - Python Runner API serialization for CombinePerKey drops arguments to CombineFn
- [BEAM-3511] - Eager evaluation of overridden transforms in Python SDK does not work
- [BEAM-3537] - Remove DirectRunner-specific internal PValue cache, allow more general eager in-process pipeline execution
- [BEAM-3569] - SpannerIO.write throws on delete mutations
- [BEAM-3570] - More elegant interface for RuntimeValueProvider
- [BEAM-3584] - Java dataflow job fails with 2.3.0 RC1, due to missing worker image
- [BEAM-3585] - Python dataflow job fails with 2.3.0 RC1, due to missing worker image
- [BEAM-3589] - Flink runner breaks with ClassCastException on UnboundedSource
- [BEAM-3592] - Spark-runner profile is broken on Nexmark after move to Spark 2.x
- [BEAM-3668] - Apache Spark Java Quickstart fails 2.3.0 RC2
- [BEAM-3958] - beam-sdks-java-io-amazon-web-services may be global pollution.
New Feature
- [BEAM-2430] - Java FnApiDoFnRunner to share across runners
- [BEAM-2500] - Add support for S3 as a Apache Beam FileSystem
- [BEAM-2718] - Add bundle retry logic to the DirectRunner
- [BEAM-2774] - Add I/O source for VCF files (python)
- [BEAM-2806] - support View.CreatePCollectionView in FlinkRunner
- [BEAM-2865] - Implement FileIO.write()
- [BEAM-3008] - BigtableIO should use ValueProviders
- [BEAM-3009] - Implement context access from user code closures
- [BEAM-3035] - Extract ReifyTimestampsAndWindows from GatherAllPanes
- [BEAM-3171] - convert a join into lookup
Improvement
- [BEAM-675] - Introduce message mapper in JmsIO
- [BEAM-1630] - Add Splittable DoFn to Python SDK
- [BEAM-1847] - KafkaIO can't specify both max records and max duration.
- [BEAM-1872] - implement Reshuffle transform in python, make it experimental in Java
- [BEAM-1920] - Add Spark 2.x support in Spark runner
- [BEAM-2482] - CodedValueMutationDetector should use the coders structural value
- [BEAM-2674] - Runner API translators should own their rehydration
- [BEAM-2804] - support TIMESTAMP in Sort
- [BEAM-2875] - Portable SDK harness containers
- [BEAM-3005] - Provision API should optionally include SDK harness memory/cpu/disk limits
- [BEAM-3018] - Remove duplicated methods in StructuredCoder
- [BEAM-3041] - Add portable Python SDK container setup support
- [BEAM-3063] - VoidCoder should implement structuralValue instead of forcing encoding of zero bytes
- [BEAM-3111] - Upgrade ElasticsearchIO elastic dependences to 5.6.3
- [BEAM-3112] - improve logs in ElasticsearchIO test utils
- [BEAM-3113] - Disable stack trace optimization in java container
- [BEAM-3114] - Do not generate ApiService text proto string manually in containers
- [BEAM-3133] - Staging files on Dataflow should not block forever without producing updates
- [BEAM-3135] - Adding futures dependency to beam python SDK
- [BEAM-3170] - support topicPartition in BeamKafkaTable
- [BEAM-3184] - Not able to access GCS API when submitting Python jobs behind corporate firewall
- [BEAM-3189] - Python Fnapi - SDK harness speedup
- [BEAM-3209] - Update documentation on support for reading/writing BQ date partitioned tables
- [BEAM-3238] - [SQL] Add builder to BeamRecordSqlType
- [BEAM-3239] - Enable debug server for python sdk workers
- [BEAM-3267] - Return file names from TFRecordIO write
- [BEAM-3275] - Support Kafka 1.0.0
- [BEAM-3308] - Improve Go exec runtime error handling
- [BEAM-3340] - Update Flink Runner to Flink 1.4.0
- [BEAM-3347] - Add worker ID to the provisioning API
- [BEAM-3356] - Add varint coder
- [BEAM-3373] - Add serviceEndpoint parameter to KinesisIO
- [BEAM-3384] - Make DataflowRunner.replaceTransforms protected
- [BEAM-3388] - Reduce Go runtime reflective overhead
- [BEAM-3404] - Update KinesisIO to use AWS SDK 1.11.255 and KCL 1.8.8
- [BEAM-3405] - Make maxNumRecords a long and validate if stream exists for KinesisIO
- [BEAM-3454] - Make Unbounded to Bounded conversions on IOs able to trigger on maxNumRecords and maxReadTime combined
- [BEAM-3502] - Avoid use of proto.Builder.clone() in DatastoreIO
- [BEAM-3507] - [JdbcIO] Allow to define the batch size in WriteFn
- [BEAM-3533] - Replace hard-coded UTF-8 Strings
- [BEAM-3539] - BigtableIO.Write javadoc of some methods is incorrect
- [BEAM-3560] - Switch to use BigInteger/BigDecimal.ZERO/ONE/TEN
Test
- [BEAM-3392] - Verify default container image names in Python SDK
Wish
- [BEAM-3551] - Add -parameters flag to javac (and test)
Task
- [BEAM-2994] - Refactor TikaIO
- [BEAM-3243] - multiple anonymous DoFn lead to conflicting names
- [BEAM-3294] - Move to graph.External and remove Source/Sink
- [BEAM-3426] - Java 8 support
Edit/Copy Release Notes
The text area below allows the project release notes to be edited and copied to another document.