Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WritePendingException over HTTP/2 tunnel #10145

Closed
sbordet opened this issue Jul 25, 2023 · 1 comment · Fixed by #10146
Closed

WritePendingException over HTTP/2 tunnel #10145

sbordet opened this issue Jul 25, 2023 · 1 comment · Fixed by #10146
Labels
Bug For general bugs on Jetty side

Comments

@sbordet
Copy link
Contributor

sbordet commented Jul 25, 2023

Jetty version(s)
10+

Description
HttpClient communicating to a https server through a forward proxy via HTTP/2 tunnel.

Caused by: java.nio.channels.WritePendingException
	at org.eclipse.jetty.http2.HTTP2StreamEndPoint.write(HTTP2StreamEndPoint.java:404)
	at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onIncompleteFlush(SslConnection.java:1291)
	at org.eclipse.jetty.io.AbstractEndPoint$2.onIncompleteFlush(AbstractEndPoint.java:53)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:283)
	at org.eclipse.jetty.io.WriteFlusher.write(WriteFlusher.java:254)
	at org.eclipse.jetty.io.AbstractEndPoint.write(AbstractEndPoint.java:386)
	at org.eclipse.jetty.http2.HTTP2Flusher.process(HTTP2Flusher.java:292)
	at org.eclipse.jetty.util.IteratingCallback.processing(IteratingCallback.java:243)
	at org.eclipse.jetty.util.IteratingCallback.iterate(IteratingCallback.java:224)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.flush(HTTP2Session.java:2332)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.createLocalStream(HTTP2Session.java:2230)
	at org.eclipse.jetty.http2.HTTP2Session$StreamsState.newLocalStream(HTTP2Session.java:2134)
	at org.eclipse.jetty.http2.HTTP2Session.newStream(HTTP2Session.java:603)
	at org.eclipse.jetty.http2.client.http.HttpSenderOverHTTP2.sendHeaders(HttpSenderOverHTTP2.java:127)
	at org.eclipse.jetty.client.HttpSender$ContentConsumer.onContent(HttpSender.java:496)
	at org.eclipse.jetty.client.util.AbstractRequestContent$AbstractSubscription.notifyContent(AbstractRequestContent.java:207)
	at org.eclipse.jetty.client.util.AbstractRequestContent$AbstractSubscription.processContent(AbstractRequestContent.java:181)
	at org.eclipse.jetty.client.util.BytesRequestContent$SubscriptionImpl.produceContent(BytesRequestContent.java:83)
	at org.eclipse.jetty.client.util.AbstractRequestContent$AbstractSubscription.produce(AbstractRequestContent.java:117)
	at org.eclipse.jetty.client.util.AbstractRequestContent$AbstractSubscription.demand(AbstractRequestContent.java:93)
	at org.eclipse.jetty.client.HttpSender.demand(HttpSender.java:237)
	at org.eclipse.jetty.client.HttpSender.send(HttpSender.java:83)
	at org.eclipse.jetty.http2.client.http.HttpChannelOverHTTP2.send(HttpChannelOverHTTP2.java:102)
	at org.eclipse.jetty.client.HttpChannel.send(HttpChannel.java:122)
	at org.eclipse.jetty.client.HttpConnection.send(HttpConnection.java:111)
	at org.eclipse.jetty.http2.client.http.HttpConnectionOverHTTP2.send(HttpConnectionOverHTTP2.java:104)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:440)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:416)
	at org.eclipse.jetty.client.HttpDestination.process(HttpDestination.java:371)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:354)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:348)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:325)
	at org.eclipse.jetty.client.HttpDestination.send(HttpDestination.java:304)
	at org.eclipse.jetty.client.HttpClient.send(HttpClient.java:591)
	at org.eclipse.jetty.client.HttpRequest.sendAsync(HttpRequest.java:819)
	at org.eclipse.jetty.client.HttpRequest.send(HttpRequest.java:807)
@sbordet sbordet added the Bug For general bugs on Jetty side label Jul 25, 2023
@sbordet
Copy link
Contributor Author

sbordet commented Jul 25, 2023

This is ultimately caused by the HTTP2StreamEndPoint.flush() implementation.

The semantic of EndPoint.flush() is that it should attempt to send some bytes from the buffers, and report back whether it was able to fully send them or not, but when flush() returns there should be no pending operations trying to send those bytes.

In particular, SslConnection.DecryptedEndPoint.onIncompleteFlush() is called when the flush is incomplete, for example because the HTTP/2 stream is flow-control congested, and -- given the flush was incomplete -- tries to write() the buffers.
This causes the WritePendingException because flush() switched to PENDING state after issuing a stream.data(), and the subsequent write() finds the PENDING state.

sbordet added a commit that referenced this issue Jul 25, 2023
@sbordet
Fixes #10145 - WritePendingException over HTTP/2 tunnel
e9f4013 
Method HTTP2StreamEndPoint.flush() has a "no pending operation" semantic, but the previous implementation was calling stream.data(), which may become a pending operation if the stream is congested.

Changed the implementation of flush() to return false in the IDLE and PENDING cases.
Now every flush() is converted to a write(), which has the same semantic as stream.data().

Signed-off-by: Simone Bordet <[email protected]>
@sbordet sbordet linked a pull request Jul 25, 2023 that will close this issue
Fixes #10145 - WritePendingException over HTTP/2 tunnel #10146
Merged
sbordet added a commit that referenced this issue Jul 26, 2023
@sbordet
Fixes #10145 - WritePendingException over HTTP/2 tunnel (#10146)
a5a0a6c 
Method HTTP2StreamEndPoint.flush() has a "no pending operation" semantic, but the previous implementation was calling stream.data(), which may become a pending operation if the stream is congested.

Changed the implementation of flush() to return false in the IDLE and PENDING cases.
Now every flush() is converted to a write(), which has the same semantic as stream.data().

Signed-off-by: Simone Bordet <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug For general bugs on Jetty side
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fixes #10145 - WritePendingException over HTTP/2 tunnel jetty/jetty.project
1 participant
@sbordet