GCS : Spring Boot service gets blocked with virtual thread and library version >= 2.25.0 during file upload

[Liveitabhi14]

I was upgrading a codebase from Java 11 to Java 21 + Spring Boot 3.2.5
I then enabled virtual threads.
The codebase contains interactions with GCS via the Java Client library google-cloud-storage. So I upgraded the library version to the latest available one (2.37.0)
The codebase has mainly two interactions with Google Cloud storage, one for downloading and the other for uploading a file.

With the above-mentioned changes, the download functionality is working fine (even though there might be pinning of virtual threads). But the upload functionality is broken.

We are using the below line to upload an excel file to the GCS bucket.
storage.createFrom(blobInfo, new ByteArrayInputStream(excelBytes));

After enabling debug logs, these are the major logs that i am able to see :-

First is a post request

c.google.api.client.http.HttpTransport : -------------- REQUEST --------------
POST https://storage.googleapis.com/upload/storage/v1/b/&uploadType=resumable
Accept-Encoding: gzip
Authorization:
User-Agent: gcloud-java/2.37.0 Google-HTTP-Java-Client/1.44.1 (gzip)
x-goog-api-client: gl-java/21.0.2__Oracle-Corporation gccl/2.37.0 gax/2.47.0 gccl-invocation-id/"invocation-id"
x-upload-content-type: application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
x-goog-gcs-idempotency-token:
Content-Type: application/json; charset=UTF-8
Content-Length: 145

Following this is the response log for the above request

c.google.api.client.http.HttpTransport : -------------- RESPONSE --------------
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Server: UploadServer
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
X-GUploader-UploadID: "upload-id"
Vary: Origin
Vary: X-Origin
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Content-Length: 0
Date: Wed, 15 May 2024 09:28:43 GMT
Content-Type: text/plain; charset=utf-8
Location: https://storage.googleapis.com/upload/storage/v1/b/&uploadType=resumable&upload_id="upload-id"

After this, there is another log but for a PUT request

c.google.api.client.http.HttpTransport : -------------- REQUEST --------------
PUT https://storage.googleapis.com/upload/storage/v1/b/&uploadType=resumable&upload_id=
Accept-Encoding: gzip
Authorization:
Content-Range: bytes 0-3596/3597
User-Agent: gcloud-java/2.37.0 Google-HTTP-Java-Client/1.44.1 (gzip)
x-goog-api-client: gl-java/21.0.2__Oracle-Corporation gccl/2.37.0 gax/2.47.0 gccl-invocation-id/
x-goog-gcs-idempotency-token:
Content-Length: 3597

And then there is this log at the end

c.google.api.client.http.HttpTransport : PK - [Content_Types].xml�S�n�0 ����*6�PU �C � �� {�X�%����] 8�R�
�� ��(�A� �p(� t S6��9�)�JG���sBQ�4���0 �eX� 9 u�1��� dg9�2�ω^�� + ��#o�)� r� �ʝ4���WH�� ���/ ��/� D1,�C 1|��7PK �,(�; PK - rels/.rels���J 1 �%̽�m iڋ ��� ��ݰ�LHFݾ����lA��03��1�v?�I�S.���uӂ�h��� x9>��@ ��p�H NT`��>ӄRW��SQ� ��A$�k]�@ KÉb�t� J-s� � {қ����' Ι�� �[�:b�I ̓��<�2�M���)�oB�뼥 �o��,d_L�^v�|�8�O��&���24 EGn�j e���W�n �,g����Gс
Q/��� �} PK n2 K� J PK - docProps/app.xmlM��
�0 D�EȽ��ADҔ� ��A? ��6�lB�J?ߜ���0���ͯ�)�@��׍ H6���V> ��$;�SC
�GS�b�� ��
~ � PK �|wؑ � PK - docProps/core.xmlm�]K�0 ��J�}���d��C���8p2�.$Ƕ�|�D��{�:+�wI��<����{� >t�T�� #0Ҫ�4 ~�o�5F!
��� �!
xl/styles.xml���n� ��{
��� lۻ� �| �,
�=�� {��v�K W \VS��hc���ޣ� )� ���AߏY���4��+�� � �瘏��Z^F�� �5FϮ�� PK g뢨� 4 PK - xl/worksheets/sheet1.xml��Qo�0 ���),�� S��JRuP
� xl/styles.xmlPK - Pv�� \ > xl/workbook.xmlPK - g뢨� 4 \ xl/_rels/workbook.xml.relsPK - |lx(� q � xl/worksheets/sheet1.xmlPK ? �

After this, the execution keeps on going and the service becomes unresponsive.
Then I need to restart the spring boot service to be able to execute any other request on it.

I tried downgrading the google-cloud-storage library and at 2.24.0 version, its working fine.
One thing to note is that with 2.24.0 version where the flow is working fine, i am seeing the same order of logs as above 4.
The difference is that after the 4th one, there is a response log as well.

c.google.api.client.http.HttpTransport : -------------- RESPONSE --------------
HTTP/1.1 200 OK
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Server: UploadServer
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
ETag: CIjKv8uvj4YDEAE=
X-GUploader-UploadID: "upload-id"
Vary: Origin
Vary: X-Origin
Expires: Mon, 01 Jan 1990 00:00:00 GMT
Pragma: no-cache
Content-Length: 870
Date: Wed, 15 May 2024 09:45:10 GMT
Content-Type: application/json; charset=UTF-8

And this is followed by other logs trivial logs.
Also, the execution is completed immediately post which i am able to hit the service with any other requests.

[BenWhitehead]

I spent some time today trying to repro with just the JDK, skipping the Spring Boot Part.

I did this by submitting several (tested up to 1000 a few times) upload tasks similar to the following (attempting to approximate what a server might be doing processing requests).

    try (Storage s = StorageOptions.http().build().getService();
        ExecutorService exec = Executors.newVirtualThreadPerTaskExecutor()) {
      for (int i = 0; i < 1000; i++) {
        exec.submit(() -> {
          try {
            byte[] bytes = Files.readAllBytes(Paths.get("path/to/my/file"));
            BlobInfo info = BlobInfo.newBuilder("my_bucket", UUID.randomUUID().toString()).build();
            BlobInfo gen1 = s.createFrom(info, new ByteArrayInputStream(bytes));
            System.out.println("gen1 = " + gen1.getBlobId().toGsUtilUriWithGeneration());
          } catch (IOException e) {
            throw new RuntimeException(e);
          }
        });
      }
    }

Unfortunately, I was not able to reproduce the hang and all uploads completed successfully.
I was running Eclipse Temurin 21 downloaded from https://adoptium.net/temurin/releases/?os=linux&arch=x64 and running on Debian 11 on a GCE VM.

openjdk version "21.0.3" 2024-04-16 LTS
OpenJDK Runtime Environment Temurin-21.0.3+9 (build 21.0.3+9-LTS)
OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (build 21.0.3+9-LTS, mixed mode, sharing)

Can you provide any more details about where you're running (GCE, GKE, On Prem, etc) and if on GCE/GKE, what instance type and size you are using?
Would it be possible to get a thread dump from one of your hangs so we can see where it's hanging? jstack <pid> > threaddump.txt (Please grep out any stack frames you would not want to share)

[Liveitabhi14]

Unfortunately, I was not able to reproduce the hang and all uploads completed successfully

Did you enable virtual threads while trying to repro the case?
I checked by disabling virtual threads and didn't face the issue. So it should be something specific to virtual threads.

Can you provide any more details about where you're running (GCE, GKE, On Prem, etc) and if on GCE/GKE, what instance type and size you are using?

I am running the case in my local setup (mac m1).

Would it be possible to get a thread dump from one of your hangs so we can see where it's hanging?

sure, pfa the related thread dump
threaddump.txt

[BenWhitehead]

Yes Virtual Threads where enabled by using Executors.newVirtualThreadPerTaskExecutor() instead of Executors.newCachedThreadPool(). Unfortunately no hang was encountered.

sure, pfa the related thread dump
threaddump.txt

Thanks for sharing. Unfortunately, there isn't anything for com.google.cloud.storage in any of the stack frames. My guess is the runtime has already moved that virtual thread off any running thread.

I've got a PR up to reduce virtual thread pinning in Java 21 #2553 hopefully this can help. I'm hoping to have it merged in time for the next release.

[BenWhitehead]

Version 2.40.0 was released late last week that included the fix I mentioned previously.

Can you give the new version a try and see if it helps in your environment?

[Liveitabhi14]

Seems to be working fine now with 2.40.0 👍🏽
Thanks !

[BenWhitehead]

Awesome, thanks for confirming @Liveitabhi14!