Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need the new "unreachable" field to be supported by BigQuery::listDatasets API #3120

Open
danielferraz-git opened this issue Feb 5, 2024 · 10 comments
Labels
api: bigquery Issues related to the googleapis/java-bigquery API.

Comments

@danielferraz-git
Copy link

We are facing the same problem as issue googleapis/google-api-ruby-client#16987.

We need java API support for the new "unreachable" property in the BigQuery::listDatasets API response.

This is urgent, as this change will take place starting on Feb 15th, as announced by Google.

@product-auto-label product-auto-label bot added the api: bigquery Issues related to the googleapis/java-bigquery API. label Feb 5, 2024
@PhongChuong
Copy link
Contributor

FYI, we're currently discussing how to approach this internally.
I'll post if there are any updates (ETA Feb [12, 13, 14] 2024).

@sjmisterm
Copy link

Thanks @PhongChuong .

Given we're getting an answer (and probably not a release) the earliest by Feb 12th, would it be possible for you to contact the BigQuery team and hold off this change while the community has time to adapt their software? The fact we're the only ones who submitted an issue shows a lot of lack of unawareness about the topic in general and certainly users won't upgrade timely and handle the new API results.

@PhongChuong
Copy link
Contributor

Hi,
Sorry for the delay. We're still in discussion on how to forward the unreachable information to the users in a manageable manner.
I can pass the information to the BigQuery team to hold off this change. However, can you help me understand why that is needed?
In particular, what is concern regarding enabling the feature.

@sjmisterm
Copy link

@PhongChuong , sorry for the delay as I was on vacation.

As stated by Google itself in the announcement sent to users:

"Today, in the uncommon case of one or more GCP regions being unreachable for a prolonged amount of time, datasets.list and jobs.list API calls will fail if the response should contain data from the unreachable region.

Starting February 15, 2024, these calls will succeed, with the additional field unreachable added to mark the locations in which data might have been skipped from dataset.list and jobs.list response in case of a prolonged unreachability of the entire region or multi-region.

Under normal circumstances, the field will be empty. However, if you rely on getting a full list of all your datasets and jobs from dataset.list and jobs.list API, you should start interpreting values of the new field if present."

Our use case - and most users, I guess - depends on this call being 100% reliable, i.e., either failing with an exception or returning all the datasets. Now an empty return in a region might mean "temporarily off" or no "datasets at all", which completely break the API.

As suggested by Google itself:

"Check the content of this field to see if there are any unreachable locations. If your API integration with BigQuery depends on getting a full list of datasets/jobs from all regions/multi-regions for which relevant data exists, make sure to adjust it depending on specific circumstances.

You may choose to fail the workflow, issue a warning, or continue without warning in case an unreachable field is populated."

We must fail the workflow given our use case - and I guess that's what most users want to do -, but currently there is no way to find out the empty response is due to an unreachable location and to do so.

@PhongChuong
Copy link
Contributor

@sjmisterm, thanks for the context.
We'll add the unreachable field to this library. I'll update with an ETA soon.

@sjmisterm
Copy link

Hi @PhongChuong ,

It's been a while since the last message. Is there an ETA already?

@PhongChuong
Copy link
Contributor

Sorry for the delay. We're still trying to find staffing to work on this issue. I'll update as soon as we have a clear ETA.

@sjmisterm
Copy link

Hi @PhongChuong . It's been over 3 months since this change has been deployed and any partial results delivered by the API are wrongly assumed to be the whole data by all Java clients.

I understand you're not the one to blame, but this can potentially cause us a major problem at any time. Does Google really intend to fix this API issue or to revert this backend change?

@sjmisterm
Copy link

Hi @PhongChuong . Just pinging you again since it's been over three months from your last message and my previous one was ignored. Any news?

@PhongChuong
Copy link
Contributor

PhongChuong commented Jul 31, 2024

Hi, @sjmisterm ,
Sorry. I should have been more on top of this issue and keep everyone posted.
Unfortunately, my original proposal for Unreachable field fell through. The issue is that listDatasets and listJobs returns Page of dataset, job respectively which requires a longer term solution to how we handle additional fields in a paginated resource. As a result, this change is not exclusive to BigQuery.

I'll bring this issue up again to see if we can raise the priority.

Meanwhile, if you have a support contract with Google, please create this issue in the support console will help us prioritize the effort.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: bigquery Issues related to the googleapis/java-bigquery API.
Projects
None yet
Development

No branches or pull requests

3 participants