-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Bucket name sometimes missing in the S3 URL - causing failures for the same bucket and code that was previously successful for operations like get object and put object. #4187
Comments
Hi @mjoeydba thanks for reaching out. Here is a guide on troubleshooting Access Denied errors in S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/troubleshoot-403-errors.html That error is likely occurring due to your settings, policies, permissions, or profile configuration. But if you'd like us to investigate this further on the SDK side, please share a complete code snippet to reproduce the issue, as well as debug logs (with any sensitive info redacted) by adding |
Hi @tim-finnigan. Thanks for the response. The only difference I see is the bucket name is missing in the S3 URL when there is a failure. AWS support also confirmed that when the error occurs the bucket name being sent is the first level folder under the bucket. |
Thanks for following up and your patience here. Looking at the logs, I see several references to Java, which makes me wonder if the issue here has something to do with |
Hi Tim, We are seeing the issue across different clusters we are using and the pattern is the same. It works fine for a while after cluster startup but breaks and the only fix is to restart the cluster. While we are running this in databricks/spark environment would that cause boto3 to use the wrong S3 URL? It looks to me like it is switching from virtual hosted url to path url but is missing the bucket name in the path. |
Here is Boto3 documentation on ![]() As noted there it is |
Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one. |
Describe the bug
S3 access failing for the same bucket and code that was previously successful. Debug trace shows that the URL used during failure does not include the bucket name either as host or in the path.
Success
2024-06-27 21:21:12,116 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Bucket': 'xxxx', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Key': 'xxxx/xxxx.xlsx', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-06-27 21:21:12,116 botocore.regions [DEBUG] Endpoint provider result: https://xxxx.s3.amazonaws.com
Failure
2024-07-02 18:22:11,094 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-07-02 18:22:11,095 botocore.regions [DEBUG] Endpoint provider result: https://s3.amazonaws.com
The URL in the getobject call is also showing same behavior which seems to cause the access denied error.
Expected Behavior
Successfully download object.
Current Behavior
Failure with Access Denied after it worked successfully for the same code.
Reproduction Steps
#Note : The issue occurrence is unpredictable.
import pandas as pd
import boto3
from io import BytesIO
from pyspark.sql.functions import upper
import logging
from botocore.config import Config
boto3.set_stream_logger('', logging.DEBUG)
#boto3.set_stream_logger('')
Initialize S3 client
s3 = boto3.client('s3')
INBOUND_S3_BUCKET = "xxxx"
INBOUND_FILE_PATH = 'xxx/xxxx.xlsx'
obj = s3.get_object(Bucket = INBOUND_S3_BUCKET, Key = INBOUND_FILE_PATH)
Possible Solution
Unknown
Additional Information/Context
No response
SDK version used
1.34.137
Environment details (OS name and version, etc.)
Linux, databricks
The text was updated successfully, but these errors were encountered: