Bucket name sometimes missing in the S3 URL - causing failures for the same bucket and code that was previously successful for operations like get object and put object.

[mjoeydba]

Describe the bug

S3 access failing for the same bucket and code that was previously successful. Debug trace shows that the URL used during failure does not include the bucket name either as host or in the path.

Success

2024-06-27 21:21:12,116 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Bucket': 'xxxx', 'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'Key': 'xxxx/xxxx.xlsx', 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-06-27 21:21:12,116 botocore.regions [DEBUG] Endpoint provider result: https://xxxx.s3.amazonaws.com

Failure

2024-07-02 18:22:11,094 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'us-east-1', 'UseFIPS': False, 'UseDualStack': False, 'ForcePathStyle': False, 'Accelerate': False, 'UseGlobalEndpoint': True, 'DisableMultiRegionAccessPoints': False, 'UseArnRegion': True}
2024-07-02 18:22:11,095 botocore.regions [DEBUG] Endpoint provider result: https://s3.amazonaws.com

The URL in the getobject call is also showing same behavior which seems to cause the access denied error.

Expected Behavior

Successfully download object.

Current Behavior

Failure with Access Denied after it worked successfully for the same code.

Reproduction Steps

#Note : The issue occurrence is unpredictable.
import pandas as pd
import boto3
from io import BytesIO
from pyspark.sql.functions import upper
import logging
from botocore.config import Config
boto3.set_stream_logger('', logging.DEBUG)
#boto3.set_stream_logger('')

Initialize S3 client

s3 = boto3.client('s3')
INBOUND_S3_BUCKET = "xxxx"
INBOUND_FILE_PATH = 'xxx/xxxx.xlsx'
obj = s3.get_object(Bucket = INBOUND_S3_BUCKET, Key = INBOUND_FILE_PATH)

Possible Solution

Unknown

Additional Information/Context

No response

SDK version used

1.34.137

Environment details (OS name and version, etc.)

Linux, databricks

[tim-finnigan]

Hi @mjoeydba thanks for reaching out. Here is a guide on troubleshooting Access Denied errors in S3: https://docs.aws.amazon.com/AmazonS3/latest/userguide/troubleshoot-403-errors.html

That error is likely occurring due to your settings, policies, permissions, or profile configuration. But if you'd like us to investigate this further on the SDK side, please share a complete code snippet to reproduce the issue, as well as debug logs (with any sensitive info redacted) by adding boto3.set_stream_logger('') to your script.

[mjoeydba]

Hi @tim-finnigan. Thanks for the response.
Please find attached logs and program. The issue is non deterministic. The code is running against the same bucket and EC2 instance profile.

The only difference I see is the bucket name is missing in the S3 URL when there is a failure. AWS support also confirmed that when the error occurs the bucket name being sent is the first level folder under the bucket.

program.txt
failure_log.txt
success_log.txt

[tim-finnigan]

Thanks for following up and your patience here. Looking at the logs, I see several references to Java, which makes me wonder if the issue here has something to do with py4j, spark or your VM rather than Boto3. Have you tried reproducing in other environments, or do you have any other updates on your end?

[mjoeydba]

Hi Tim,

We are seeing the issue across different clusters we are using and the pattern is the same. It works fine for a while after cluster startup but breaks and the only fix is to restart the cluster.

While we are running this in databricks/spark environment would that cause boto3 to use the wrong S3 URL?

It looks to me like it is switching from virtual hosted url to path url but is missing the bucket name in the path.

[tim-finnigan]

Here is Boto3 documentation on addressing_style: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.html

As noted there it is auto by default. Virtual style is recommended per https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html. Have you trying setting the addressing_style to virtual?

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.