AWS Glue returns 'ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden' when using Boto3

I am running a glue job in AWS account 'A', and accessing a file from s3 in AWS account 'B'. I have access to both accounts. When I run the code to pull sed file:

bucket = 'bucket-name'filename = 's3-file-name.rdb'local_filename = 'temp.rdb's3 = boto3.client('s3')s3.download_file(bucket, filename, local_filename)

I get the following error:

  File "/tmp/myScript.py", line 127, in <module>    s3.download_file(bucket, filename, local_filename)  File "/home/spark/.local/lib/python3.10/site-packages/boto3/s3/inject.py", line 190, in download_file    return transfer.download_file(  File "/home/spark/.local/lib/python3.10/site-packages/boto3/s3/transfer.py", line 320, in download_file    future.result()  File "/home/spark/.local/lib/python3.10/site-packages/s3transfer/futures.py", line 103, in result    return self._coordinator.result()  File "/home/spark/.local/lib/python3.10/site-packages/s3transfer/futures.py", line 266, in result    raise self._exception  File "/home/spark/.local/lib/python3.10/site-packages/s3transfer/tasks.py", line 269, in _main    self._submit(transfer_future=transfer_future, **kwargs)  File "/home/spark/.local/lib/python3.10/site-packages/s3transfer/download.py", line 354, in _submit    response = client.head_object(  File "/home/spark/.local/lib/python3.10/site-packages/botocore/client.py", line 508, in _api_call    return self._make_api_call(operation_name, kwargs)  File "/home/spark/.local/lib/python3.10/site-packages/botocore/client.py", line 915, in _make_api_call    raise error_class(parsed_response, operation_name)botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

I am aware of the iAM access needed, and have checked and confirmed I conform with everything mentioned in the accepted answer of boto.exception.S3ResponseError: S3ResponseError: 403 Forbidden, however, to no avail.

I can confirm that the iAM policy attached to the glue job has s3:read, and s3:get access, and have double checked the policy on the bucket and it contains the following:

{"Sid": "ReadOnlyFromAccountA","Effect": "Allow","Principal": {"AWS": "arn:aws:iam::ACCOUNT_ID:root"    },"Action": ["s3:List*","s3:Get*"    ],"Resource": ["arn:aws:s3:::required-bucket","arn:aws:s3:::required-bucket/*"    ]}

Also, interestingly, when I run the following code locally it works without issue:

bucket = 'bucket-name'filename = 's3-file-name.rdb'local_filename = 'temp.rdb'session = boto3.Session(profile_name='AWS_ACCOUNT_B')s3 = session.client('s3')s3.download_file(bucket, filename, local_filename)

Since I am running this locally I have to specify to use the account 'B' credentials from my ~/.aws/credentials. Again, this works fine and has no issue or error in downloading the file, leading me to believe that everything should work fine in glue since the iAM role will provide boto3 with the required credentials, since it is allowed to access account 'B'.

I am unsure why this is not working and if anyone with a bigger brain than me could figure this out it would be much appreciated. Any other info required just drop a comment and I'll respond straight away.