Acknowledgements
Describe the bug
When we use transferManager.UploadObject to upload a file under high write contention (being written to as it's being uploaded), and the upload is retried, we get a BadDigest / checksum mismatch error:
operation error S3: PutObject, https response error StatusCode: 400, RequestID: , HostID: , api error BadDigest: The X-Amz-Checksum-Crc32 you specified did not match what we received.
It looks like the checksum is being cached and reused on retry.
Regression Issue
Expected Behavior
I expect the checksum to be computed again on retry.
Current Behavior
I think the checksum is being cached and reused on retry.
I let Copilot dig into it, and it identified code in middleware_compute_input_checksum.go, where the checksum looks like it is cached to m.checksum and then reused on retry.
Reproduction Steps
In our code, we pass a read-only file handle as an io.Reader into transferManager.UploadObject.
This is done in parallel with other handles which write to the file during upload.
This issue only occurs when the upload is retried. Unfortunately, I don't know how to make that retry happen for testing or issue reproduction.
Possible Solution
Compute a fresh checksum every time the stream is rewound for a new attempt.
If this is a performance issue, maybe a flag could be added to UploadObjectInput to direct the SDK to compute fresh checksums on every attempt, for cases like this, where the stream source can be written to in parallel.
Additional Information/Context
I'm not thrilled with the fact that we're uploading a file in high write contention, but our code prioritizes fast writes over this concern. And waiting for write contention to die down before uploading might not always be possible.
AWS Go SDK V2 Module Versions Used
github.com/aws/aws-sdk-go-v2 v1.41.5
github.com/aws/aws-sdk-go-v2/config v1.32.14
github.com/aws/aws-sdk-go-v2/credentials v1.19.14
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.1.14
github.com/aws/aws-sdk-go-v2/service/s3 v1.98.0
github.com/aws/smithy-go v1.24.3
Compiler and Version used
go version go1.26.2 linux/amd64
Operating System and version
Ubuntu 24.04
Acknowledgements
go get -u github.com/aws/aws-sdk-go-v2/...)Describe the bug
When we use transferManager.UploadObject to upload a file under high write contention (being written to as it's being uploaded), and the upload is retried, we get a BadDigest / checksum mismatch error:
operation error S3: PutObject, https response error StatusCode: 400, RequestID: , HostID: , api error BadDigest: The X-Amz-Checksum-Crc32 you specified did not match what we received.It looks like the checksum is being cached and reused on retry.
Regression Issue
Expected Behavior
I expect the checksum to be computed again on retry.
Current Behavior
I think the checksum is being cached and reused on retry.
I let Copilot dig into it, and it identified code in middleware_compute_input_checksum.go, where the checksum looks like it is cached to m.checksum and then reused on retry.
Reproduction Steps
In our code, we pass a read-only file handle as an io.Reader into transferManager.UploadObject.
This is done in parallel with other handles which write to the file during upload.
This issue only occurs when the upload is retried. Unfortunately, I don't know how to make that retry happen for testing or issue reproduction.
Possible Solution
Compute a fresh checksum every time the stream is rewound for a new attempt.
If this is a performance issue, maybe a flag could be added to UploadObjectInput to direct the SDK to compute fresh checksums on every attempt, for cases like this, where the stream source can be written to in parallel.
Additional Information/Context
I'm not thrilled with the fact that we're uploading a file in high write contention, but our code prioritizes fast writes over this concern. And waiting for write contention to die down before uploading might not always be possible.
AWS Go SDK V2 Module Versions Used
github.com/aws/aws-sdk-go-v2 v1.41.5
github.com/aws/aws-sdk-go-v2/config v1.32.14
github.com/aws/aws-sdk-go-v2/credentials v1.19.14
github.com/aws/aws-sdk-go-v2/feature/s3/transfermanager v0.1.14
github.com/aws/aws-sdk-go-v2/service/s3 v1.98.0
github.com/aws/smithy-go v1.24.3
Compiler and Version used
go version go1.26.2 linux/amd64
Operating System and version
Ubuntu 24.04