logging - Append to a log file in S3 (or any other AWS service) - Stack Overflow
"What you are looking for is possible with S3. S3 does not need to have the complete file to begin storing data. You can use the multipart upload to store chunks (5mb blocks) at a time. Once you have finished, say a day's run, you can send the final block and close the multipart upload. and begin logging the next day.
S3 allows up to 10,000 parts. So by choosing a part-size of 5MiB you will be able to upload dynamic files of up to 50GiB. Should be enough for most use-cases.
Initiate S3 Multipart Upload. Gather data into a buffer until that buffer reaches S3's lower chunk-size limit (5MB). Generate MD5 checksum while building up the buffer. Upload that buffer as a Part, store the ETag. Once you reach EOF of your data, upload the last chunk (which can be smaller than 5MiB). Finalize the Multipart Upload.
This is a way better solution as it doesn't cost you and EC2 instance or EBS storage costs"
No comments:
Post a Comment