Find solution
How to sequence events while uploading large files to Amazon S3?

Do Dropbox clients upload users' files directly to Amazon's S3 service?

  • I want to know if my files are uploaded to s3 servers directly or uploaded to Dropbox's EC2 instances first and then get backed up to s3.

  • Answer:

    Refer to the document: http://eprints.eemcs.utwente.nl/22286/01/imc140-drago.pdf (Inside Dropbox: Understanding Personal Cloud Storage Services) Section 2.4 depicts the data storage flow. Dropbox clients always communicate with EC2 instances, and never with S3 storage directly. Dropbox server (EC2 instances) need to do some processing on file before transferring them to s3 - encrypting file using server side encryption technology. - calculating checksum on the content of file for de-duplication check. - resuming broken uploads. - calculating meta data information from the file contents (e.g file size).    For these processing, dropbox server need to be aware of file content. So dropbox client would be uploading contents to dropbox servers, and then they get further transferred to s3.

Lalit Kumar Bhasin at Quora Visit the source

Was this solution helpful to you?

Other answers

If I wrote the code I would enable the files to be uploaded directly to S3, provided there was no other work to be performed on the files prior to storing them. One of the reasons to use a service like S3 is to avoid having to provision infrastructure (like EC2 instances) to stage the uploads. How it's actually implemented depends on the way DropBox wrote its code. Some of the work such as encryption, checksums, meta data could likely be performed "in-line." In other words, some analysis of the file could be performed by the application, yet the application could direct the upload to S3 without storing the file at an intermediate location. At the scale on which DropBox is operating performing any kind of "pre-processing" inline is a very compelling approach.

Brian McCallion

Related Q & A:

Just Added Q & A:

Find solution

For every problem there is a solution! Proved by Solucija.

  • Got an issue and looking for advice?

  • Ask Solucija to search every corner of the Web for help.

  • Get workable solutions and helpful tips in a moment.

Just ask Solucija about an issue you face and immediately get a list of ready solutions, answers and tips from other Internet users. We always provide the most suitable and complete answer to your question at the top, along with a few good alternatives below.