How to make a Daemon application?

What are the ways of to make an application more resilient when RabbitMQ is down?

• We are trying to make a more resilient system. We have an architecture based on a message broker, in this case, RabbitMQ is the message broker used. Our application is basically  a video sharing web site. We have a web application written in Rails and a simple ruby application to edit videos (add a watermark, upload to youtube). The main use case of this application is to allow the user to upload videos. To complete this task the following steps are needed: 1 - upload video to S3 (directly from the web application) 2 - add a watermark to the video (ruby console application subscribes to RabbitMQ queue) 3 - upload this video to youtube (ruby console application) To put a watermark on a video, the ruby console application subscribes to a queue in RabbitMQ. When the process is completed, RabbitMQ will notify the Rails application that the video process is completed. What do you do if RabbitMQ is down? Imagine this happening in the middle of the editing process. In this case the rails app will never be notified when the editing process of a video is completed. Is there some way to avoid this scenario? Change the ruby  console to executing a series of retrying until the RabbitMQ is live again? Thanks

• Answer:

  Why does your RabbitMQ go down? What does your application do when RabbiqMQ is unavailable? What should it do when its unavailable? If you want your application to keep accepting work, even when the queue is unavailable, then you could: a) write the message to a local database, or queue, or file. b) write the message to a remote database, or file then have a process, as part of the application, that checks for these orphaned messages... and make sure that they are idempotent operations, so that it's ok if a message is accidentally processed multiple times.