What is the best way to design a chat server and make it scalable?

As you said you are asking from a job interview perspective, the reality is that more often than not, particular techniques like Ejabberd may not be discussed. Instead, interviewers care more about how you would  design the system in general and how to scale the system to million of users.Basically, one of the most common ways to build a chat system is to have a chat server that acts as the core of the whole system. When a message comes, it won’t be sent to the receiver directly. Instead, it goes to the chat server and is stored there first. And then, based on the receiver’s status, the server may send the message immediately to him or send a push notification.This usually works well in small scale. When you want to support millions of users, there will a bunch of scale issues you need to solve. For instance, if you want to reduce cost of message delivery, one solution is to use https://en.wikipedia.org/wiki/HTTP_persistent_connection. In a nutshell, receivers can make an HTTP GET request over a persistent connection that doesn’t return until the chat server provides any data back. Each request will be re-established when it’s timed out or interrupt. This approach provides a lot of advantages in terms of response time, throughput and cost.http://blog.gainlo.co/index.php/2016/04/19/design-facebook-chat-function/?utm_source=quora&utm_medium=What+is+the+best+way+to+design+a+chat+server+and+make+it+scalable%3F&utm_campaign=quora has a detailed discussion about this topic from interview perspective. Basically, everything tends to break when scaling to certain level and a lot of optimization is needed.

What we use is a modified customized version of Tigase put on scalable AWS (Amazon) infrastructure. You need to understand how AWS works and set up special scripts which allow your system to spawn additional instances when there are spikes of activity. Also need to set up load balancer, proxy etc. We run a cloud backend platform so it has been tested with millions of users and thousands of concurrent connections.

Perhaps it's even worth taking a leaf from the books of good old IRC if you're really talking about millions of users. Using multiple servers connected in a peer-to-peer manner can help with scaling quite a lot. It all gets fun when servers get disconnected though (netsplits!)

(Why the sad face on Erlang?) ejabberd is the best way. Compared to building a chat server from scratch, it is economically cheaper to just learn Erlang and contribute to ejabberd.

You may write entire scalable chat server from scratch using Erlang. Just use Erlang with XMPP, enough to write simple server. Ejabberd is a nice option..don't know why you are frowning, but that's the best choice among all.

Try node.js that is very fast(event-loop non-blocking) and also has very speedy native bindings(C). There are some chat applications using node, i.e. http://Talk.io