How does a non-designer CEO hire an amazing web designer?

Ask a designer that you trust (but can't / doesn't want to join your startup) to join the interview and have him express his opinion afterwards.

I would ask for references and a fee schedule.  That's really all you need.  Draw the terms of the position, and carry out a 30-day probationary period.  If after 30 days, you don't like the services you are receiving, then move on to someone else.

I have personally been a part of 3 startups from the inception to failure, and joined 3 other small ventures as well as met with 2-3 "concept/pitch" level ventures, all under the auspices of a technical web developer lending my expertise to a venture. First decide what you want in a candidate, I suggest UI, self-motivation, the ability to work long hours without direction and an innate understanding of what will work (as opposed to having to figure it out through focus groups). remember that web developers/designers (or artists) have two strengths (sometimes equally) the graphics/design/front end side and the back-end side. You may have to consider bringing in two. Secondly, consider the candidate as you would a partner in your venture. Weed out the 9-to-5ers, get signed NDA's, and pitch the venture tot he others to get a feel for the candidates response and passion for your idea. Consider offering more than just money (or alternatives to money), since they are taking on higher risk and you are competing with established companies that offer benefits packages and established culture. I do not agree with Whitney Cann's answer. References are old school and often merely a formality as candidates give three that will usually give you three people that will give positive reviews and if you are only interested in someone who works from detailed specs and direction, then you should look at the contract programmer sites like Project4hire or Freelancer.

Good web design has two components, UI which is look, feel and flow, and technology which is platform, architecture and code. If you have a clear vision of your idea, in terms of who the users are, how they will use it and the results they should expect, then you can handle the UI side of things. If your CTO is worthy of the title, then he/she should be able to answer the platform, architecture and coding questions in their sleep. If both of you need a future employee to answer the questions for you, I wish you the very best of luck because you will need it. Sure, many heads are often better than one, but when it hits the fan, and it will hit the fan, you need to be able to close ranks and charge. If the answers to the critical questions lie outside of your closed ranks, you will either find out how much you already know, or you will fail. You are not looking for an employee, you are looking for a third founder. Somebody who shares the vision, passion and has the same degree of staying power as you do. The technical stuff is secondary, but the right candidate will be more than happy to demonstrate his commitment by providing a sample of what will be needed.