Company B is flying me out to interview, how do I handle this with Company A?

This is perfectly normal and not an ethical dilemma. Also, keep in mind that it's too your advantage to communicate to these companies that you're actively interviewing with their competitors. I was job-hunting for two months this fall, and kept my prospects (consulting firms A thru D) appraised of where I was in the process with each once things got serious. Since I was also employed at the time, they were very welcoming of the idea of me "bundling" interview trips to make it easier on my work schedule. TLDR: don't worry about this.

If Company B flew you out, would it be ethically problematic for you to spend non-interview time with your family? As long as you're available for whatever interviews and meetings they need, what you do with the rest of your time is your business.

I don't see how either talking to Company A while in town to talk to Company B or not telling Company A would be a breach of ethics in the least. Why do you think that either of those options is in any way unethical?

You're overthinking this. Company B is paying for you to fly out there. You have no obligation to tell Company A you're there if you're not charging Company A for any of your time. If Company A wants to meet with you, they will pay for you to fly out too, just at a separate time. Why would you give them a freebie? If they want you badly enough they'll pay for it. If Company A somehow found out you were in town, you would simply say you were there for personal reasons and did not have time to meet with them. Which is true. Because you're busy being wooed by Company B.

just because company B is flying you out for an interview does not mean they own your ass 24/7 while you are there. what you do on your own time (i.e.: time that you are not scheduled to be interviewing, touring, etc with company B) is up to you. so go and meet with company A. not a big deal or a breach of ethics.

I don't see any ethical problem. Assuming Company B has not actually paid for all of your time, they are only reimbursing you for your travel costs and possibly a per diem. That is because it is cheaper for them to do that than it is for them to send their people to meet with you. You are the one doing them the favor, and not the other way around. While there's no need to be a jerk about this, or put on airs, neither should you feel like you owe B anything for the opportunity to minimize their cost to talk to you. Second, your free time is your free time. If you want to call on Company A, let your contact or the decision-maker know you're in town on unspecified other "business" but that you have some free time and would hate to miss the chance to have lunch/dinner/cocktails or whatever. Your making the most of your lost travel time makes sense and, in the absence of other facts, it appears fine to me.

Another person voting that there is no problem here.

If you don't need to hide from either company that you're interviewing with both, I would tell both about it and see if they can come to some sort of cost-sharing agreement with respect to you flying out there. I have done a similar thing many times. Of course, this doesn't apply if Company A isn't interested in paying for you to come and interview in person (but if that's the case, then I question their desire to hire you).