Which is better? an integrated amp or a stereo amp and can they both work for a home theater system?

The answer depends largely on where you are getting your AV content and your budget. Home theater amplifiers (NOT theater-in-a-box) typically have the number and type of inputs and outputs that allow you to connect all the typical entertainment devices, ie, game console, dvd or bluray player, Netflix, cable box, over-the-air TV signal. Upstairs I have the big screen lcd, satellite dish, bluray play, and vcr connected through an Onkyo home theater receiver. I get local channels over the air, so I have to run a digital audio cable back from the TV to the receiver. Downstairs, I'm using an old analog TV receiving over-the-air signal from an antenna dvd player/recorder with a digital tuner. I route the audio through an old stereo receiver. Both provide sound a couple orders of magnitude better than the TV speakers. You need to assess what sort of inputs you need your amplifier to handle and start your decision process at that point. Only then can you start looking at actual specs.

You don't see that many integrated amp in the market today and stereo amp does not have an optical audio input. Keep in mind that home theater requires multi channel sound and stereo only have a left and right channel. Receiver of 5.1 or 7.1 is very common for theater use. Go online to Home Theater Magazine and read the reviews on receivers. Hope this will help you out.