Recommend a programmer a book that teaches high school maths step by step

Not a book, but have you tried Khan Academy?

I'm not sure if a single book containing all of your listed requirements exists. I would recommend that you get a book on discrete mathematics (Rosen's http://www.amazon.com/exec/obidos/ASIN/0072880082/metafilter-20/ref=nosim/ is a classic undergraduate book) and a comprehensive precalculus guide (I really like http://www.amazon.com/exec/obidos/ASIN/0071508643/metafilter-20/ref=nosim/ for refreshers, although I cannot vouch for the quality of that specific one). You say that you need to understand limits and matrix manipulation; would you be able to give us some examples of when you are running into these subjects? Which materials are you using to teach yourself algorithms/data structures?

You said... "high school maths step by step?" Get http://www.amazon.co.uk/Engineering-Mathematics-K-A-Stroud/dp/1403942463/ref=sr_1_1?ie=UTF8&qid=1342975947&sr=8-1. The first part covers foundation topics, including everything you listed except proof by induction. If you want to keep going, you can continue with part two. I cannot emphasize this enough: this book saved my life. It is the most didactally perfect mathematics textbook I have ever seen. No, not for math majors -- it's methods focused. But it works. If it could teach me, it can teach anybody. The discrete stuff -- like set theory, proof by induction, etc -- you will have to get elsewhere. As yet, I have not found a really good discrete math textbook. Rosen is so expensive it's breathtaking. After being raped by US textbook publishers during my first stint at university, I was astounded when I discovered how much better (while at the same time costing a third the price) most UK textbooks are. If you are searching for something for self-study, that's what I'd stick to. The next http://Stroud%20book covers matrix algebra and has some nice treatments of numerical methods for computing, so if you like Stroud I, you might want to consider it.

The Idiot Guides to Algebra and Calculus are pretty decent. I used them to review algebra and calculus before a stats class I took umpteen years after my last official math class. They aren't super detailed but the overview is very good and they are easy and fast to read. If you have any kind of aptitude for math or basic knowledge via the mists of time they're a good first review.

My guess is you want Rosen, as mentioned above. The caveat is that it may be simply too big for your purposes, though. I have a paperback international edition and it bears a certain resemblance to a phonebook. I just checked the table of contents. It doesn't have basic algebra. There's an appendix covering logs and exponents, but the book assumes you know it already. There is a matrix section. I think the course I took using Rosen had calculus as a pre-req, but the book doesn't assume it. The http://mathcentre.ac.uk/ website is designed to solve your problem, at least in theory. It probably imagines you've finished school relatively recently, though, so there might not be enough detail if you've totally forgotten things.

The 'The humungous book of [various kind of math] problems' books will teach you the basics in various areas of math. I've got the one about statistics and my only problem with it is that, apart from learning how to solve the statistics problems, I would have liked to know more about why they're solved in that particular way.

Maybe a different algorithms book would help? I'm finishing my CS degree and I got an A in my algorithms class, but I didn't find it very mathy except in the very abstract sense. Algorithmic analysis is...subtle, and maybe a different perspective would help. Incidentally, you are welcome to MeMail me with questions and I'd be glad to try and help!

Oops -- somehow that last link got munged. You can found Stroud's Advanced Engineering Mathematics http://www.amazon.co.uk/Advanced-Engineering-Mathematics-K-A-Stroud/dp/0230275486/ref=sr_1_1?ie=UTF8&qid=1342977804&sr=8-1.

Stroud (as above) plus http://www.amazon.com/exec/obidos/ASIN/053494728X/metafilter-20/ref=nosim/ might be a good fit for you. The Sipser text is a well-written intro to computational complexity theory, so it's directly relevant to what you're interested in, and it starts with an introduction to proofs ("Chapter 0") that includes proof by induction. Stroud should get you up through matrix algebra.