How do I perform matrix rotation in 3-space?

This is called diagonalizing a matrix. First solve for the eigenvalues $\{\lambda_1, \lambda_2,\lambda_3\}$ of $A$. Then perform a change of basis to the basis of eigenvectors $\{\vec{v_1}, \vec{v_2}, \vec{v_3}\}$. Let $P$ be the $3x3$ matrix $[\vec{v_1}\; \vec{v_2}\; \vec{v_3}]$. Then $\[A=P^{-1}DP\]$ where D is the diagonal matrix with diagonal $\{\lambda_1, \lambda_2,\lambda_3\}$. This will only amount to a rigid rotation in $\mathbb{R}^3$ if the columns of A were already orthogonal to one another, but note that the process above is a rotation in the sense that it is multiplication by two orthogonal matrices of determinant 1, which can be seen if we normalize the eigenvectors of $A$ to create $P$. Also worth noting that not all matrices have 3 real eigenvalues. Real symmetric $n$x$n$ matrices have an orthonormal basis of eigenvectors. For more google "spectral theorem".

All rotations in three space correspond to quaternions. The rotation about an axis through the origin corresponds to the quaternion with i,j,k coordinates equal to the coordinates with the remaining real coordinate specifying angle magnitude and then the coordinate must be multiplied by a scalar factor so the length of the coordinate is one. Then for any such axis and any angle We can find a quaternion corresponding to the rotation. Now in your problem the rotation about the axis perpendicular to the z coordinate and the third row of the matrix will do the trick. That axis is just the product of the normalized quaternions whose coordinates which are derived from the z coordinate (simply k) and the quaternion corresponding to the third row of the matrix with the scalar term discarded. Now looking at this I see I don't need the matrix to be invertible so I have the stronger result: Let A be a 3x3 matrix. Show that you can always perform a rotation of 3-space to make the last row of A be [0 0 A33].