What are the best tools to analyze a competitor's website?

What algorithms/programming tools are used for textual analysis-based webapps like 'I Write Like' or '750Words'?

• The website "I Write Like" (http://iwl.me/) asks you to paste in a chunk of text and it will analyze which writer you write like. I was wondering about the accuracy of the algorithm and what they use to determine this information. Do they have a database of all the writers text and do a cross-analysis based on the linguistic structure your writing? If so, how is something like this implemented from a programmer's standpoint? Another example of a webapp that uses textual analysis is 750Words (http://750words.com), which uses Regressive Imagery Dictionary and LIWC, but how do all these these textual analysis-based tools work on a subatomic level in regard to computer science/programming/algorithms in general.

• Answer:

  I agree with that the first step in comparing two texts is typically generating a word-frequency (easy) or maybe a bi-gram or n-gram frequency (harder) as described in http://en.wikipedia.org/wiki/Language_model. You precompute these frequencies for several authors (let's say Stephen King and Shakespeare), and get vectors of probabilities, which the language model article describe as P(w_1,...,w_n). You should imagine and implement these effectively as hashes/collections/associations something like:  \vec{P}_{king}=["cujo"=0.25,"shining"=0.25] and  \vec{P}_{shakespeare}=["yonder"=0.50,"thy"=0.60] but populated with many more words or n-grams and their occurrence frequencies. Then when somebody submits a writing sample, you compute a vector/hash for them \vec{P}_{user}=["the"=0.25,"omg"=0.50, "gaga"=0.33].  What you then do with these vectors, the comparison of a user's many-dimensional vector with the pre-computed vectors of authors... ah, well it turns out that is pretty much what personalization and recommendation on the internet is all about, comparing vectors of information about people, clustering or grouping people so that you can recommend things to one person based on what other people in their cluster have done, bought, read, seen, shopped-for, heard, played, or commented on, finding likely connections between people based on their vectors.  Although there are deep algorithms in machine-learning and elsewhere in computer science to really go bananas with these "vector" comparisons, a very typical quickie first "matching" algorithm in situations like "I Write Like" is simply a dot-product comparison, aka a \Theta-angle comparison - computing the angle between the user's vector and each of the author's vector by using the dot-product equation:  \vec{P}_{user} \cdot \vec{P}_{author} =\parallel \vec{P}_{user} \parallel \parallel \vec{P}_{author} \parallel \cos\Theta so  \cos\Theta={\vec{P}_{user} \cdot \vec{P}_{author} \over \parallel \vec{P}_{user} \parallel \parallel \vec{P}_{author} \parallel} and matching the user to the author with the smallest \Theta distance.  (If it's not clear how the words are "dimensions" and probabilities are "magnitudes" in these "vectors" and how you would compute their values, I can add some additional pointers. The first time you manipulate data in this way is always the biggest mental leap -- then you start finding linear algebra under every rock in computer science.)