How do I fingerprint a dynamic URL?

What model is the best for URL Similarity/URL Pattern Recognition?

• I am trying to come up with a model that will go over a number of URLs in a training set and identify new patterns in the test set. Let's say I have 2 sets of URLs that comes to my website First set includes the following: http://www.google.com/url?somerthing=a http://www.google.com/url?somerthing2=b http://www.google.com/url?somerthing3=c Second set will include Non human traffic(junk, bot traffic) http://www.google.com/url?somerthing5=a http://www.google.com/url?somerthing6=b http://www.google.com/url?somerthing7=c http://www.google.com/url?somerthing7=e&I_am_junk http://www.google.com/url?somerthing8=e&I_am_junk http://www.google.com/url?somerthing9=e&I_am_junk So I will build a model, such that it can compare the first set (Training set) with the second one and tell me something like, "I am seeing new traffic coming from a pattern like http://www.google.com%&l_am_junk%". What are some of your suggestions to tackle this problem? I did a lot of research on URL similarity but could not get to anywhere. Currently I am thinking about building clusters from k-means or something similar but don't really feel like it will do a good job. Thanks!

• Answer:

  I would frame it as a classification problem and use character n-grams.