How can I use SSL with django?

How to use Apache proxy to snoop my machine's HTTPS/SSL communication?

• The short version of my question is this: For reasons explained more fully in the Background section below I want to be able to snoop HTTPS/SSL traffic between my Linux machine and assorted remote secure servers. To do this, I want to set up the following pipeline: browser <-> HTTP proxy <--> Apache proxy <-> secure server The part of this pipeline I need to understand is the Apache proxy, whose role is to en/decrypt the secure communication from/to the HTTP proxy. How do I set up and operate an Apache proxy on my (non-root) Linux account? Note that I have not installed Apache yet, so the answer I need requires instructions on how to configure the Apache installation on my home directory to allow for what I want to do. Background: I'm a skilled Perl programmer, though I have very little experience in the area of network programming, and in particular the area of HTTPS/SSL transactions, which I need to know more about for my current personal project. My project is this: I have several accounts (credit cards, bank, etc.) that offer online access. I want to write a robot that periodically visits these (secure) sites (using my passwords, of course), collects information of interest to me, and e-mails me a report. In the past, when I've wanted to write a Perl robot to automate the fetching of information from the Web, I've made heavy use of a little HTTP proxy written in Perl that I downloaded from the Web some time ago. I've modified the proxy so that it logs all the communication between my browser and the outside. So I configure my browser to use the proxy, and proceed to access websites of interest "manually". Then I use the details logged by the proxy to write a Perl/LWP script that automates the browser end of this communication. This strategy doesn't work for my current project, because the proxy script mentioned above misses the HTTPS communication between browser and server. I understand that it is possible to set up some other third-party proxies (e.g. Apache) as an additional proxy to handle the HTTPS en/decrypting: browser <-> HTTP proxy <--> Apache proxy <-> secure server I suppose that the Apache proxy speaks HTTPS to the server and speaks HTTP to the HTTP proxy, performing the en/decryption between the two on the fly. NOTE: I would accept as valid answers to my question a detailed description of an alternative way to accomplish my ultimate goal (i.e. writing a web robot to automate the gathering of my secure information) different from the one outlined above.

• Answer:

  Hello! I'll show you another way to do the research for your program. Instead of sniffing what your browser does, I'll explain you how to understand HTML forms to know what to send to each step of the authentication process and to get the info you're looking for. First of all, this method is not less powerfull than the one with the proxy. It takes more time, but it's better, because you have to understand what does your browser do on each step. ------------------------ Web Developers use HTML forms to send information that the user enter to the server. Forms have input elements, for example, a Text box, a Check box, a Button, etc. Each element has a name and a value (may be entered by the user or be a fixed value) and when the user clicks on the 'Submit' button, the browser sends the information to the server. An example of a simple HTML form: <form action='login.php' method='post'> Username <input type='text' name='username' /><br /> Password <input type='password' name='password' /><br /> <input type='submit' value='Send' /> </form> When this form is rendered by a browser it will show 2 text boxes and a button. Then, when the user clicks the 'Submit' button, the browser will send the information to the program 'login.php' on the server side. There're 2 ways to send the data to the server, specified by the 'method' attribute on the form tag: via POST or via GET GET: When using GET the browser pass parameters after asking for the file. For example, if the previous form used 'GET' to send the information after clicking on the button, the browser will show on the Location Bar: http://www.mysite.com/login.php?username=entered_username&password=entered_password If you want to make a robot to login into that server, you only have to tell it to ask for that URL!! POST: This is more 'secure' because the sent data is not shown on the Location bar. Probably you will need to log in to your account using one of this forms. If you want to make your robot to login into this site (using the above form), check this example: use LWP; use HTTP::Request::Common; $ua = LWP::UserAgent->new; $ua->request(POST 'http://www.mysite.com/login.php', ["username" => "my_username", "password" => "password"]); Please note that every element in a form that has a name will be passed as a parameter to the 'action' script. [ http://www.w3.org/TR/REC-html40/interact/forms.html ] ---- To get the information you're looking for, the steps are: 1) Login into the system 2) Go to the page that has the information 3) Parse it using regex 4) Mail it, print it, etc ( To show you how to understand the forms, I'll develop a little program to log into Google Answers and get the status of the account ( https://answers.google.com/answers/main?cmd=myinvoices )) 1) Login To start, go to the main page of the site you're looking for and click until you get on the 'login page'. In my case, this will be https://answers.google.com/answers/main?cmd=login Then, click with the right button on the page and click on 'View Source'. Find where the form starts (<form ...) and check if it's using GET or POST and where's the information submitted after clicking on the submit button. In my case, the forms starts like this: <form method="post" action="main?cmd=login"> Note that 'action' doesn't have the full URI of the file, so you have to prepend the current directory. After clicking on the 'Login' button, the information will be sent to https://answers.google.com/answers/main?cmd=login Now, check which are the input elements on the form: look for <input.. <textarea... and <select.. tags. Get their names and figure out the value to send to the server. In my case, the input elements are: <input type="text" name="email" size="20"> <input type="password" name="password" size="20"> <input type="submit" name="submit" value="Login"> So, I have to send 3 variables to the server: 'email' with my email address 'password' with my password 'submit' with the value 'Login' Please note, that 'submit' is a button, so the defaul value ('Login') can't be changed. But if this variable is not sent, you won't be logged in Try this little program: ------------------------------------ use LWP; use HTTP::Request::Common; use HTTP::Cookies; $email='[email protected]'; $pass='my_google_answers_password'; $ua = LWP::UserAgent->new; $ua->cookie_jar(HTTP::Cookies->new); $req=$ua->request(POST 'https://answers.google.com/answers/main?cmd=login', ['email'=>$email, 'password'=>$pass, 'submit'=>'Login']); if ($req->content=~ /Invalid login/){ print "invalid login!\n"; }else{ print "welcome to google answers :)\n"; } ----------------------------------- In the 8th line, I tell LWP to request 'https://answers.google.com/answers/main?cmd=login' and pass the parameters 'email'=$email, 'password'=$pass and 'submit='Login' Set $email and $pass with your info and try it! 2) Getting the info Now you're into the system, you have to go to the page where the info you're looking for is. Click on the link that takes you there and write down the address on your browser's Location bar when you're there. For example, if I want to get the status of my account, I'll have to go to https://answers.google.com/answers/main?cmd=myinvoices So, after login into the system, I'll go to that address: $req=$ua->request(GET 'https://answers.google.com/answers/main?cmd=myinvoices'); and inside $req->content I'll have the contents of the page. Then, I have to parse it: $req->content=~/<td> Current Earnings \(what you will be paid\) for Answering Questions: <\/td> <td width="1%"> \$([0-9]+(?:.[0-9]+)?)/; $ear=$1; $req->content=~/<td> Current Balance \(what you will be charged\) for Asked Questions: <\/td> <td width="1%"> \$([0-9]+(?:.[0-9]+)?)/; $char=$1; print "Will be paid: $ear \nWill be charged: $char\n"; -------------------------- The finished script will be: use LWP; use HTTP::Request::Common; use HTTP::Cookies; $email='[email protected]'; $pass='my_google_answers_password'; $ua = LWP::UserAgent->new; $ua->cookie_jar(HTTP::Cookies->new); $req=$ua->request(POST 'https://answers.google.com/answers/main?cmd=login', ['email'=>$email, password=>$pass, 'submit'=>'Login']); if ($req->content=~ /Invalid login/){ print "invalid login!\n"; }else{ print "welcome to google answers :)\n"; $req=$ua->request(GET 'https://answers.google.com/answers/main?cmd=myinvoices'); $req->content=~/<td> Current Earnings \(what you will be paid\) for Answering Questions: <\/td> <td width="1%"> \$([0-9]+(?:.[0-9]+)?)/; $ear=$1; $req->content=~/<td> Current Balance \(what you will be charged\) for Asked Questions: <\/td> <td width="1%"> \$([0-9]+(?:.[0-9]+)?)/; $char=$1; print "Will be paid: $ear \nWill be charged: $char\n"; } ------------------------- Probably it won't be this straightfoward on a Bank (you know, their HTML will be very messy: they don't understand the beauty of the simple things, as google ;) but it won't be very hard if you have patience :) Good luck with your program, and feel free to ask all the clarifications you need! Aditional links: LWP [ http://www.linpro.no/lwp/ ] HTML Forms [ http://www.w3.org/TR/REC-html40/interact/forms.html ] Search Strategy: Personal experience