85 thoughts on “Python Requests – 2. Using Python Requests to Log Into a Website

  1. ripperx444 Post author

    U best question.. I know Python has the module called requests that you can import. My question is always how do people actually find the correct url to post the request?

    Reply
  2. AKDodgeball Post author

    What happens if the response name contains symbols that python recognizes as an expression? For example, the login form has two inputs, the email field which is called admin_user[email] and the password field is called admin_user[password]. This is not acceptable in putting it into the dictionary. Any help would be great!

    Cheers! 🙂 

    Reply
  3. beckerdennis Post author

    Hi Chris,

    this tutorial was awesome… exactly what I needed. So I am logged on successfully and now I want to make another request with a form on the page that triggers a file download. How do I get the response file saved? 

    If I run the download manually chrome shows me that the post request carries the following Response Headers:

    Cache-Control:private
    Connection:close
    Content-Disposition:attachment; filename=invxml.txt;
    Content-Type:text/xml;charset=UTF-8
    Date:Thu, 16 Apr 2015 23:01:25 GMT
    Server:Microsoft-IIS/7.5
    Set-Cookie:lngOpt=1; expires=Sat, 16-May-2015 23:01:24 GMT; path=/
    Set-Cookie:isCountryID=DE; path=/
    Set-Cookie:BLalertCnt=1184; domain=bricklink.com; path=/
    Set-Cookie:BLdiscussFlag=; domain=bricklink.com; path=/
    Set-Cookie:viewCurrencyID=2; path=/
    X-Powered-By:ASP.NET

    Can you please help me with getting the response file saved?

    Reply
  4. Calvin van Laaren Post author

    Nice tutorial man! U are really good at explaining! But i have one question, the c.get(url) doesn't work with mine, do you know what the problem is? 
    (sorry for bad english)   🙂

    Reply
  5. blutril Post author

    Hey, nice tutorial thought. I'm having a problem with the login button integrated on the website. Its name is "continue" which seems to be reserved by python itself. Any idea how i could fix that?

    Reply
  6. glitcher880 Post author

    I have the same code as you do but, it prints a django error page. Do you know why?
    ———————————
    import requests
    with requests.Session() as c:
        url = 'http://www.noobmovies.com/accounts/login/?next=/'
        USERNAME = 'bob'
        PASSWORD = 'booob'
        c.get(url)
        login_data = dict(username=USERNAME, password=PASSWORD, next='/')
        c.post(url, data=login_data, headers={"Referer": "http://www.noobmovies.com/"})
        page = c.get('http://www.noobmovies.com/user/profile/0/')
        print (page.content)
    ———————————–
    TypeError at /user/profile/0/
    int() argument must be a string or a number, not 'SimpleLazyObject'
    Request Method: GET
    Request URL: http://www.noobmovies.com/user/profile/0/
    Django Version: 1.7.7
    Exception Type: TypeError
    Exception Value:
    int() argument must be a string or a number, not 'SimpleLazyObject'
    Exception Location: /home/chrshawkes/noobmoviesenv/lib/python2.7/site-packages/django/db/models/fields/__init__.py in get_prep_value, line 915
    Python Executable: /usr/bin/python
    Python Version: 2.7.6
    Python Path:
    ['/home/chrshawkes/webapps/movies/myproject',
     '/home/chrshawkes/noobmoviesenv/lib/python2.7/site-packages',
     '/usr/lib/python2.7',
     '/usr/lib/python2.7/plat-x86_64-linux-gnu',
     '/usr/lib/python2.7/lib-tk',
     '/usr/lib/python2.7/lib-old',
     '/usr/lib/python2.7/lib-dynload',
     '/usr/local/lib/python2.7/dist-packages',
     '/usr/lib/python2.7/dist-packages',
     '/home/chrshawkes/webapps/myproject',
     '/home/chrshawkes/webapps/movies/myproject/myproject']
    Server time: Sun, 12 Jul 2015 19:16:36 -0400

    Reply
  7. Adithya Naresh Post author

    Thanks for sharing this. Your post is definitely a good addition to the requests website. I tried it on 'https://login.comcast.net/login' and it works. Note that, I did not have to use headers in the login_data dict. Also I used wakari.io's ipython notebook for testing which is really easy and has a lot of python libraries built-in.

    Reply
  8. Meathead Online Post author

    How can I get Requests to open certain classes on my page.

    Reply
  9. Elavazhagan Ila Post author

    I just gave the data as my cookie and I simply omitted the username and password.. Yet am getting the post response as 200. How is this possible when I dont give the user name and password?

    Reply
  10. MooreTour Post author

    ur great..i really wanna learn this programming language

    Reply
  11. invinciblekd Post author

    Nice tutorial, I have a website from where we download reports (excel file) once a month. At the moment this process is manual and hence cumbersome. Is it possible using python to login and download this file every month automatically ? Can you direct me to right direction if it is possible ?

    Reply
  12. Deepanshu Kapoor Post author

    i tried this on fb but it has to many parameters like isd , qsstamp etc..
    can anyone guide me for doing this on fb?

    Reply
  13. Mete Dagsuyu Post author

    how do i change the value of the input on a web page then click a button?

    Reply
  14. SREEJITH S Post author

    I want to do the same thing on a portal of web server. And its not working..
    Can you help??

    Reply
  15. twilighttucson Post author

    Why are you using such a high resolution for a screencast? Your videos are generally unusable since I would need to view it in atleast 720p along with a high res monitor to catch the code (for it to be legible). I believe you're cutting out a sizeable enough demographic with lower specced computers. Not to mention that even if I had all that extra bandwidth to kill its sooooooooooo hard to read the code. It's a difference between downloading a 175mb webm or a ~600mb webm.

    Would it be too dificult to record your videos using a region select tool for your screencaster, or presentation mode (if your IDE supports it).

    Reply
  16. df c Post author

    what should i do if the website have verification code?

    Reply
  17. ᄅ호ᅩᅥᅡᅢ노ᅮᅢᅧᄅ호ᅩᅥᅡᅢᄂ서ᅢᅡᄂᄒᄎᄂ ᄇᄃ퍼ᅢᅭᄉᄂᄐ츄ᅥᅮᅥᅡᅵᅢ Post author

    thanks brother for your education videos 🙂

    Reply
  18. Sean Munter Post author

    Thank you!!! The part about the csrftoken was especially helpful, as I was trying to log into a website all day with requests and couldn't get it to work, but that's what I was missing (not csrftoken specifically, but another type of info that needed to be sent in the POST request).

    Reply
  19. Penclick Productions Post author

    Hello Chris, I have been attempting to use this tutorial for a project, however I can't seem to get my script logging in to the site I am interested in. I would very much appreciate any advice or guidance you could give me. I do not have a lot of experience with programing, however I have gotten by with some other scripts in python before. Please let me know if you have a moment to help me out and perhaps we can touch bases through another medium.

    Reply
  20. Aviv Zvuluny Post author

    if the site im trying to login to doesnt have a csrfmiddlewaretoken, what do i need to do?

    Reply
  21. edantonio505 Post author

    Great video… I use linux. I prefer it over windows because it already has python installed.

    Reply
  22. Anthony sanchez Post author

    could never get the cookie part. pretty ill explained on that part. would have liked it a little more detailed.
    thumbs up tho

    Reply
  23. Shubham Sawant Post author

    import requests as r

    with r.Session() as c:
    url = 'https://lagunita.stanford.edu/login'
    emailid=' '
    passw=' ''
    c.get(url)
    csrftoken=c.cookies['csrftoken']
    login_data=dict(csrfmiddlewaretoken=csrftoken,email=emailid, password=passw)
    c.post(url,data=login_data,headers={"Referer":"https://lagunita.stanford.edu/login"})
    page=c.get('https://lagunita.stanford.edu/dashboard')
    print(page.text)

    hey chris why the above code doesn't work
    please help me

    Reply
  24. Joe Chong Post author

    hi thank you for the time spent on recording these tutorials.
    I have a question here. Can I direct to another page in the website to scarp other data after logging in?

    Reply
  25. amunera Post author

    I've been trying to do this, however the site I'm trying to acces does not seem to have a csrftoken, I tried with an authenticity_token (which I found with the inspector) but also gave me nothing, one that did not gave me an error was session_id but did not work anyways, any other ideas what I can try?

    Great video btw. Just starting with python a bit and thought it would be a good exercise to get some info out of a forum (which you have to access using a password) but so far no succes, I have the code doing what i want except for the log in 😛

    Reply
  26. Trần Bảo Lộc Post author

    Thanks Chris,wow it really help, now i can login on any website that i want.

    Reply
  27. Code-Tap Post author

    Hi chris.. im currently beating my head against a wall trying to log into an ASP.NET site. A video tutorial of doing that will be very helpful if you're up for it 🙂 thanks

    Reply
  28. mercedbread Post author

    has anyone come across a website that doesn't have the csrmiddlewaretoken ? This site i'm using only uses SHA-1 encryption.

    Reply
  29. Behdad Ahmadi Post author

    Thank you. I want to login to m.vk.com , I try too much but failed. How can I login to that website? It has not any csrfmiddlewaretoken , but has some keys and values like that. Thank you so much.

    Reply
  30. Justin Morse Post author

    I think I'm close – but I get an error regarding my csrf tokens. Fairly stuck at this point, not seeing much in forums.

    Form Data
    username:MYEMAIL
    password:MYPASSWORD
    mfa_input:
    login_token:LT-xxxxxx-xxxxxxx
    service:
    csrf_token:xxxxxx-xxxxxxx
    form:
    csrf_token:xxxxxx-xxxxxxx

    Error:

    Traceback (most recent call last):
    File "C:.py", line 8, in <module>
    csrftoken = c.cookies['csrf_token']
    File "C:Python27libsite-packagesrequestscookies.py", line 329, in _getitem_
    return self._find_no_duplicates(name)
    File "C:Python27libsite-packagesrequestscookies.py", line 400, in _find_no_duplicates
    raise KeyError('name=%r, domain=%r, path=%r' % (name, domain, path))
    KeyError: "name='csrf_token', domain=None, path=None"

    Code:

    import requests

    c = requests.Session()
    url = 'https://site-I'm-Accessing'
    USERNAME = '[email protected]'
    PASSWORD = 'Super Secret Password'
    c.get(url)
    csrftoken = c.cookies['csrf_token']
    logintoken = c.cookies['login_token']
    login_data = dict(username=USERNAME, password=PASSWORD, login_token=logintoken, csrf_token=csrftoken, next='/')
    c.post(url, data=login_data, headers=dict(Referer=url))
    page = c.get('https://site-I'm-Accessing')
    print(page.content)

    Reply
  31. Constantine Ketskalo Post author

    KeyError: "name='csrftoken', domain=None, path=None"

    Reply
  32. Rabi Kar Post author

    Hello Chris. Excellent tutorial. I tried the same process but I am getting response 200 as a return. Need your help regarding this. Thank you.

    Reply
  33. Michael Currie Post author

    Great video!!
    My problem is the Form Data is different, and it doesnt look like it has a CSRF code, which is suspect is the _xfToken.

    Do you have any suggestions for the below form data:

    login:XXXX
    password:XXXX
    remember:1
    tos:Y
    cookie_check:1
    redirect:https://hotcopper.com.au/
    _xfToken:

    Reply
  34. ines santos Post author

    do you can do a 2017 version with python 3.6?

    Reply
  35. Emad Ezzeldin Post author

    Hey Chris,
    on your website "HipsterCode" on the tutorial series of "web scraping with python" , it was mentioned that after the end of the video series there shall be a quiz. For those who pass that quiz shall receive a certificate. I have finished the series and neither found the quiz nor a way to earn the certificate. Can you please elaborate on that.
    Thank you very much for your most appreciated efforts to teach people programming.

    Reply
  36. Jamel Stringer Post author

    Thanks for the video, I'm getting the following error: NameError: name 'username' is not defined . Does anyone have a fix ?

    Reply
  37. Ben Gatley Post author

    What do you do if your usename has a dollar sign in it, like 'zzz$m_password'

    Reply
  38. MODESTO Post author

    Gosh you sound tired as fuck, lol, get some rest dude.

    Reply
  39. Katharina Huang Post author

    Exactly what I need! You deserve more likes!!

    Reply
  40. The Destroyer Post author

    That was hilarious. 2 times – Let me see if I can make this any bigger.
    After 15 minutes of Googling, you come back "Yeah, not sure how to make that bigger"

    Anyway, excellent tutorial.

    Reply
  41. Tirthankar Das Post author

    Great Tutorial!
    Could you please help with

    1. I am using session object and able to login successfully to a site post authentication I need to do another POST that returns a 500 Error invalid cookies

    having the same issues with another site but I donot get any error messges. I guess cookie authentication is the problem. Any pointers how we can fix it?

    2. Also i need to set the user agent for all traffic to mozilla/chrome
    but when i do session.headers.update({'User-Agent':'my user agent string…'})
    only the next subsequesnt request is reflecting and then it reverts back to pytho/cpython user agent sting!

    Reply
  42. tracy soul Post author

    I dont really know much about hacking after so many tries i met Cyberhacking lord who later help me find out my husband has been cheating on me and stealing from my bank account, he had this scheme going for 6 months. He gave me access to his mail,social media account,phone(could see deleted messages) and even track his location, still going to sue to him. Having doubts in your relationship? contact him ([email protected])

    Reply
  43. Jake Ambrose Post author

    lol why does every one make tutorials that are soooooo small???????? do u guys have laswers for eyes? command ++++++++++++++

    Reply
  44. Newascap Post author

    I really appreciate the effort you put into trying to explain this without giving up (and now i kinda now what to do) but man you need to organize more. Good vid 🙂

    Reply
  45. Jesper Lindberg Post author

    Hey man, really like this video. I have a question tho, when I try to get the post login parameters from a website I throw me so fast into the next site and clears my network tab in the console and fill it with new stuff. Is there anyway to run it as a debugger or save it to a file??

    Reply
  46. TechnologicalPinnacle Post author

    I tried the post method on login into Quora and Twitter, websites don't accept this method anymore I think, I received two HTTP responses one 405 and other 403, one suggests that the method is not allowed the other one suggests that request is forbidden..any workaround, except selenium?

    Reply
  47. Keplar Post author

    'Lemme see how to make this bigger' 1 day later 'Alright so I dont know how to make this bigger.' 😂😂

    Reply
  48. Agar HT Post author

    I cant see any explaine even in requests documentations. I can log in and log in authenticate user and i want to get that user authenticated values. Like username, email etc.

    Reply
  49. David George Post author

    What should you do if when you do inspect the element, there is no token under form data?

    Reply
  50. Chrysler Panaguiton Post author

    Hello Chris I'm trying your but but i get an response saying couldn't not authenticate you… I'm hoping to access a json from zendesk

    Reply
  51. MatthewLugoTV Post author

    Hey is there any way I can email you or DM you for you can help me with a quick question with requests?

    Reply
  52. Daammon Post author

    Hi Chris, thank you very much for your tutorial. I can't get to login on this ssl2 website. https://manybooks.net/. I would like to download all the books to use them in Machine learning but I can't login!! Any advice?

    Reply
  53. hanqie11 Post author

    Thank you! Have spent two days with no result until I found your video!

    Reply
  54. Oliver Zain Post author

    Great tutorial, quick question though:
    Couldn't you just print page.ok or page.status_code to verify it was successful?
    The ok method yielding TRUE if it was successful and the status_code method yielding 200 or 300 response if it was successful

    Reply
  55. Experimental Post author

    just curious ……. was your machine lagging OR my video lagging ???

    Reply
  56. Dick Tracey Post author

    Chris, Thank You for this video!!!! You had saved my sanity. Keep up the great work!!!!

    Reply
  57. Amit Mishra Post author

    Hey! Have you tried scraping authorised sites? Have you used requests-oAuth?

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *