Python Tutorial: Web Scraping with Requests-HTML

///Python Tutorial: Web Scraping with Requests-HTML

Python Tutorial: Web Scraping with Requests-HTML

FavoriteLoadingAdd to favorites

In this Python Programming Tutorial, we will be learning how to scrape websites using the Requests-HTML library. Requests-HTML is an excellent tool for parsing HTML code and grabbing exactly the information you need. So whether you’re pulling down headlines from news sites, scores from sports websites, or prices from an online store… Requests-HTML and Python will help you get this done quickly and easily. Let’s get started…

The code from this video can be found at:

File Objects Tutorial –
Requests Tutorial –
F-Strings Tutorial –
Try/Except Tutorial –
CSV Tutorial –

✅ Support My Channel Through Patreon:

✅ Become a Channel Member:

✅ One-Time Contribution Through PayPal:

✅ Cryptocurrency Donations:
Bitcoin Wallet – 3MPH8oY2EAgbLVy7RBMinwcBntggi7qeG3
Ethereum Wallet – 0x151649418616068fB46C3598083817101d3bCD33
Litecoin Wallet – MPvEBY5fxGkmPQgocfJbxP6EmTo5UUXMot

✅ Corey’s Public Amazon Wishlist

✅ Equipment I Use and Books I Recommend:

▶️ You Can Find Me On:
My Website –
My Second Channel –
Facebook –
Twitter –
Instagram –

#Python #Requests

source

By |2019-12-02T00:18:58+00:00December 2nd, 2019|Python Video Tutorials|47 Comments

47 Comments

  1. Preeti Narayan December 2, 2019 at 12:19 am - Reply

    Hey Corey, How can i implement this web scraping in django? Thanks for your pervious videos.. They're of great help!

  2. farmakoxeris December 2, 2019 at 12:19 am - Reply

    Great video buddy

  3. farmakoxeris December 2, 2019 at 12:19 am - Reply

    45:53 How can I get the number of the html links contained inside r.html.links?

  4. Randy Diffenderfer December 2, 2019 at 12:19 am - Reply

    as your source is a known url, why not use a url parsing library to do the parsing? can get last part of the route with little chicanery, i'd think! 😀
    plus, you get a sanity check on what you've pulled out for free!

  5. 123 Boy December 2, 2019 at 12:19 am - Reply

    Hey nice tutorial, but I have a question. How do you parse if the website uses Ajax calls to populate the data?
    More exactly, I am trying to parse a bet website.
    Any info or links would really help me. Have a nice day Corey.

  6. taro December 2, 2019 at 12:19 am - Reply

    This is so understandable and amazing. Thanks a lot Corey!

  7. Samir Sarkar December 2, 2019 at 12:19 am - Reply

    I like your tutorial most. I just have one question. For parsing the article headline and summary you used ".entry-title-link" and & ".entry-content p" in find method but at the time of vid_src you used "iframe" . So how we decide when to use . and when not ?

  8. Samir Sarkar December 2, 2019 at 12:19 am - Reply

    You rock man 🤘🏻

  9. Rob Carreon December 2, 2019 at 12:19 am - Reply

    Thank you for your videos! They are very well organized, easy to follow and extremely helpful!
    I followed through on the above video and got it working perfectly on a site that uses javascript. But I only got it to work when running via python command line or shell.
    When I put the code inside a very simple "hello world" django project, the .render() function causes a Thread Loop-1 error; if I comment out the r.html.render() line, I don't get the error, but the
    information I get back is incomplete. I searched all over for results-HTML and django and this error and couldn't find much on a cause/solution. Just curious if you've run into this and know
    why it doesn't work? Thanks again!

  10. Duke Wellington December 2, 2019 at 12:19 am - Reply

    which python version were you using for this? The docs say it only works with 3.6 but I'm hoping they mean 3.6 and up.

  11. NAveeN Kumar Reddy December 2, 2019 at 12:19 am - Reply

    Hello Sir, can you please look into this, i am facing issue with django framework when i tried to implement requests_html while processing that for javascript execution result.html.render() i am getting this error : There is no current event loop in thread 'Thread-10'. This is my view method.

    def search_movie(request):

    if request.method=='POST':
    movie_name = request.POST.get('movie_name').replace(' ','+')

    session = HTMLSession()
    results = session.get(f"https://isongs.info/search/?q={movie_name}")
    results.html.render()

    return HttpResponse(results.html.html)

    return render(request,'site/search_form.html')

  12. Debby Gram December 2, 2019 at 12:19 am - Reply

    real heroes don't wear capes they teach like Corey Schafer. You literarily make programming simpler than A, B, C.

  13. paulseldn December 2, 2019 at 12:19 am - Reply

    i keep getting this error. Can you please help Corey? MagicPython – Webscrape.py:9

    Traceback (most recent call last):

    File "C:UsersPAULDesktopTestWebscrape.py", line 1, in <module>

    from requests_html import HTML, HTMLSession

    File "C:Python35-32libsite-packagesrequests_html.py", line 6, in <module>

    import pyppeteer

    File "C:Python35-32libsite-packagespyppeteer__init__.py", line 30, in <module>

    from pyppeteer.launcher import connect, launch, executablePath # noqa: E402

    File "C:Python35-32libsite-packagespyppeteerlauncher.py", line 20, in <module>

    from pyppeteer.browser import Browser

    File "C:Python35-32libsite-packagespyppeteerbrowser.py", line 11, in <module>

    from pyppeteer.target import Target

    File "C:Python35-32libsite-packagespyppeteertarget.py", line 4, in <module>

    from typing import Any, Callable, Coroutine, Dict, List, Optional

    ImportError: cannot import name 'Coroutine'

    [Finished in 0.872s]

  14. paulseldn December 2, 2019 at 12:19 am - Reply

    Which python version are you using as i am getting errors all over the place?

  15. paulseldn December 2, 2019 at 12:19 am - Reply

    Hi Corey. Can you please tell us what IDE you are using. it is so nice to be able to read large fonts and also resize the console. I cannot do this in Atom

  16. alberto medina December 2, 2019 at 12:19 am - Reply

    Hi Corey, this explanation is great, as always! I'd love to see a more difficult scraping example. Are you planning on doing something like that? Or do you have any recommendations of what to read/watch?
    Thanks!!

  17. Alex Berk December 2, 2019 at 12:19 am - Reply

    Can we, please, have a tutorial on async/await and yield from? I kinda get it, but I really want to hear your explanation.
    Thank you so much for your vids!

  18. Daywalker December 2, 2019 at 12:19 am - Reply

    I am a big fan of you showcasing what it looks like first, then breaking it down in detail.

  19. ori61 51 December 2, 2019 at 12:19 am - Reply

    31:00 why not just use find("embed/") then find("?") like this:
    url[url.find('embed/')+6:url.find('?')]

  20. manpaal singh December 2, 2019 at 12:19 am - Reply

    Can you use python to automate and export every iteration of a searchable or filterable website?

  21. code_ coder__ December 2, 2019 at 12:19 am - Reply

    python guru (India support)

  22. Earl Tan December 2, 2019 at 12:19 am - Reply

    Awesome!!!! Been looking for this man!

  23. Hugo Gradvohl December 2, 2019 at 12:19 am - Reply

    Hi Corey, I was hitting a wall with asynchrone programmation and the last part has help me a lot. Since "await" doesn't work with all functions and the methods name have been upgraded, I was looking for something consistent. Has often I end up finding the right way to use the functionality in your video. I know that you have many projects but asynchrone and thread programmation are a very interesting topic, if you run out of projects (not happening ^^) it would be great if you could make a light series of video (by the way I reduce my download time from 58 sec to 18sec by applying async functionality). Thank you!

  24. Tang Chee Ming December 2, 2019 at 12:19 am - Reply

    Great video but would like to see more example on scrapping dynamic image etc using the render() function or something.

  25. Anunay Anu December 2, 2019 at 12:19 am - Reply

    what is difference between using Beautifulsoup and Requests-HTML

  26. Rahul Soni December 2, 2019 at 12:19 am - Reply

    Sir can you please teach me how to use render() function properly.I am facing huge problem for scrapping data from a web which loads results dynamically using jacascript

  27. Carlos Matos Fanpage December 2, 2019 at 12:19 am - Reply

    How do I display results of scraping on my own website using Django.I want to make a website that compares prices of products across different websites but don’t know how to put the data which I have scraped onto my website. Thanks

  28. Hell 101 December 2, 2019 at 12:19 am - Reply

    46:32 Javascript rendering
    49:42 Asynchronous requests

  29. Rocco Cocolombo December 2, 2019 at 12:19 am - Reply

    Great videos and thanks for using large readable fonts.

  30. Whilelab December 2, 2019 at 12:19 am - Reply

    Just want to add that you can access a prettified version of the HTML using

    r.html.html

    BTW Great video.

  31. Šarūnas Savickas December 2, 2019 at 12:19 am - Reply

    Very informative and useful tutorial. Thank you, Corey! Would love to see your approach on asyncio or multithreading 🙂

  32. Eric Klungle December 2, 2019 at 12:19 am - Reply

    Great tutorials. Most people I have watched also will take requests. I have been trying to figure out how to change an element on a webpage and as yet have not seen anything very useful THAT WORKS!! It would be nice if someone would do a tutorial on how to do the following:
    change the following:
    <option selected="selected" value="1"> &nbsp;&nbsp;Single Draw</option>
    to this: (Notice the value="" parameter)
    <option selected="selected" value="100"> &nbsp;&nbsp;Single Draw</option>

  33. ramzan sheikh December 2, 2019 at 12:19 am - Reply

    great work. it helps me a lot . can you tell me how we extract the data from different pages without click or add link into the code from next pages tags.

  34. David Emanuel Sandoval December 2, 2019 at 12:19 am - Reply

    Nice video!

  35. David Emanuel Sandoval December 2, 2019 at 12:19 am - Reply

    Nice video!

  36. Hinstea December 2, 2019 at 12:19 am - Reply

    any idea how we can implement it in dynamic websites?

  37. Preeti Singh December 2, 2019 at 12:19 am - Reply

    hii.. i need a code for how to convert a pdf file to html file using python 3? kindly help..

  38. Dheeraj Kura December 2, 2019 at 12:19 am - Reply

    Corey, you're a gift for the programmers. long live and spread the knowledge

    Can you please do the Vidoe's on OpenCV as well which is useful for Computer Vision

  39. rasterops1083 December 2, 2019 at 12:19 am - Reply

    time.perf_counter!?!?! Thanks!

  40. Roman Khripunov December 2, 2019 at 12:19 am - Reply

    Как всегда круто! Самый качественный контент ) спасибо, бро!)

  41. AMV king December 2, 2019 at 12:19 am - Reply

    Hi, i need some help. Im trying to get an object from an html div that contains multiple classes but it returns a none type value back. Is there anyway to get that object?

  42. raffaele rimorso December 2, 2019 at 12:19 am - Reply

    big lezion

  43. Maxwell December 2, 2019 at 12:19 am - Reply

    Can you please make a video about threads? I want to scrape a website and get pictures from it and I want to download several at the same time. I'm not sure if it would be better using threads or async.

  44. Benedikt Hollenhorst December 2, 2019 at 12:19 am - Reply

    I usually use Selenium if I want to scrape JavaScript content. When would you use Requests-HTML instead of Selenium? As far as I know, Requests-HTML uses headless Chromium to scrape JS data.

  45. Sumit chatterjee December 2, 2019 at 12:19 am - Reply

    Very useful video! I have a question though if I want to scrape a webpage where I have sublinks and that sublinks have many sublinks, how do I scrape that? Any suggestions would be appreciated!

  46. WAI CHAN December 2, 2019 at 12:19 am - Reply

    I don't usually comment in fact never, but I just wanna say thanks! the content you produce is by far the best I have seen

  47. joseph sowah December 2, 2019 at 12:19 am - Reply

    Thanks Corey.. you're the best…love all your tutorials.
    Looking forward to your tuts on rest api in python.

Leave A Comment

*