{"id":8391,"date":"2019-08-07T11:30:21","date_gmt":"2019-08-07T18:30:21","guid":{"rendered":"https:\/\/www.springboard.com\/?p=8391"},"modified":"2023-07-05T22:38:58","modified_gmt":"2023-07-06T05:38:58","slug":"web-scraping-basics","status":"publish","type":"post","link":"https:\/\/www.springboard.com\/blog\/data-analytics\/web-scraping-basics\/","title":{"rendered":"Web Scraping Basics: What You Need to Know"},"content":{"rendered":"\n

Web scraping<\/span>: it\u2019s used all the time online, yet you may not know what it is. With the internet being a treasure trove of content, web scrapers provide us with the proper tools to extract valuable information from the web\u2019s innumerable pages. This information is then saved as a local file on your computer. The extracted data can be used for open source projects, web interfaces, various APIs, or just your own recordkeeping. <\/span><\/p>\n\n\n\n

On that note, let\u2019s dig into web scraping and find out what it\u2019s all about. <\/span><\/p>\n\n\n\n

What Is Website Scraping?<\/h2>\n\n\n\n

Simply put, web scraping allows us to download specific data from web pages based on certain parameters. Intelligent bots today do much of this work, crawling websites and storing the information needed in databases. Moreover, Data Analysts<\/a> also perform web scrapping to extract the relevant data for analysis purposes.<\/span> Therefore, web crawling is an important component of scraping. <\/span><\/p>\n\n\n\n

The web scraping definition and process are pretty simple to understand. First, web pages that match certain criteria are found. The pages are then downloaded and fetched for processing, where they are searched, reformatted, copied, and so on. Web scrapers can, among other things, extract images, videos, text, contact information, product items, and much more from a website. <\/span><\/p>\n\n\n\n

Web scraping today is a core component of much of our digital infrastructure. For example, all web indexing relies heavily on data scrapers. Changes in online activity between the over 1 billion websites can thus be easily tracked using scraping methods. Internet scraping is necessary to make sense of the vast expanse of data available online. As such, the technique has proven fundamental to big data analytics<\/a>, machine learning<\/a>, and artificial intelligence.\u00a0<\/span><\/p>\n\n\n\n

(If you\u2019re interested in reading more about big data, check out this list of <\/span><\/i>the top data scientists to follow<\/span><\/i><\/a>.)\u00a0<\/span><\/i><\/p>\n\n\n\n

With more intelligent scripts, web scraping has become much easier to do and omnipresent online. The parameters for what these scripts look for has also become more precise, which has led to a whole host of ever-growing data science projects<\/a>.<\/span><\/p>\n\n\n\n

How Does Web Scraping Work?<\/h2>\n\n\n\n

Almost all data scrapers on the web today are actually just intelligent bots. Generally speaking, these scrapers are responsible for extracting the HTML code of a website and then compiling it into structured data. How it works is simple to explain.<\/span><\/p>\n\n\n\n

    \n
  1. First, a GET request is sent using an HTTP protocol to the site the scraper is targeting. <\/span><\/li>\n\n\n\n
  2. The web server processes the request and, if legitimate, the scraper is then allowed to read and extract the HTML of the web page. <\/span><\/li>\n\n\n\n
  3. A web scrape locates the targeted elements and saves these in the set variables. <\/span><\/li>\n<\/ol>\n\n\n\n

    That’s the process in a nutshell, but you have to imagine this basic process multiplied by millions, even billions, of data points. As data scrapers become more sophisticated, the potential of big data and machine learning grows with it. Moreover, with dynamic web pages becoming more common, scrapers are being forced to adapt to the changing times.<\/span><\/p>\n\n\n\n

    Common Libraries Used for Web Scraping<\/h2>\n\n\n\n

    The world of internet scraping is vast. However, there are a few key libraries and tools that are commonly used by all. Most web scraping requires some knowledge of Python, so you may want to pick up <\/span>some books on the topic<\/span><\/a> and start reading.\u00a0<\/span><\/p>\n\n\n\n

    BeautifulSoup, <\/b>for example, is a popular Python package that extracts information from HTML and XML documents. It then creates parsed trees, which are useful for sifting through large amounts of data. It is currently available for both Python 2.7 and Python 3. <\/span><\/p>\n\n\n\n

    Pandas<\/b> is a software library written in Python that specializes in data manipulation and indexing. Its main benefit is that it allows users to carry out data analysis all within Python, so there\u2019s no need to switch to a language like R. <\/span><\/p>\n\n\n\n

    Selenium <\/b>is an automation tool built into your web browser. As such, it can enter forms, click buttons, and search for bits of information like a bot would. On top of it, you can <\/span>build repositories<\/span><\/a> which will give you the power to scrape websites.  <\/span><\/p>\n\n\n\n

    There are many more libraries well suited for web scraping out there, but these are three that warrant initial attention. <\/span><\/p>\n\n\n

    Get To Know Other\tData Analytics Students<\/h4>
    \"Joel<\/a>

    Joel Antolijao<\/p>

    Data Analyst at FanDuel<\/p><\/div>

    <\/div>

    Read Story<\/a><\/p><\/div><\/div>

    \"Rahil<\/a>

    Rahil Jetly<\/p>

    Sales Operations Manager at Springboard<\/p><\/div>

    Read Story<\/a><\/p><\/div><\/div>

    \"Bart<\/a>

    Bart Teeuwen<\/p>

    Global Business Analyst, Global Talent Intelligence (GTI) at Meta<\/p><\/div>

    Read Story<\/a><\/p><\/div><\/div><\/div><\/div>\n\n\n\n

    The Super-Simplified Way to Scrape<\/h2>\n\n\n\n

    Just a note: We\u2019re not going to run through the details on how to specifically use any particular web scraper. Instead, these are the basic steps that everyone needs to follow. So, plan accordingly. <\/span><\/p>\n\n\n\n

    Find the URL You Want to Scrape<\/h4>\n\n\n\n

    This step is self-explanatory. You need to zero in on the niche you are researching. For example, if you’re looking into the competitive pricing of laptops, it might be smart to compile a list of all the sites that contain valuable information before starting. <\/span><\/p>\n\n\n\n

    Inspect Page and Check Tags<\/h4>\n\n\n\n

    Your web scraper needs to be told what to do. So, you need to carefully figure out which <\/span>elements <\/span><\/i>you will be looking at as well as <\/span>tags. <\/span><\/i>Right-click any element of the page and choose \u201cinspect\u201d to be taken to the page\u2019s backend. The successive box will allow you to see the details of said element, including tags and the metadata that will prove crucial for your scraper. <\/span><\/p>\n\n\n\n

    Once you\u2019ve identified which elements you want to target, along with which tag represents each, it\u2019s time to start scraping. <\/span><\/p>\n\n\n\n

    Fire Up the Scraper<\/h4>\n\n\n\n

    Now, you can scrape online in a few ways. If you’re feeling up for it (and have the knowledge), you can write it from scratch with Python. You will need to tap into libraries like BeautifulSoup to get this operational. Feel free to read Ethan Jarrell\u2019s great guide on <\/span>HackerNoon<\/span><\/a>.<\/p>\n\n\n\n

    If you think Python is over your head, you\u2019re going to want to use software that simplifies this process. There are plenty available today, most of which are not free. They are often used by enterprises as part of SaaS web data platforms. <\/span><\/p>\n\n\n\n

    If you\u2019re only planning to scrape a few sites, then you\u2019re better off creating your own scraper. However, for more complex functions, try looking for software solutions that suit you.<\/span><\/p>\n\n\n\n

    Unpack Your Data<\/h4>\n\n\n\n

    After letting the scraper run for a bit, you will have a healthy data collection ready for analysis. How you go about this is your prerogative, but you may need to use \u201cregular expressions\u201d (Regex) to turn it into readable text. This last stage all depends on how much data you have collected, determining whether you need to take any additional steps to better parse your findings. <\/span><\/p>\n\n\n\n

    Some Things to Consider Before Web Scraping<\/h2>\n\n\n\n

    As you might expect, the use of web scrapes does not mean you can simply extract any information online without any restrictions. There are naturally both <\/span>beneficial <\/b>and <\/span>malicious <\/b>actors online who use web scraping. <\/span><\/p>\n\n\n\n

    For example, Google and other search engine bots analyze content to properly categorize and rank it. This is an example of a perfectly expected, normal use of internet scraping. Market research companies also employ these same methods to gauge sentiments and preferences for an intended audience. <\/span><\/p>\n\n\n\n

    Conversely, there are many malicious actors in the data scraping world. For example, the internet is rife with <\/span>content theft<\/b>, which is done through web scrapers. A massive amount of content is stolen and republished illegally this way. Some companies also employ internet scrapers to undercut the prices of rivals, using these tools to access competing business databases. This is another example of malicious data scraping. <\/span><\/p>\n\n\n\n

    If you\u2019re looking to get started web scraping, make sure you are complying with protocols: <\/span><\/p>\n\n\n\n

    Adhere to robots.txt Instructions<\/i><\/b><\/p>\n\n\n\n

    This text file (called the \u201crobots exclusion standard\u201d) is responsible for giving instructions to intelligent bots. There are provisions located in it which you should consider before scraping a website. <\/span><\/p>\n\n\n\n

    Know What Elements You Are Targeting<\/i><\/b><\/p>\n\n\n\n

    If you don\u2019t limit yourself to specific elements on your target, then you are going to end up with too much data. Also, be sure to understand <\/span>how HTML tags work<\/span><\/a>. <\/span><\/p>\n\n\n\n

    Figure Out How to Best Store the Data<\/i><\/b><\/p>\n\n\n\n

    Various tools exist to store your data effectively. This nifty <\/span>step-by-step guide by How PC Rules<\/span><\/a> explains how to scrape and save the collected information in a proper database. <\/span><\/p>\n\n\n\n

    Consider Copyright Limitations<\/i><\/b><\/p>\n\n\n\n

    Internet data scraping has been getting a bad rap recently because of the clear financial motive so often behind it. Many seem to also ignore the basic terms of service (ToS). However, this does not mean you should be lax when it comes to copyright; in fact, Linkedin <\/span>sued dozens of people<\/span><\/a> for scraping its website. <\/span><\/p>\n\n\n\n

    When in doubt, always read through the ToS and respect the rules of <\/span>robots.txt. <\/span><\/i><\/p>\n\n\n\n

    Make the Extracted Clean Text More Readable<\/i><\/b><\/p>\n\n\n\n

    The Regex Python module can be used to extract a \u201ccleaner\u201d version of a given data set from a web scraper. This can be instrumental in making the database readable. Laura Turner O\u2019Hara, as part of the \u201cProgramming Historian\u201d<\/span> series, <\/span>explains how she used Regex<\/span><\/a> to extract a readable version of a \u201cCongressional Directory\u201d text file. It\u2019s worth a read if you want to know more about this often-needed process. <\/span><\/p>\n\n\n\n

    Don\u2019t Overload the Site with Your Scrapers<\/i><\/b><\/p>\n\n\n\n

    If you\u2019re just a single person trying to extract data from a website, then obviously you can\u2019t do much damage. Yet, imagine you are running a Python script that executes thousands of requests. Suddenly, your web scraping could have a serious impact and potentially bring down the entire site for a brief period. <\/span><\/p>\n\n\n\n

    When employing program web scrapers, you should limit your request to one per page. This way you don\u2019t bring down the host site when extracting information from it. <\/span><\/p>\n\n\n\n

    Final Thoughts<\/h2>\n\n\n\n

    Web scraping has opened up the door to big data, allowing us to compile billions of pieces of information through intelligent scripts and design. If you\u2019ve ever used AWS or Google Analytics, you have already encountered web scraping tools, whether you’ve been aware of it or not. As the vast collection of knowledge online increases, scrapers will only become more complex.<\/span><\/p>\n\n\n\n

    So, if you\u2019re looking to jump into web scraping, be it big or small, just remember to plan beforehand or else you will end up with a jumbled mess of data. <\/span>Set your parameters, have a clear plan on how to best store the data and know exactly what you are looking for before you start. <\/b><\/p>\n\n\n\n

    Web scraping without a plan will lead you down a long, confusing road. Luckily, with the help of intelligent bots, internet scraping can make your life a whole lot easier if you do it right.<\/span><\/p>\n\n\n\n

    Contemplating a career in tech? Springboard can help you land your dream job with our self-guided, mentor-led, career-focused bootcamps in data science<\/a>, data analytics<\/a>, machine learning<\/a>, UX design<\/a>, and digital marketing<\/a>.<\/b><\/p>\n\n\n\n

    Since you’re here…<\/strong>
    Considering a career in data analytics? We can get you there. Don\u2019t take our word for it \u2013 check out our     student reviews<\/a>. After just 6 months of study in our fully flexible Data Analytics Bootcamp<\/a>, we\u2019ll boost you into a job in the field or your tuition back. Get started now with our free data analytics course<\/a>.\u00a0<\/p>\n","protected":false},"excerpt":{"rendered":"

    Web scraping: it\u2019s used all the time online, yet you may not know what it is. With the internet being a treasure trove of content, web scrapers provide us with the proper tools to extract valuable information from the web\u2019s innumerable pages. This information is then saved as a local file on your computer. The […]<\/p>\n","protected":false},"author":87,"featured_media":8392,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_eb_attr":"","_eb_data_table":"","footnotes":""},"categories":[134],"tags":[],"marketing_tags":[],"class_list":{"0":"post-8391","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-data-analytics"},"acf":[],"_links":{"self":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts\/8391"}],"collection":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/users\/87"}],"replies":[{"embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/comments?post=8391"}],"version-history":[{"count":3,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts\/8391\/revisions"}],"predecessor-version":[{"id":47327,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/posts\/8391\/revisions\/47327"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/media\/8392"}],"wp:attachment":[{"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/media?parent=8391"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/categories?post=8391"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/tags?post=8391"},{"taxonomy":"marketing_tags","embeddable":true,"href":"https:\/\/www.springboard.com\/blog\/wp-json\/wp\/v2\/marketing_tags?post=8391"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}