Proxy locations

Europe

North America

South America

Asia

Africa

Oceania

See all locations

Network statusCareers

Back to blog

How to Scrape Google Search Results: Python Tutorial

google

Roberta Aukstikalnyte

2023-10-196 min read
Share

With Google being the largest search engine, naturally, it holds immeasurable amounts of useful information. However, if you need to scrape Google search data automatically and on a large scale, you may run into some difficulties. In today’s article, we’ll explore what exactly those difficulties are, how to overcome them, and successfully scrape Google search results at scale. 

What is a Google SERP? 

Upon any discussion of scraping Google search results, you’ll likely run into the “SERP” abbreviation. SERP stands for Search Engine Results Page; it’s the page you get after entering a query into the search bar. Back in the day, Google would return a list of links for your query. Today, it looks completely different – to make your search experience quick and convenient, SERPs contain various features and elements. There are quite a few of them, so let’s just take a look at the most important ones.

1. Featured snippets

2. Paid ads

3. Video carousel

Image source: fiveblocks.com

4. People also ask

5. Local pack

6. Related searches

The legality of scraping Google search data is largely discussed in the scraping field. As a matter of fact, scraping publicly available data on the internet – including Google SERP data – is legal. However, it may vary from one situation to another, so it’s best to seek legal advice about your specific case. 

Google search scraping: the difficulties 

As said in the beginning of this article, scraping Google search results data isn't easy. Google has implemented several techniques to prevent malicious bots from harvesting its data. Here’s where the issues begin – since it’s difficult to tell the difference between a malicious bot and a non-malicious one, the latter gets flagged, too. In other words, even if your scraper isn’t a threat, it may still get banned. 

Let’s take a closer look into the difficulties of scraping public Google search results. 

1. CAPTCHAs

Prompting CAPTCHAs is the way Google differentiates real users from bots. The tests are difficult for bots to do, but humans can complete them relatively easily. If the visitor cannot complete CAPTCHA after a few tries, it often results in IP blocks. Luckily, advanced web scraping tools, such as our SERP Scraper API, are adept at dealing with CAPTCHAs without blocks. 

2. IP blocks 

Whenever you’re doing something online, your IP is visible to websites you visit. When you’re scraping Google SERP data (or data from any other website), it receives a massive number of requests. The website may see it as suspicious activity and, consequently, ban your IP, preventing the visitor from accessing it again. 

3. Disorganized data 

The whole point of gathering data from Google on a large scale is analyzing it and gaining insights. For instance, with data from Google at scale, you can build a solid search engine optimization strategy. 

However, to be able to analyze search results data, it has to be well-organized and easy to read. That said, your data-gathering tool should be able to return it in a structured format – JSON or CSV, for example.

With these challenges in mind, you’ll require an advanced web scraping tool that can overcome them all. Oxylabs’ Google Search API is constructed to bypass the technical challenges Google has implemented. It smoothly delivers public Google search results without users having to take care of scraper maintenance. 

In fact, it takes just a few steps to scrape search results with our SERP API– let’s take a closer look at that very process. Additionally, check out our other guide if you want to scrape Google Shopping results.

Scraping public Google search results with Python using our API

1. Install required Python libraries

To follow this guide on scraping Google search results, you’ll need the following: 

First, sign up for Oxylabs' Google Search Results API and save your username and password. Throughout this guide, you’ll see the username and password entities – these are the places where you’ll need to insert your own credentials. 

Then, download and install Python 3.8 or above from the python.org website. 

Finally, install the Request library, which is a popular third-party Python library for making HTTP requests. The Request library has a simple and intuitive interface for sending HTTP requests to web servers and receiving responses. 

To install this library, use the following command:

$python3 -m pip install requests 

If you’re using Windows, choose Python instead of Python3. The rest of the command remains the same: 

d:\amazon>python -m pip install requests

2. Set up a payload and send a POST request

Create a new file and enter the following code:

import requests
from pprint import pprint

payload = {
    'source': 'google',
    'url': 'https://www.google.com/search?hl=en&q=newton'  # search for newton
}

response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload,
)

pprint(response.json())

Here’s what the result should look like:

{
    "results": [
        {
            "content": "<!doctype html><html>...</html>",
            "created_at": "YYYY-DD-MM HH:MM:SS",
            "updated_at": "YYYY-DD-MM HH:MM:SS",
            "page": 1,
            "url": "https://www.google.com/search?hl=en&q=newton",
            "job_id": "1234567890123456789",
            "status_code": 200
        }
    ]
}

Notice how the url in the payload dictionary is a Google search results page. In this example, the keyword is newton.

As you can see, the query is executed and the page result in HTML is returned in the content key of the response.

Customizing query parameters

Let's review the payload dictionary from the above example for scraping Google search data.

payload = {
    'source': 'google',
    'url': 'https://www.google.com/search?hl=en&q=newton'
}

The dictionary keys are parameters used to inform Google Scraper API about required customization. 

The first parameter is the source, which is really important because it sets the scraper we’re going to use. 

The default value is Google – when you use it, you can set the url as any Google search page, and all the other parameters will be extracted from the URL. 

Although in this guide we’ll be using the google_search parameter, there are many others –  google_ads, google_hotels, google_images, google_suggest, and more. To see the full parameter list, head to our documentation

Keep in mind that if you set the source as google_search, you cannot use the url parameter. Luckily, you can use several different parameters for acquiring public Google SERP data without having to create multiple URLs. Let’s find out more about these parameters, shall we? 

Basic parameters

We’ll build the payload by adding the parameters one by one. First, begin with setting the source as google_search.

payload = {
    'source': 'google_search',
}

Now, let’s add query – a crucial parameter that determines what search results you’ll be retrieving. In our example, we’ll use newton as our search query. At this stage, the payload dictionary looks like this:

payload = {
    'source': 'google_search',
    'query': 'newton',
}

That said, google_search and query are the two essential parameters for scraping public Google search data. If you want the API to return Google search results at this stage, you can use payload. Now, let’s move to the next parameter. 

Location query parameters

You can work with a domain parameter if you want to use a localized domain – for example, 'domain':'de' will fetch results from google.de. If you want to see the results from Germany, use the geo_location parameter— 'geo_location':'Germany'. See the documentation for the geo_location parameter to learn more about the correct values.

Also, here’s what changing the locale parameter looks like: 

payload = {
'source':'google_search',
'query':'newton',
'domain':'de' ,
'geo_location': 'Germany',
'locale' : 'en-us'
}

To learn more about the potential values of the locale parameter, check the documentation, as well.  

If you send the above payload, you’ll receive search results in American English from google.de, just like anyone physically located in Germany would.

Controlling the number of results

By default, you’ll see the first ten results from the first page. If you want to customize this, you can use these parameters: start_page, pages, and limit.

The start_page parameter determines which page of search results to return. The pages parameter specifies the number of pages. Finally, the limit parameter sets the number of results on each page.

For example, the following set of parameters fetch results from pages 11 and 12 of the search engine results, with 20 results on each page:

payload = {
    'start_page': 11,
    'pages': 2,
    'limit': 20,
    ...  # other parameters
}

Apart from the search parameters we’ve covered so far, there are a few more you can use to fine-tune your results – see our documentation on collecting public Google Search data. 

Python code for scraping Google search data

Now, let’s put together everything we’ve learned so far – here’s what the final script with the shoes keyword looks like:

import requests
from pprint import pprint
payload = {
    'source': 'google_search',
    'query': 'shoes',
    'domain': 'de',
    'geo_location': 'Germany',
    'locale': 'en-us',
    'parse': True,
    'start_page': 1,
    'pages': 5,
    'limit': 10,
}


# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload,
)


if response.status_code != 200:
    print("Error - ", response.json())
    exit(-1)


pprint(response.json())

3. Export scraped data to a CSV

One of the best Google Scraper API features is the ability to parse an HTML page into JSON. For that, you don't need to use BeautifulSoup or any other library – just send the parse parameter as True. 

Here is a sample payload:

payload = {
    'source': 'google_search',
    'query': 'adidas',
    'parse': True,
}

When sent to the Google Scraper API, this payload will return the results in JSON. To see a detailed JSON data structure, see our documentation

The key highlights: 

  • The results are in the dedicated results list. Here, each page gets a new entry.

  • Each result contains the content in a dictionary key named content

  • The actual results are in the results key.

  • Note that there’s a job_id in the results.

The easiest way to save the data is by using the Pandas library, since it can normalize JSON quite effectively.

import pandas as pd
...
data = response.json()
df = pd.json_normalize(data['results'])
df.to_csv('export.csv', index=False)

Alternatively, you can also take note of the job_id and send a GET request to the following URL, along with your credentials.

http://data.oxylabs.io/v1/queries/{job_id}/results/normalized?format=csv

Handling errors and exceptions

When scraping Google, you can run into several challenges: network issues,  invalid query parameters, or API quota limitations.

To handle these, you can use try-except blocks in your code. For example, if an error occurs when sending the API request, you can catch the exception and print an error message:

try:
    response = requests.request(
        'POST',
        'https://realtime.oxylabs.io/v1/queries',
        auth=('USERNAME', 'PASSWORD'),
        json=payload,
    )
except requests.exceptions.RequestException as e:
    print("Error:", e)

If you send an invalid parameter, Google Scraper API will return the 400 response code.

To catch these errors, check the status code: 

if response.status_code != 200:
    print("Error - ", response.json())

Conclusion

We hope this extensive guide on how to scrape Google search results with Python helps you get the data you need to get your operations going. In case you run into any questions or uncertainties, don’t hesitate to reach out to our support team via email or live chat – our professional team will be happy to consult you about any matter related to scraping Google SERPs.

Frequently asked questions

Is scraping Google allowed? 

Google search results are considered publicly available data, so scraping them is allowed. However, there are some types of data you cannot scrape (i.e., personal information, copyrighted content) so it’s best if you consult a legal professional beforehand. 

Can I scrape Google Events results?

You can easily search for events, such as concerts, festivals, exhibitions, gatherings, etc., happening worldwide via Google. By entering events-related keywords, you'll see an additional table of events above the search engine result page that gives you the exact location, title of event, band or artists, date, etc. And, of course, it's possible to scrape this public data. However, extracting information from Google is legal if you follow all the rules and regulations. Getting professional legal advice is a great idea before collecting public data from Google, especially on a large scale. 

Can I scrape Google Local results?

A combination of relevance and distance factors helps Google find the best match for a user's search. For example, if you're searching for coffee places, Google will show you all places near you and even how to get there. These results are called Google Local results and differ from Google Maps results. The latter is a location search tool that calculates routes to your specified location. You can scrape public Google Local results for your use case as long as you follow all the regulations. We also suggest consulting a lawyer for professional advice about scraping Google.

Can I scrape the "About this result" sections?

Google provides more information about a website on which a result is found by clicking on the three dots next to the search result's right side. If you follow all the rules and regulations, you can scrape this public data for your use case. Still, getting legal advice, especially if you're planning to extract public data on a large scale, is a good idea.

Can I scrape Google Video results?

It's considered legal to scrape public Google Video results, but you should keep in mind that you must follow all the regulations and rules. Gathering meta titles, descriptions of a video, URL, etc., might be helpful for your use case. But consulting with a lawyer before gathering such public data is a good idea, especially if you're planning to collect vast amounts of information.

What are the main ways to scrape Google search pages?

You have two options for gathering information from Google search pages: either by URL or by search query. Extracting data by URL lets you retrieve information from a Google Search results page. You can obtain Google data by pasting a URL copied from any Google country domain (e.g., google.co.uk). You have the freedom to include as many URLs as needed.

About the author

Roberta Aukstikalnyte

Senior Content Manager

Roberta Aukstikalnyte is a Senior Content Manager at Oxylabs. Having worked various jobs in the tech industry, she especially enjoys finding ways to express complex ideas in simple ways through content. In her free time, Roberta unwinds by reading Ottessa Moshfegh's novels, going to boxing classes, and playing around with makeup.

All information on Oxylabs Blog is provided on an "as is" basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on Oxylabs Blog or any third-party websites that may be linked therein. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website's terms of service or receive a scraping license.

Related articles

Get the latest news from data gathering world

IN THIS ARTICLE:


  • What is a Google SERP? 

  • Is it legal to scrape Google results? 

  • Google search scraping: the difficulties 

  • Scraping public Google search results with Python using our API

  • Conclusion

Try Google Search Scraper API

Choose Oxylabs' Google Search Scraper API to unlock real-time search data hassle-free.

Scale up your business with Oxylabs®