How to Set Up a Proxy Server on Screaming Frog
Building a solid SEO strategy requires fresh and accurate search engine results data. One of the ways to get this data in bulk is by using web scraping tools and proxies. In fact, proxies are an inseparable part of web scraping, helping you bypass certain anti-bot measures. Scraping public SEO data is no exception – this time, we’ll demonstrate how to set up Oxylabs’ Residential and Datacenter Proxies with the Screaming Frog SEO Spider tool.
1486.png)
What is Screaming Frog?
Screaming Frog is a platform designed to scrape public web data for SEO-related purposes (specifics below), available on macOS, Linux, and Windows devices. The tool offers public data scraping from up to 500 URLs; if you need more, you’ll need to upgrade to a bigger plan.
Here are some common SEO-related initiatives professionals use this tool for:
Redirect searches
Broken link searches
Content duplicate searches
Analyzing metadata
Analyzing site architecture
…and more.
Screaming Frog SEO Spider proxy integration
For this tutorial, we’ll be using macOS 12.6.1
Before we get started with the integration, download and install the program. If you have the paid version, navigate to the menu bar above and click Enter License.
A window will pop up, requiring you to enter your username and license key. Enter those and click OK and Restart. You’ll have received these details in your email upon registration.
1. Locate the Configuration menu > select System > select Proxy.
2. Insert necessary proxy details.
A new window will pop up where you’ll have to enter proxy details. Make sure to tick the Use Proxy Server box.
2.1. Residential Proxies
If you’re using Oxylabs’ Residential Proxies, enter pr.oxylabs.io in the Address field and 7777 in the Port field. Then, click OK and Restart to complete the connection:
You can also connect to a proxy in a specific country of your choice. For example, if you enter us-pr.oxylabs.io in the Address field and 10000 under Port, you’ll connect to a US-based proxy server. There are plenty of other countries you can connect to, including sticky session options – check out our documentation for more information on that.
2.2. Dedicated Datacenter Proxies
Enterprise Dedicated Datacenter Proxies
The process of connecting to Dedicated Datacenter Proxies purchased through sales is fairly similar – you’ll just need to fill in different details. In the Address field, you’ll need to enter a specific numerical IP address from the acquired list (refer to our documentation for more information.) We’ll use 1.2.3.4 for the sake of this tutorial. In the port field, please enter 60000.
Self-Service Dedicated Datacenter Proxies
For Dedicated Datacenter Proxies acquired via the Oxylabs’ dashboard, you need to use the ddc.oxylabs.io address and a specific port to indicate the IP address from your proxy list. For example, you can use port 8001 to make a request by using the first proxy IP in your list. Visit our documentation for more information.
2.3. Shared Datacenter Proxies
For Shared Datacenter Proxies, you’ll need to enter dc.pr.oxylabs.io in the Address field and 10000 under Port, then, click OK and Restart.
Once again, you can use a country-specific entry. If you enter dc.de-pr.oxylabs.io under Address and 40000 under Port, you’ll connect to a Germany-based proxy server. To see the full list of country-specific entry nodes, head to our documentation.
That’s it! You can now start scraping public SEO data without worrying about blocks.
Wrapping up
Combined with Oxylabs proxies, Screaming Frog can deliver public search engine data in real time while bypassing anti-bot measures at the same time.
As the tutorial shows, setting up proxies on this platform is fairly simple, so you’ll be able to start working on your SEO projects right away. If you have questions about integrating Oxylabs products, don’t hesitate to drop us a message at support@oxylabs.io or reach out via live chat.
Please be aware that this is a third-party tool not owned or controlled by Oxylabs. Each third-party provider is responsible for its own software and services. Consequently, Oxylabs will have no liability or responsibility to you regarding those services. Please carefully review the third party's policies and practices and/or conduct due diligence before accessing or using third-party services.
Useful resources
Datacenter Proxies vs. Residential Proxies
A question might have arisen whilst scrolling through the infinite pools of proxy information - “what’s the difference between data center and residential proxies?” This article tackles this question.
How Does a Proxy Server Improve Security
Find out the numerous features of proxies and how they improve security.
Dedicated IP vs Shared IP
When it comes to hiding your identity online, using proxies is a great way to stay anonymous and keep your real IP address hidden. In this article, you’ll get an answer to the question, why dedicated proxies are usually better than shared ones.
Get the latest news from data gathering world
Get Screaming Frog proxies for $10/GB
GET IN TOUCH
General:
hello@oxylabs.ioSupport:
support@oxylabs.ioCareer:
career@oxylabs.io
Certified data centers and upstream providers
Connect with us
Advanced proxy solutions
Resources
Innovation hub