Proxy locations

Europe

North America

South America

Asia

Africa

Oceania

See all locations

Network statusCareers

Proxy Integration With Node-Fetch

Node-fetch is a popular package that brings window.fetch functionality to NodeJS code. Window.fetch is a client-side API for making async HTTP requests in web browsers. Using this API, the developers can fetch JSON files, images, or any other data from the server.

Node Fetch brings the same functionality to the server-side applications using Node.JS. With this package, developers can send HTTP GET and POST requests. It also allows setting headers, credentials, and many other options that make it perfect for web scraping.

This guide will show you how to integrate Oxylabs Residential and Datacenter Proxies with Node-Fetch.

Requirements

The first step is installing node.js if you don't have it. Head over to the downloads page of Nodejs.org and download Node. 

The next step is to install the node-fetch package. To install this package, we can use the Node Package Manager tool.

Important note: Node Fetch from version 3 is an ESM-only module. In this tutorial, we will be using version 2 so that it remains compatible with CommonJS.

Open the terminal and navigate to the directory where you want to keep your source code. After that, run the following command to install node-fetch:

npm install node-fetch@2

Node fetch code example

Create a new file and save it as check-ip.js.

Then, load the node-fetch module. To load the module, add the following line of code:

const fetch = require("node-fetch");

You can now use either the then-catch syntax as follows:

fetch("https://ip.oxylabs.io/location")
  .then((response) => response.text())
  .then((data) => console.log(data))
  .catch((error) => console.error(error));

Alternatively, you can use the newer try-catch syntax as follows:

(async () => {
  try {
    const response = await fetch("https://ip.oxylabs.io/location");
    const data = await response.text();
    console.log(data);
  } catch (e) {
    console.error(e.message);
  }
})();

Save this file and open the terminal. Enter the following command to run it:

node check-ip.js

The output is your IP address.

Integrating proxies

To use proxies, we first need to load the https-proxy-agent package. 

Create a new file and save it as proxies.js. Add the following lines to load both the required packages:

const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');

Next, we must create a variable for the http or https proxy URL. 

Most proxy servers, such as Oxylabs' Residential and Datacenter proxy servers, require you to send the credentials. In such cases, you can construct the http proxy user as follows:

const proxyUrl = `http://${username}:${password}@${proxyServer}`;

We are using three other local variables to create the proxy URL here. These local variables store your username, password, and proxy server.

Residential Proxies

The following example shows how the https proxy or the http proxy URL would be for Oxylabs Residential Proxies:

const proxyUrl = `http://<USERNAME>:<PASSWORD>@pr.oxylabs.io:7777`;

Here, USERNAME and PASSWORD are your Oxylabs proxy user's credentials.

Here, you can use country-specific entries. For example, if you use the proxy server as au-pr.oxylabs.io:40000 instead of pr.oxylabs.io:7777, you'll get an IP in Australia. 

Please see our documentation for a complete list of country-specific entry nodes and sticky session details.

Datacenter Proxies

The following table summarizes the proxy server configuration for Datacenter Proxies.

Proxies Proxy type Proxy address Port User credentials Notes
Enterprise Dedicated Datacenter Proxies HTTP or SOCKS5 A selected IP from the acquired list 60000 Oxylabs proxy user’s username and password Purchase via sales
Self-Service Dedicated Datacenter Proxies HTTP or HTTPS ddc.oxylabs.io 8001 Oxylabs proxy user’s username and password Purchase via the dashboard
Shared Datacenter Proxies HTTP dc.pr.oxylabs.io 10000 Oxylabs proxy user’s username and password Purchase via the dashboard

Testing proxies

Now, it's time to test the proxies. This step would be the same for both Residential Proxies and Datacenter Proxies.

Create an instance of the HttpsProxyAgent class. The constructor of this class takes the proxy URL we have just created:

const proxyAgent = new HttpsProxyAgent(proxyUrl);

Finally, we can send this proxy agent to one of the optional parameters of the fetch method—agent. This agent represents an http(s) agent instance, which we have created using the http-proxy-agent package.

Putting together everything, the node fetch proxy code looks as follows:

const fetch = require("node-fetch");
const { HttpsProxyAgent } = require("https-proxy-agent"); 

const proxyUrl = `http://<USERNAME>:<PASSWORD>@pr.oxylabs.io:7777`;


(async () => {
  try {
    const proxyAgent = new HttpsProxyAgent(proxyUrl);
    const response = await fetch("https://ip.oxylabs.io/location", {
      agent: proxyAgent,
    });
    const data = await response.text();
    console.log(data);
  } catch (e) {
    console.error(e.message);
  }
})();

You can run this code to see the IP of the proxy address.

How to rotate proxies with node fetch

Notably, while using proxies, most websites will ban the IP if you use the same proxy. The solution is to rotate the IPs.

Oxylabs Residential and Shared Datacenter Proxies perform proxy rotation automatically and don't require external rotation.

Our Residential Proxies can either randomly change the proxy address for each request or use the same proxy IP for up to 30 minutes. Shared Datacenter Proxies also offer the above mentioned options but can keep the same IP indefinitely.

See our documentation for Residential and Shared Datacenter Proxies to find out more.

Dedicated Datacenter Proxies don't have an in-built rotation feature, but they can be implemented with our Proxy Rotator. With this tool, you can easily automate the rotation of our Dedicated Datacenter Proxies.

Selecting a proxy randomly 

Assuming you have a proxy list, you can use the following code to rotate the proxies from the given proxy list. Within the example, we run the code three times, each time picking one of the proxies randomly.

const fetch = require("node-fetch");
const HttpsProxyAgent = require("https-proxy-agent");


proxyUrls =[
    '127.0.0.1:60000',
    '127.0.0.2:60000',
    '127.0.0.3:60000'
]; 




(async () => {
  try {
    for (let i = 0; i < 3; i++) {
      const randomIndex = Math.floor(Math.random() * proxyUrls.length); // Generate a random index
      const proxyAgent = new HttpsProxyAgent(proxyUrls[randomIndex]); // Select a proxy URL using the random index
      const response = await fetch("https://ip.oxylabs.io/location", {
        agent: proxyAgent,
      });
      const data = await response.text();
      console.log(data);
    }
  } catch (e) {
    console.error(e.message);
  }
})();

Iterating over the proxy list

You can modify the above code so that it goes over all the proxies in a sequence too:

(async () => {
  try {
    for (let i = 0; i < proxyUrls.length; i++) {
      const proxyAgent = new HttpsProxyAgent(proxyUrls[i]);
      const response = await fetch("https://ip.oxylabs.io/location", {
        agent: proxyAgent,
      });
      const data = await response.text();
      console.log(data);
    }
  } catch (e) {
    console.error(e.message);
  }
})();

Conclusion

You can easily incorporate proxies into Node Fetch using the https-proxy-agent package. This integration enables you to initiate web scraping projects with Node-Fetch while avoiding concerns related to IP blocks and geo-restrictions caused by proxies.

If you have any questions or need assistance, don't hesitate to get in touch with us.

Please be aware that this is a third-party tool not owned or controlled by Oxylabs. Each third-party provider is responsible for its own software and services. Consequently, Oxylabs will have no liability or responsibility to you regarding those services. Please carefully review the third party's policies and practices and/or conduct due diligence before accessing or using third-party services.

Get the latest news from data gathering world

I'm interested

Get Node-Fetch proxies for $10/GB