Proxy Configuration in Puppeteer: Authentication, Rotation, and Bypass Techniques

Proxies are essential for Puppeteer automation. They hide your IP, bypass geo-restrictions, and help avoid bans. Here's what you'll learn about using proxies in Puppeteer:

Set Up Proxies: Add proxies to your browser and authenticate them with ease.
Rotate Proxies: Distribute requests across multiple IPs to avoid detection.
Handle Errors: Retry requests, manage blocks, and switch proxies when needed.
Optimize Speed: Block unnecessary resources and rotate user agents for faster, stealthier scraping.

Quick Comparison of Proxy Types

Proxy Type	Speed	Anonymity	Cost	Best For
Datacenter	Fast	Low	Affordable	High-volume scraping
Residential	Moderate	High	Expensive	Sensitive or geo-specific tasks
Mobile	Slow	Very High	Very Expensive	Ad verification, CAPTCHAs

Puppeteer Proxy Integration With Bright Data - Full Tutorial 2024

Basic Proxy Setup

Setting up a proxy is a key step to optimize your Puppeteer automation. Here's how you can get started.

Adding a Proxy Server

You can configure a proxy server during the browser launch in Puppeteer like this:

<span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>({
    <span class="hljs-attr">args</span>: [<span class="hljs-string">'--proxy-server=http://157.230.255.230:8118'</span>]
});
<span class="hljs-keyword">const</span> page = <span class="hljs-keyword">await</span> browser.<span class="hljs-title function_">newPage</span>();

It's also important to handle errors when navigating with a proxy:

<span class="hljs-keyword">try</span> {
    <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">goto</span>(<span class="hljs-string">'https://books.toscrape.com/'</span>, {
        <span class="hljs-attr">timeout</span>: <span class="hljs-number">70000</span>,
        <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'networkidle2'</span>
    });
} <span class="hljs-keyword">catch</span> (err) {
    <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(err);
}

Next, you'll want to secure your connection by setting up proxy authentication.

Proxy Authentication Steps

If your proxy requires authentication, you can use the following method:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">authenticate</span>({
    <span class="hljs-attr">username</span>: <span class="hljs-string">'your_proxy_username'</span>,
    <span class="hljs-attr">password</span>: <span class="hljs-string">'your_proxy_password'</span>
});

Here's a quick comparison of different authentication methods:

Authentication Method	Best Use Case	Implementation Complexity
page.authenticate()	Standard proxy auth	Low
proxy-chain package	Advanced routing	Medium
Proxy-Authorization header	HTTP-only sites	High

Once you've set up your proxy and authentication, it's time to test it.

Testing Proxy Settings

Make sure your proxy is working by following these steps:

Test the connection with a sample URL.
Confirm the proxy's IP is being used.
Monitor the performance of the setup.

Here's an example script to test your proxy:

<span class="hljs-keyword">async</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">testProxy</span>(<span class="hljs-params"></span>) {
    <span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>({
        <span class="hljs-attr">args</span>: [<span class="hljs-string">'--proxy-server=http://157.230.255.230:8118'</span>]
    });
    <span class="hljs-keyword">const</span> page = <span class="hljs-keyword">await</span> browser.<span class="hljs-title function_">newPage</span>();
    <span class="hljs-keyword">try</span> {
        <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">goto</span>(<span class="hljs-string">'https://books.toscrape.com/'</span>, {
            <span class="hljs-attr">timeout</span>: <span class="hljs-number">70000</span>,
            <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'networkidle2'</span>
        });
        <span class="hljs-keyword">const</span> content = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">content</span>();
        <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">'Proxy connection successful'</span>);
    } <span class="hljs-keyword">catch</span> (err) {
        <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">'Proxy connection failed:'</span>, err);
    }
    <span class="hljs-keyword">await</span> browser.<span class="hljs-title function_">close</span>();
}

Testing ensures your proxy configuration is ready for use in your automation tasks.

Proxy Rotation Methods

Managing proxy rotation is essential to prevent IP bans and avoid rate limiting while scraping or automating tasks online.

Building a Proxy List

To start, create a list of proxies that can be rotated efficiently. Here's an example of how to set up and use a proxy pool:

<span class="hljs-keyword">const</span> proxyList = [
    {
        <span class="hljs-attr">host</span>: <span class="hljs-string">'157.230.255.230'</span>,
        <span class="hljs-attr">port</span>: <span class="hljs-string">'8118'</span>,
        <span class="hljs-attr">username</span>: <span class="hljs-string">'user1'</span>,
        <span class="hljs-attr">password</span>: <span class="hljs-string">'pass1'</span>
    },
    <span class="hljs-comment">// Add more proxy configurations</span>
];

<span class="hljs-keyword">function</span> <span class="hljs-title function_">getRandomProxy</span>(<span class="hljs-params"></span>) {
    <span class="hljs-keyword">return</span> proxyList[<span class="hljs-title class_">Math</span>.<span class="hljs-title function_">floor</span>(<span class="hljs-title class_">Math</span>.<span class="hljs-title function_">random</span>() * proxyList.<span class="hljs-property">length</span>)];
}

When setting up your proxy list, it's important to understand the different types of proxies and their trade-offs:

Proxy Type	Response Time	Success Rate	Cost Range
Datacenter	100-300ms	85-95%	$0.50-$2/IP
Residential	500-1000ms	90-98%	$2-$8/IP
Mobile	800-1500ms	95-99%	$5-$15/IP

Each type has its strengths and weaknesses, so choose based on your specific needs, such as speed, reliability, or budget.

Switching Proxies Between Requests

To rotate proxies automatically, you can use the following setup:

<span class="hljs-keyword">const</span> <span class="hljs-title class_">ProxyChain</span> = <span class="hljs-built_in">require</span>(<span class="hljs-string">'proxy-chain'</span>);

<span class="hljs-keyword">const</span> server = <span class="hljs-keyword">new</span> <span class="hljs-title class_">ProxyChain</span>.<span class="hljs-title class_">Server</span>({
    <span class="hljs-attr">port</span>: <span class="hljs-number">8080</span>,
    <span class="hljs-attr">prepareRequestFunction</span>: <span class="hljs-function">(<span class="hljs-params">{ request }</span>) =></span> {
        <span class="hljs-keyword">const</span> proxy = <span class="hljs-title function_">getRandomProxy</span>();
        <span class="hljs-keyword">return</span> {
            <span class="hljs-attr">upstreamProxyUrl</span>: <span class="hljs-string">`http://<span class="hljs-subst">${proxy.username}</span>:<span class="hljs-subst">${proxy.password}</span>@<span class="hljs-subst">${proxy.host}</span>:<span class="hljs-subst">${proxy.port}</span>`</span>
        };
    }
});

<span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>({
    <span class="hljs-attr">args</span>: [<span class="hljs-string">'--proxy-server=http://localhost:8080'</span>]
});

This approach ensures that each request gets routed through a different proxy without restarting the browser, making the process more efficient.

Proxy Rotation Guidelines

Here are some key practices to improve your proxy rotation strategy:

1. Timing Strategy

Introduce random delays between requests to simulate natural browsing behavior:

<span class="hljs-keyword">function</span> <span class="hljs-title function_">getRotationDelay</span>(<span class="hljs-params"></span>) {
    <span class="hljs-comment">// Random delay between 2-5 seconds</span>
    <span class="hljs-keyword">return</span> <span class="hljs-title class_">Math</span>.<span class="hljs-title function_">floor</span>(<span class="hljs-title class_">Math</span>.<span class="hljs-title function_">random</span>() * <span class="hljs-number">3000</span>) + <span class="hljs-number">2000</span>;
}

<span class="hljs-keyword">async</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">rotateProxy</span>(<span class="hljs-params"></span>) {
    <span class="hljs-keyword">await</span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Promise</span>(<span class="hljs-function"><span class="hljs-params">resolve</span> =></span> <span class="hljs-built_in">setTimeout</span>(resolve, <span class="hljs-title function_">getRotationDelay</span>()));
    <span class="hljs-keyword">const</span> proxy = <span class="hljs-title function_">getRandomProxy</span>();
    <span class="hljs-comment">// Apply new proxy configuration</span>
}

2. Error Handling

If a proxy fails, handle the error gracefully by switching to a new one and retrying the request:

<span class="hljs-keyword">async</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">handleProxyError</span>(<span class="hljs-params">error, page</span>) {
    <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">`Proxy error: <span class="hljs-subst">${error.message}</span>`</span>);
    <span class="hljs-keyword">await</span> <span class="hljs-title function_">rotateProxy</span>();
    <span class="hljs-keyword">return</span> page.<span class="hljs-title function_">reload</span>({ <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'networkidle2'</span> });
}

3. Session Management

For tasks requiring session consistency, use the same proxy for related requests:

<span class="hljs-keyword">const</span> sessionMap = <span class="hljs-keyword">new</span> <span class="hljs-title class_">Map</span>();

<span class="hljs-keyword">function</span> <span class="hljs-title function_">getProxyForSession</span>(<span class="hljs-params">sessionId</span>) {
    <span class="hljs-keyword">if</span> (!sessionMap.<span class="hljs-title function_">has</span>(sessionId)) {
        sessionMap.<span class="hljs-title function_">set</span>(sessionId, <span class="hljs-title function_">getRandomProxy</span>());
    }
    <span class="hljs-keyword">return</span> sessionMap.<span class="hljs-title function_">get</span>(sessionId);
}

With these strategies, you can efficiently manage proxies and prepare for more advanced challenges like handling blocks, CAPTCHAs, and other obstacles.

sbb-itb-23997f1

Advanced Proxy Techniques

Residential vs. Datacenter Proxies

Selecting the right proxy is key when working with Puppeteer automation. Here’s a comparison to help you decide:

Feature	Residential Proxies	Datacenter Proxies
Source	Provided by ISPs to residential properties	Sourced from data centers
Speed	Slower due to ISP routing	Faster with dedicated infrastructure
Detection Risk	Lower - mimics real user traffic	Higher - easier to detect
Cost	More expensive	Cost-effective
Geographic Coverage	Broad, across multiple locations	Often limited
Best Use Cases	High-anonymity tasks, sensitive scraping, ad verification	High-volume data mining, batch processing

Residential proxies are a solid choice for tasks that demand high anonymity, as they closely resemble real user activity [1].

Handling Blocks and CAPTCHAs

To tackle website blocks and CAPTCHA challenges, use proxy rotation and retry mechanisms. Here’s an example of how to manage these issues:

<span class="hljs-keyword">const</span> proxyManager = {
    <span class="hljs-attr">rotationDelay</span>: <span class="hljs-number">2000</span>,
    <span class="hljs-keyword">async</span> <span class="hljs-title function_">handleBlock</span>(<span class="hljs-params">page</span>) {
        <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForTimeout</span>(<span class="hljs-variable language_">this</span>.<span class="hljs-property">rotationDelay</span>);
        <span class="hljs-keyword">const</span> newProxy = <span class="hljs-keyword">await</span> <span class="hljs-variable language_">this</span>.<span class="hljs-title function_">getNextProxy</span>();
        <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">authenticate</span>(newProxy);
        <span class="hljs-keyword">return</span> page.<span class="hljs-title function_">reload</span>({ <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'networkidle0'</span> });
    },
    <span class="hljs-keyword">async</span> <span class="hljs-title function_">detectBlock</span>(<span class="hljs-params">response</span>) {
        <span class="hljs-keyword">return</span> response.<span class="hljs-title function_">status</span>() === <span class="hljs-number">403</span> || response.<span class="hljs-title function_">status</span>() === <span class="hljs-number">429</span>;
    }
};

When a block is detected, retrying with a new proxy can keep your automation running smoothly:

page.<span class="hljs-title function_">on</span>(<span class="hljs-string">'response'</span>, <span class="hljs-title function_">async</span> (response) => {
    <span class="hljs-keyword">if</span> (<span class="hljs-keyword">await</span> proxyManager.<span class="hljs-title function_">detectBlock</span>(response)) {
        <span class="hljs-keyword">await</span> proxyManager.<span class="hljs-title function_">handleBlock</span>(page);
    }
});

These strategies help maintain workflow continuity, even when encountering challenges.

Managing Proxy Errors

Handling proxy errors effectively requires timeout adjustments, validation checks, and backup proxies. For example:

<span class="hljs-keyword">async</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">handleProxyError</span>(<span class="hljs-params">error, page</span>) {
    <span class="hljs-keyword">if</span> (error.<span class="hljs-property">message</span>.<span class="hljs-title function_">includes</span>(<span class="hljs-string">'net::ERR_PROXY_CONNECTION_FAILED'</span>)) {
        <span class="hljs-keyword">const</span> newProxy = <span class="hljs-keyword">await</span> <span class="hljs-title function_">getBackupProxy</span>();
        <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">authenticate</span>(newProxy);
        <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setExtraHTTPHeaders</span>({
            <span class="hljs-string">'User-Agent'</span>: <span class="hljs-title function_">generateRandomUserAgent</span>()
        });
        <span class="hljs-keyword">return</span> page.<span class="hljs-title function_">reload</span>({ <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'networkidle2'</span> });
    }
}

Follow these guidelines for better error management:

Set connection timeouts to avoid prolonged waits.
Continuously monitor proxy performance.
Keep a pool of backup proxies ready.

Configuring your browser correctly also plays a role in ensuring seamless performance:

<span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>({
    <span class="hljs-attr">args</span>: [<span class="hljs-string">'--proxy-server=http://proxy-server:port'</span>],
    <span class="hljs-attr">timeout</span>: <span class="hljs-number">60000</span>,
    <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'networkidle0'</span>
});

Lastly, always validate your proxies to ensure they’re functional:

<span class="hljs-keyword">async</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">validateProxy</span>(<span class="hljs-params">proxyConfig</span>) {
    <span class="hljs-keyword">try</span> {
        <span class="hljs-keyword">const</span> response = <span class="hljs-keyword">await</span> <span class="hljs-title function_">fetch</span>(<span class="hljs-string">'https://api.ipify.org?format=json'</span>, {
            <span class="hljs-attr">proxy</span>: <span class="hljs-string">`http://<span class="hljs-subst">${proxyConfig.host}</span>:<span class="hljs-subst">${proxyConfig.port}</span>`</span>
        });
        <span class="hljs-keyword">return</span> response.<span class="hljs-property">ok</span>;
    } <span class="hljs-keyword">catch</span> (error) {
        <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">error</span>(<span class="hljs-string">`Proxy validation failed: <span class="hljs-subst">${error.message}</span>`</span>);
        <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
    }
}

Proxy Performance Tips

Want to get the most out of Puppeteer? These tips can help fine-tune your proxy setup and improve overall performance.

Speed Testing and Optimization

Cut down on loading times by blocking unnecessary requests like images, fonts, and stylesheets:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setRequestInterception</span>(<span class="hljs-literal">true</span>);
page.<span class="hljs-title function_">on</span>(<span class="hljs-string">'request'</span>, <span class="hljs-function">(<span class="hljs-params">req</span>) =></span> {
    <span class="hljs-keyword">if</span> (req.<span class="hljs-title function_">resourceType</span>() === <span class="hljs-string">'stylesheet'</span> || req.<span class="hljs-title function_">resourceType</span>() === <span class="hljs-string">'font'</span> || req.<span class="hljs-title function_">resourceType</span>() === <span class="hljs-string">'image'</span>) {
        req.<span class="hljs-title function_">abort</span>();
    } <span class="hljs-keyword">else</span> {
        req.<span class="hljs-title function_">continue</span>();
    }
});

Pair page.goto with page.waitForSelector for quicker navigation:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">goto</span>(url, {
    <span class="hljs-attr">timeout</span>: <span class="hljs-number">30000</span>,
    <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'domcontentloaded'</span>
});
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForSelector</span>(<span class="hljs-string">'.main-content'</span>);

"When optimizing Puppeteer, remember that there are only so many ways to speed up the startup/shutdown performance of Puppeteer itself. Most likely, the biggest gains will come from getting your target pages to render faster."
– Jon Yongfook, Founder, Bannerbear [3]

Proxy Load Distribution

Distribute traffic across multiple proxies using Puppeteer Cluster:

<span class="hljs-keyword">const</span> proxyCluster = {
    <span class="hljs-keyword">async</span> <span class="hljs-title function_">distributeLoad</span>(<span class="hljs-params">urls, proxyList</span>) {
        <span class="hljs-keyword">const</span> cluster = <span class="hljs-keyword">await</span> <span class="hljs-title class_">Cluster</span>.<span class="hljs-title function_">launch</span>({
            <span class="hljs-attr">concurrency</span>: <span class="hljs-title class_">Cluster</span>.<span class="hljs-property">CONCURRENCY_CONTEXT</span>,
            <span class="hljs-attr">maxConcurrency</span>: <span class="hljs-number">5</span>
        });

        <span class="hljs-keyword">await</span> cluster.<span class="hljs-title function_">task</span>(<span class="hljs-title function_">async</span> ({ page, <span class="hljs-attr">data</span>: url }) => {
            <span class="hljs-keyword">const</span> proxy = proxyList.<span class="hljs-title function_">getNext</span>();
            <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">authenticate</span>(proxy);
            <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">goto</span>(url);
        });
    }
};

This approach balances traffic across proxies, complementing earlier rotation methods to avoid overloading any single proxy.

Common Problems and Solutions

Here are some common challenges and how to handle them effectively:

Network Latency Use the following code to manage timeouts and avoid unnecessary delays:

<span class="hljs-keyword">const</span> <span class="hljs-title function_">performRequest</span> = <span class="hljs-keyword">async</span> (<span class="hljs-params">page, url</span>) => {
    <span class="hljs-keyword">try</span> {
        <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">goto</span>(url, {
            <span class="hljs-attr">timeout</span>: <span class="hljs-number">30000</span>,
            <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'networkidle2'</span>
        });
    } <span class="hljs-keyword">catch</span> (error) {
        <span class="hljs-keyword">if</span> (error.<span class="hljs-property">name</span> === <span class="hljs-string">'TimeoutError'</span>) {
            <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> <span class="hljs-variable language_">window</span>.<span class="hljs-title function_">stop</span>());
            <span class="hljs-keyword">return</span> <span class="hljs-title function_">handleTimeout</span>();
        }
        <span class="hljs-keyword">throw</span> error;
    }
};

Rate Limiting Detect and respond to rate limits with this snippet:

<span class="hljs-keyword">const</span> <span class="hljs-title function_">checkRateLimit</span> = <span class="hljs-keyword">async</span> (<span class="hljs-params">response</span>) => {
    <span class="hljs-keyword">if</span> (response.<span class="hljs-title function_">status</span>() === <span class="hljs-number">429</span> || response.<span class="hljs-title function_">headers</span>()[<span class="hljs-string">'retry-after'</span>]) {
        <span class="hljs-keyword">await</span> <span class="hljs-title function_">handleRateLimit</span>(response.<span class="hljs-title function_">headers</span>()[<span class="hljs-string">'retry-after'</span>]);
    }
};

Authentication Failures Handle proxy authentication errors by retrying with new credentials:

<span class="hljs-keyword">const</span> <span class="hljs-title function_">handleAuthError</span> = <span class="hljs-keyword">async</span> (<span class="hljs-params">response, page, proxy</span>) => {
    <span class="hljs-keyword">if</span> (response.<span class="hljs-title function_">status</span>() === <span class="hljs-number">407</span>) {
        <span class="hljs-keyword">await</span> <span class="hljs-title function_">retryWithNewCredentials</span>(page, proxy);
        <span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>;
    }
    <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
};

"By optimizing your Puppeteer script, you can ensure smooth and efficient operation with accurate and consistent results."
– ScrapeOps [2]

Keep an eye on response times and success rates to ensure your proxies remain efficient and undetected.

Conclusion

Key Configuration Methods

Setting up Puppeteer with proxies involves a few important steps. For private proxies, use the authenticate function to provide credentials:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">authenticate</span>({
    <span class="hljs-attr">username</span>: <span class="hljs-string">'proxyUser'</span>,
    <span class="hljs-attr">password</span>: <span class="hljs-string">'proxyPass'</span>
});

To ensure the page fully loads, use the waitUntil option (e.g., 'networkidle2') and set a timeout (e.g., 30 seconds):

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">goto</span>(url, {
    <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'networkidle2'</span>,
    <span class="hljs-attr">timeout</span>: <span class="hljs-number">30000</span>
});

These configurations form the foundation of a reliable proxy setup.

Best Practices

To maintain secure and efficient proxy operations, follow these strategies:

Connection Management

Adjust timeouts based on proxy speed.
Handle authentication errors effectively.
Monitor performance metrics to identify bottlenecks.

Security Optimization

Rotate user agents along with proxies to avoid detection.
Add a VPN layer for extra protection.
Intercept and filter unnecessary requests to save bandwidth.

Here’s a quick summary of important considerations for proxy implementation:

Aspect	Implementation	Benefit
Authentication	Use authenticate function	Secures access to private proxies
Load Management	Enable request interception	Reduces bandwidth and speeds up loading
Error Handling	Implement try-catch blocks	Manages connection issues gracefully
Identity Protection	Rotate proxies and user agents	Improves success rates and avoids blocks