.png)
.svg){kind=icon}
Ready-to-go AI agents are available on Latenode 🚀
Intercepting and modifying network requests in Puppeteer is a powerful way to manage web automation tasks. It allows you to monitor, block, or adjust HTTP requests and responses in real-time. Here's what you can do:
await page.setRequestInterception(true);
page.on('request', async (request) => {
if (request.url().includes('/api/test')) {
await request.respond({
status: 200,
contentType: 'application/json',
body: JSON.stringify({ success: true })
});
} else {
await request.continue();
}
});
This guide explains how to use Puppeteer’s API for request interception, including practical examples for monitoring, modifying, and testing network interactions.
Puppeteer makes it possible to monitor network requests, giving you a clear view of web traffic during automation tasks. Here's how you can track and analyze network activity effectively.
Start by enabling request interception and attaching a listener. Here's an example:
await page.setRequestInterception(true);
page.on('request', async (request) => {
if (request.isInterceptResolutionHandled()) return;
console.log(`Intercepted: ${request.url()}`);
await request.continue();
});
Inside the 'request' event listener, you can access and even modify intercepted requests. Just make sure to call request.continue() so the original request isn't blocked.
Puppeteer lets you filter network requests based on their type, making it easier to focus on specific traffic. Here's how you can handle requests selectively:
page.on('request', interceptedRequest => {
if (interceptedRequest.isInterceptResolutionHandled()) return;
const resourceType = interceptedRequest.resourceType();
switch(resourceType) {
case 'image':
// Handle image requests
break;
case 'xhr':
// Handle API calls
break;
default:
interceptedRequest.continue();
}
});
By categorizing and prioritizing resource types, you can streamline your network monitoring efforts and focus on what matters most.
Once you've filtered the requests, logging their key details can help you uncover useful patterns. Here's a quick overview of some important request properties:
| Request Property | Description | Access Method |
|---|---|---|
| URL | Full request URL | request.url() |
| Method | HTTP method (e.g., GET, POST) | request.method() |
| Headers | Request headers | request.headers() |
| Post Data | Payload sent with the request | request.postData() |
For example, if you want to monitor API requests for pricing data:
page.on('request', async (request) => {
if (request.url().includes('/api/prices')) {
const requestData = {
url: request.url(),
method: request.method(),
postData: request.postData()
};
console.log('Price Request:', requestData);
}
await request.continue();
});
This method allows you to analyze traffic in detail. Always check request.isInterceptResolutionHandled() to prevent conflicts with other handlers.
With Puppeteer, you can tweak outgoing requests to better suit your needs when interacting with web servers.
You can adjust HTTP headers to include tokens, change user agents, or add custom values.
await page.setExtraHTTPHeaders({
'Authorization': 'Bearer YOUR_TOKEN',
'Custom-Header': 'CustomValue'
});
For more granular control over individual requests:
page.on('request', async (request) => {
if (request.isInterceptResolutionHandled()) return;
const headers = request.headers();
headers['Authorization'] = 'Bearer YOUR_TOKEN';
await request.continue({ headers });
});
"Request interception facilitates modification of outgoing requests' properties, such as setting custom headers, altering request methods, or adjusting the request payload." - Saairaamprasad [1]
Next, let’s look at rerouting requests by redirecting URLs.
Redirect specific requests using the following approach:
await page.setRequestInterception(true);
page.on('request', async (request) => {
if (request.url().includes('/api/original')) {
await request.continue({
url: request.url().replace('/api/original', '/api/new')
});
} else {
await request.continue();
}
});
This method helps you change API endpoints or reroute requests dynamically.
You can also modify payloads for POST and PUT requests. Here’s how:
page.on('request', async (request) => {
if (request.isInterceptResolutionHandled()) return;
if (request.url().includes('/api/prices') && request.method() === 'POST') {
let postData = request.postData();
if (postData) {
postData = postData.replace(/11001/g, '90210');
await request.continue({ postData });
return;
}
}
await request.continue();
});
Here’s a quick reference for modifying different request properties:
| Request Property | How to Modify | Common Use Cases |
|---|---|---|
| Headers | continue({ headers }) |
Adding authentication tokens, custom identifiers |
| URL | continue({ url }) |
Redirecting endpoints, rerouting requests |
| Post Data | continue({ postData }) |
Updating form data, altering API payloads |
Note: Changing requests can affect performance, so it’s best to intercept only when necessary. These methods provide you with precise control over outgoing requests in your automation workflows.
Puppeteer allows you to handle and modify server responses, making it a powerful tool for web automation. This section builds on earlier techniques for request modification and focuses on controlling server responses to give you full control over network interactions.
You can track incoming server responses by setting up a response event listener:
page.on('response', async (response) => {
const url = response.url();
const status = response.status();
console.log(`Response from ${url}: ${status}`);
if (response.url().includes('/api/prices')) {
const data = await response.json();
// Process response data
}
});
This snippet logs details like the URL, status codes, and headers, helping you analyze how the server behaves. Beyond monitoring, you can modify response content dynamically to suit your needs.
To customize how responses are handled, use interception. Here's an example:
page.on('request', async (request) => {
if (request.url().includes('/api/prices')) {
await request.respond({
status: 200,
contentType: 'application/json',
body: JSON.stringify({
price: 99.99,
currency: 'USD',
zipCode: '90210'
})
});
return;
}
await request.continue();
});
This technique is particularly useful for testing APIs by mocking responses or tweaking data in specific scenarios.
Once you’ve mastered modifying content, you can simulate various HTTP status codes to test error handling. Here’s how:
| Status Code | Use Case | Implementation Example |
|---|---|---|
| 200 | Success response | status: 200, body: JSON.stringify(successData) |
| 404 | Missing resource | status: 404, body: 'Not Found' |
| 500 | Server error | status: 500, body: 'Internal Server Error' |
For example:
page.on('request', async (request) => {
if (request.url().includes('/api/test-endpoint')) {
await request.respond({
status: 404,
contentType: 'text/plain',
body: 'Not Found!'
});
return;
}
await request.continue();
});
"Request interception in Puppeteer allows you to observe, modify, or block outgoing HTTP requests and incoming responses. This feature is handy when optimizing page loading, simulating various network conditions, or handling dynamic content loading." – Saairaamprasad [1]
Pro Tip: While intercepting responses can be powerful, use it sparingly. Overusing interception might slow down your automation scripts or introduce unnecessary complexity.
Request interception is a powerful tool, but it requires careful use to ensure smooth performance and maintain reliability. Below are practical steps for managing sessions, improving speed, and resolving common issues.
Here’s how you can handle user sessions effectively:
await page.setRequestInterception(true);
page.on('request', async (request) => {
if (!request.isInterceptResolutionHandled()) {
const headers = request.headers();
headers['Authorization'] = 'Bearer ' + yourAuthToken;
await request.continue({ headers });
}
});
Always check the interception status before modifying headers to avoid conflicts with other interceptors.
Blocking unnecessary resources can significantly cut down page load times - sometimes by as much as 500 milliseconds [3]. Here’s an example:
await page.route('**/*', (route) => {
const resourceType = route.request().resourceType();
if (['image', 'stylesheet', 'font'].includes(resourceType)) {
route.abort();
return;
}
route.continue();
});
| Resource Type | Action | Impact |
|---|---|---|
| Images | Block | Speeds up page loading |
| Stylesheets | Selective loading | Saves bandwidth |
| Analytics | Block | Reduces network overhead |
| API Calls | Allow | Keeps essential functionality |
When performance issues occur, these resource management strategies can help.
Improper handling is a common source of issues. Here are some practical fixes:
"Once request interception is enabled, every request will stall unless it's continued, responded or aborted."
isInterceptResolutionHandled checks to prevent race conditions.await request.continue({
priority: 1,
headers: modifiedHeaders
});
page.on('request', async (request) => {
try {
if (!request.isInterceptResolutionHandled()) {
await request.continue();
}
} catch (error) {
console.error('Request handling error:', error);
await request.abort();
}
});
Keep your interception logic straightforward. Overcomplicated rules can slow things down and make your code harder to maintain.
Puppeteer's request interception provides developers with powerful tools to refine web automation tasks. These tools allow for:
Request interception has proven helpful in many scenarios. For instance, in a recent e-commerce case, modifying product price requests based on zip codes demonstrated its utility [2].
Some common uses include:
These examples highlight how request interception can address both development and operational challenges, paving the way for advanced automation techniques.
Latenode simplifies implementing these strategies. Here's a sample workflow to show how Puppeteer integrates with Latenode:
// Example workflow setup in Latenode
await page.setRequestInterception(true);
page.on('request', async (request) => {
if (request.resourceType() === 'fetch') {
const modifiedHeaders = {
...request.headers(),
'Custom-Header': 'Modified-Value'
};
await request.continue({ headers: modifiedHeaders });
}
});
This example demonstrates how you can modify request headers dynamically to suit your specific needs.
Swap Apps
Application 1
Application 2
Step 1: Choose a Trigger
Step 2: Choose an Action
No credit card needed
Without restriction
