User-Agent Management in Puppeteer for Device Emulation

Controlling how your browser appears to websites is crucial for web automation and testing. Puppeteer makes this possible by allowing you to modify the User-Agent string - a digital ID that websites use to detect your browser and device.

Here’s what you’ll learn:

• What is a User-Agent string? It tells websites about your browser, OS, and device.
• Why customize it? To avoid detection, test websites on different devices, or ensure mobile responsiveness.
• How to use Puppeteer: Modify the User-Agent with page.setUserAgent() and emulate devices with built-in profiles like iPhone or custom settings.

Key tips include:

• Always set the User-Agent before loading a page.
• Match the User-Agent with device-specific settings (e.g., viewport size, touch support).
• Use tools like the puppeteer-extra-stealth-plugin to avoid bot detection.

Quick Example:

await page.setUserAgent(
  'Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1'
);
await page.setViewport({ width: 375, height: 812, isMobile: true, hasTouch: true });

This ensures your browser mimics an iPhone, helping you test websites or scrape data without being flagged.

Puppeteer Tutorial #13 | setUserAgent and setViewPort | Test ...

Setting Up Puppeteer

Get Puppeteer ready by installing the necessary dependencies and writing your first emulation script.

Installation Steps

Make sure you're using Node.js v14 or newer. Then, choose one of these installation methods:

# Option 1: Install Puppeteer with Chrome included
npm i puppeteer

# Option 2: Install Puppeteer without Chrome
npm i puppeteer-core

If you're on Linux, check for missing dependencies with:

ldd chrome | grep not

Once Puppeteer is installed, you're ready to write your first device emulation script. This setup is key for controlling User-Agent and simulating devices effectively.

First Device Emulation Script

Here's an example script that emulates an iPhone X and loads a webpage:

const puppeteer = require('puppeteer');

(async () => {
  // Launch browser in non-headless mode
  const browser = await puppeteer.launch({ 
    headless: false 
  });

  // Create a new page
  const page = await browser.newPage();

  // Set iPhone X User-Agent
  await page.setUserAgent(
    'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) ' +
    'AppleWebKit/604.1.38 (KHTML, like Gecko) ' +
    'Version/11.0 Mobile/15A372 Safari/604.1'
  );

  // Configure viewport for iPhone X
  await page.setViewport({
    width: 375,
    height: 812,
    deviceScaleFactor: 3,
    isMobile: true,
    hasTouch: true
  });

  // Navigate to a webpage
  await page.goto('https://pptr.dev');

  // Pause to view the result
  await page.waitForTimeout(3000);

  // Close the browser
  await browser.close();
})();

This script covers the essentials for device emulation:

• Browser launch: Starts Puppeteer in non-headless mode for visibility.
• Page setup: Creates a new page in the browser.
• User-Agent settings: Mimics an iPhone X browser.
• Viewport configuration: Matches the screen dimensions and capabilities of an iPhone X.
• Navigation: Loads a specified webpage.

Configuration Tips

• Puppeteer stores its browser cache at ~/.cache/puppeteer.
• To use a custom cache directory, set the PUPPETEER_CACHE_DIR environment variable.
• If you're working in Docker or WSL, ensure all required system dependencies are installed.
• For security, avoid running Chrome without sandboxing unless absolutely necessary.

Setting User-Agent Strings

This section explains how to configure and fine-tune User-Agent settings effectively.

Using setUserAgent()

The page.setUserAgent() method is your go-to for customizing how the browser identifies itself. Here's an example of how to use it:

const page = await browser.newPage();
await page.setUserAgent(
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 ' +
  '(KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'
);

Make sure to set the User-Agent before navigating to a page to ensure consistency.

Choosing User-Agent Strings

Pick a User-Agent string that aligns with your use case:

For Testing:

// Desktop Chrome on Windows 10
await page.setUserAgent(
  'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ' +
  'AppleWebKit/537.36 (KHTML, like Gecko) ' +
  'Chrome/120.0.0.0 Safari/537.36'
);

For Mobile Emulation:

// iPhone 14 Safari
await page.setUserAgent(
  'Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) ' +
  'AppleWebKit/605.1.15 (KHTML, like Gecko) ' +
  'Version/16.0 Mobile/15E148 Safari/604.1'
);

User-Agent Configuration Tips

Here are some tips to ensure smooth User-Agent management:

• Match settings: Pair your User-Agent with corresponding browser properties. For example, use mobile User-Agents with mobile viewport settings.
• Avoid detection issues: Puppeteer's default User-Agent includes "HeadlessChrome", which can trigger bot detection mechanisms.

// Default User-Agent (not recommended)

// Custom User-Agent (recommended)
await page.setUserAgent(
  'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) ' +
  'AppleWebKit/537.36 (KHTML, like Gecko) ' +
  'Chrome/120.0.0.0 Safari/537.36'
);

• Be consistent: Set the User-Agent for every new page or tab to avoid inconsistencies.

For better reliability, you can also configure additional settings alongside your User-Agent:

await page.setExtraHTTPHeaders({
  'Accept-Language': 'en-US,en;q=0.9',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
});

Proper User-Agent management is just one piece of device emulation. Pair it with accurate viewport, network, and browser settings to create a more realistic browsing experience.

sbb-itb-23997f1

Advanced Device Emulation

Built-in Device Profiles

Puppeteer makes device emulation easier with its pre-configured device profiles. These profiles bundle essential settings into one, so you can quickly emulate specific devices.

const puppeteer = require('puppeteer');
const iPhone15Pro = puppeteer.KnownDevices['iPhone 15 Pro'];

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.emulate(iPhone15Pro);
  // The page now behaves like an iPhone 15 Pro.
})();

These profiles automatically set up important parameters, making it simple to emulate devices accurately. They work well alongside User-Agent configurations and allow for additional tweaks to screen and viewport settings.

Screen and Viewport Settings

To emulate devices effectively, you need to set up the viewport properly. Puppeteer's default viewport (800×600 pixels) doesn't match most real devices, so you'll want to customize it:

await page.setViewport({
  width: 1920,
  height: 1080,
  deviceScaleFactor: 2,
  isMobile: false,
  hasTouch: false,
  isLandscape: true
});

You can also adjust the viewport dynamically to capture full-page screenshots with ease:

await page.setViewport({
  width: 1440,
  height: 900,
  deviceScaleFactor: 1
});

const options = {
  fullPage: true,
  captureBeyondViewport: true
};
await page.screenshot(options);

Network and Touch Simulation

Once the viewport is configured, you can take it a step further by simulating network conditions and touch interactions:

// Simulate 3G network conditions
await page.emulateNetworkConditions({
  offline: false,
  downloadThroughput: (750 * 1024) / 8, // 750 kb/s
  uploadThroughput: (250 * 1024) / 8, // 250 kb/s
  latency: 100 // 100 ms
});

await page.setViewport({
  width: 375,
  height: 812,
  isMobile: true,
  hasTouch: true
});

await page.tap('#button-element');

Fixing Common Problems

Emulation Error Solutions

When emulating, mismatched user-agent strings and browser features can trigger detection. To avoid this, make sure the user-agent is consistent across all pages:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch();
const page = await browser.newPage();
await page.setUserAgent('Mozilla/5.0 (iPhone; CPU iPhone OS 17_3_1 like Mac OS X)');

For every new page, ensure the user-agent is set:

const newPage = await browser.newPage();
await newPage.setUserAgent(currentUserAgent);

Additionally, refine your setup to handle more advanced detection techniques.

Avoiding Browser Detection

Websites often use sophisticated methods to identify automation tools. To counter this, use the puppeteer-extra-stealth-plugin:

const stealthPlugin = StealthPlugin();
stealthPlugin.enabledEvasions.add('user-agent-override');
puppeteer.use(stealthPlugin);

To further minimize detection, apply these practical tactics:

• Manage Request Rates: Introduce random delays to mimic human behavior.

async function naturalDelay() {
  const delay = Math.floor(Math.random() * (3000 - 1000) + 1000);
  await page.waitForTimeout(delay);
}

• Ensure Header Consistency: Align HTTP headers with the user-agent.

await page.setExtraHTTPHeaders({
  'Accept-Language': 'en-US,en;q=0.9',
  'Accept': 'text/html,application/xhtml+xml',
  'User-Agent': currentUserAgent
});

Once detection risks are addressed, focus on improving performance and resource efficiency.

Speed and Resource Usage

Streamline your setup to enhance speed and reduce resource consumption. Start by optimizing browser launch parameters:

const browser = await puppeteer.launch({
  args: [
    '--disable-gpu',
    '--disable-dev-shm-usage',
    '--disable-setuid-sandbox',
    '--no-first-run',
    '--no-sandbox',
    '--no-zygote'
  ],
  userDataDir: './cache'
});

For screenshot tasks, use efficient settings:

const screenshot = await page.screenshot({
  type: 'jpeg',
  quality: 80,
  fullPage: true,
  encoding: 'binary'
});

Block unnecessary resources like images and fonts to save memory and bandwidth:

await page.setRequestInterception(true);
page.on('request', (request) => {
  if (request.resourceType() === 'image' || request.resourceType() === 'font') {
    request.abort();
  } else {
    request.continue();
  }
});

These adjustments improve performance while maintaining reliable emulation.

Conclusion

Key Tips Review

Fine-tuning User-Agent strings and managing headers effectively can make a big difference when it comes to avoiding detection. Here’s a quick overview of strategies for better device emulation:

Aspect	Best Practice	Impact
User-Agent Setup	Assign a User-Agent for each new tab or page	40% decrease in bot traffic
Request Patterns	Mimic browser fingerprint patterns	Reduces chances of detection
Header Alignment	Match HTTP headers to the assigned User-Agent	Ensures consistent emulation

For example, Farfetch implemented these methods in February 2023 and saw a 40% drop in bot traffic along with a 15% improvement in load times. These results show how small adjustments can lead to big performance gains.

Next Steps with Puppeteer

Once you’ve nailed the basics, take your Puppeteer scripts to the next level by tweaking launch settings for more realistic behavior. Here’s a sample configuration to get started:

// Advanced browser launch setup
const browser = await puppeteer.launch({
  args: [
    '--enable-webgl',
    '--use-gl=desktop',
    '--disable-automation'
  ],
  ignoreDefaultArgs: ['--enable-automation']
});

This setup enables features like WebGL while disabling automation flags, helping your scripts blend in more naturally.

Related posts

• What is a Headless Browser and Why Do You Need It?
• Installing and Configuring Puppeteer: Solving Common Dependency and Chromium Issues
• Form Automation with Puppeteer: Text Input, Form Filling, and User Simulation
• Browser Automation with Puppeteer and JavaScript: Practical Implementation in Node.js