User-Agent Management in Puppeteer for Device Emulation

Controlling how your browser appears to websites is crucial for web automation and testing. Puppeteer makes this possible by allowing you to modify the User-Agent string - a digital ID that websites use to detect your browser and device.

Here’s what you’ll learn:

• What is a User-Agent string? It tells websites about your browser, OS, and device.
• Why customize it? To avoid detection, test websites on different devices, or ensure mobile responsiveness.
• How to use Puppeteer: Modify the User-Agent with page.setUserAgent() and emulate devices with built-in profiles like iPhone or custom settings.

Key tips include:

• Always set the User-Agent before loading a page.
• Match the User-Agent with device-specific settings (e.g., viewport size, touch support).
• Use tools like the puppeteer-extra-stealth-plugin to avoid bot detection.

Quick Example:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setUserAgent</span>(
  <span class="hljs-string">'Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.0 Mobile/15E148 Safari/604.1'</span>
);
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setViewport</span>({ <span class="hljs-attr">width</span>: <span class="hljs-number">375</span>, <span class="hljs-attr">height</span>: <span class="hljs-number">812</span>, <span class="hljs-attr">isMobile</span>: <span class="hljs-literal">true</span>, <span class="hljs-attr">hasTouch</span>: <span class="hljs-literal">true</span> });

This ensures your browser mimics an iPhone, helping you test websites or scrape data without being flagged.

Puppeteer Tutorial #13 | setUserAgent and setViewPort | Test ...

Setting Up Puppeteer

Get Puppeteer ready by installing the necessary dependencies and writing your first emulation script.

Installation Steps

Make sure you're using Node.js v14 or newer. Then, choose one of these installation methods:

<span class="hljs-comment"># Option 1: Install Puppeteer with Chrome included</span>
npm i puppeteer

<span class="hljs-comment"># Option 2: Install Puppeteer without Chrome</span>
npm i puppeteer-core

If you're on Linux, check for missing dependencies with:

ldd chrome | grep not

Once Puppeteer is installed, you're ready to write your first device emulation script. This setup is key for controlling User-Agent and simulating devices effectively.

First Device Emulation Script

Here's an example script that emulates an iPhone X and loads a webpage:

<span class="hljs-keyword">const</span> puppeteer = <span class="hljs-built_in">require</span>(<span class="hljs-string">'puppeteer'</span>);

(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-comment">// Launch browser in non-headless mode</span>
  <span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>({ 
    <span class="hljs-attr">headless</span>: <span class="hljs-literal">false</span> 
  });

  <span class="hljs-comment">// Create a new page</span>
  <span class="hljs-keyword">const</span> page = <span class="hljs-keyword">await</span> browser.<span class="hljs-title function_">newPage</span>();

  <span class="hljs-comment">// Set iPhone X User-Agent</span>
  <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setUserAgent</span>(
    <span class="hljs-string">'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) '</span> +
    <span class="hljs-string">'AppleWebKit/604.1.38 (KHTML, like Gecko) '</span> +
    <span class="hljs-string">'Version/11.0 Mobile/15A372 Safari/604.1'</span>
  );

  <span class="hljs-comment">// Configure viewport for iPhone X</span>
  <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setViewport</span>({
    <span class="hljs-attr">width</span>: <span class="hljs-number">375</span>,
    <span class="hljs-attr">height</span>: <span class="hljs-number">812</span>,
    <span class="hljs-attr">deviceScaleFactor</span>: <span class="hljs-number">3</span>,
    <span class="hljs-attr">isMobile</span>: <span class="hljs-literal">true</span>,
    <span class="hljs-attr">hasTouch</span>: <span class="hljs-literal">true</span>
  });

  <span class="hljs-comment">// Navigate to a webpage</span>
  <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">goto</span>(<span class="hljs-string">'https://pptr.dev'</span>);

  <span class="hljs-comment">// Pause to view the result</span>
  <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForTimeout</span>(<span class="hljs-number">3000</span>);

  <span class="hljs-comment">// Close the browser</span>
  <span class="hljs-keyword">await</span> browser.<span class="hljs-title function_">close</span>();
})();

This script covers the essentials for device emulation:

• Browser launch: Starts Puppeteer in non-headless mode for visibility.
• Page setup: Creates a new page in the browser.
• User-Agent settings: Mimics an iPhone X browser.
• Viewport configuration: Matches the screen dimensions and capabilities of an iPhone X.
• Navigation: Loads a specified webpage.

Configuration Tips

• Puppeteer stores its browser cache at ~/.cache/puppeteer.
• To use a custom cache directory, set the PUPPETEER_CACHE_DIR environment variable.
• If you're working in Docker or WSL, ensure all required system dependencies are installed.
• For security, avoid running Chrome without sandboxing unless absolutely necessary.

Setting User-Agent Strings

This section explains how to configure and fine-tune User-Agent settings effectively.

Using setUserAgent()

The page.setUserAgent() method is your go-to for customizing how the browser identifies itself. Here's an example of how to use it:

<span class="hljs-keyword">const</span> page = <span class="hljs-keyword">await</span> browser.<span class="hljs-title function_">newPage</span>();
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setUserAgent</span>(
  <span class="hljs-string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '</span> +
  <span class="hljs-string">'(KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36'</span>
);

Make sure to set the User-Agent before navigating to a page to ensure consistency.

Choosing User-Agent Strings

Pick a User-Agent string that aligns with your use case:

For Testing:

<span class="hljs-comment">// Desktop Chrome on Windows 10</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setUserAgent</span>(
  <span class="hljs-string">'Mozilla/5.0 (Windows NT 10.0; Win64; x64) '</span> +
  <span class="hljs-string">'AppleWebKit/537.36 (KHTML, like Gecko) '</span> +
  <span class="hljs-string">'Chrome/120.0.0.0 Safari/537.36'</span>
);

For Mobile Emulation:

<span class="hljs-comment">// iPhone 14 Safari</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setUserAgent</span>(
  <span class="hljs-string">'Mozilla/5.0 (iPhone; CPU iPhone OS 16_0 like Mac OS X) '</span> +
  <span class="hljs-string">'AppleWebKit/605.1.15 (KHTML, like Gecko) '</span> +
  <span class="hljs-string">'Version/16.0 Mobile/15E148 Safari/604.1'</span>
);

User-Agent Configuration Tips

Here are some tips to ensure smooth User-Agent management:

• Match settings: Pair your User-Agent with corresponding browser properties. For example, use mobile User-Agents with mobile viewport settings.
• Avoid detection issues: Puppeteer's default User-Agent includes "HeadlessChrome", which can trigger bot detection mechanisms.

<span class="hljs-comment">// Default User-Agent (not recommended)</span>

<span class="hljs-comment">// Custom User-Agent (recommended)</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setUserAgent</span>(
  <span class="hljs-string">'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) '</span> +
  <span class="hljs-string">'AppleWebKit/537.36 (KHTML, like Gecko) '</span> +
  <span class="hljs-string">'Chrome/120.0.0.0 Safari/537.36'</span>
);

• Be consistent: Set the User-Agent for every new page or tab to avoid inconsistencies.

For better reliability, you can also configure additional settings alongside your User-Agent:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setExtraHTTPHeaders</span>({
  <span class="hljs-string">'Accept-Language'</span>: <span class="hljs-string">'en-US,en;q=0.9'</span>,
  <span class="hljs-string">'Accept'</span>: <span class="hljs-string">'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'</span>
});

Proper User-Agent management is just one piece of device emulation. Pair it with accurate viewport, network, and browser settings to create a more realistic browsing experience.

sbb-itb-23997f1

Advanced Device Emulation

Built-in Device Profiles

Puppeteer makes device emulation easier with its pre-configured device profiles. These profiles bundle essential settings into one, so you can quickly emulate specific devices.

<span class="hljs-keyword">const</span> puppeteer = <span class="hljs-built_in">require</span>(<span class="hljs-string">'puppeteer'</span>);
<span class="hljs-keyword">const</span> iPhone15Pro = puppeteer.<span class="hljs-property">KnownDevices</span>[<span class="hljs-string">'iPhone 15 Pro'</span>];

(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>();
  <span class="hljs-keyword">const</span> page = <span class="hljs-keyword">await</span> browser.<span class="hljs-title function_">newPage</span>();
  <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">emulate</span>(iPhone15Pro);
  <span class="hljs-comment">// The page now behaves like an iPhone 15 Pro.</span>
})();

These profiles automatically set up important parameters, making it simple to emulate devices accurately. They work well alongside User-Agent configurations and allow for additional tweaks to screen and viewport settings.

Screen and Viewport Settings

To emulate devices effectively, you need to set up the viewport properly. Puppeteer's default viewport (800×600 pixels) doesn't match most real devices, so you'll want to customize it:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setViewport</span>({
  <span class="hljs-attr">width</span>: <span class="hljs-number">1920</span>,
  <span class="hljs-attr">height</span>: <span class="hljs-number">1080</span>,
  <span class="hljs-attr">deviceScaleFactor</span>: <span class="hljs-number">2</span>,
  <span class="hljs-attr">isMobile</span>: <span class="hljs-literal">false</span>,
  <span class="hljs-attr">hasTouch</span>: <span class="hljs-literal">false</span>,
  <span class="hljs-attr">isLandscape</span>: <span class="hljs-literal">true</span>
});

You can also adjust the viewport dynamically to capture full-page screenshots with ease:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setViewport</span>({
  <span class="hljs-attr">width</span>: <span class="hljs-number">1440</span>,
  <span class="hljs-attr">height</span>: <span class="hljs-number">900</span>,
  <span class="hljs-attr">deviceScaleFactor</span>: <span class="hljs-number">1</span>
});

<span class="hljs-keyword">const</span> options = {
  <span class="hljs-attr">fullPage</span>: <span class="hljs-literal">true</span>,
  <span class="hljs-attr">captureBeyondViewport</span>: <span class="hljs-literal">true</span>
};
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">screenshot</span>(options);

Network and Touch Simulation

Once the viewport is configured, you can take it a step further by simulating network conditions and touch interactions:

<span class="hljs-comment">// Simulate 3G network conditions</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">emulateNetworkConditions</span>({
  <span class="hljs-attr">offline</span>: <span class="hljs-literal">false</span>,
  <span class="hljs-attr">downloadThroughput</span>: (<span class="hljs-number">750</span> * <span class="hljs-number">1024</span>) / <span class="hljs-number">8</span>, <span class="hljs-comment">// 750 kb/s</span>
  <span class="hljs-attr">uploadThroughput</span>: (<span class="hljs-number">250</span> * <span class="hljs-number">1024</span>) / <span class="hljs-number">8</span>, <span class="hljs-comment">// 250 kb/s</span>
  <span class="hljs-attr">latency</span>: <span class="hljs-number">100</span> <span class="hljs-comment">// 100 ms</span>
});

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setViewport</span>({
  <span class="hljs-attr">width</span>: <span class="hljs-number">375</span>,
  <span class="hljs-attr">height</span>: <span class="hljs-number">812</span>,
  <span class="hljs-attr">isMobile</span>: <span class="hljs-literal">true</span>,
  <span class="hljs-attr">hasTouch</span>: <span class="hljs-literal">true</span>
});

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">tap</span>(<span class="hljs-string">'#button-element'</span>);

Fixing Common Problems

Emulation Error Solutions

When emulating, mismatched user-agent strings and browser features can trigger detection. To avoid this, make sure the user-agent is consistent across all pages:

<span class="hljs-keyword">const</span> puppeteer = <span class="hljs-built_in">require</span>(<span class="hljs-string">'puppeteer-extra'</span>);
<span class="hljs-keyword">const</span> <span class="hljs-title class_">StealthPlugin</span> = <span class="hljs-built_in">require</span>(<span class="hljs-string">'puppeteer-extra-plugin-stealth'</span>);
puppeteer.<span class="hljs-title function_">use</span>(<span class="hljs-title class_">StealthPlugin</span>());

<span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>();
<span class="hljs-keyword">const</span> page = <span class="hljs-keyword">await</span> browser.<span class="hljs-title function_">newPage</span>();
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setUserAgent</span>(<span class="hljs-string">'Mozilla/5.0 (iPhone; CPU iPhone OS 17_3_1 like Mac OS X)'</span>);

For every new page, ensure the user-agent is set:

<span class="hljs-keyword">const</span> newPage = <span class="hljs-keyword">await</span> browser.<span class="hljs-title function_">newPage</span>();
<span class="hljs-keyword">await</span> newPage.<span class="hljs-title function_">setUserAgent</span>(currentUserAgent);

Additionally, refine your setup to handle more advanced detection techniques.

Avoiding Browser Detection

Websites often use sophisticated methods to identify automation tools. To counter this, use the puppeteer-extra-stealth-plugin:

<span class="hljs-keyword">const</span> stealthPlugin = <span class="hljs-title class_">StealthPlugin</span>();
stealthPlugin.<span class="hljs-property">enabledEvasions</span>.<span class="hljs-title function_">add</span>(<span class="hljs-string">'user-agent-override'</span>);
puppeteer.<span class="hljs-title function_">use</span>(stealthPlugin);

To further minimize detection, apply these practical tactics:

• Manage Request Rates: Introduce random delays to mimic human behavior.

<span class="hljs-keyword">async</span> <span class="hljs-keyword">function</span> <span class="hljs-title function_">naturalDelay</span>(<span class="hljs-params"></span>) {
  <span class="hljs-keyword">const</span> delay = <span class="hljs-title class_">Math</span>.<span class="hljs-title function_">floor</span>(<span class="hljs-title class_">Math</span>.<span class="hljs-title function_">random</span>() * (<span class="hljs-number">3000</span> - <span class="hljs-number">1000</span>) + <span class="hljs-number">1000</span>);
  <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForTimeout</span>(delay);
}

• Ensure Header Consistency: Align HTTP headers with the user-agent.

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setExtraHTTPHeaders</span>({
  <span class="hljs-string">'Accept-Language'</span>: <span class="hljs-string">'en-US,en;q=0.9'</span>,
  <span class="hljs-string">'Accept'</span>: <span class="hljs-string">'text/html,application/xhtml+xml'</span>,
  <span class="hljs-string">'User-Agent'</span>: currentUserAgent
});

Once detection risks are addressed, focus on improving performance and resource efficiency.

Speed and Resource Usage

Streamline your setup to enhance speed and reduce resource consumption. Start by optimizing browser launch parameters:

<span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>({
  <span class="hljs-attr">args</span>: [
    <span class="hljs-string">'--disable-gpu'</span>,
    <span class="hljs-string">'--disable-dev-shm-usage'</span>,
    <span class="hljs-string">'--disable-setuid-sandbox'</span>,
    <span class="hljs-string">'--no-first-run'</span>,
    <span class="hljs-string">'--no-sandbox'</span>,
    <span class="hljs-string">'--no-zygote'</span>
  ],
  <span class="hljs-attr">userDataDir</span>: <span class="hljs-string">'./cache'</span>
});

For screenshot tasks, use efficient settings:

<span class="hljs-keyword">const</span> screenshot = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">screenshot</span>({
  <span class="hljs-attr">type</span>: <span class="hljs-string">'jpeg'</span>,
  <span class="hljs-attr">quality</span>: <span class="hljs-number">80</span>,
  <span class="hljs-attr">fullPage</span>: <span class="hljs-literal">true</span>,
  <span class="hljs-attr">encoding</span>: <span class="hljs-string">'binary'</span>
});

Block unnecessary resources like images and fonts to save memory and bandwidth:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setRequestInterception</span>(<span class="hljs-literal">true</span>);
page.<span class="hljs-title function_">on</span>(<span class="hljs-string">'request'</span>, <span class="hljs-function">(<span class="hljs-params">request</span>) =></span> {
  <span class="hljs-keyword">if</span> (request.<span class="hljs-title function_">resourceType</span>() === <span class="hljs-string">'image'</span> || request.<span class="hljs-title function_">resourceType</span>() === <span class="hljs-string">'font'</span>) {
    request.<span class="hljs-title function_">abort</span>();
  } <span class="hljs-keyword">else</span> {
    request.<span class="hljs-title function_">continue</span>();
  }
});

These adjustments improve performance while maintaining reliable emulation.

Conclusion

Key Tips Review

Fine-tuning User-Agent strings and managing headers effectively can make a big difference when it comes to avoiding detection. Here’s a quick overview of strategies for better device emulation:

Aspect	Best Practice	Impact
User-Agent Setup	Assign a User-Agent for each new tab or page	40% decrease in bot traffic [1]
Request Patterns	Mimic browser fingerprint patterns	Reduces chances of detection
Header Alignment	Match HTTP headers to the assigned User-Agent	Ensures consistent emulation

For example, Farfetch implemented these methods in February 2023 and saw a 40% drop in bot traffic along with a 15% improvement in load times [1]. These results show how small adjustments can lead to big performance gains.

Next Steps with Puppeteer

Once you’ve nailed the basics, take your Puppeteer scripts to the next level by tweaking launch settings for more realistic behavior. Here’s a sample configuration to get started:

<span class="hljs-comment">// Advanced browser launch setup</span>
<span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>({
  <span class="hljs-attr">args</span>: [
    <span class="hljs-string">'--enable-webgl'</span>,
    <span class="hljs-string">'--use-gl=desktop'</span>,
    <span class="hljs-string">'--disable-automation'</span>
  ],
  <span class="hljs-attr">ignoreDefaultArgs</span>: [<span class="hljs-string">'--enable-automation'</span>]
});

This setup enables features like WebGL while disabling automation flags, helping your scripts blend in more naturally.

Related posts

• What is a Headless Browser and Why Do You Need It?
• Installing and Configuring Puppeteer: Solving Common Dependency and Chromium Issues
• Form Automation with Puppeteer: Text Input, Form Filling, and User Simulation
• Browser Automation with Puppeteer and JavaScript: Practical Implementation in Node.js