Debugging Puppeteer Scripts: From slowMo Mode to Advanced Techniques

Table of contents

Debugging Puppeteer Scripts: From slowMo Mode to Advanced Techniques

Debugging Puppeteer scripts can feel overwhelming, but mastering a few key techniques can save you time and frustration. Here's a quick rundown of what you'll learn:

Start with Visual Debugging: Use slowMo mode and headful browser mode to watch your script in action.
Capture Errors: Add try-catch blocks and automate error screenshots for better troubleshooting.
Leverage Console Logs: Track page errors, failed requests, and custom messages for deeper insights.
Handle Selectors and Timing: Use robust selector strategies and manage timeouts to avoid common pitfalls.
Organize Your Code: Break scripts into modules for actions, selectors, and validations to simplify maintenance.

Quick Setup Example:

const browser = await puppeteer.launch({
    headless: false,
    slowMo: 100,
    devtools: true
});

Debugging Tips:

Enable Logging: Run with DEBUG="puppeteer:*" for detailed logs.
Use Screenshots: Capture page states to see what went wrong.
Manage Resources: Always clean up browser and page instances to prevent crashes.

By combining these techniques, you'll streamline your debugging process and improve your Puppeteer scripts' reliability.

Debug Environment Setup

Basic Debug Settings

Set up Puppeteer for debugging with these launch options:

const browser = await puppeteer.launch({
    headless: false,
    slowMo: 20,
    devtools: true
});

To enable detailed logging, run your script with the following command:

DEBUG="puppeteer:*" node script.js

Once configured, use debugging tools to analyze and refine your script's behavior.

Required Debug Tools

Here are some tools to help you resolve issues effectively:

Tool	Purpose	Key Feature
Chrome DevTools	Inspect scripts	Console and network tools
VS Code Debugger	Manage breakpoints	Step-by-step execution
Screenshot Utility	Visual troubleshooting	Capture page states

You can also add checkpoints in your code for better visibility:

await page.screenshot({ path: 'before_click.png' });
await page.click('#button');
await page.screenshot({ path: 'after_click.png' });

Script Organization

Good organization is just as important as the tools you use. Break your script into logical modules and include error handling for smoother debugging:

try {
    await page.waitForSelector('#target-element');
    await page.click('#target-element');
} catch (error) {
    console.error(`Navigation failed: ${error.message}`);
    await page.screenshot({ path: 'error-state.png' });
}

To reduce the chance of bot detection, integrate the stealth plugin:

const puppeteer = require('puppeteer-extra');
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());

Finally, ensure proper resource management by implementing cleanup procedures:

async function cleanup() {
    if (page) await page.close();
    if (browser) await browser.close();
}

This setup provides a solid foundation for visual and console debugging in your projects.

Puppeteer - 3 Approaches to Consider at Debugging

Puppeteer

Visual Debug Methods

Watching your scripts in action can reveal problems that traditional code analysis might miss. These methods expand your debugging options beyond console logs and error tracking.

SlowMo Mode Guide

SlowMo introduces a delay between Puppeteer actions, making it easier to follow what's happening:

const browser = await puppeteer.launch({
    headless: false,
    slowMo: 250,
    devtools: true
});

The slowMo value (in milliseconds) controls the delay between actions. Adjust it based on what you're testing:

Operation Type	Recommended SlowMo (ms)	Use Case
Simple clicks	100–250	Basic navigation steps
Form filling	250–500	Testing input validation
Dynamic content	500–1000	Checking loading states

Once you've set up SlowMo, pair it with Browser View Mode to monitor how the UI behaves during script execution.

Browser View Mode

Browser View Mode lets you see your script run in a visible browser window, which is especially helpful for debugging dynamic content and complex interactions.

const browser = await puppeteer.launch({
    headless: false,
    defaultViewport: { width: 1700, height: 800 },
    args: ['--start-maximized']
});

For instance, Acme Corp's QA team used this mode in June 2024 to troubleshoot a web scraping script. They spotted incorrect selectors and fixed them, cutting debugging time by 40%.

To complement this, capture screenshots of important visual states for further analysis.

Visual Recording

Screenshots and videos can create a clear record of your script's execution, making debugging easier:

// Screenshot of a specific element
await page.screenshot({
    path: 'element-state.png',
    clip: {
        x: 0,
        y: 0,
        width: 500,
        height: 300
    }
});

// Full-page screenshot
await page.screenshot({
    path: 'full-page.png',
    fullPage: true
});

Start by enabling Browser View Mode, use SlowMo for detailed tracking, and document key moments with screenshots. Together, these steps create a thorough visual debugging process.

Console Debug Methods

Console methods provide a straightforward way to gain text-based insights into how your scripts are behaving. These outputs work alongside visual debugging to give you precise details about script execution.

Console Message Tracking

Puppeteer makes it easy to capture browser messages with event handlers like these:

page.on('console', msg => {
    console.log('PAGE LOG:', msg.text());
});

page.on('pageerror', err => {
    console.error('PAGE ERROR:', err.message);
});

page.on('requestfailed', request => {
    console.error('REQUEST FAILED:', request.url());
});

This setup creates a logging system that tracks console messages, page errors, and failed requests. To make things clearer, you can categorize messages by type:

Message Type	Purpose	Example Output
Log	General information	Standard execution flow
Error	Major issues	Failed operations
Warning	Potential concerns	Performance slowdowns
Info	Status updates	Task completion

Console.log Best Practices

Using console.log wisely can make debugging much easier. Place logs strategically to track progress and identify issues:

// Log before attempting to find an element
console.log(`Looking for element: ${selector}`);
const element = await page.$(selector);
// Log after confirming the element exists
console.log(`Element found: ${!!element}`);

// Log form data before filling it out
console.log(`Form data: ${JSON.stringify(formData)}`);
await page.type('#email', formData.email);

Extended Logging Methods

For more complex issues, advanced logging techniques can be a game-changer:

// Enable detailed debugging for Puppeteer
process.env.DEBUG = 'puppeteer:*';
process.env.DEBUG_MAX_STRING_LENGTH = null;

// Monitor pending protocol calls
const browser = await puppeteer.launch({
    dumpio: true
});
console.log(browser.debugInfo.pendingProtocolErrors);

One team saw a 40% drop in test failures after adopting detailed protocol logging.

// Filter out specific network domain messages
// Command: DEBUG="puppeteer:*" DEBUG_COLORS=true node script.js 2>&1 | grep -v '"Network'

These methods add a text-based layer to your debugging process, helping you catch and resolve issues more effectively.

sbb-itb-23997f1

Advanced Debug Methods

Debugging complex Puppeteer scripts involves using effective error-handling strategies and advanced techniques to ensure scripts run smoothly.

Try-Catch for Error Handling

Use try-catch blocks to manage errors effectively and keep your script running:

async function navigateAndScreenshot(url, selector) {
    try {
        await page.goto(url, { waitUntil: 'networkidle0' });
        const element = await page.waitForSelector(selector, { timeout: 5000 });
        await element.screenshot({ path: 'element.png' });
    } catch (error) {
        if (error instanceof TimeoutError) {
            console.error(`Element ${selector} not found within timeout`);
            // Add recovery logic if needed
            await page.reload();
        } else {
            console.error(`Navigation failed: ${error.message}`);
            throw error; // Re-throw unexpected errors
        }
    }
}

You can enhance error handling by combining try-catch blocks with custom error classes for better categorization and response.

Custom Error Classes

Creating custom error classes helps you pinpoint and classify issues more efficiently:

class PuppeteerScriptError extends Error {
    constructor(message, details = {}) {
        super(message);
        this.name = 'PuppeteerScriptError';
        this.details = details;
        this.timestamp = new Date().toISOString();
    }
}

class SelectorError extends PuppeteerScriptError {
    constructor(selector, context) {
        super(`Failed to find selector: ${selector}`, {
            selector,
            context,
            type: 'SELECTOR_ERROR'
        });
        this.name = 'SelectorError';
    }
}

These classes allow you to track and debug asynchronous operations with more clarity.

Debugging Asynchronous Code

Asynchronous code often introduces timing issues and unresolved promises. Tackle these problems with the following techniques:

// Enable detailed debugging for protocol calls
const browser = await puppeteer.launch({
    dumpio: true
});

// Monitor unresolved promises periodically
setInterval(() => {
    const pending = browser.debugInfo.pendingProtocolErrors;
    if (pending.length > 0) {
        console.log('Pending protocol calls:', pending);
    }
}, 5000);

// Handle async errors with a timeout mechanism
async function safeExecute(promiseFn) {
    try {
        return await Promise.race([
            promiseFn(),
            new Promise((_, reject) => 
                setTimeout(() => reject(new Error('Operation timed out')), 30000)
            )
        ]);
    } catch (error) {
        console.error(`Operation failed: ${error.message}`);
        throw new PuppeteerScriptError('Execution timeout', {
            originalError: error,
            operation: promiseFn.name
        });
    }
}

By using the debugInfo interface, you can monitor pending callbacks and identify unresolved promises during browser protocol communication.

Debug Level	Purpose	Implementation
Basic	Handle common errors	Standard try-catch blocks
Intermediate	Classify errors	Custom error class hierarchy
Advanced	Track protocol issues	Debug interface monitoring

Common Issue Solutions

This section dives into frequent challenges with Puppeteer and provides clear fixes to keep your automation scripts running smoothly.

Selector Issues

Selector problems can often disrupt script execution. Here's how to handle them effectively:

async function findElement(page) {
  try {
    const element = await page.waitForSelector('[data-testid="target"]', {
      timeout: 5000
    });
    return element;
  } catch {
    return page.waitForSelector('.target-class', {
      timeout: 5000
    });
  }
}

For elements inside iframes or Shadow DOM, use these approaches:

// Access iframe content
const frame = await page.frames().find(f => f.name() === 'content-frame');
const button = await frame.$('button[data-hook="create"]');

// Handle Shadow DOM elements
await page.evaluateHandle(selector => {
  const element = document.querySelector('parent-element')
    .shadowRoot
    .querySelector(selector);
  return element;
}, 'target-selector');

Properly handling selectors ensures your scripts locate elements reliably.

Timing Problems

Once selectors are stable, managing timing is crucial for smooth execution:

await page.setDefaultNavigationTimeout(30000);
await page.setDefaultTimeout(10000);

async function waitForContent(page) {
  await Promise.all([
    page.waitForNavigation({ waitUntil: 'networkidle0' }),
    page.click('#load-more-button')
  ]);
}

Here’s a quick reference for timing controls:

Timing Issue	Solution	Implementation
Page Load	waitForNavigation	Wait for network idle
Dynamic Content	waitForSelector	Use with appropriate timeout
AJAX Updates	waitForResponse	Monitor specific network requests

These strategies help align your script's timing with page behavior.

Browser Crash Fixes

Even with solid selector and timing strategies, browser crashes can still occur. Here’s how to minimize and recover from them:

const browser = await puppeteer.launch({
  args: [
    '--disable-dev-shm-usage',
    '--enable-gpu',
    '--no-first-run',
    '--disable-extensions'
  ]
});

For crash recovery:

let browser;
try {
  browser = await puppeteer.launch();
  const page = await browser.newPage();

  page.on('error', err => {
    console.error('Page crashed:', err);
  });

  await page.goto('https://example.com');
} catch (error) {
  console.error('Browser error:', error);
} finally {
  if (browser) {
    await browser.close();
  }
}

If you're working on Linux, check for missing dependencies:

ldd chrome | grep not

To optimize resource usage, adjust browser flags:

const browser = await puppeteer.launch({
  args: [
    '--disable-dev-shm-usage',
    '--disable-accelerated-2d-canvas',
    '--disable-gpu'
  ]
});

Set up automatic recovery for added resilience:

async function checkAndRecoverPage(page) {
  if (!page.isClosed()) {
    try {
      await page.reload();
    } catch {
      page = await browser.newPage();
    }
  }
  return page;
}

Script Debug Optimization

Enhance your scripts for easier maintenance and quicker error resolution by building on proven debugging techniques.

Code Clarity

Keep your code readable by grouping configurations and using clear, descriptive names:

// Group related configurations
const browserConfig = {
  headless: false,
  defaultViewport: { width: 1920, height: 1080 },
  args: ['--no-sandbox', '--disable-setuid-sandbox']
};

// Use descriptive function names
async function validatePageContent(page) {
  const pageTitle = await page.title();
  console.log(`Validating content for page: ${pageTitle}`);

  const contentExists = await page.evaluate(() => {
    const mainContent = document.querySelector('.main-content');
    return {
      hasHeader: !!document.querySelector('header'),
      hasContent: !!mainContent,
      contentLength: mainContent?.textContent.length || 0
    };
  });

  return contentExists;
}

Module Organization

Divide your scripts into separate modules to simplify debugging. This approach isolates selectors, actions, and validations, making it easier to locate and fix errors.

// selectors.js
export const SELECTORS = {
  loginForm: '#login-form',
  submitButton: '[data-testid="submit-btn"]',
  errorMessage: '.error-notification'
};

// actions.js
export async function performLogin(page, credentials) {
  await page.type(SELECTORS.loginForm + ' input[name="username"]', credentials.username);
  await page.type(SELECTORS.loginForm + ' input[name="password"]', credentials.password);
  await Promise.all([
    page.waitForNavigation(),
    page.click(SELECTORS.submitButton)
  ]);
}

// validators.js
export async function checkLoginStatus(page) {
  const errorElement = await page.$(SELECTORS.errorMessage);
  if (errorElement) {
    throw new Error('Login failed: ' + await page.evaluate(el => el.textContent, errorElement));
  }
}

This modular structure not only organizes your code but also helps streamline error tracking.

Error Tracking Setup

Set up error tracking to identify issues quickly and provide detailed context for debugging:

class PuppeteerError extends Error {
  constructor(message, action, selector) {
    super(message);
    this.name = 'PuppeteerError';
    this.action = action;
    this.selector = selector;
    this.timestamp = new Date().toISOString();
  }
}

async function executeWithTracking(page, action, description) {
  try {
    await action();
  } catch (error) {
    const screenshot = await page.screenshot({
      path: `error-${Date.now()}.png`,
      fullPage: true
    });

    throw new PuppeteerError(
      `Failed to ${description}`,
      error.message,
      error.selector
    );
  }
}

You can also automate logging of console errors and warnings:

page.on('console', message => {
  const type = message.type();
  const text = message.text();

  if (type === 'error' || type === 'warning') {
    console.log(`[${type.toUpperCase()}] ${text}`);

    // Log to external service or file
    logger.log({
      level: type,
      message: text,
      timestamp: new Date().toISOString(),
      url: page.url()
    });
  }
});

Validation for Critical Operations

Add validation checks to ensure critical operations complete successfully:

async function validateOperation(page, action) {
  const beforeState = await page.evaluate(() => ({
    url: window.location.href,
    elements: document.querySelectorAll('*').length
  }));

  await action();

  const afterState = await page.evaluate(() => ({
    url: window.location.href,
    elements: document.querySelectorAll('*').length
  }));

  return {
    urlChanged: beforeState.url !== afterState.url,
    elementsDelta: afterState.elements - beforeState.elements
  };
}

These techniques, combined with earlier debugging methods, help you quickly identify and resolve issues while keeping your scripts maintainable.

Conclusion

Key Debugging Techniques

Using visual debugging in headful mode with slowMo allows for immediate feedback on scripts and precise timing adjustments. For more detailed scenarios, the DevTools protocol provides step-by-step debugging and access to process logs for deeper insights.

const browser = await puppeteer.launch({
    headless: false,
    slowMo: 100,
    devtools: true,
    dumpio: true
});

To improve your workflow, consider incorporating continuous monitoring and resource management practices alongside these debugging methods.

Next Steps

Now that you have a solid foundation in debugging techniques, here’s how you can optimize and maintain your Puppeteer scripts:

Performance Monitoring: Use detailed logging to track execution times and resource usage. This helps pinpoint bottlenecks and makes debugging more efficient.
Error Prevention: Add the puppeteer-extra-plugin-stealth plugin to minimize automation detection and reduce script failures.
Resource Management: Focus on efficient memory usage and implement cleanup routines to keep your scripts running smoothly.

Here’s an example of a cleanup function to manage resources effectively:

async function cleanupResources(page) {
    await page.evaluate(() => {
        if (window.performance.memory) {
            console.log(`Heap size limit: ${(window.performance.memory.jsHeapSizeLimit / 1024 / 1024).toFixed(2)} MB`);
        }
    });
    await page.close();
}

Stay ahead by regularly checking the Puppeteer GitHub repository for updates, new features, and best practices. Keeping your toolkit current ensures your scripts remain efficient and adaptable as web technologies change.