Custom Wait Conditions with waitForFunction in Puppeteer

Want your Puppeteer scripts to handle dynamic websites better? Here's the key: waitForFunction lets you define custom wait rules to ensure your automation works seamlessly with changing content.

Why Use waitForFunction?

• Control Timing: Wait until specific conditions are met (e.g., elements load, text changes).
• Handle Dynamic Content: Perfect for modern, dynamic websites.
• Avoid Errors: Skip unnecessary delays or premature interactions.

How It Works:

• Write JavaScript conditions that return true when ready.
• Use options like polling frequency (raf, mutation, or milliseconds) and timeouts for better performance.
• Combine DOM checks with API responses for advanced scenarios.

Example:

await page.waitForFunction(
  () => document.querySelector('.status').textContent === 'Ready'
);

Key Features:

• Monitor element visibility, text content, or custom properties.
• Combine multiple conditions for complex workflows.
• Handle timeouts and errors gracefully.

Tip: Use waitForFunction to replace outdated sleep calls and improve efficiency.

Quick Overview:

Feature	What It Does
Polling Modes	Checks conditions via 'raf', 'mutation', or custom intervals
Custom Conditions	Wait for specific DOM states or API responses
Timeout Management	Set operation-specific or global timeouts

This method ensures your scripts are reliable, efficient, and ready for any dynamic content challenge.

How to wait / sleep for N seconds in puppeteer?

waitForFunction Basics

The waitForFunction method in Puppeteer allows you to set up custom conditions for your scripts to wait before proceeding. Here's how you can use it effectively.

How to Write Basic Wait Functions

Here's a simple way to create a wait function:

// Wait until the input element has a specific value
await page.waitForFunction(
  'document.getElementById("loginUsername").value === "hello"'
);

// Using an arrow function for a condition
await page.waitForFunction(
  () => document.querySelector('.status').textContent === 'Ready'
);

The key here is that your function needs to return a truthy value before the script can move forward.

Now, let’s look at how to configure it for better performance.

Setting Up Function Parameters

You can fine-tune waitForFunction by passing an options object. Key parameters include:

Parameter	Description	Common Values
polling	How often the condition is checked	'raf', 'mutation', or milliseconds
timeout	Maximum time to wait	Default: 30,000ms (30 seconds)

Polling Modes:

1. RequestAnimationFrame ('raf')
   Best for monitoring style changes, as it checks conditions during every animation frame.
2. Mutation Observer ('mutation')
   Useful for tracking changes in the DOM structure. It triggers checks whenever the DOM updates.
3. Custom Interval (milliseconds)
   For example, this checks the condition every second:

   await page.waitForFunction(
     () => document.readyState === 'complete',
     { polling: 1000, timeout: 5000 }
   );
   

Choose the polling mode based on your needs:

• Use 'raf' for animations or style updates.
• Opt for 'mutation' for DOM-related changes.
• Set a custom interval (milliseconds) for broader use cases.

Creating Custom Wait Rules

Checking Element Display Status

To confirm an element is visible, you can check its presence and dimensions:

await page.waitForFunction(() => {
    const element = document.querySelector('.tv-lightweight-charts');
    return element && element.offsetHeight > 0 && element.offsetWidth > 0;
});

This ensures the element exists and has visible dimensions on the page. It's especially handy for dynamic content that takes time to load properly.

Testing Text and Element Properties

Beyond visual checks, you can monitor text content or specific properties of elements:

// Wait for specific text content
await page.waitForFunction(
    selector => {
        const element = document.querySelector(selector);
        return element && element.textContent.includes('Ready');
    },
    {},
    '.status-message'
);

For more detailed property checks, pass additional arguments:

const expectedValue = 'completed';
const selector = '.status';

await page.waitForFunction(
    (sel, val) => {
        const element = document.querySelector(sel);
        return element && element.getAttribute('data-state') === val;
    },
    {},
    selector,
    expectedValue
);

Combining Multiple Wait Rules

Once you've tested individual conditions, you can combine them for more complex scenarios. For example:

await page.waitForFunction(() => {
    // Check multiple elements and conditions
    const button = document.querySelector('.donate-button');
    const searchBox = document.querySelector('.search-button');

    return button && 
           searchBox && 
           button.offsetHeight > 0 && 
           searchBox.offsetHeight > 0 &&
           !button.disabled;
});

For handling multiple elements with independent conditions, use Promise combinations:

Wait Pattern	Use Case	Implementation
All Elements	Wait for multiple required elements	Promise.all()
Any Element	Continue when the first element appears	Promise.race()
Custom Logic	Handle complex conditional checks	Combined wait functions

For asynchronous operations, you can create advanced wait conditions:

await page.waitForFunction(
    async () => {
        const response = await fetch('/api/status');
        const data = await response.json();
        return data.isReady && document.querySelector('.content').offsetHeight > 0;
    },
    { polling: 'mutation' }
);

This method combines API response validation with DOM element checks, ensuring that both the data and the visual content are ready.

sbb-itb-23997f1

Advanced waitForFunction Methods

Writing Complex JavaScript Conditions

For more dynamic web applications, you can use waitForFunction to create detailed JavaScript conditions. Here's an example:

await page.waitForFunction(
    (threshold) => {
        const chart = document.querySelector('.tv-lightweight-charts');
        const dataPoints = chart?.querySelectorAll('.data-point');

        return chart && 
               dataPoints?.length > threshold && 
               Array.from(dataPoints).every(point => 
                   point.getBoundingClientRect().height > 0 &&
                   !point.classList.contains('loading')
               );
    },
    { polling: 'mutation' },
    5
);

This script ensures:

• The chart container exists.
• A minimum number of data points are present.
• All data points are visible.
• None of the points are in a loading state.

You can also combine asynchronous checks with DOM evaluations for more complex scenarios:

await page.waitForFunction(
    async () => {
        // Check if the container is ready
        const container = document.querySelector('.dynamic-content');
        if (!container || container.offsetHeight === 0) return false;

        // Validate API response
        try {
            const response = await fetch('/api/status');
            const data = await response.json();
            return data.isReady && container.children.length > 0;
        } catch {
            return false;
        }
    },
    { 
        polling: 'raf',
        timeout: 30000
    }
);

This approach combines DOM checks with an API call to ensure both the UI and backend are in sync.

Handling Timeouts and Errors

Timeout management is crucial when working with waitForFunction. Here's an example of handling timeouts effectively:

try {
    await page.setDefaultTimeout(60000); // Set a global timeout of 60 seconds

    await page.waitForFunction(
        () => {
            const element = document.querySelector('.dynamic-element');
            return element?.complete === true;
        },
        {
            timeout: 45000, // Specific timeout for this operation
            polling: 'mutation'
        }
    );
} catch (error) {
    if (error.name === 'TimeoutError') {
        console.error('Element state check timed out:', error.message);
        await page.reload(); // Reload page as a fallback
    }
    throw error;
}

Here's a quick overview of timeout strategies:

Timeout Strategy	Use Case	Configuration
Default Timeout	General operations	page.setDefaultTimeout()
Navigation Timeout	Page loads	page.setDefaultNavigationTimeout()
Operation-Specific	Detailed checks	Use the timeout option in method
Infinite Wait	Known delays	timeout: 0

To improve error handling:

• Adjust timeout settings based on the complexity of each operation.
• Use try-catch blocks to recover gracefully from errors.
• Monitor network activity to identify bottlenecks.
• Implement fallback actions, like reloading the page, when timeouts occur.
• Double-check selectors to avoid unnecessary delays.

These practices will help ensure your scripts are both reliable and efficient.

Making waitForFunction Run Faster

Speed and Resource Tips

To get waitForFunction running more efficiently, focus on smart waiting strategies and proper resource management. Use browser developer tools to measure load times and set precise timeouts.

// Optimize waiting with a networkidle strategy
await page.goto('https://example.com', {
    waitUntil: 'networkidle2',
    timeout: 30000
});

// Combine checks in a single evaluate call
await page.evaluate(() => {
    const element = document.querySelector('.dynamic-content');
    const isVisible = element?.offsetHeight > 0;
    const hasData = element?.children.length > 0;
    return isVisible && hasData;
});

To reduce resource usage:

• Block unnecessary assets like images or fonts.
• Use waitForSelector or waitForFunction instead of outdated waitForTimeout.
• Combine multiple checks in a single evaluate call to cut down on browser-to-Node communication.

Strategy	Performance Impact	Best Use Case
networkidle2	Moderate	Page navigation
waitForSelector	Fast	Single element checks
waitForFunction	Variable	Complex conditions
Combined evaluate	Fastest	Multiple element checks

These methods can help address common bottlenecks, which are covered in the next section.

Fixing Common Problems

Performance issues often stem from inefficient waiting patterns. Here's how to handle them:

Selector Issues
Overly rigid selectors can cause failures. Simplify them for better reliability:

// Avoid rigid selectors like this
await page.waitForSelector('div.container > div:nth-child(2) > span.text-red');

// Use a more flexible approach
await page.waitForFunction(
    () => document.querySelector('.text-red')?.offsetParent !== null
);

Resource Management
Manage resources and avoid unnecessary delays:

try {
    await page.waitForFunction(
        () => document.readyState === 'complete' &&
              performance.now() > 1000
    );
} catch (error) {
    console.error('Page load timeout:', error.message);
}

"Puppeteer has event-driven architecture, which removes a lot of potential flakiness. There's no need for evil sleep[undefined] calls in puppeteer scripts." - Puppeteer's readme

Always wrap waiting methods in try...catch blocks to handle errors gracefully and provide fallback options. This approach ensures your scripts remain robust and reliable.

Common Uses for Custom Wait Rules

Online Store Product Loading

Making sure products load properly is a must for collecting accurate data. Use a custom wait rule to pause execution until product items are fully loaded:

await page.waitForFunction(() => {
    const products = document.querySelectorAll('.product-card');
    return products.length > 0 && all images and prices fully load;
});

For more precision, you can use this approach:

await page.waitForFunction(() => {
    const productDetails = document.querySelector('.product-details');
    return productDetails && 
           productDetails.offsetHeight > 0 && 
           productDetails.querySelector('.inventory-status') !== null;
}, {timeout: 30000});

This ensures your script waits for all necessary elements to load, improving the reliability of data collection in e-commerce scenarios.

Loading Content in Modern Web Apps

Dynamic web apps often require specific wait conditions to handle content loading. For example, you can wait for a particular element to become fully visible:

await page.waitForFunction(() => {
    const element = document.querySelector('.tv-lightweight-charts');
    return element && element.offsetHeight > 0 && element.offsetWidth > 0;
});

If multiple sections need to load, combine conditions like this:

await page.waitForFunction(() => {
    const contentLoaded = document.querySelector('.content').children.length > 0;
    const dataUpdated = document.body.textContent.includes('Last updated:');
    return contentLoaded && dataUpdated;
}, {polling: 1000});

This method helps ensure your automation scripts interact seamlessly with dynamic content.

Form Error Message Detection

Detecting form errors uses a similar logic to element visibility checks. Here's how you can monitor for error messages:

await page.waitForFunction(() => {
    const errorContainer = document.querySelector('.error-messages');
    const hasErrors = errorContainer?.children.length > 0;
    const isVisible = errorContainer?.offsetParent !== null;
    return hasErrors && isVisible;
});

You can also track various form validation states using specific wait conditions:

Validation Type	Wait Condition	Use Case
Field Errors	Check for presence of an error class	Individual field validation
Form-wide Errors	Monitor the error container	Overall form status
Success Messages	Watch for confirmation displays	Submission completion
Loading States	Track submit button state	Processing indication

Conclusion

Key Takeaways

The waitForFunction method in Puppeteer evaluates JavaScript conditions until they return true, offering precise control over dynamic page interactions.

Here are some key benefits of using waitForFunction:

• Flexible Evaluation: Handles asynchronous functions to monitor complex page states.
• Context Integration: Allows direct passing of Node.js arguments into the browser context.
• Custom Logic: Enables tailored automation based on specific page conditions.

This approach is particularly handy in cases where standard wait methods aren't enough. For example, in advanced single-page applications, multiple elements may load simultaneously, or specific JavaScript states might need to be confirmed before moving forward.

Puppeteer and Latenode in Action

Latenode takes advantage of waitForFunction to enhance workflow automation. By integrating this method, Latenode has created a custom monitoring node that checks website statuses and captures screenshots when certain conditions aren't satisfied.

Here’s an example of how Latenode uses waitForFunction to ensure critical elements are fully rendered before proceeding:

await page.waitForFunction(() => {
    const element = document.querySelector('.tv-lightweight-charts');
    return element && element.offsetHeight > 0 && element.offsetWidth > 0;
});

This snippet waits for an element with the class .tv-lightweight-charts to not only appear in the DOM but also be fully rendered.

For best results when using Latenode with Puppeteer:

• Set appropriate timeout values using page.setDefaultTimeout().
• Use try-catch blocks for robust error handling.
• Track execution times to fine-tune your wait conditions.

Related Blog Posts

• Installing and Configuring Puppeteer: Solving Common Dependency and Chromium Issues
• Mastering Page Navigation with Puppeteer: Effective Use of goto and Navigation Options
• Puppeteer Click Operations: Handling Complex Elements, Double Clicks, and Troubleshooting
• Browser Automation with Puppeteer and JavaScript: Practical Implementation in Node.js