Custom Wait Conditions with waitForFunction in Puppeteer

Want your Puppeteer scripts to handle dynamic websites better? Here's the key: waitForFunction lets you define custom wait rules to ensure your automation works seamlessly with changing content.

Why Use waitForFunction?

• Control Timing: Wait until specific conditions are met (e.g., elements load, text changes).
• Handle Dynamic Content: Perfect for modern, dynamic websites.
• Avoid Errors: Skip unnecessary delays or premature interactions.

How It Works:

• Write JavaScript conditions that return true when ready.
• Use options like polling frequency (raf, mutation, or milliseconds) and timeouts for better performance.
• Combine DOM checks with API responses for advanced scenarios.

Example:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
  <span class="hljs-function">() =></span> <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.status'</span>).<span class="hljs-property">textContent</span> === <span class="hljs-string">'Ready'</span>
);

Key Features:

• Monitor element visibility, text content, or custom properties.
• Combine multiple conditions for complex workflows.
• Handle timeouts and errors gracefully.

Tip: Use waitForFunction to replace outdated sleep calls and improve efficiency.

Quick Overview:

Feature	What It Does
Polling Modes	Checks conditions via 'raf', 'mutation', or custom intervals
Custom Conditions	Wait for specific DOM states or API responses
Timeout Management	Set operation-specific or global timeouts

This method ensures your scripts are reliable, efficient, and ready for any dynamic content challenge.

How to wait / sleep for N seconds in puppeteer?

waitForFunction Basics

The waitForFunction method in Puppeteer allows you to set up custom conditions for your scripts to wait before proceeding. Here's how you can use it effectively.

How to Write Basic Wait Functions

Here's a simple way to create a wait function:

<span class="hljs-comment">// Wait until the input element has a specific value</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
  <span class="hljs-string">'document.getElementById("loginUsername").value === "hello"'</span>
);

<span class="hljs-comment">// Using an arrow function for a condition</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
  <span class="hljs-function">() =></span> <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.status'</span>).<span class="hljs-property">textContent</span> === <span class="hljs-string">'Ready'</span>
);

The key here is that your function needs to return a truthy value before the script can move forward [1].

Now, let’s look at how to configure it for better performance.

Setting Up Function Parameters

You can fine-tune waitForFunction by passing an options object. Key parameters include:

Parameter	Description	Common Values
polling	How often the condition is checked	'raf', 'mutation', or milliseconds
timeout	Maximum time to wait	Default: 30,000ms (30 seconds)

Polling Modes [4]:

1. RequestAnimationFrame ('raf')
   Best for monitoring style changes, as it checks conditions during every animation frame.

2. Mutation Observer ('mutation')
   Useful for tracking changes in the DOM structure. It triggers checks whenever the DOM updates.

3. Custom Interval (milliseconds)
   For example, this checks the condition every second:

   <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
     <span class="hljs-function">() =></span> <span class="hljs-variable language_">document</span>.<span class="hljs-property">readyState</span> === <span class="hljs-string">'complete'</span>,
     { <span class="hljs-attr">polling</span>: <span class="hljs-number">1000</span>, <span class="hljs-attr">timeout</span>: <span class="hljs-number">5000</span> }
   );
   

Choose the polling mode based on your needs:

• Use 'raf' for animations or style updates.
• Opt for 'mutation' for DOM-related changes.
• Set a custom interval (milliseconds) for broader use cases.

Creating Custom Wait Rules

Checking Element Display Status

To confirm an element is visible, you can check its presence and dimensions:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">const</span> element = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.tv-lightweight-charts'</span>);
    <span class="hljs-keyword">return</span> element && element.<span class="hljs-property">offsetHeight</span> > <span class="hljs-number">0</span> && element.<span class="hljs-property">offsetWidth</span> > <span class="hljs-number">0</span>;
});

This ensures the element exists and has visible dimensions on the page. It's especially handy for dynamic content that takes time to load properly [2].

Testing Text and Element Properties

Beyond visual checks, you can monitor text content or specific properties of elements:

<span class="hljs-comment">// Wait for specific text content</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
    <span class="hljs-function"><span class="hljs-params">selector</span> =></span> {
        <span class="hljs-keyword">const</span> element = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(selector);
        <span class="hljs-keyword">return</span> element && element.<span class="hljs-property">textContent</span>.<span class="hljs-title function_">includes</span>(<span class="hljs-string">'Ready'</span>);
    },
    {},
    <span class="hljs-string">'.status-message'</span>
);

For more detailed property checks, pass additional arguments:

<span class="hljs-keyword">const</span> expectedValue = <span class="hljs-string">'completed'</span>;
<span class="hljs-keyword">const</span> selector = <span class="hljs-string">'.status'</span>;

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
    <span class="hljs-function">(<span class="hljs-params">sel, val</span>) =></span> {
        <span class="hljs-keyword">const</span> element = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(sel);
        <span class="hljs-keyword">return</span> element && element.<span class="hljs-title function_">getAttribute</span>(<span class="hljs-string">'data-state'</span>) === val;
    },
    {},
    selector,
    expectedValue
);

Combining Multiple Wait Rules

Once you've tested individual conditions, you can combine them for more complex scenarios. For example:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-comment">// Check multiple elements and conditions</span>
    <span class="hljs-keyword">const</span> button = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.donate-button'</span>);
    <span class="hljs-keyword">const</span> searchBox = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.search-button'</span>);

    <span class="hljs-keyword">return</span> button && 
           searchBox && 
           button.<span class="hljs-property">offsetHeight</span> > <span class="hljs-number">0</span> && 
           searchBox.<span class="hljs-property">offsetHeight</span> > <span class="hljs-number">0</span> &&
           !button.<span class="hljs-property">disabled</span>;
});

For handling multiple elements with independent conditions, use Promise combinations [5]:

Wait Pattern	Use Case	Implementation
All Elements	Wait for multiple required elements	Promise.all()
Any Element	Continue when the first element appears	Promise.race()
Custom Logic	Handle complex conditional checks	Combined wait functions

For asynchronous operations, you can create advanced wait conditions:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
    <span class="hljs-title function_">async</span> () => {
        <span class="hljs-keyword">const</span> response = <span class="hljs-keyword">await</span> <span class="hljs-title function_">fetch</span>(<span class="hljs-string">'/api/status'</span>);
        <span class="hljs-keyword">const</span> data = <span class="hljs-keyword">await</span> response.<span class="hljs-title function_">json</span>();
        <span class="hljs-keyword">return</span> data.<span class="hljs-property">isReady</span> && <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.content'</span>).<span class="hljs-property">offsetHeight</span> > <span class="hljs-number">0</span>;
    },
    { <span class="hljs-attr">polling</span>: <span class="hljs-string">'mutation'</span> }
);

This method combines API response validation with DOM element checks, ensuring that both the data and the visual content are ready [1].

sbb-itb-23997f1

Advanced waitForFunction Methods

Writing Complex JavaScript Conditions

For more dynamic web applications, you can use waitForFunction to create detailed JavaScript conditions. Here's an example:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
    <span class="hljs-function">(<span class="hljs-params">threshold</span>) =></span> {
        <span class="hljs-keyword">const</span> chart = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.tv-lightweight-charts'</span>);
        <span class="hljs-keyword">const</span> dataPoints = chart?.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'.data-point'</span>);

        <span class="hljs-keyword">return</span> chart && 
               dataPoints?.<span class="hljs-property">length</span> > threshold && 
               <span class="hljs-title class_">Array</span>.<span class="hljs-title function_">from</span>(dataPoints).<span class="hljs-title function_">every</span>(<span class="hljs-function"><span class="hljs-params">point</span> =></span> 
                   point.<span class="hljs-title function_">getBoundingClientRect</span>().<span class="hljs-property">height</span> > <span class="hljs-number">0</span> &&
                   !point.<span class="hljs-property">classList</span>.<span class="hljs-title function_">contains</span>(<span class="hljs-string">'loading'</span>)
               );
    },
    { <span class="hljs-attr">polling</span>: <span class="hljs-string">'mutation'</span> },
    <span class="hljs-number">5</span>
);

This script ensures:

• The chart container exists.
• A minimum number of data points are present.
• All data points are visible.
• None of the points are in a loading state.

You can also combine asynchronous checks with DOM evaluations for more complex scenarios:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
    <span class="hljs-title function_">async</span> () => {
        <span class="hljs-comment">// Check if the container is ready</span>
        <span class="hljs-keyword">const</span> container = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.dynamic-content'</span>);
        <span class="hljs-keyword">if</span> (!container || container.<span class="hljs-property">offsetHeight</span> === <span class="hljs-number">0</span>) <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;

        <span class="hljs-comment">// Validate API response</span>
        <span class="hljs-keyword">try</span> {
            <span class="hljs-keyword">const</span> response = <span class="hljs-keyword">await</span> <span class="hljs-title function_">fetch</span>(<span class="hljs-string">'/api/status'</span>);
            <span class="hljs-keyword">const</span> data = <span class="hljs-keyword">await</span> response.<span class="hljs-title function_">json</span>();
            <span class="hljs-keyword">return</span> data.<span class="hljs-property">isReady</span> && container.<span class="hljs-property">children</span>.<span class="hljs-property">length</span> > <span class="hljs-number">0</span>;
        } <span class="hljs-keyword">catch</span> {
            <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>;
        }
    },
    { 
        <span class="hljs-attr">polling</span>: <span class="hljs-string">'raf'</span>,
        <span class="hljs-attr">timeout</span>: <span class="hljs-number">30000</span>
    }
);

This approach combines DOM checks with an API call to ensure both the UI and backend are in sync.

Handling Timeouts and Errors

Timeout management is crucial when working with waitForFunction. Here's an example of handling timeouts effectively:

<span class="hljs-keyword">try</span> {
    <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setDefaultTimeout</span>(<span class="hljs-number">60000</span>); <span class="hljs-comment">// Set a global timeout of 60 seconds</span>

    <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
        <span class="hljs-function">() =></span> {
            <span class="hljs-keyword">const</span> element = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.dynamic-element'</span>);
            <span class="hljs-keyword">return</span> element?.<span class="hljs-property">complete</span> === <span class="hljs-literal">true</span>;
        },
        {
            <span class="hljs-attr">timeout</span>: <span class="hljs-number">45000</span>, <span class="hljs-comment">// Specific timeout for this operation</span>
            <span class="hljs-attr">polling</span>: <span class="hljs-string">'mutation'</span>
        }
    );
} <span class="hljs-keyword">catch</span> (error) {
    <span class="hljs-keyword">if</span> (error.<span class="hljs-property">name</span> === <span class="hljs-string">'TimeoutError'</span>) {
        <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">error</span>(<span class="hljs-string">'Element state check timed out:'</span>, error.<span class="hljs-property">message</span>);
        <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">reload</span>(); <span class="hljs-comment">// Reload page as a fallback</span>
    }
    <span class="hljs-keyword">throw</span> error;
}

Here's a quick overview of timeout strategies:

Timeout Strategy	Use Case	Configuration
Default Timeout	General operations	page.setDefaultTimeout()
Navigation Timeout	Page loads	page.setDefaultNavigationTimeout()
Operation-Specific	Detailed checks	Use the timeout option in method
Infinite Wait	Known delays	timeout: 0

To improve error handling:

• Adjust timeout settings based on the complexity of each operation.
• Use try-catch blocks to recover gracefully from errors.
• Monitor network activity to identify bottlenecks.
• Implement fallback actions, like reloading the page, when timeouts occur.
• Double-check selectors to avoid unnecessary delays.

These practices will help ensure your scripts are both reliable and efficient.

Making waitForFunction Run Faster

Speed and Resource Tips

To get waitForFunction running more efficiently, focus on smart waiting strategies and proper resource management. Use browser developer tools to measure load times and set precise timeouts.

<span class="hljs-comment">// Optimize waiting with a networkidle strategy</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">goto</span>(<span class="hljs-string">'https://example.com'</span>, {
    <span class="hljs-attr">waitUntil</span>: <span class="hljs-string">'networkidle2'</span>,
    <span class="hljs-attr">timeout</span>: <span class="hljs-number">30000</span>
});

<span class="hljs-comment">// Combine checks in a single evaluate call</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">const</span> element = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.dynamic-content'</span>);
    <span class="hljs-keyword">const</span> isVisible = element?.<span class="hljs-property">offsetHeight</span> > <span class="hljs-number">0</span>;
    <span class="hljs-keyword">const</span> hasData = element?.<span class="hljs-property">children</span>.<span class="hljs-property">length</span> > <span class="hljs-number">0</span>;
    <span class="hljs-keyword">return</span> isVisible && hasData;
});

To reduce resource usage:

• Block unnecessary assets like images or fonts.
• Use waitForSelector or waitForFunction instead of outdated waitForTimeout.
• Combine multiple checks in a single evaluate call to cut down on browser-to-Node communication.

Strategy	Performance Impact	Best Use Case
networkidle2	Moderate	Page navigation
waitForSelector	Fast	Single element checks
waitForFunction	Variable	Complex conditions
Combined evaluate	Fastest	Multiple element checks

These methods can help address common bottlenecks, which are covered in the next section.

Fixing Common Problems

Performance issues often stem from inefficient waiting patterns. Here's how to handle them:

Selector Issues
Overly rigid selectors can cause failures. Simplify them for better reliability:

<span class="hljs-comment">// Avoid rigid selectors like this</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForSelector</span>(<span class="hljs-string">'div.container > div:nth-child(2) > span.text-red'</span>);

<span class="hljs-comment">// Use a more flexible approach</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
    <span class="hljs-function">() =></span> <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.text-red'</span>)?.<span class="hljs-property">offsetParent</span> !== <span class="hljs-literal">null</span>
);

Resource Management
Manage resources and avoid unnecessary delays:

<span class="hljs-keyword">try</span> {
    <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(
        <span class="hljs-function">() =></span> <span class="hljs-variable language_">document</span>.<span class="hljs-property">readyState</span> === <span class="hljs-string">'complete'</span> &&
              performance.<span class="hljs-title function_">now</span>() > <span class="hljs-number">1000</span>
    );
} <span class="hljs-keyword">catch</span> (error) {
    <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">error</span>(<span class="hljs-string">'Page load timeout:'</span>, error.<span class="hljs-property">message</span>);
}

"Puppeteer has event-driven architecture, which removes a lot of potential flakiness. There's no need for evil sleep[undefined] calls in puppeteer scripts." - Puppeteer's readme [6]

Always wrap waiting methods in try...catch blocks to handle errors gracefully and provide fallback options. This approach ensures your scripts remain robust and reliable.

Common Uses for Custom Wait Rules

Online Store Product Loading

Making sure products load properly is a must for collecting accurate data. Use a custom wait rule to pause execution until product items are fully loaded:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">const</span> products = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'.product-card'</span>);
    <span class="hljs-keyword">return</span> products.<span class="hljs-property">length</span> > <span class="hljs-number">0</span> && all images and prices fully load;
});

For more precision, you can use this approach:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">const</span> productDetails = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.product-details'</span>);
    <span class="hljs-keyword">return</span> productDetails && 
           productDetails.<span class="hljs-property">offsetHeight</span> > <span class="hljs-number">0</span> && 
           productDetails.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.inventory-status'</span>) !== <span class="hljs-literal">null</span>;
}, {<span class="hljs-attr">timeout</span>: <span class="hljs-number">30000</span>});

This ensures your script waits for all necessary elements to load, improving the reliability of data collection in e-commerce scenarios.

Loading Content in Modern Web Apps

Dynamic web apps often require specific wait conditions to handle content loading. For example, you can wait for a particular element to become fully visible:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">const</span> element = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.tv-lightweight-charts'</span>);
    <span class="hljs-keyword">return</span> element && element.<span class="hljs-property">offsetHeight</span> > <span class="hljs-number">0</span> && element.<span class="hljs-property">offsetWidth</span> > <span class="hljs-number">0</span>;
});

If multiple sections need to load, combine conditions like this:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">const</span> contentLoaded = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.content'</span>).<span class="hljs-property">children</span>.<span class="hljs-property">length</span> > <span class="hljs-number">0</span>;
    <span class="hljs-keyword">const</span> dataUpdated = <span class="hljs-variable language_">document</span>.<span class="hljs-property">body</span>.<span class="hljs-property">textContent</span>.<span class="hljs-title function_">includes</span>(<span class="hljs-string">'Last updated:'</span>);
    <span class="hljs-keyword">return</span> contentLoaded && dataUpdated;
}, {<span class="hljs-attr">polling</span>: <span class="hljs-number">1000</span>});

This method helps ensure your automation scripts interact seamlessly with dynamic content.

Form Error Message Detection

Detecting form errors uses a similar logic to element visibility checks. Here's how you can monitor for error messages:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">const</span> errorContainer = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.error-messages'</span>);
    <span class="hljs-keyword">const</span> hasErrors = errorContainer?.<span class="hljs-property">children</span>.<span class="hljs-property">length</span> > <span class="hljs-number">0</span>;
    <span class="hljs-keyword">const</span> isVisible = errorContainer?.<span class="hljs-property">offsetParent</span> !== <span class="hljs-literal">null</span>;
    <span class="hljs-keyword">return</span> hasErrors && isVisible;
});

You can also track various form validation states using specific wait conditions:

Validation Type	Wait Condition	Use Case
Field Errors	Check for presence of an error class	Individual field validation
Form-wide Errors	Monitor the error container	Overall form status
Success Messages	Watch for confirmation displays	Submission completion
Loading States	Track submit button state	Processing indication

Conclusion

Key Takeaways

The waitForFunction method in Puppeteer evaluates JavaScript conditions until they return true, offering precise control over dynamic page interactions [3].

Here are some key benefits of using waitForFunction:

• Flexible Evaluation: Handles asynchronous functions to monitor complex page states [3].
• Context Integration: Allows direct passing of Node.js arguments into the browser context [3].
• Custom Logic: Enables tailored automation based on specific page conditions [2].

This approach is particularly handy in cases where standard wait methods aren't enough. For example, in advanced single-page applications, multiple elements may load simultaneously, or specific JavaScript states might need to be confirmed before moving forward.

Puppeteer and Latenode in Action

Latenode takes advantage of waitForFunction to enhance workflow automation. By integrating this method, Latenode has created a custom monitoring node that checks website statuses and captures screenshots when certain conditions aren't satisfied [7].

Here’s an example of how Latenode uses waitForFunction to ensure critical elements are fully rendered before proceeding:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">const</span> element = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.tv-lightweight-charts'</span>);
    <span class="hljs-keyword">return</span> element && element.<span class="hljs-property">offsetHeight</span> > <span class="hljs-number">0</span> && element.<span class="hljs-property">offsetWidth</span> > <span class="hljs-number">0</span>;
});

This snippet waits for an element with the class .tv-lightweight-charts to not only appear in the DOM but also be fully rendered [2].

For best results when using Latenode with Puppeteer:

• Set appropriate timeout values using page.setDefaultTimeout().
• Use try-catch blocks for robust error handling.
• Track execution times to fine-tune your wait conditions.

Related posts

• Installing and Configuring Puppeteer: Solving Common Dependency and Chromium Issues
• Mastering Page Navigation with Puppeteer: Effective Use of goto and Navigation Options
• Puppeteer Click Operations: Handling Complex Elements, Double Clicks, and Troubleshooting
• Browser Automation with Puppeteer and JavaScript: Practical Implementation in Node.js