Executing JavaScript in Page Context with page.evaluate in Puppeteer

page.evaluate() is a key Puppeteer method that lets you run JavaScript directly in the browser context. It bridges Node.js and the browser, enabling tasks like DOM manipulation, data extraction, and automation of dynamic web pages. Here's what you need to know:

• What It Does: Executes JavaScript in the browser, as if you were using the browser's console.
• How It Works: Converts a function to a string, sends it to the browser, executes it, and returns the result.
• Key Uses:
  • Extracting data from websites (e.g., text, tables, JSON).
  • Automating form submissions and user interactions.
  • Handling dynamic content like infinite scrolling or AJAX updates.
• Limitations: Functions must be JSON-serializable, and Node.js variables are not directly accessible in the browser context.

Quick Example:

<span class="hljs-keyword">const</span> title = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> <span class="hljs-variable language_">document</span>.<span class="hljs-property">title</span>);

This retrieves the page title directly from the browser.

Comparison: Node.js vs. Browser Context

Feature	Node.js Context	Browser Context
Global Objects	process, require	window, document
Script Location	Local machine	Target webpage
API Access	Node.js APIs	Browser Web APIs

Use page.evaluate() for precise, efficient automation tasks, especially when working with JavaScript-heavy websites.

NodeJS : Nodejs/Puppeteer - How to use page.evaluate

Page Context Explained

When working with Puppeteer for web automation, it's crucial to grasp the distinction between the Node.js context and the browser context. These two environments are isolated, each with its own rules for running code and exchanging data.

Comparing Node.js and Browser Contexts

Puppeteer operates across two environments: the Node.js context, where your main script runs, and the browser context, where interactions with the webpage occur. These are separate processes, each with its own virtual machine [3].

Here's a quick comparison of their key characteristics:

Feature	Node.js Context	Browser Context
Global Objects	process, require, __dirname	window, document, localStorage
Script Location	Local machine	Target webpage
Variable Scope	Puppeteer script scope	Page context scope
API Access	Node.js APIs	Browser Web APIs
Memory Space	Separate process	Browser process

How Context Communication Works

Data exchange between these contexts involves a series of steps, relying heavily on serialization:

1. The function is converted to a string using Function.prototype.toString() [1].
2. This string is sent to the browser via the Chrome DevTools Protocol [1].
3. The browser evaluates the function within its environment.
4. Results are serialized into JSON and sent back to the Node.js context [1].

Key limitations: Functions in the browser context cannot directly access variables from the Node.js scope. Puppeteer offers specific tools to address these challenges:

• page.evaluateHandle(): Returns references to objects in the browser context [1].
• page.exposeFunction(): Allows the browser to call Node.js functions [1].
• evaluateOnNewDocument(): Executes code before any page scripts load [1].

However, JSON serialization may strip certain properties, especially with complex objects like DOM nodes [2]. To avoid issues, pass data as function arguments instead of relying on Node.js variables [3].

Mastering these communication techniques ensures you can use page.evaluate effectively for automation tasks. Next, we'll dive into practical examples to see these concepts in action.

Getting Started with page.evaluate

Method Structure and Parameters

Syntax:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(pageFunction, ...args)

Parameter	Type	Description
pageFunction	Function or string	JavaScript code to execute in the browser context
args	Optional parameters	Values passed from Node.js to the browser context
Return value	Promise	Resolves with the function's return value

The pageFunction can be a function or a string containing JavaScript code. Using a function is generally better for debugging and TypeScript compatibility. Below are some examples to demonstrate how it works.

Basic Code Examples

Examples:

• Extract text from the first <h1> directly from the DOM:

<span class="hljs-keyword">const</span> headingText = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">return</span> <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'h1'</span>).<span class="hljs-property">textContent</span>;
});

• Automate form submission by passing parameters:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">(<span class="hljs-params">username, password</span>) =></span> {
    <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">getElementById</span>(<span class="hljs-string">'username'</span>).<span class="hljs-property">value</span> = username;
    <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">getElementById</span>(<span class="hljs-string">'password'</span>).<span class="hljs-property">value</span> = password;
    <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#login-form'</span>).<span class="hljs-title function_">submit</span>();
}, <span class="hljs-string">'myUsername'</span>, <span class="hljs-string">'myPassword'</span>);

• Manipulate the DOM by adding a new element:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">const</span> div = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">createElement</span>(<span class="hljs-string">'div'</span>);
    div.<span class="hljs-property">textContent</span> = <span class="hljs-string">'Added by Puppeteer'</span>;
    <span class="hljs-variable language_">document</span>.<span class="hljs-property">body</span>.<span class="hljs-title function_">appendChild</span>(div);
    <span class="hljs-keyword">return</span> div.<span class="hljs-property">textContent</span>;
});

Key Notes for Development

• Functions run in isolation from your Node.js code.
• Arguments passed to the function must be JSON-serializable.
• Returned values are automatically wrapped in a Promise.
• Handling complex objects like DOM nodes requires extra care.

Debugging Tip: Use the following configuration to enable debugging during development:

<span class="hljs-keyword">const</span> browser = <span class="hljs-keyword">await</span> puppeteer.<span class="hljs-title function_">launch</span>({
    <span class="hljs-attr">headless</span>: <span class="hljs-literal">false</span>,
    <span class="hljs-attr">slowMo</span>: <span class="hljs-number">100</span> <span class="hljs-comment">// Adds a 100ms delay to each operation</span>
});

Next, we'll dive into techniques for exchanging data between Node.js and browser contexts.

Data Exchange Between Contexts

Input Parameters

When transferring data with page.evaluate, stick to JSON-serializable values for input arguments.

Here's a quick breakdown of supported parameter types:

Parameter Type	Supported?	Example
Primitives	✓ Fully	'text', 42, true
Arrays/Objects	✓ JSON-compatible	{ key: 'value' }, [1, 2, 3]
Functions	✗ Not directly	Use page.exposeFunction
DOM Elements	✓ Through JSHandle	Use page.evaluateHandle

Now, let's see how these values are returned from the browser context.

Output Handling

When using page.evaluate, the returned values are automatically serialized to JSON. Here's how it works:

<span class="hljs-comment">// Returning a simple value</span>
<span class="hljs-keyword">const</span> pageTitle = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> <span class="hljs-variable language_">document</span>.<span class="hljs-property">title</span>);

<span class="hljs-comment">// Returning a complex object</span>
<span class="hljs-keyword">const</span> metrics = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> ({
    <span class="hljs-attr">viewport</span>: <span class="hljs-variable language_">window</span>.<span class="hljs-property">innerWidth</span>,
    <span class="hljs-attr">scrollHeight</span>: <span class="hljs-variable language_">document</span>.<span class="hljs-property">body</span>.<span class="hljs-property">scrollHeight</span>,
    <span class="hljs-attr">timestamp</span>: <span class="hljs-title class_">Date</span>.<span class="hljs-title function_">now</span>()
}));

"As a rule of thumb, if the return value of the given function is more complicated than a JSON object (e.g., most classes), then evaluate will likely return some truncated value (or {}). This is because we are not returning the actual return value, but a deserialized version as a result of transferring the return value through a protocol to Puppeteer." [1]

Once you've retrieved the output, you may encounter serialization-related challenges. Here's how to tackle them.

Handling Serialization Issues

Some common scenarios require specific workarounds:

• Working with DOM Elements

<span class="hljs-keyword">const</span> bodyHandle = <span class="hljs-keyword">await</span> page.$(<span class="hljs-string">'body'</span>);
<span class="hljs-keyword">const</span> html = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function"><span class="hljs-params">body</span> =></span> body.<span class="hljs-property">innerHTML</span>, bodyHandle);
<span class="hljs-keyword">await</span> bodyHandle.<span class="hljs-title function_">dispose</span>(); <span class="hljs-comment">// Always clean up to avoid memory leaks</span>

• Using Node.js Functions

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">exposeFunction</span>(<span class="hljs-string">'md5'</span>, <span class="hljs-function"><span class="hljs-params">text</span> =></span>
    crypto.<span class="hljs-title function_">createHash</span>(<span class="hljs-string">'md5'</span>).<span class="hljs-title function_">update</span>(text).<span class="hljs-title function_">digest</span>(<span class="hljs-string">'hex'</span>)
);

<span class="hljs-keyword">const</span> hash = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-title function_">async</span> () => {
    <span class="hljs-keyword">return</span> <span class="hljs-keyword">await</span> <span class="hljs-variable language_">window</span>.<span class="hljs-title function_">md5</span>(<span class="hljs-string">'test-string'</span>);
});

• Adjusting Transpiler Settings

If you're working with TypeScript, ensure your transpiler is set up correctly:

<span class="hljs-comment">// tsconfig.json</span>
{
    <span class="hljs-string">"compilerOptions"</span>: {
        <span class="hljs-string">"target"</span>: <span class="hljs-string">"es2018"</span>
    }
}

These strategies will help you handle data exchange effectively in various contexts.

sbb-itb-23997f1

Practical Examples

Here’s how you can use page.evaluate in real-world scenarios, complete with practical code snippets.

Extracting Data

Example: Scraping product details

This script collects details like title, price, rating, and stock status from product cards on a webpage:

<span class="hljs-keyword">const</span> productData = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">const</span> products = <span class="hljs-title class_">Array</span>.<span class="hljs-title function_">from</span>(<span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'.product-card'</span>));
  <span class="hljs-keyword">return</span> products.<span class="hljs-title function_">map</span>(<span class="hljs-function"><span class="hljs-params">product</span> =></span> ({
    <span class="hljs-attr">title</span>: product.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.title'</span>).<span class="hljs-property">textContent</span>.<span class="hljs-title function_">trim</span>(),
    <span class="hljs-attr">price</span>: product.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.price'</span>).<span class="hljs-property">textContent</span>.<span class="hljs-title function_">trim</span>(),
    <span class="hljs-attr">rating</span>: <span class="hljs-built_in">parseFloat</span>(product.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.rating'</span>).<span class="hljs-property">dataset</span>.<span class="hljs-property">value</span>),
    <span class="hljs-attr">inStock</span>: product.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.stock'</span>).<span class="hljs-property">textContent</span>.<span class="hljs-title function_">includes</span>(<span class="hljs-string">'Available'</span>)
  }));
});

Example: Extracting table data

This approach retrieves data from a table by iterating through its rows and columns:

<span class="hljs-keyword">const</span> tableData = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">const</span> rows = <span class="hljs-title class_">Array</span>.<span class="hljs-title function_">from</span>(<span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'table tr'</span>));
  <span class="hljs-keyword">return</span> rows.<span class="hljs-title function_">map</span>(<span class="hljs-function"><span class="hljs-params">row</span> =></span> {
    <span class="hljs-keyword">const</span> columns = row.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'td'</span>);
    <span class="hljs-keyword">return</span> <span class="hljs-title class_">Array</span>.<span class="hljs-title function_">from</span>(columns, <span class="hljs-function"><span class="hljs-params">column</span> =></span> column.<span class="hljs-property">innerText</span>);
  });
});

Automating Forms

Basic form automation

Here’s how to fill out form fields, trigger events, and submit the form:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-comment">// Fill form fields</span>
  <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#username'</span>).<span class="hljs-property">value</span> = <span class="hljs-string">'testuser'</span>;
  <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#password'</span>).<span class="hljs-property">value</span> = <span class="hljs-string">'secretpass'</span>;

  <span class="hljs-comment">// Trigger input events for dynamic forms</span>
  <span class="hljs-keyword">const</span> event = <span class="hljs-keyword">new</span> <span class="hljs-title class_">Event</span>(<span class="hljs-string">'input'</span>, { <span class="hljs-attr">bubbles</span>: <span class="hljs-literal">true</span> });
  <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#username'</span>).<span class="hljs-title function_">dispatchEvent</span>(event);

  <span class="hljs-comment">// Submit form</span>
  <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'form'</span>).<span class="hljs-title function_">submit</span>();
});

Handling complex forms

For tasks like selecting dropdown options or checking radio buttons:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-comment">// Select dropdown option</span>
  <span class="hljs-keyword">const</span> select = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#country'</span>);
  select.<span class="hljs-property">value</span> = <span class="hljs-string">'US'</span>;
  select.<span class="hljs-title function_">dispatchEvent</span>(<span class="hljs-keyword">new</span> <span class="hljs-title class_">Event</span>(<span class="hljs-string">'change'</span>, { <span class="hljs-attr">bubbles</span>: <span class="hljs-literal">true</span> }));

  <span class="hljs-comment">// Check radio button</span>
  <span class="hljs-keyword">const</span> radio = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'input[value="express"]'</span>);
  radio.<span class="hljs-property">checked</span> = <span class="hljs-literal">true</span>;
  radio.<span class="hljs-title function_">dispatchEvent</span>(<span class="hljs-keyword">new</span> <span class="hljs-title class_">Event</span>(<span class="hljs-string">'change'</span>, { <span class="hljs-attr">bubbles</span>: <span class="hljs-literal">true</span> }));
});

Managing Dynamic Elements

Example: Infinite scrolling

This script scrolls through a page until it collects at least 100 items:

<span class="hljs-keyword">const</span> items = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-keyword">const</span> <span class="hljs-title function_">delay</span> = ms => <span class="hljs-keyword">new</span> <span class="hljs-title class_">Promise</span>(<span class="hljs-function"><span class="hljs-params">resolve</span> =></span> <span class="hljs-built_in">setTimeout</span>(resolve, ms));
  <span class="hljs-keyword">const</span> items = <span class="hljs-keyword">new</span> <span class="hljs-title class_">Set</span>();

  <span class="hljs-keyword">while</span> (items.<span class="hljs-property">size</span> < <span class="hljs-number">100</span>) {
    <span class="hljs-comment">// Scroll to bottom</span>
    <span class="hljs-variable language_">window</span>.<span class="hljs-title function_">scrollTo</span>(<span class="hljs-number">0</span>, <span class="hljs-variable language_">document</span>.<span class="hljs-property">body</span>.<span class="hljs-property">scrollHeight</span>);

    <span class="hljs-comment">// Wait for new content</span>
    <span class="hljs-keyword">await</span> <span class="hljs-title function_">delay</span>(<span class="hljs-number">1000</span>);

    <span class="hljs-comment">// Collect items</span>
    <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'.item'</span>).<span class="hljs-title function_">forEach</span>(<span class="hljs-function"><span class="hljs-params">item</span> =></span> 
      items.<span class="hljs-title function_">add</span>(item.<span class="hljs-property">textContent</span>.<span class="hljs-title function_">trim</span>())
    );
  }

  <span class="hljs-keyword">return</span> <span class="hljs-title class_">Array</span>.<span class="hljs-title function_">from</span>(items);
});

Example: Handling AJAX content

To load more content dynamically, this script clicks a "Load More" button and waits for new elements to appear:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-comment">// Click load more button</span>
  <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#loadMore'</span>).<span class="hljs-title function_">click</span>();

  <span class="hljs-comment">// Wait for content update</span>
  <span class="hljs-keyword">await</span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Promise</span>(<span class="hljs-function"><span class="hljs-params">resolve</span> =></span> {
    <span class="hljs-keyword">const</span> observer = <span class="hljs-keyword">new</span> <span class="hljs-title class_">MutationObserver</span>(<span class="hljs-function">(<span class="hljs-params">mutations, obs</span>) =></span> {
      <span class="hljs-keyword">if</span> (<span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'.item'</span>).<span class="hljs-property">length</span> > <span class="hljs-number">10</span>) {
        obs.<span class="hljs-title function_">disconnect</span>();
        <span class="hljs-title function_">resolve</span>();
      }
    });

    observer.<span class="hljs-title function_">observe</span>(<span class="hljs-variable language_">document</span>.<span class="hljs-property">body</span>, {
      <span class="hljs-attr">childList</span>: <span class="hljs-literal">true</span>,
      <span class="hljs-attr">subtree</span>: <span class="hljs-literal">true</span>
    });
  });
});

These examples showcase how to handle diverse scenarios like scraping, form automation, and dynamic content. Adjustments can be made based on the specific structure and behavior of the webpage you're working with.

Using page.evaluate in Latenode

Latenode incorporates Puppeteer's core features into its automation workflows, making it easier to execute JavaScript directly in the browser. With page.evaluate, users can manipulate the DOM and extract data efficiently. This approach allows for seamless integration of advanced data handling and DOM operations within Latenode's automation environment.

Browser Scripts in Latenode

Latenode's browser automation module uses page.evaluate to handle everything from simple DOM tasks to more complex JavaScript execution. Here's how it works in different scenarios:

<span class="hljs-comment">// Basic DOM interaction</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">const</span> loginButton = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#login'</span>);
  loginButton.<span class="hljs-title function_">click</span>();

  <span class="hljs-comment">// Trigger a custom event</span>
  loginButton.<span class="hljs-title function_">dispatchEvent</span>(<span class="hljs-keyword">new</span> <span class="hljs-title class_">Event</span>(<span class="hljs-string">'customClick'</span>));
});

<span class="hljs-comment">// Processing data with exposed functions</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">exposeFunction</span>(<span class="hljs-string">'processData'</span>, <span class="hljs-title function_">async</span> (data) => {
  <span class="hljs-comment">// Process data in Node.js context</span>
  <span class="hljs-keyword">return</span> transformedData;
});

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-keyword">const</span> rawData = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#data'</span>).<span class="hljs-property">textContent</span>;
  <span class="hljs-keyword">const</span> processed = <span class="hljs-keyword">await</span> <span class="hljs-variable language_">window</span>.<span class="hljs-title function_">processData</span>(rawData);
  <span class="hljs-keyword">return</span> processed;
});

Latenode also keeps a log of execution history, making it easier to debug scripts.

Automation Examples

Latenode is well-equipped to handle dynamic content and complex automation tasks. Here's an example of processing dynamic content on a page:

<span class="hljs-keyword">const</span> extractProductData = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-keyword">const</span> <span class="hljs-title function_">delay</span> = ms => <span class="hljs-keyword">new</span> <span class="hljs-title class_">Promise</span>(<span class="hljs-function"><span class="hljs-params">resolve</span> =></span> <span class="hljs-built_in">setTimeout</span>(resolve, ms));

  <span class="hljs-comment">// Wait for dynamic content to load</span>
  <span class="hljs-keyword">while</span> (!<span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.product-grid'</span>)) {
    <span class="hljs-keyword">await</span> <span class="hljs-title function_">delay</span>(<span class="hljs-number">100</span>);
  }

  <span class="hljs-keyword">return</span> <span class="hljs-title class_">Array</span>.<span class="hljs-title function_">from</span>(<span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'.product'</span>))
    .<span class="hljs-title function_">map</span>(<span class="hljs-function"><span class="hljs-params">product</span> =></span> ({
      <span class="hljs-attr">name</span>: product.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.name'</span>).<span class="hljs-property">textContent</span>,
      <span class="hljs-attr">price</span>: product.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.price'</span>).<span class="hljs-property">textContent</span>,
      <span class="hljs-attr">availability</span>: product.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.stock'</span>).<span class="hljs-property">dataset</span>.<span class="hljs-property">status</span>
    }));
});

For more advanced operations, page.exposeFunction allows seamless interaction between Node.js and the browser:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">exposeFunction</span>(<span class="hljs-string">'md5'</span>, <span class="hljs-function"><span class="hljs-params">text</span> =></span>
  crypto.<span class="hljs-title function_">createHash</span>(<span class="hljs-string">'md5'</span>).<span class="hljs-title function_">update</span>(text).<span class="hljs-title function_">digest</span>(<span class="hljs-string">'hex'</span>)
);

<span class="hljs-keyword">const</span> processedData = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-keyword">const</span> sensitiveData = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#secure-data'</span>).<span class="hljs-property">value</span>;
  <span class="hljs-keyword">return</span> <span class="hljs-keyword">await</span> <span class="hljs-variable language_">window</span>.<span class="hljs-title function_">md5</span>(sensitiveData);
});

To maintain references to DOM elements across steps, Latenode uses page.evaluateHandle:

<span class="hljs-keyword">const</span> elementHandle = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluateHandle</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">return</span> <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.dynamic-content'</span>);
});

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function"><span class="hljs-params">element</span> =></span> {
  element.<span class="hljs-title function_">scrollIntoView</span>();
}, elementHandle);

These techniques ensure Latenode can handle dynamic content effectively while maintaining reliable performance. For users on the Prime plan, the platform supports up to 1.5 million scenario runs each month, providing extensive automation capabilities.

Error Resolution Guide

When working with page.evaluate in browser automation, you might encounter various issues. Here are practical solutions to address them and ensure smoother execution.

Fixing Context Errors

Properly configure your TypeScript settings to avoid issues caused by transpilation. For example:

<span class="hljs-comment">// Use direct, non-transpiled functions</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#button'</span>).<span class="hljs-title function_">click</span>();
});

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-string">`(async () => {
  document.querySelector('#button').click();
})()`</span>);

Avoid returning DOM elements directly from page.evaluate. Instead, use ElementHandle for better handling:

<span class="hljs-comment">// Incorrect: Returning a DOM element</span>
<span class="hljs-keyword">const</span> element = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">return</span> <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.dynamic-element'</span>);
});

<span class="hljs-comment">// Correct: Using ElementHandle</span>
<span class="hljs-keyword">const</span> element = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluateHandle</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">return</span> <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.dynamic-element'</span>);
});

Solving Timing Issues

Scripts may run before the page is fully loaded, leading to timing errors. Use these strategies to handle such cases:

<span class="hljs-comment">// Wait for navigation after an action</span>
<span class="hljs-keyword">await</span> <span class="hljs-title class_">Promise</span>.<span class="hljs-title function_">all</span>([
  page.<span class="hljs-title function_">waitForNavigation</span>(),
  page.<span class="hljs-title function_">click</span>(<span class="hljs-string">'#submit-button'</span>)
]);

<span class="hljs-comment">// Wait for a specific condition</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForFunction</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">const</span> element = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.lazy-loaded'</span>);
  <span class="hljs-keyword">return</span> element && element.<span class="hljs-property">dataset</span>.<span class="hljs-property">loaded</span> === <span class="hljs-string">'true'</span>;
}, { <span class="hljs-attr">timeout</span>: <span class="hljs-number">5000</span> });

For dynamic websites, adopt more targeted waiting mechanisms:

<span class="hljs-comment">// Wait for specific network requests</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForResponse</span>(
  <span class="hljs-function"><span class="hljs-params">response</span> =></span> response.<span class="hljs-title function_">url</span>().<span class="hljs-title function_">includes</span>(<span class="hljs-string">'/api/data'</span>)
);

<span class="hljs-comment">// Ensure elements are both present and visible</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">waitForSelector</span>(<span class="hljs-string">'.dynamic-content'</span>, {
  <span class="hljs-attr">visible</span>: <span class="hljs-literal">true</span>,
  <span class="hljs-attr">timeout</span>: <span class="hljs-number">3000</span>
});

Managing DOM References

To prevent memory leaks, carefully manage DOM references. Here’s how:

<span class="hljs-comment">// Use and dispose ElementHandles</span>
<span class="hljs-keyword">const</span> handle = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluateHandle</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">return</span> <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.temporary-element'</span>);
});
<span class="hljs-keyword">await</span> handle.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function"><span class="hljs-params">element</span> =></span> {
  <span class="hljs-comment">// Perform operations</span>
});
<span class="hljs-keyword">await</span> handle.<span class="hljs-title function_">dispose</span>(); <span class="hljs-comment">// Dispose of handle after use</span>

When working with multiple elements, pass data safely between contexts:

<span class="hljs-comment">// Extract data from the DOM</span>
<span class="hljs-keyword">const</span> selector = <span class="hljs-string">'.product-price'</span>;
<span class="hljs-keyword">const</span> price = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">(<span class="hljs-params">sel</span>) =></span> {
  <span class="hljs-keyword">const</span> element = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(sel);
  <span class="hljs-keyword">return</span> element ? element.<span class="hljs-property">textContent</span>.<span class="hljs-title function_">trim</span>() : <span class="hljs-literal">null</span>;
}, selector);

For event listeners, ensure proper cleanup to avoid lingering handlers:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">const</span> <span class="hljs-title function_">handler</span> = (<span class="hljs-params"></span>) => <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">log</span>(<span class="hljs-string">'clicked'</span>);
  <span class="hljs-keyword">const</span> button = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#button'</span>);
  button.<span class="hljs-title function_">addEventListener</span>(<span class="hljs-string">'click'</span>, handler);

  <span class="hljs-comment">// Store cleanup references</span>
  <span class="hljs-variable language_">window</span>.<span class="hljs-property">_cleanupHandlers</span> = <span class="hljs-variable language_">window</span>.<span class="hljs-property">_cleanupHandlers</span> || [];
  <span class="hljs-variable language_">window</span>.<span class="hljs-property">_cleanupHandlers</span>.<span class="hljs-title function_">push</span>(<span class="hljs-function">() =></span> {
    button.<span class="hljs-title function_">removeEventListener</span>(<span class="hljs-string">'click'</span>, handler);
  });
});

Implementation Guidelines

To get the best results with page.evaluate, you need to focus on improving performance, reducing unnecessary context switching, and ensuring security. Here’s how you can fine-tune your browser automation workflows.

Performance Optimization

Running code efficiently within the page context saves time and system resources. Below are some techniques to make your scripts faster:

<span class="hljs-comment">// Block unnecessary resources like images and stylesheets</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">setRequestInterception</span>(<span class="hljs-literal">true</span>);
page.<span class="hljs-title function_">on</span>(<span class="hljs-string">'request'</span>, <span class="hljs-function"><span class="hljs-params">request</span> =></span> {
  <span class="hljs-keyword">if</span> ([<span class="hljs-string">'image'</span>, <span class="hljs-string">'stylesheet'</span>].<span class="hljs-title function_">includes</span>(request.<span class="hljs-title function_">resourceType</span>())) {
    request.<span class="hljs-title function_">abort</span>();
  } <span class="hljs-keyword">else</span> {
    request.<span class="hljs-title function_">continue</span>();
  }
});

<span class="hljs-comment">// Batch operations to reduce overhead</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
  <span class="hljs-keyword">const</span> results = [];
  <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'.product-item'</span>).<span class="hljs-title function_">forEach</span>(<span class="hljs-function"><span class="hljs-params">item</span> =></span> {
    results.<span class="hljs-title function_">push</span>({
      <span class="hljs-attr">title</span>: item.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.title'</span>).<span class="hljs-property">textContent</span>,
      <span class="hljs-attr">price</span>: item.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.price'</span>).<span class="hljs-property">textContent</span>,
      <span class="hljs-attr">stock</span>: item.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'.stock'</span>).<span class="hljs-property">dataset</span>.<span class="hljs-property">value</span>
    });
  });
  <span class="hljs-keyword">return</span> results;
});

Choosing the right selectors also plays a big role in performance:

Selector Type	Speed	Example
ID	Fastest	#main-content
Class	Fast	.product-item
Tag	Moderate	div > span
Complex XPath	Slowest	//div[@class='wrapper']//span

Context Switch Management

Context switching between Node.js and the browser environment can slow things down. Here's how to minimize it:

<span class="hljs-comment">// Example of inefficient context switching</span>
<span class="hljs-keyword">for</span> (<span class="hljs-keyword">const</span> item <span class="hljs-keyword">of</span> items) {
  <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">(<span class="hljs-params">i</span>) =></span> {
    <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">`#item-<span class="hljs-subst">${i}</span>`</span>).<span class="hljs-title function_">click</span>();
  }, item);
}

<span class="hljs-comment">// Better: Batch operations in a single context switch</span>
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">(<span class="hljs-params">itemsList</span>) =></span> {
  itemsList.<span class="hljs-title function_">forEach</span>(<span class="hljs-function"><span class="hljs-params">i</span> =></span> {
    <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">`#item-<span class="hljs-subst">${i}</span>`</span>).<span class="hljs-title function_">click</span>();
  });
}, items);

If you need to process data in Node.js and pass it back to the browser, expose functions instead of repeatedly switching contexts:

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">exposeFunction</span>(<span class="hljs-string">'processData'</span>, <span class="hljs-title function_">async</span> (data) => {
  <span class="hljs-comment">// Process data in Node.js</span>
  <span class="hljs-keyword">return</span> transformedData;
});

<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-keyword">const</span> result = <span class="hljs-keyword">await</span> <span class="hljs-variable language_">window</span>.<span class="hljs-title function_">processData</span>(documentData);
  <span class="hljs-comment">// Use the processed data in the browser</span>
});

Security Guidelines

Once performance and context switching are optimized, focus on keeping your scripts secure. Here are some best practices:

<span class="hljs-comment">// Always sanitize inputs before using them</span>
<span class="hljs-keyword">const</span> sanitizedInput = <span class="hljs-title function_">sanitizeHtml</span>(userInput);
<span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">(<span class="hljs-params">input</span>) =></span> {
  <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelector</span>(<span class="hljs-string">'#search'</span>).<span class="hljs-property">value</span> = input;
}, sanitizedInput);

<span class="hljs-comment">// Use error handling for critical operations</span>
<span class="hljs-keyword">try</span> {
  <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-function">() =></span> {
    <span class="hljs-keyword">if</span> (!<span class="hljs-variable language_">window</span>.<span class="hljs-property">__securityCheck</span>) {
      <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> <span class="hljs-title class_">Error</span>(<span class="hljs-string">'Security check failed'</span>);
    }
    <span class="hljs-comment">// Continue with the operation</span>
  });
} <span class="hljs-keyword">catch</span> (error) {
  <span class="hljs-variable language_">console</span>.<span class="hljs-title function_">error</span>(<span class="hljs-string">'Security violation:'</span>, error);
}

For Latenode workflows, consider these additional tips:

• Use userDataDir to cache resources and improve performance across sessions.
• Close unused pages and browser instances to save memory.
• Handle screenshots with buffers instead of relying on file system operations.
• Implement robust error handling and thorough security checks.

Summary

Key Points Review

The page.evaluate method connects Node.js and browser contexts by sending a stringified JavaScript function to execute in the browser. This function operates independently of the Node.js environment, so you need to handle data transfer carefully.

Here's a common example for extracting data:

<span class="hljs-keyword">const</span> data = <span class="hljs-keyword">await</span> page.<span class="hljs-title function_">evaluate</span>(<span class="hljs-title function_">async</span> () => {
  <span class="hljs-keyword">const</span> results = <span class="hljs-variable language_">document</span>.<span class="hljs-title function_">querySelectorAll</span>(<span class="hljs-string">'.data-item'</span>);
  <span class="hljs-keyword">return</span> <span class="hljs-title class_">Array</span>.<span class="hljs-title function_">from</span>(results, <span class="hljs-function"><span class="hljs-params">item</span> =></span> ({
    <span class="hljs-attr">id</span>: item.<span class="hljs-property">dataset</span>.<span class="hljs-property">id</span>,
    <span class="hljs-attr">value</span>: item.<span class="hljs-property">textContent</span>.<span class="hljs-title function_">trim</span>()
  }));
});

Things to keep in mind:

• Arguments must be JSON-serializable.
• Return values are automatically deserialized.
• Browser APIs are available only within the evaluate context.
• Node.js variables are not accessible in the browser context.

These basics lay the groundwork for using Puppeteer effectively. Additional tools can further streamline your automation tasks.

Additional Puppeteer Tools

Puppeteer offers several tools to expand the capabilities of page.evaluate:

Tool	Purpose	Best Use Case
page.evaluateHandle	Returns object references	Interacting with DOM elements directly
page.exposeFunction	Makes Node.js functions usable in the browser	Managing complex server-side logic
page.evaluateOnNewDocument	Runs scripts before a page loads	Preparing the browser environment in advance

For example, exposing Node.js functions to the browser can simplify advanced data processing in workflows like those in Latenode. While page.evaluate works well for handling primitive types and JSON-serializable objects, page.evaluateHandle is essential for dealing with complex browser objects that can't be serialized.

Related posts

• Puppeteer Click Operations: Handling Complex Elements, Double Clicks, and Troubleshooting
• Form Automation with Puppeteer: Text Input, Form Filling, and User Simulation
• Custom Wait Conditions with waitForFunction in Puppeteer
• Browser Automation with Puppeteer and JavaScript: Practical Implementation in Node.js