Mastering Headless Browser Automation with Latenode

Mastering Headless Browser Automation with Latenode

Hi there, it's L! Headless browsers have immense potential when it comes to automating your workflows. However, if you’ve never worked with one, it might be challenging to figure out how to use it effectively. That’s the purpose of this guide. Over the next few minutes, I’ll show you how to leverage headless browsers using Latenode. We’ll cover basic features like capturing screenshots and passing data from a webpage into the Latenode automation platform. We will also look at some practical beginners' use cases and one we use at Latenode.

Fetching Data from Web Pages

Let's start with a simple use case. The headless browser can extract data from web pages, and that's what this scenario demonstrates. All the action happens right here. Let’s dive in.

In this code, the headless browser navigates to the Bank of England’s webpage to scrape the exchange rate of a currency. I send the currency via a webhook. Watching it in action provides a better understanding, so let’s do this. First, we copy our webhook link and switch over to Postman. Here, we insert our link, choose POST, then go to Body, select raw and JSON, and write something like this:

(JSON example)

Next, we click 'Send'. Now, let's wait. Here it is. Checking our script reveals one success. This is where the magic happens. After the headless browser completes its task, the webhook response returns a JSON object. I only use that response to demonstrate the result. You can input this JSON into a table, Google Sheets, or whichever tool you prefer.

Performing Automated Searches

Here’s another quick use case. This time, the headless browser carries out a search query in Google and retrieves the results. Let’s see it in action. First, activate the webhook link by clicking 'Activate', then we use our link. Since I use the webhook response, the search results now appear here. They’re here due to the webhook response. Here’s an inside look at how it’s done.

Scheduling and Filtering Data

You can also automate tasks on a schedule. In this example, I have two trigger nodes. One activates the script on a schedule—every Monday at 9:00 AM. Here's how it looks in a crontab expression. The other webhook activates the scenario whenever I send a signal to the link. Next, comes the Airtable node where you insert your API key in the connections and choose the database you want to interact with. In this instance, I’ve added a filter to select data only from the last seven days. Remember to click 'Save' every time you make changes to your automation. This allows you to revert to previous versions if necessary.

Afterward, we move to the JavaScript node. The code here creates JSON objects for each weekday. There's an AI co-pilot that can assist you. Simply prompt it with what you need, and it will help generate the code. Once your code is ready, you’ll be creating global variables with this node. Save all changes and activate this scenario.

Creating Charts from Data

The next script generates an HTML form and uses global variables to create a chart within the JavaScript node. You can ask the AI to incorporate free NPM packages in your prompt. Here’s how the chart looks. Take the webhook link, and you’ll see the output.

Capturing and Sharing Screenshots

Now, let’s introduce the headless browser again. This time, the headless browser takes a screenshot of the chart we created. The script then sends this screenshot to our Telegram group chat, using a token and chat ID.

Here’s how it works in action. Save the scenario, activate it, and then trigger the webhook link. The result appears in our chat.

If you encounter any issues or have questions while using Latenode, don’t hesitate to ask for help in the comments section or our Discord channel.