Automating without API: Headless Browser on low-code platform Latenode

Hello, It’s Daniel from Latenode 👋

Today, we’re going to discuss an automation tool that’s not widely used among no-code experts. Moreover, this feature, native to Latenode, isn’t available on platforms like Zapier, Make, and others, but it has strong potential for enhancing your workflows.

By the way, each time you browse the web via ChatGPT, you're using it! It's a Headless Browser.

Headless Browser Explanation

Let’s step back and look at the world of business process automation. In most cases, businesses use only two automation approaches that fully satisfy their needs:

1. Automate processes via API.

This is a solid approach for building data flows between different applications that have a public API. You either go to the developer hub yourself and figure out how to make a correct API call to do precisely what you need, or ask the JavaScript AI Assistant on Latenode to build integration with any app you need in seconds by simply describing your request.

Unfortunately, not all actions on the web have underlying API infrastructure to do the same things automatically. That’s where businesses rely on the second method that doesn’t require any API endpoints.

2. Headless Browser to Automate without APIs

This article is all about it, so take your seat, relax, and let’s dive into a new automation field that will later cover your back.

What is a Headless Browser?

Imagine a regular internet browser like Chrome or Firefox. Now, remove all the visible interface parts - buttons, address bar, bookmarks. What's left? The "brains" that can navigate the internet, open websites, and interact with them. This is what we call a "headless" browser. It can automate various tasks on websites without displaying visual content and can do this at breakneck speed. A script, not a human, controls the process.

A crucial part of the Headless Browser is its ability to run JavaScript custom scripts. It allows you to simulate user actions like clicking buttons, filling out forms, and navigating through a site’s menu. This is crucial for tasks like automated testing, web scraping, and automating repetitive tasks on web portals.

How Does a Headless Browser Work?

Operating a headless browser involves a few key steps, and it's essential to understand these to start using its potential:

1. URL to Browse: First, you need to specify the URL of the web page you want to interact with. This is like entering a website address in a regular browser, but here, you do it through a script.
2. Selectors to Navigate: Selectors are crucial in telling the headless browser what elements on a web page you want to interact with. These can be CSS selectors, XPath, or JavaScript commands. They allow you to pinpoint specific elements like buttons, text fields, images, etc.
3. Additional Parameters: Depending on your task, you might need to input text into forms, upload files, or click buttons. These actions are handled by scripts that you write, which specify what to do and when to do it.

Basically, you should tell the Headless Browser where to go, what to find, where to click, which text to type or copy, and so on.

What Can a Headless Browser Do?

There are some basic actions a Headless Browser supports under your control:

• Automated Navigation and Interaction: Headless browsers can perform tasks like filling out forms, clicking on links, scraping data, and even taking screenshots of web pages.
• Data Extraction and Handling: Often, the goal is to extract data from web pages. Headless browsers can parse the HTML and text of a page and extract the information you need, which can then be used in your application or stored for later use.
• Running Custom Scripts: Since headless browsers can run JavaScript, you can execute custom scripts to interact with web pages in complex ways, such as handling dynamic content or dealing with authentication.

For What Purposes Can I Use a Headless Browser?

Considering the basic actions, Headless Browsers offer a range of advanced actions that can be incredibly useful. To make it real, you need to integrate the Headless browser into low-code scenarios on Latenode. That allows you to implement headless browser into the following use-cases:

Use-Cases of Headless Browser

Now let’s jump to particular use-cases that you can simply copy-paste and hotwire the ignition of Headless Browser by yourself, even without prior experience.

Use-Case #1: Search the Web like GPT plugins do with Headless Browser

Fun fact: when you use ChatGPT and ask it to browse the web, a Headless Browser comes into play! Let’s build an MVP of something similar and ask the Headless Browser to make a search query for us.

Here is a brief overview of the following scenario:

• The Webhook URL is looking for a post request with a search request.
• Headless Browser gets that request, opens Google, and returns titles of the first 10 search positions.
• Webhook response sends back to the webhook the result.

Let’s dive a bit deeper into the headless browser code, where we map data from the webhook, so our Headless Browser node knows what exactly it should search for in Google.

After that, let’s make a POST request and send our search query as a ‘Search’ Key in the Body. Just in a few seconds, we’re able to see the result of scenario execution.

👉Want to test it by yourself? Copy the ready-to-use template and browse the web with Headless Browser!

Use-Case #2: Parse Data From Webpages with Headless Browser

Now straight to parsing data from websites! It can be used so widely: from real-time sync of prices from marketplaces to bulk scraping SEO elements for further analysis.

First example: what about scraping all titles (H1, H2, H3) from Latenode’s landing page? Let’s do it!

• The scenario looks the same:
• The Webhook listens for a URL to parse.
• Headless Browser navigates to the provided URL, finds, and retrieves H1, H2, H3 Titles.
• Webhook Response shows the list of titles as a result of calling the Webhook trigger.

Then we make a POST request again to our scenario and send the website to parse in the body section as shown below:

As a result, we get the list of H1, H2, and H3 titles from the webpage we sent to the headless browser.

Note: What’s important, you can operate with that info inside the Latenode scenario for any further data transformation or sending info wherever you need.

Second example: if you don’t have an API for the website from which you need to retrieve some crucial info, such as an exchange rate from US Dollar and Euro to England Pound – use a headless browser to get direct access to it.

By making one more POST request with two currencies, you inevitably get two exchange rates as a response from the Latenode scenario.

Use-Case #3: Fill out Forms Using Headless Browser

Now we switch our focus from searching and retrieving to filling. What about filling out a web form automatically and without an API?

• Provide the headless browser with the link to the web form.
• Give it text to input.
• Provide the path to the HTML, CSS, or XPath element to type your text.

The code for filling out such a simple HTML code looks like this:

const targetUrl = '<https://webhook.latenode.com/368/dev/SIMPLE_FORM>';

await page.setUserAgent('Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.108 Safari/537.36');
await page.goto(targetUrl);

// Using 'id' selectors as per your HTML structure
await page.type('#subject', data["{{$3.body.subject}}"]);
await page.type('#title', data["{{$3.body.title}}"]);
await page.type('#overview', data["{{$3.body.overview}}"]);
await page.type('#platform-updates', data["{{$3.body.updates}}"]);
await page.type('#useful-materials', data["{{$3.body.materials}}"]);
await page.type('#other-news', data["{{$3.body.news}}"]);

// The 'value' in the select option should match the 'value' attribute of the option
await page.select('#platform', 'TestEmail');
await page.click('#submit'); // Updated to use the 'id' of the submit button

// Assuming the popup is the confirmation of submission
await page.waitForSelector('#popup', {visible: true, timeout: 5000});

return { message: 'FORM SUBMITTED' };

Use-Case #4: Make Screenshots with Headless Browser

Last but not least, making screenshots of whatever you may find on the web.

To make a showcase, we build a dynamic chart on the Latenode platform that updates itself each week with new info. All done with the help of a JavaScript node and Global Variables.

It looks like this. But what if we want to share this chart with someone else each week when it's updated? To make it real, we can ask Headless Browsers for help to make a screenshot and send the file where it needs to go.

The Headless Browser node returns you a screenshot in base64 format. After that, the JavaScript node allows you to transform it as your further system needs to get this file.

👉To customize this ready-to-use template and start making screenshots with further sending to your Telegram chat, follow these steps:

• Copy the template from our gallery.
• In the JavaScript node, replace the Telegram Bot Token & ChatID with your data.
• Deploy your scenario and call the webhook URL manually.

Conclusion 

In this article, we've uncovered the power of Headless Browsers on Latenode, a powerful tool for low-code automations. These browsers, without the usual interface of Chrome or Firefox, offer a fast, script-driven way to use the web. They're perfect for tasks like filling out forms, grabbing data from websites, and automated testing, especially on complex, dynamic sites.

What makes our Headless Browser node stand out is its ease of use in a low-code environment. This means even those without deep coding knowledge can still use its capabilities. From automating simple tasks to handling complex web interactions, Headless Browser is a robust tool for various needs.

Enjoy using Latenode, and for any questions about the platform, join our Discord community of low-code experts.

For a visual representation of Headless Browser Automation, watch the Latenode tutorial on low-code automation with the help of our ready-to-use Headless Browser node on our platform.

Related articles:

• JavaScript + AI: a Perfect Duo for Low-code Automations
• Create Integration With Any App: Two Methods
• First-Ever Lifetime Deal for Early Birds!