+1 (726) 227-3027
ETL Advisors

Extracting Data via HTTP with HtmlUnit 4 (Part 2)

This is a 2026 rewrite of Part 2 of our HTTP extraction series. The 2014 version used HtmlUnit 2.13 with com.gargoylesoftware.htmlunit imports, the long-removed closeAllWindows() method, and a staging URL that no longer exists. HtmlUnit moved to the org.htmlunit package in version 3 (2023) and is at 4.x today, so the old code no longer compiles. Read Part 1 for the regex-based approach; this part uses a proper HTML parser. The original screenshots have been removed because they no longer match the current code.

Welcome to Part 2 of our HTML extraction tutorial. Here we pull structured data out of a web page using the HtmlUnit Java library: a headless browser that parses HTML into a DOM you can query with XPath, which is far more robust than matching tags with regular expressions.

Before you start: a note on etiquette

Scrape only pages you are allowed to. Check the site's robots.txt and terms of use, identify your client with a descriptive user agent, rate-limit your requests, and prefer an API when one exists. This example targets our own tutorials page, which is the only reason we are comfortable hammering it.

Step 1: Get HtmlUnit 4.x

HtmlUnit is published to Maven Central under the org.htmlunit group. Releases and changelogs are on GitHub; at the time of writing the current 4.x line is 4.21 with a 5.0 release also available. The coordinates are:

<dependency>
  <groupId>org.htmlunit</groupId>
  <artifactId>htmlunit</artifactId>
  <version>4.21.0</version>
</dependency>

You have two ways to bring it into a Talend job.

Option A: Maven coordinates in tLibraryLoad (recommended). In Talend Studio 7.3+ and 8, tLibraryLoad accepts a Maven URI instead of a local jar. Drop a tLibraryLoad on the canvas and set Library to:

mvn:org.htmlunit/htmlunit/4.21.0

Studio resolves the artifact and its transitive dependencies (Apache HttpClient, NekoHtml, Rhino, and the rest) from Maven Central or your configured repository. This replaces the old ritual of loading twenty jars one at a time.

Option B: local jars. Download the htmlunit-4.21.0-bin.zip from the releases page, unzip it, and add each jar from lib/ through separate tLibraryLoad components chained with OnComponentOk. It works, but Option A is far less error-prone.

Either way, put the library loading in its own small job (call it HTML_Unit) and reuse it through tRunJob from any job that needs the library, exactly as the 2014 tutorial suggested.

Step 2: Create the job

Create a job called HU_Demo. Drag the HTML_Unit job from the repository onto the canvas to create a tRunJob, then add a tJavaFlex (we use tJavaFlex rather than tJavaRow so the component can generate rows instead of needing an input row) and a tLogRow. Connect tRunJob --OnSubjobOk--> tJavaFlex --Main--> tLogRow.

Edit the tJavaFlex schema to a single String column title.

Step 3: Imports

In tJavaFlex's Advanced settings > Import, add:

import org.htmlunit.WebClient;
import org.htmlunit.BrowserVersion;
import org.htmlunit.html.HtmlPage;
import org.htmlunit.html.DomNode;
import java.util.List;

Note the package: org.htmlunit, not com.gargoylesoftware.htmlunit. Every class moved in version 3; if you see the old package in an example online, it is pre-2023.

Step 4: The code

Start code (runs once):

WebClient webClient = new WebClient(BrowserVersion.CHROME);
webClient.getOptions().setJavaScriptEnabled(false);
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setRedirectEnabled(true);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
webClient.getOptions().setTimeout(15000);
webClient.addRequestHeader("User-Agent",
    "ETLAdvisorsTutorialBot/1.0 (+https://etladvisors.com/contact)");

HtmlPage page = webClient.getPage("https://etladvisors.com/talend-tutorials/");
List<DomNode> headers = page.getByXPath("//h2");
int count = headers.size();
for (int i = 0; i < count; i++) {
    DomNode h = headers.get(i);
    String text = h.getTextContent().trim();
    if (text.isEmpty()) continue;

Main code (runs per generated row):

    row2.title = text;

End code:

}
webClient.close();

Line by line:

  • new WebClient(BrowserVersion.CHROME) creates the headless browser. Picking a browser version controls the user agent and a few parsing quirks; BrowserVersion.BEST_SUPPORTED is the other common choice. (The 2014 text mentioned Internet Explorer; that browser was retired in 2022 and HtmlUnit no longer emulates it.)
  • The getOptions() calls turn off JavaScript and CSS, which we do not need for static HTML and which make fetches much faster. Leave JavaScript on if the content you want is rendered client-side.
  • setThrowExceptionOnFailingStatusCode(false) lets you inspect a 404 page rather than crash; check page.getWebResponse().getStatusCode() if you need to branch on it.
  • getPage(...) fetches and parses the page. In HtmlUnit 4 it returns the concrete page type; assigning to HtmlPage works for HTML responses.
  • getByXPath("//h2") returns every h2 element as a List<DomNode> (in HtmlUnit 4 it is generic, so the unchecked cast from the old code is gone). On our tutorials index each tutorial title is an h2; adjust the XPath for your target page (//article/h2/a, //table[@id='prices']//tr, and so on).
  • getTextContent() returns the visible text of the node and its children. The old code indexed into getChildNodes().get(0), which breaks the moment the markup gains a wrapper element; getTextContent() does not.
  • webClient.close() releases the connection pool and any open windows. closeAllWindows() was removed years ago; if your code still calls it, that is the compile error you are seeing.

Because tJavaFlex emits one row per iteration of the loop in the start/main/end structure, there is no need for the string-concatenate-then-tNormalize dance from the original tutorial. Each heading becomes a row directly.

Step 5: Run the job

Open the Run tab and click Run. tLogRow prints one tutorial title per row. If you get zero rows, print page.asXml() to the console in the start code and confirm the XPath against what the server actually returned (JavaScript-rendered sites return a skeleton when JS is disabled).

Going further

  • Links as well as titles: use //h2/a and read ((HtmlAnchor) h).getHrefAttribute() (import org.htmlunit.html.HtmlAnchor).
  • Forms and logins: HtmlForm form = page.getFormByName("login"); form.getInputByName("user").type("...") and form.getInputByName("submit").click(); HtmlUnit maintains cookies across requests.
  • Pagination: loop over page.getAnchorByText("Next") until it throws, with a Thread.sleep between pages to be polite.
  • Memory: for long crawls call webClient.getCurrentWindow().getHistory() sparingly and close pages you are done with; setHistorySizeLimit(0) on the options helps.

The HtmlUnit getting-started guide and API docs cover the rest.

That concludes the HTTP extraction series. Part 1 showed that you can get by with regexes; this part shows why you usually should not.