ArtsAutosBooksBusinessEducationEntertainmentFamilyFashionFoodGamesGenderHealthHolidaysHomeHubPagesPersonal FinancePetsPoliticsReligionSportsTechnologyTravel

How To Parse Html In Java

Updated on February 1, 2013
Source

Libraries To Parse Html In Java

If you need to parse Html in Java there are different options available. In this article I will show you 3 ways to parse the same Html code in Java using the following libraries:

  • Internal Swing SDK parser
  • Jsoup
  • HtmlCleaner

In the second part will do the same using the libraries:

  • TagSoup
  • HTML Parser
  • JTidy
  • NekoHtml

Of course, these examples don't cover all the possibilities that these libraries offer you but should give you an idea of the kind of syntax and flexibility that is available. There are many other solutions out there as I will show you at the end. Hopefully it will help you to find the solution that suits you best!

Please Show Me The Code!

import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
import java.util.Enumeration;

import javax.swing.text.MutableAttributeSet;
import javax.swing.text.html.HTML.Tag;
import javax.swing.text.html.HTMLEditorKit;
import javax.swing.text.html.parser.ParserDelegator;

import org.htmlcleaner.HtmlCleaner;
import org.htmlcleaner.TagNode;
import org.htmlcleaner.XPatherException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;


public class HtmlParserSamples {
	
	private final static String HTML = 
			"<html xmlns=\"http://www.w3.org/1999/xhtml\">\n" + 
			"<head>\n" + 
			"  <title>This is a test page</title>\n" + 
			"</head>\n" +
			"<body>\n" + 
			"  <h1>This is a simple Html page to test:</h1>\n" + 
			"  <table>\n" + 
			"    <tr>\n" + 
			"      <td>Hello</td>\n" + 
			"      <td>World!</td>\n" + 
			"    </tr>\n" + 
			"  </table>\n" + 
			"</body>\n" + 
			"</html>";
	
	public static void main(String[] args) throws Exception {
		parseWithSwing();
		parseWithJsoup();
		parseWithHtmlCleaner();
	}

	// the remaining code will be showed separately
	...
	....

}

The JDK's Swing Html Parser

As I usually do, before looking for solutions outside, I look in house. By referring to the JDK 6 only, a possible solution can be HTMLEditorKit.

In my humble opinion, the code I produced with HTMLEditorKit looks fine to traverse an unknown html structure file, but makes the thing harder to read, understand and maintain especially if you need to extract separate information from the single html page (and multi-thread environment).

private static void parseWithSwing() throws IOException {
	System.out.println("*** SWING ***");
	Reader reader = new StringReader(HTML);
	HTMLEditorKit.Parser parser = new ParserDelegator();
	parser.parse(reader, new SwingHtmlParser(), true);
	reader.close();
	System.out.println();
}

....

class SwingHtmlParser extends HTMLEditorKit.ParserCallback {

	public void handleSimpleTag(Tag tag, MutableAttributeSet attrSet, int pos) {
		printTagWithAttributes(tag, attrSet);
	}

	private void printTagWithAttributes(Tag tag, MutableAttributeSet attrSet) {
		printTagWithAttributes(tag, attrSet);
	}

	public void handleStartTag(Tag tag, MutableAttributeSet attrSet, int pos) {
		System.out.println("Tag => " + tag.toString());
		Enumeration attributeNames = attrSet.getAttributeNames();
		while(attributeNames.hasMoreElements()){
			String attrName = attributeNames.nextElement().toString();
			System.out.println("Tag With Attributes => " +  attrName + " : " + attrSet.getAttribute(attrName));
		}
	}

	public void handleText(char[] data, int pos) {
		System.out.println("Tag Text => " + new String(data));
	}
	
	public void handleComment(char[] data, int pos) {
		System.out.println("Comment Text => " + new String(data));
	}

	public void handleEndOfLineString(String data) {
		// This is invoked after the stream has been parsed, but before flush. 
		// eol will be one of \n, \r or \r\n, which ever is 
		// encountered the most in parsing the stream.
		System.out.println("End of Line String => " + data);
	}

	public void handleEndTag(Tag tag, int pos) {
		System.out.println("End Of Tag for => " + tag.toString());
	}

	public void handleError(String err, int pos) {
		System.out.println("Error => " + err);
	}

}

Jsoup

Jsoup is my favourite choice! I consider it very flexible and easy to read. If you go to the project homepage there are different statements about what Jsoup's features actually are, like a convenient API, implementing the WHATWG HTML5 specification, outputting tidy HTML, etc.

And this is the code I produced for my example:

private static void parseWithJsoup() {
	System.out.println("*** JSOUP ***");
	Document doc = Jsoup.parse(HTML);
	System.out.println("Title: " + doc.getElementsByTag("title").text());
	System.out.println("H1: " + doc.getElementsByTag("h1").text());
	Element table = doc.getElementsByTag("table").first();
	Elements trs = table.getElementsByTag("tr");
	for (Element tr : trs) {
		System.out.println("TR: " + tr.text());
		for (Element td : tr.getAllElements()) {
			System.out.println("TD: " + td.text());
		}
	}
	System.out.println();
}

I have created another example of how to use Jsoup here.

HtmlCleaner

HtmlCleaner is another good library that I am suggesting to you. These are some of its main features as reported in the official website:

  • parses input HTML.
  • is thread safe.
  • can be used from Java code, from command line or as Ant task.

This is my sample code:

private static void parseWithHtmlCleaner() throws XPatherException {
	System.out.println("*** HTMLCLEANER ***");
	HtmlCleaner cleaner = new HtmlCleaner();
	TagNode node = cleaner.clean(HTML);
	System.out.println("Title: " + ((TagNode)(node.evaluateXPath("//title")[0])).getText());
	System.out.println("H1: " + ((TagNode)(node.evaluateXPath("//h1")[0])).getText());
	for (Object o : node.evaluateXPath("//tr")) {
		System.out.println("TR: " + ((TagNode)(o)).getText());			
	}
	for (Object o : node.evaluateXPath("//td")) {
		System.out.println("TD: " + ((TagNode)(o)).getText());			
	}
	System.out.println();
}

Other Libraries To Parse Html In Java

As I said in my introduction, there are many other libraries that help you to parse Html in Java. This is a list of other libraries I found (in alphabetical order):

Favourite Library Poll

What Is Your Favourite Java library Html Parser?

See results

Comments

    0 of 8192 characters used
    Post Comment
    • profile image

      thashi 

      7 years ago

      Still i didn't get a proper way to get print a HTML page which contains dynamic contents (iframes, nested iframes)

    • profile image

      zqb 

      7 years ago

      my email is 6637152@qq.com

    • profile image

      zqb 

      7 years ago

      hello 'parser.parse(reader, new SwingHtmlParser(), true);'

      this line Error

      : No enclosing instance of type HtmlParserSamples is accessible. Must qualify the allocation with an

      enclosing instance of type HtmlParserSamples (e.g. x.new A() where x is an instance of

      HtmlParserSamples).

      SwingHtmlParser s=new SwingHtmlParser();

      please see this problem ,and tell me How to solve

    • NitroD profile imageAUTHOR

      NitroD 

      8 years ago from London

      I would try using Apache Commons Lang, the method unescapeHtml4 in StringEscapeUtils.

      http://commons.apache.org/proper/commons-lang/java...

    • profile image

      Jim Wurster 

      8 years ago

      What would you do if the html string had the raw HTML codes for some of the less than and greater than signs?

    • NitroD profile imageAUTHOR

      NitroD 

      8 years ago from London

      Hi Wayne, thank you for your comment!

      I am agree using regex to parse HTML is not the best choice (i.e. open/close tag)

    • Wayne Riesterer profile image

      Wayne Riesterer 

      8 years ago

      Awesome tutorial. Thanks for this. It's often difficult to determine the best method of parsing HTML.

      I was leaning toward the regex way of doing things, but have read a lot of comments suggesting that this is not a good option at all.

      I'll be sure to check out some of your other tutorials.

    working

    This website uses cookies

    As a user in the EEA, your approval is needed on a few things. To provide a better website experience, hubpages.com uses cookies (and other similar technologies) and may collect, process, and share personal data. Please choose which areas of our service you consent to our doing so.

    For more information on managing or withdrawing consents and how we handle data, visit our Privacy Policy at: https://corp.maven.io/privacy-policy

    Show Details
    Necessary
    HubPages Device IDThis is used to identify particular browsers or devices when the access the service, and is used for security reasons.
    LoginThis is necessary to sign in to the HubPages Service.
    Google RecaptchaThis is used to prevent bots and spam. (Privacy Policy)
    AkismetThis is used to detect comment spam. (Privacy Policy)
    HubPages Google AnalyticsThis is used to provide data on traffic to our website, all personally identifyable data is anonymized. (Privacy Policy)
    HubPages Traffic PixelThis is used to collect data on traffic to articles and other pages on our site. Unless you are signed in to a HubPages account, all personally identifiable information is anonymized.
    Amazon Web ServicesThis is a cloud services platform that we used to host our service. (Privacy Policy)
    CloudflareThis is a cloud CDN service that we use to efficiently deliver files required for our service to operate such as javascript, cascading style sheets, images, and videos. (Privacy Policy)
    Google Hosted LibrariesJavascript software libraries such as jQuery are loaded at endpoints on the googleapis.com or gstatic.com domains, for performance and efficiency reasons. (Privacy Policy)
    Features
    Google Custom SearchThis is feature allows you to search the site. (Privacy Policy)
    Google MapsSome articles have Google Maps embedded in them. (Privacy Policy)
    Google ChartsThis is used to display charts and graphs on articles and the author center. (Privacy Policy)
    Google AdSense Host APIThis service allows you to sign up for or associate a Google AdSense account with HubPages, so that you can earn money from ads on your articles. No data is shared unless you engage with this feature. (Privacy Policy)
    Google YouTubeSome articles have YouTube videos embedded in them. (Privacy Policy)
    VimeoSome articles have Vimeo videos embedded in them. (Privacy Policy)
    PaypalThis is used for a registered author who enrolls in the HubPages Earnings program and requests to be paid via PayPal. No data is shared with Paypal unless you engage with this feature. (Privacy Policy)
    Facebook LoginYou can use this to streamline signing up for, or signing in to your Hubpages account. No data is shared with Facebook unless you engage with this feature. (Privacy Policy)
    MavenThis supports the Maven widget and search functionality. (Privacy Policy)
    Marketing
    Google AdSenseThis is an ad network. (Privacy Policy)
    Google DoubleClickGoogle provides ad serving technology and runs an ad network. (Privacy Policy)
    Index ExchangeThis is an ad network. (Privacy Policy)
    SovrnThis is an ad network. (Privacy Policy)
    Facebook AdsThis is an ad network. (Privacy Policy)
    Amazon Unified Ad MarketplaceThis is an ad network. (Privacy Policy)
    AppNexusThis is an ad network. (Privacy Policy)
    OpenxThis is an ad network. (Privacy Policy)
    Rubicon ProjectThis is an ad network. (Privacy Policy)
    TripleLiftThis is an ad network. (Privacy Policy)
    Say MediaWe partner with Say Media to deliver ad campaigns on our sites. (Privacy Policy)
    Remarketing PixelsWe may use remarketing pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to advertise the HubPages Service to people that have visited our sites.
    Conversion Tracking PixelsWe may use conversion tracking pixels from advertising networks such as Google AdWords, Bing Ads, and Facebook in order to identify when an advertisement has successfully resulted in the desired action, such as signing up for the HubPages Service or publishing an article on the HubPages Service.
    Statistics
    Author Google AnalyticsThis is used to provide traffic data and reports to the authors of articles on the HubPages Service. (Privacy Policy)
    ComscoreComScore is a media measurement and analytics company providing marketing data and analytics to enterprises, media and advertising agencies, and publishers. Non-consent will result in ComScore only processing obfuscated personal data. (Privacy Policy)
    Amazon Tracking PixelSome articles display amazon products as part of the Amazon Affiliate program, this pixel provides traffic statistics for those products (Privacy Policy)
    ClickscoThis is a data management platform studying reader behavior (Privacy Policy)