How To Parse Html In Java

Source

Libraries To Parse Html In Java

If you need to parse Html in Java there are different options available. In this article I will show you 3 ways to parse the same Html code in Java using the following libraries:

  • Internal Swing SDK parser
  • Jsoup
  • HtmlCleaner

In the second part will do the same using the libraries:

  • TagSoup
  • HTML Parser
  • JTidy
  • NekoHtml

Of course, these examples don't cover all the possibilities that these libraries offer you but should give you an idea of the kind of syntax and flexibility that is available. There are many other solutions out there as I will show you at the end. Hopefully it will help you to find the solution that suits you best!

Please Show Me The Code!

import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
import java.util.Enumeration;

import javax.swing.text.MutableAttributeSet;
import javax.swing.text.html.HTML.Tag;
import javax.swing.text.html.HTMLEditorKit;
import javax.swing.text.html.parser.ParserDelegator;

import org.htmlcleaner.HtmlCleaner;
import org.htmlcleaner.TagNode;
import org.htmlcleaner.XPatherException;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;


public class HtmlParserSamples {
	
	private final static String HTML = 
			"<html xmlns=\"http://www.w3.org/1999/xhtml\">\n" + 
			"<head>\n" + 
			"  <title>This is a test page</title>\n" + 
			"</head>\n" +
			"<body>\n" + 
			"  <h1>This is a simple Html page to test:</h1>\n" + 
			"  <table>\n" + 
			"    <tr>\n" + 
			"      <td>Hello</td>\n" + 
			"      <td>World!</td>\n" + 
			"    </tr>\n" + 
			"  </table>\n" + 
			"</body>\n" + 
			"</html>";
	
	public static void main(String[] args) throws Exception {
		parseWithSwing();
		parseWithJsoup();
		parseWithHtmlCleaner();
	}

	// the remaining code will be showed separately
	...
	....

}

The JDK's Swing Html Parser

As I usually do, before looking for solutions outside, I look in house. By referring to the JDK 6 only, a possible solution can be HTMLEditorKit.

In my humble opinion, the code I produced with HTMLEditorKit looks fine to traverse an unknown html structure file, but makes the thing harder to read, understand and maintain especially if you need to extract separate information from the single html page (and multi-thread environment).

private static void parseWithSwing() throws IOException {
	System.out.println("*** SWING ***");
	Reader reader = new StringReader(HTML);
	HTMLEditorKit.Parser parser = new ParserDelegator();
	parser.parse(reader, new SwingHtmlParser(), true);
	reader.close();
	System.out.println();
}

....

class SwingHtmlParser extends HTMLEditorKit.ParserCallback {

	public void handleSimpleTag(Tag tag, MutableAttributeSet attrSet, int pos) {
		printTagWithAttributes(tag, attrSet);
	}

	private void printTagWithAttributes(Tag tag, MutableAttributeSet attrSet) {
		printTagWithAttributes(tag, attrSet);
	}

	public void handleStartTag(Tag tag, MutableAttributeSet attrSet, int pos) {
		System.out.println("Tag => " + tag.toString());
		Enumeration attributeNames = attrSet.getAttributeNames();
		while(attributeNames.hasMoreElements()){
			String attrName = attributeNames.nextElement().toString();
			System.out.println("Tag With Attributes => " +  attrName + " : " + attrSet.getAttribute(attrName));
		}
	}

	public void handleText(char[] data, int pos) {
		System.out.println("Tag Text => " + new String(data));
	}
	
	public void handleComment(char[] data, int pos) {
		System.out.println("Comment Text => " + new String(data));
	}

	public void handleEndOfLineString(String data) {
		// This is invoked after the stream has been parsed, but before flush. 
		// eol will be one of \n, \r or \r\n, which ever is 
		// encountered the most in parsing the stream.
		System.out.println("End of Line String => " + data);
	}

	public void handleEndTag(Tag tag, int pos) {
		System.out.println("End Of Tag for => " + tag.toString());
	}

	public void handleError(String err, int pos) {
		System.out.println("Error => " + err);
	}

}

Jsoup

Jsoup is my favourite choice! I consider it very flexible and easy to read. If you go to the project homepage there are different statements about what Jsoup's features actually are, like a convenient API, implementing the WHATWG HTML5 specification, outputting tidy HTML, etc.

And this is the code I produced for my example:

private static void parseWithJsoup() {
	System.out.println("*** JSOUP ***");
	Document doc = Jsoup.parse(HTML);
	System.out.println("Title: " + doc.getElementsByTag("title").text());
	System.out.println("H1: " + doc.getElementsByTag("h1").text());
	Element table = doc.getElementsByTag("table").first();
	Elements trs = table.getElementsByTag("tr");
	for (Element tr : trs) {
		System.out.println("TR: " + tr.text());
		for (Element td : tr.getAllElements()) {
			System.out.println("TD: " + td.text());
		}
	}
	System.out.println();
}

I have created another example of how to use Jsoup here.

HtmlCleaner

HtmlCleaner is another good library that I am suggesting to you. These are some of its main features as reported in the official website:

  • parses input HTML.
  • is thread safe.
  • can be used from Java code, from command line or as Ant task.

This is my sample code:

private static void parseWithHtmlCleaner() throws XPatherException {
	System.out.println("*** HTMLCLEANER ***");
	HtmlCleaner cleaner = new HtmlCleaner();
	TagNode node = cleaner.clean(HTML);
	System.out.println("Title: " + ((TagNode)(node.evaluateXPath("//title")[0])).getText());
	System.out.println("H1: " + ((TagNode)(node.evaluateXPath("//h1")[0])).getText());
	for (Object o : node.evaluateXPath("//tr")) {
		System.out.println("TR: " + ((TagNode)(o)).getText());			
	}
	for (Object o : node.evaluateXPath("//td")) {
		System.out.println("TD: " + ((TagNode)(o)).getText());			
	}
	System.out.println();
}

Other Libraries To Parse Html In Java

As I said in my introduction, there are many other libraries that help you to parse Html in Java. This is a list of other libraries I found (in alphabetical order):

Favourite Library Poll

What Is Your Favourite Java library Html Parser?

See results without voting

More by this Author


Comments 7 comments

Wayne Riesterer profile image

Wayne Riesterer 3 years ago

Awesome tutorial. Thanks for this. It's often difficult to determine the best method of parsing HTML.

I was leaning toward the regex way of doing things, but have read a lot of comments suggesting that this is not a good option at all.

I'll be sure to check out some of your other tutorials.


NitroD profile image

NitroD 3 years ago from London Author

Hi Wayne, thank you for your comment!

I am agree using regex to parse HTML is not the best choice (i.e. open/close tag)


Jim Wurster 3 years ago

What would you do if the html string had the raw HTML codes for some of the less than and greater than signs?


NitroD profile image

NitroD 3 years ago from London Author

I would try using Apache Commons Lang, the method unescapeHtml4 in StringEscapeUtils.

http://commons.apache.org/proper/commons-lang/java...


zqb 3 years ago

hello 'parser.parse(reader, new SwingHtmlParser(), true);'

this line Error

: No enclosing instance of type HtmlParserSamples is accessible. Must qualify the allocation with an

enclosing instance of type HtmlParserSamples (e.g. x.new A() where x is an instance of

HtmlParserSamples).

SwingHtmlParser s=new SwingHtmlParser();

please see this problem ,and tell me How to solve


zqb 3 years ago

my email is 6637152@qq.com


thashi 2 years ago

Still i didn't get a proper way to get print a HTML page which contains dynamic contents (iframes, nested iframes)

    Sign in or sign up and post using a HubPages Network account.

    0 of 8192 characters used
    Post Comment

    No HTML is allowed in comments, but URLs will be hyperlinked. Comments are not for promoting your articles or other sites.


    Click to Rate This Article
    working