public final class WebBotItem
extends Object
implements java.io.Closeable
WebBot.
There are three types of WebBotItem:
WebBotItem.Type.HTML - an HTML page or page fragment
WebBotItem.Type.CSS - a CSS file
WebBotItem.Type.RESOURCE - any other resource, including script files, images or any other media.
The main distinction between the three types is how they are parsed for links and other references to expand the crawl.
| Modifier and Type | Class and Description |
|---|---|
static class |
WebBotItem.Type
An enumerated type representing a type of
WebItem. |
| Modifier and Type | Method and Description |
|---|---|
void |
close() |
java.util.Queue<LogItem> |
getHTMLParsingLog() |
int |
getHTTPResponseCode() |
String |
getHTTPResponseMessage() |
java.net.HttpURLConnection |
getHttpURLConnection() |
java.io.IOException |
getIOException() |
java.net.URI |
getLinkedFromURI() |
java.nio.charset.Charset |
getOutputCharset() |
OutputDocument |
getOutputDocument() |
java.nio.file.Path |
getOutputPath()
Returns the output path of the generated archive file of this item, relative to the output base path.
|
java.util.Queue<LogItem> |
getProcessingLog() |
java.util.Queue<LogItem> |
getRefURILog() |
Source |
getSource() |
String |
getSourcePathFromBase() |
java.net.URI |
getSourceURI() |
WebBotItem.Type |
getType() |
boolean |
isErrorHTTPResponseCode()
Indicates whether this item's HTTP response code is an error code.
|
void |
output() |
void |
process() |
void |
setOutputCharset(java.nio.charset.Charset outputCharset) |
public java.net.URI getSourceURI()
public String getSourcePathFromBase()
public java.net.URI getLinkedFromURI()
public WebBotItem.Type getType()
public java.nio.file.Path getOutputPath()
The generated archive file is part of the website archive, which is only generated if an output base path has been specified.
An output path is created for each item even if no website archive is being generated, as it is used as a unique identifier to determine whether a source URI has already been added to the queue The unique output path is used for this purpose as the source URI itself is not necessarily a unique identifier. For example, the URLs http://jericho.htmlparser.net/docs/index.html and http://jericho.htmlparser.net/docs/ both reference the same document.
The return value of this method is never null.
******************* add documentation of how getUniqueOutputPath and getOutputPath work here
null.public Source getSource()
public OutputDocument getOutputDocument()
public void setOutputCharset(java.nio.charset.Charset outputCharset)
public java.nio.charset.Charset getOutputCharset()
public int getHTTPResponseCode()
public boolean isErrorHTTPResponseCode()
An error code theoretically has a value of 400 or higher, with the most common being 404 Not Found.
This method returns true if the code is anything other than 200 OK or one of the redirect codes
301 Moved Permanently, 302 Found, 303 See Other, 307 Temporary Redirect, or 308 Permanent Redirect.
But despite this wider definition of an error code, in practice the result should always comply with the theoretical definition.
true if this item's HTTP response code is an error code, otherwise falsepublic String getHTTPResponseMessage()
public java.net.HttpURLConnection getHttpURLConnection()
public java.io.IOException getIOException()
public java.util.Queue<LogItem> getProcessingLog()
public java.util.Queue<LogItem> getHTMLParsingLog()
public java.util.Queue<LogItem> getRefURILog()
public void process()
public void output()
public void close()
close in interface java.io.Closeableclose in interface AutoCloseable