public class WebBot
extends Object
| Constructor and Description |
|---|
WebBot() |
| Modifier and Type | Method and Description |
|---|---|
void |
abort() |
void |
abort(String cause) |
void |
addRetryItem(WebBotItem item)
Retries an existing
WebBotItem whose processing resulted in an IOException, by adding it back to the front of the crawl queue. |
WebBot |
addSiteBaseURI(java.net.URI siteBaseURI)
Adds a site base URI.
|
boolean |
addSourceURI(java.net.URI sourceURI)
Adds the source URI of an HTML page to the crawl queue.
|
boolean |
addSourceURI(java.net.URI sourceURI,
WebBotItem.Type webBotItemType)
Adds a source URI of the specified type to the crawl queue.
|
String |
getAbortCause() |
java.nio.file.Path |
getAbsoluteCommonSiteOutputPath() |
java.net.URI |
getCommonSiteBaseURI()
Returns the common base URI of all added site base URIs.
|
int |
getDelaySeconds() |
static java.util.Set<String> |
getHTMLExtensionSet()
Gets a modifiable set of file extensions that indicate HTML pages.
|
int |
getInProcessCount()
Gets the number of items currently in process.
|
java.nio.file.Path |
getLogFilePath() |
WebBotItem |
getNextItem()
Gets the next
WebBotItem from the crawl queue. |
java.nio.file.Path |
getOutputBasePath()
Returns the base path of the website archive generated during the crawl.
|
int |
getPercentComplete()
|
int |
getProcessedCount()
Gets the number of processed items.
|
int |
getProcessedCSSCount()
|
int |
getProcessedErrorCount()
Gets the number of processed items that resulted in an error HTTP response.
|
int |
getProcessedHTMLCount()
|
int |
getProcessedResourceCount()
|
int |
getReadTimeoutSeconds()
Gets the server connection read timeout, measured in seconds.
|
static java.nio.file.Path |
getRelativeOutputPathOfFolderURI(java.net.URI uri) |
int |
getThreadCount() |
int |
getTotalCount()
|
int |
getUnprocessedCount()
Gets the number of unprocessed URIs in the crawl queue.
|
String |
getUserAgent()
Returns the user agent header included in each web server request.
|
boolean |
isIgnoreServerContentTypeHeader()
|
boolean |
isIncludeErrorPageContent()
Indicates whether error page content is to be preserved in the website archive.
|
boolean |
isOutputEnabled()
Indicates whether a website archive is to be generated during the crawl.
|
boolean |
isProcessCSS()
Indicates whether CSS files are to be processed.
|
boolean |
isProcessResources()
Indicates whether resource files are to be processed.
|
boolean |
isQueueEmpty()
Indicates whether the crawl queue is empty.
|
protected boolean |
isURIAllowed(java.net.URI sourceURI)
Overridable method determining whether the specified source URI is allowed in the crawl.
|
void |
log(String text) |
void |
resume() |
void |
run() |
WebBot |
setDelaySeconds(int delaySeconds) |
WebBot |
setIgnoreServerContentTypeHeader(boolean ignoreServerContentTypeHeader)
|
WebBot |
setIncludeErrorPageContent(boolean includeErrorPageContent)
Sets whether to include error page content in the website archive.
|
WebBot |
setLogFilePath(java.nio.file.Path logFilePath) |
WebBot |
setOutputBasePath(java.nio.file.Path outputBasePath)
Sets the base path of the website archive to be generated during the crawl.
|
WebBot |
setProcessCSS(boolean processCSS)
Sets whether to process CSS files.
|
WebBot |
setProcessResources(boolean processResources)
Sets whether to process resource files.
|
WebBot |
setReadTimeoutSeconds(int readTimeoutSeconds)
Sets the server connection read timeout, measured in seconds.
|
WebBot |
setThreadCount(int threadCount) |
WebBot |
setUserAgent(String userAgent)
Sets the user agent header to be included in each web server request.
|
public FunctionalInterfaces.Call_with_String_return_void logFunction
public FunctionalInterfaces.Call_with_WebBotItem_return_void onErrorHTTPResponseCode
public FunctionalInterfaces.Call_with_WebBotItem_return_boolean preOutput
public FunctionalInterfaces.Call_with_WebBotItem_Source_OutputDocument_return_void customiseOutputHTML
public FunctionalInterfaces.Call_with_WebBotItem_Source_OutputDocument_return_void customiseOutputCSS
public FunctionalInterfaces.Call_with_WebBotItem_return_void onSuccessfulOutput
public FunctionalInterfaces.Call_with_noargs_return_String userAbort
public FunctionalInterfaces.Call_with_noargs_return_void finalReport
public FunctionalInterfaces.Call_with_String_WebBotItem_return_void progressReport
public FunctionalInterfaces.Call_with_WebBotItem_return_String getItemProcessingNotifications
public WebBot setOutputBasePath(java.nio.file.Path outputBasePath)
One of the main features of this class is the ability to create a functioning static archive of a website while it is being crawled.
This method enables that functionality by specifying the local filesystem path where the website archive should be created. The specified path is the base path of the archive, with files created under a folder structure reflecting the structure of the crawled website.
The folder(s) immediately under the specified base path will reflect the hostname of each crawled website, followed by the canonical folder structure of the site, even if the site base URI is a subfolder under the host. This is necessary because of the ability to include multiple sites in a single crawl, in which case the archive copy of every page and resource must be stored in a canonical folder structure.
If only a single site base URI is included in the crawl, the equivalent archive filesystem path of the site base URI can be determined
using the getAbsoluteCommonSiteOutputPath() method.
For example, if this method is called with the path C:\Data\WebsiteArchives,
and only the source URI https://example.com/abc/def/homepage.html is added to the crawl queue,
then the method getAbsoluteCommonSiteOutputPath() will return the path C:\Data\WebsiteArchives\example.com\abc\def,
and the generated archive file homepage.html will be found under that folder.
See the main class documentation for details of how the website archive is generated.
outputBasePath - the base path of the website archive to be generated during the crawl, or null if no archive is to be generated.WebBot instance, allowing multiple property setting methods to be chained in a single statement.public WebBot addSiteBaseURI(java.net.URI siteBaseURI)
Web crawls include only URIs under the registered site base URIs.
The first site base URI is usually added automatically when the site's homepage is added using the addSourceURI(URI) method.
For example, the method call addSourceURI(new URI("https://example.com/abc/def/homepage.html")) automatically adds the site base URI
https://example.com/abc/def/.
Every call to the addSourceURI(URI) method automatically adds the specified URI's parent folder as a site base URI unless it is under an existing site base URI.
Adding multiple site base URIs makes it possible to archive multiple sites at the same time while preserving any links between them.
This method might be called directly, instead of calling the addSourceURI(URI) method, if it is known that pages under this site base URI will be linked to from
one of the pages already included in the crawl.
siteBaseURI - the site base URI to add.WebBot instance, allowing multiple property setting methods to be chained in a single statement.public WebBot setUserAgent(String userAgent)
Some reasons for setting this might be:
The default value is
"net.htmlparser.jericho.WebBot/3.5-dev (http://jericho.htmlparser.net/)"
userAgent - the user agent header to be included in each web server request, must not be null.WebBot instance, allowing multiple property setting methods to be chained in a single statement.public WebBot setIgnoreServerContentTypeHeader(boolean ignoreServerContentTypeHeader)
Content-Type response header from the server when determining the type of a linked item.
The crawler uses multiple techniques to determine whether a linked item is an HTML page, CSS file or general resource file.
One of the more reliable techniques is usually to look at the MIME type inside the Content-Type response header from the server,
but if the server returns incorrect MIME types due to misconfiguration then the crawl might not work properly.
Setting this property to true tells the crawler to ignore the Content-Type response header
and use other techniques to determine the types of linked items,
such as the filename extension and detection of binary data.
It should only be used in this uncommon circumstance.
The default value is false.
ignoreServerContentTypeHeader - specifies whether to ignore the Content-Type response header from the server.WebBot instance, allowing multiple property setting methods to be chained in a single statement.public WebBot setProcessCSS(boolean processCSS)
Setting this property to false prevents CSS files from being included in the website archive,
and also prevents the inclusion of any resource files that are referenced only from CSS files.
The default value is true.
processCSS - specifies whether to process CSS files.WebBot instance, allowing multiple property setting methods to be chained in a single statement.public WebBot setProcessResources(boolean processResources)
Setting this property to false prevents resource files from being included in the website archive.
The default value is true.
processResources - specifies whether to process resource files.WebBot instance, allowing multiple property setting methods to be chained in a single statement.public WebBot setReadTimeoutSeconds(int readTimeoutSeconds)
This property is used to set both the
read timeout and the
connect timeout of the
URLConnection.
If the connection times out during a crawl, the crawl is aborted, but can be resumed later when the connection problem has been resolved.
A timeout of zero is interpreted as an infinite timeout.
The default value is 10 seconds.
readTimeoutSeconds - specifies the server connection read timeout.WebBot instance, allowing multiple property setting methods to be chained in a single statement.public WebBot setIncludeErrorPageContent(boolean includeErrorPageContent)
An error page is one that returns an HTTP error code.
This is typically a code of 400 or higher, including 404 Not Found.
Such pages often return customised content which should be preserved as part of the archive.
The default value is true.
includeErrorPageContent - specifies whether to include error page content in the website archive.WebBot instance, allowing multiple property setting methods to be chained in a single statement.public boolean addSourceURI(java.net.URI sourceURI,
WebBotItem.Type webBotItemType)
In many cases this method only needs to be called once, specifying the URI of the homepage, with all other pages and resources in the site added automatically from links found recursively from the homepage.
Sometimes however there are hidden pages in the site, or resources and pages accessed via script, which the crawler does not find automatically. In these cases extra pages and resources should be added manually using this method.
This method is thread safe.
sourceURI - the source URI of a page or resource to add to the crawl queue, may be null.webBotItemType - the type of item to add, must not be null.true if the specified source URI is allowed in this crawl, otherwise false.addSourceURI(URI sourceURI)public WebBot setLogFilePath(java.nio.file.Path logFilePath)
public java.nio.file.Path getLogFilePath()
public WebBot setThreadCount(int threadCount)
public int getThreadCount()
public WebBot setDelaySeconds(int delaySeconds)
public int getDelaySeconds()
public String getAbortCause()
public void abort()
public void abort(String cause)
public void run()
throws java.io.IOException
java.io.IOExceptionpublic void resume()
throws java.io.IOException
java.io.IOExceptionpublic void log(String text)
public final java.nio.file.Path getOutputBasePath()
See the setOutputBasePath(Path) method for a full description of this property.
null if no website archive is to be generated.public final boolean isOutputEnabled()
This is equivalent to getOutputBasePath()!=null.
true if a website archive is to be generated during the crawl, otherwise false.public final java.net.URI getCommonSiteBaseURI()
In simple terms, this is the URI of the website being crawled.
In most crawls there is only one registered site base URI, but in some cases multiple site base URIs are registered to archive multiple sites at the same time while preserving any links between them.
The common base URI returned by this method is the longest URI that is above all the site base URIs.
For example, if the following two site base URIs have been added:
If site base URIs with different hostnames have been added, there is no common site base URI and this method returns null.
null if no site base URIs have been added or if site base URIs with different hostnames have been added.public final String getUserAgent()
See the setUserAgent(String) method for a full description of this property.
null.public final boolean isIgnoreServerContentTypeHeader()
Content-Type response header from the server when determining the type of a linked item.
See the setIgnoreServerContentTypeHeader(boolean) method for a full description of this property.
true if the Content-Type response header from the server is to be ignored, otherwise false.public final boolean isProcessCSS()
See the setProcessCSS(boolean) method for a full description of this property.
true if CSS files are to be processed, otherwise false.public final boolean isProcessResources()
See the setProcessResources(boolean) method for a full description of this property.
true if resource files are to be processed, otherwise false.public int getReadTimeoutSeconds()
See the setReadTimeoutSeconds(int) method for a full description of this property.
public boolean isIncludeErrorPageContent()
See the setIncludeErrorPageContent(boolean) method for a full description of this property.
true if error page content is to be preserved in the website archive, otherwise false.public static java.util.Set<String> getHTMLExtensionSet()
To make it possible to browse the complete website archive directly from to file system,
the output path of every HTML page should be given an .html extension.
The most reliable way to determine the type of a crawled item is based on the MIME type reported by the server.
But for efficiency reasons the output path must be fixed when the item is first discovered, before the item is fetched from the server. It is therefore desirable to guess the item type as accurately as possible before the server-reported MIME type can be used.
Of the many factors that can be used to guess the type of a crawled item, the file extension of the item is an important indicator.
The set returned by this method can be modified to improve the accuracy of the preliminary guess of each item type, and hence improve the "browsability" of the website archive.
This is a static property, so changing the set affects all WebBot instances.
The default set is html, htm, jsp, do, php, asp, aspx, cfm.
public final int getUnprocessedCount()
public final int getInProcessCount()
An item is counted as "in process" as soon as it is fetched from the queue using the getNextItem() method,
even if the WebBotItem.process() method hasn't yet been called.
When execution of the item's WebBotItem.process() method completes successfully it is no longer counted as in process and is counted as processed.
However if the WebBotItem.process() method fails with an IOException, the item is considered to be still in process
until it is retried.
public final int getProcessedCount()
An item is counted as processed as soon as execution of its WebBotItem.process() method is complete.
This is equivalent to getProcessedHTMLCount()+getProcessedCSSCount()+getProcessedResourceCount()+getProcessedErrorCount().
This method is not synchronized so during a multithreaded crawl the result should be treated as an approximation only.
public final int getProcessedHTMLCount()
An item is counted as processed as soon as execution of its WebBotItem.process() method is complete.
getProcessedCount()public final int getProcessedCSSCount()
An item is counted as processed as soon as execution of its WebBotItem.process() method is complete.
getProcessedCount()public final int getProcessedResourceCount()
An item is counted as processed as soon as execution of its WebBotItem.process() method is complete.
getProcessedCount()public final int getProcessedErrorCount()
An item is counted as processed as soon as execution of its WebBotItem.process() method is complete.
getProcessedCount()public final int getTotalCount()
This is equivalent to getUnprocessedCount()+getInProcessCount()+getProcessedCount().
This method is not synchronized so during a multithreaded crawl the result should be treated as an approximation only.
public final int getPercentComplete()
This is equivalent to 100*getProcessedCount()/getTotalCount().
The percent complete often goes backwards as more items are found and added to the queue, especially in the early stages of the crawl. The figure should therefore be considered an approximation only.
public boolean addSourceURI(java.net.URI sourceURI)
This is equivalent to calling addSourceURI(sourceURI,WebBotItem.Type.HTML).
This method is thread safe.
public void addRetryItem(WebBotItem item)
WebBotItem whose processing resulted in an IOException, by adding it back to the front of the crawl queue.
Items will often succeed on retry after an initial IOException failure as it is typically caused by a temporary connectivity or server issue.
This method doesn't need to be called directly if you use the high-level run() method to manage the crawl
as that will automatically keep track of failed items.
If a temporary connectivity or server issue aborts the crawl,
calling the resume() method will automatically add the failed items back to start of the queue before resuming the crawl.
If the specified item was not created by this WebBot instance, an IllegalArgumentException is thrown.
Calling this method increments the unprocessed count and decrements the in process count.
The reason for the latter is that an item that fails processing due to an IOException is still counted as being in process.
This method is thread safe.
item - a WebBotItem that previously failed to process, must not be null.public WebBotItem getNextItem()
WebBotItem from the crawl queue.
This method doesn't need to be called directly if you use the high-level run() method to manage the crawl.
A return value of null indicates the queue is empty, but that doesn't necessarily mean the crawl is finished.
This is explained in the documentation of the isQueueEmpty() method.
The recommended course of action if a single thread in a multithreaded queue processor encounters an empty queue, and other items are still processing, is to sleep for a short period of time and check for more items later.
Calling this method increments the in process count and decrements the unprocessed count if there was an item in the queue.
The WebBot class doesn't implement Iterable or Iterator
because iterators are not designed for use with collections that change during iteration, especially when concurrent use is allowed.
The problem with the Iterator interface is that its hasNext() and next() methods are separate calls,
meaning the iterator would need to be locked while calling both to avoid consistency errors.
To avoid all of this, we use this atomic method that combines these calls, returning null if there are no more items.
This method is thread safe even though it is not synchronized, as the underlying queue is thread safe.
WebBotItem from the crawl queue, or null if there are no items left.public final boolean isQueueEmpty()
An empty queue doesn't necessarily mean the crawl is finished. If items are processed in multiple threads, then the crawl is only finished when there are no more items in the queue and no more items are being processed in other threads.
The queue may be temporarily empty but then fill up again as items are processed since processing an HTML page automatically adds links and resources found in the page to the queue.
true if the crawl queue is empty, otherwise false.getNextItem()protected boolean isURIAllowed(java.net.URI sourceURI)
The default implementation returns true if the specified URI is not null, meaning no URIs are filtered from the crawl.
URIs outside of the site base URIs are filtered out before this method is called.
An example use for this method might be to implement robots.txt filtering.
It is recommended that any implementation uses a caching mechanism, as the method is likely to be called multiple times with the same URI.
sourceURI - the source URI to test.true if the specified source URI is allowed in the crawl, otherwise false.public final java.nio.file.Path getAbsoluteCommonSiteOutputPath()
public static java.nio.file.Path getRelativeOutputPathOfFolderURI(java.net.URI uri)