is.landsbokasafn.deduplicator

Class DeDuplicator

java.lang.Object

org.archive.modules.Processor

is.landsbokasafn.deduplicator.DeDuplicator

All Implemented Interfaces:

org.archive.checkpointing.Checkpointable, org.archive.spring.HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.beans.factory.InitializingBean, org.springframework.context.Lifecycle

public class DeDuplicator
extends org.archive.modules.Processor
implements org.springframework.beans.factory.InitializingBean

Heritrix compatible processor.

Will determine if CrawlURIs are duplicates.

Duplicate detection can only be performed after the fetch processors have run.

Author:

Kristinn Sigurðsson

Field Summary

Fields

Modifier and Type	Field and Description
static String	ATTR_ANALYZE_TIMESTAMP
static String	ATTR_EQUIVALENT
static String	ATTR_FILTER_MODE
static String	ATTR_JUMP_TO
static String	ATTR_MIME_FILTER
static String	ATTR_ORIGIN
static String	ATTR_ORIGIN_HANDLING
static String	ATTR_STATS_PER_HOST
static String	ATTR_USE_SPARSE_RANGE_FILTER
static String	DEFAULT_MIME_FILTER
static OriginHandling	DEFAULT_ORIGIN_HANDLING
protected boolean	lookupByURL
protected HashMap<String,is.landsbokasafn.deduplicator.Statistics>	perHostStats
protected org.apache.lucene.search.IndexSearcher	searcher
protected org.archive.modules.net.ServerCache	serverCache
protected is.landsbokasafn.deduplicator.Statistics	stats
protected boolean	statsPerHost
protected boolean	useOrigin
protected boolean	useOriginFromIndex

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
DeDuplicator()

Method Summary

Methods

Modifier and Type	Method and Description
void	afterPropertiesSet()
protected void	doAnalysis(org.archive.modules.CrawlURI curi, is.landsbokasafn.deduplicator.Statistics currHostStats, boolean isDuplicate)
protected void	doTimestampAnalysis(org.archive.modules.CrawlURI curi, org.apache.lucene.document.Document urlHit, is.landsbokasafn.deduplicator.Statistics currHostStats, boolean isDuplicate)
protected void	finalTasks()
boolean	getAnalyzeTimestamp()
boolean	getBlacklist()
String	getIndexLocation()
String	getJumpTo()
is.landsbokasafn.deduplicator.MatchingMethod	getMatchingMethod()
String	getMimeFilter()
String	getOrigin()
OriginHandling	getOriginHandling()
protected static String	getPercentage(double portion, double total)
org.archive.modules.net.ServerCache	getServerCache()
boolean	getStatsPerHost()
boolean	getTryEquivalent()
boolean	getUseSparseRengeFilter()
protected void	innerProcess(org.archive.modules.CrawlURI puri)
protected org.archive.modules.ProcessResult	innerProcessResult(org.archive.modules.CrawlURI curi)
protected org.apache.lucene.document.Document	lookupByDigest(org.archive.modules.CrawlURI curi, is.landsbokasafn.deduplicator.Statistics currHostStats) Process a CrawlURI looking up in the index by content digest
protected org.apache.lucene.document.Document	lookupByURL(org.archive.modules.CrawlURI curi, is.landsbokasafn.deduplicator.Statistics currHostStats) Process a CrawlURI looking up in the index by URL
protected org.apache.lucene.search.Query	queryField(String fieldName, String value) Run a simple Lucene query for a single term in a single field.
String	report()
void	setAnalyzeTimestamp(boolean analyzeTimestamp)
void	setBlacklist(boolean blacklist)
void	setIndexLocation(String indexLocation)
void	setJumpTo(String jumpTo)
void	setMatchingMethod(is.landsbokasafn.deduplicator.MatchingMethod matchinMethod)
void	setMimeFilter(String mimeFilter)
void	setOrigin(String origin)
void	setOriginHandling(OriginHandling originHandling)
void	setServerCache(org.archive.modules.net.ServerCache serverCache)
void	setStatsPerHost(boolean statsPerHost)
void	setTryEquivalent(boolean tryEquivalent)
void	setUseSparseRengeFilter(boolean useSparseRengeFilter)
protected boolean	shouldProcess(org.archive.modules.CrawlURI curi)

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerRejectProcess, isRunning, isSuccess, process, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, start, startCheckpoint, stop, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

ATTR_JUMP_TO

public static final String ATTR_JUMP_TO

See Also:

Constant Field Values

ATTR_ORIGIN

public static final String ATTR_ORIGIN

See Also:

Constant Field Values

ATTR_EQUIVALENT

public static final String ATTR_EQUIVALENT

See Also:

Constant Field Values

ATTR_MIME_FILTER

public static final String ATTR_MIME_FILTER

See Also:

Constant Field Values

DEFAULT_MIME_FILTER

public static final String DEFAULT_MIME_FILTER

See Also:

Constant Field Values

ATTR_FILTER_MODE

public static final String ATTR_FILTER_MODE

See Also:

Constant Field Values

ATTR_ANALYZE_TIMESTAMP

public static final String ATTR_ANALYZE_TIMESTAMP

See Also:

Constant Field Values

ATTR_STATS_PER_HOST

public static final String ATTR_STATS_PER_HOST

See Also:

Constant Field Values

ATTR_USE_SPARSE_RANGE_FILTER

public static final String ATTR_USE_SPARSE_RANGE_FILTER

See Also:

Constant Field Values

ATTR_ORIGIN_HANDLING

public static final String ATTR_ORIGIN_HANDLING

See Also:

Constant Field Values

DEFAULT_ORIGIN_HANDLING

public static final OriginHandling DEFAULT_ORIGIN_HANDLING

serverCache

protected org.archive.modules.net.ServerCache serverCache

searcher

protected org.apache.lucene.search.IndexSearcher searcher

lookupByURL

protected boolean lookupByURL

statsPerHost

protected boolean statsPerHost

useOrigin

protected boolean useOrigin

useOriginFromIndex

protected boolean useOriginFromIndex

stats

protected is.landsbokasafn.deduplicator.Statistics stats

perHostStats

protected HashMap<String,is.landsbokasafn.deduplicator.Statistics> perHostStats

Constructor Detail

DeDuplicator

public DeDuplicator()

Method Detail

getIndexLocation

public String getIndexLocation()

setIndexLocation

public void setIndexLocation(String indexLocation)

getMatchingMethod

public is.landsbokasafn.deduplicator.MatchingMethod getMatchingMethod()

setMatchingMethod

public void setMatchingMethod(is.landsbokasafn.deduplicator.MatchingMethod matchinMethod)

getJumpTo

public String getJumpTo()

setJumpTo

public void setJumpTo(String jumpTo)

getOrigin

public String getOrigin()

setOrigin

public void setOrigin(String origin)

getTryEquivalent

public boolean getTryEquivalent()

setTryEquivalent

public void setTryEquivalent(boolean tryEquivalent)

getMimeFilter

public String getMimeFilter()

setMimeFilter

public void setMimeFilter(String mimeFilter)

getBlacklist

public boolean getBlacklist()

setBlacklist

public void setBlacklist(boolean blacklist)

getAnalyzeTimestamp

public boolean getAnalyzeTimestamp()

setAnalyzeTimestamp

public void setAnalyzeTimestamp(boolean analyzeTimestamp)

getStatsPerHost

public boolean getStatsPerHost()

setStatsPerHost

public void setStatsPerHost(boolean statsPerHost)

getUseSparseRengeFilter

public boolean getUseSparseRengeFilter()

setUseSparseRengeFilter

public void setUseSparseRengeFilter(boolean useSparseRengeFilter)

getOriginHandling

public OriginHandling getOriginHandling()

setOriginHandling

public void setOriginHandling(OriginHandling originHandling)

getServerCache

public org.archive.modules.net.ServerCache getServerCache()

setServerCache

public void setServerCache(org.archive.modules.net.ServerCache serverCache)

afterPropertiesSet

public void afterPropertiesSet()
                        throws Exception

Specified by:

afterPropertiesSet in interface org.springframework.beans.factory.InitializingBean

Throws:

Exception

shouldProcess

protected boolean shouldProcess(org.archive.modules.CrawlURI curi)

Specified by:

shouldProcess in class org.archive.modules.Processor

innerProcess

protected void innerProcess(org.archive.modules.CrawlURI puri)

Specified by:

innerProcess in class org.archive.modules.Processor

innerProcessResult

protected org.archive.modules.ProcessResult innerProcessResult(org.archive.modules.CrawlURI curi)
                                                        throws InterruptedException

Overrides:

innerProcessResult in class org.archive.modules.Processor

Throws:

InterruptedException

lookupByURL

protected org.apache.lucene.document.Document lookupByURL(org.archive.modules.CrawlURI curi,
                                              is.landsbokasafn.deduplicator.Statistics currHostStats)

Process a CrawlURI looking up in the index by URL

Parameters:

curi - The CrawlURI to process

currHostStats - A statistics object for the current host. If per host statistics tracking is enabled this must be non null and the method will increment appropriate counters on it.

Returns:

The result of the lookup (a Lucene document). If a duplicate is not found null is returned.

lookupByDigest

protected org.apache.lucene.document.Document lookupByDigest(org.archive.modules.CrawlURI curi,
                                                 is.landsbokasafn.deduplicator.Statistics currHostStats)

Process a CrawlURI looking up in the index by content digest

Parameters:

curi - The CrawlURI to process

currHostStats - A statistics object for the current host. If per host statistics tracking is enabled this must be non null and the method will increment appropriate counters on it.

Returns:

The result of the lookup (a Lucene document). If a duplicate is not found null is returned.

report

public String report()

Overrides:

report in class org.archive.modules.Processor

getPercentage

protected static String getPercentage(double portion,
                   double total)

doAnalysis

protected void doAnalysis(org.archive.modules.CrawlURI curi,
              is.landsbokasafn.deduplicator.Statistics currHostStats,
              boolean isDuplicate)

doTimestampAnalysis

protected void doTimestampAnalysis(org.archive.modules.CrawlURI curi,
                       org.apache.lucene.document.Document urlHit,
                       is.landsbokasafn.deduplicator.Statistics currHostStats,
                       boolean isDuplicate)

queryField

protected org.apache.lucene.search.Query queryField(String fieldName,
                                        String value)

Run a simple Lucene query for a single term in a single field.

Parameters:

fieldName - name of the field to look in.

value - The value to query for

finalTasks

protected void finalTasks()