is.landsbokasafn.deduplicator

Class DeDupFetchHTTP

java.lang.Object

org.archive.modules.Processor

org.archive.modules.fetcher.FetchHTTP

is.landsbokasafn.deduplicator.DeDupFetchHTTP

All Implemented Interfaces:

org.archive.checkpointing.Checkpointable, org.archive.spring.HasKeyedProperties, org.springframework.beans.factory.BeanNameAware, org.springframework.context.Lifecycle

public class DeDupFetchHTTP
extends org.archive.modules.fetcher.FetchHTTP

An extentsion of Heritrix's org.archive.crawler.fetcher.FetchHTTP processor for downloading HTTP documents. This extension adds a check after the content header has been downloaded that compares the 'last-modified' and or 'last-etag' values from the header against information stored in an appropriate index.

Author:

Kristinn Sigurðsson

See Also:

is.hi.bok.deduplicator.DigestIndexer, org.archive.crawler.fetcher.FetchHTTP

Field Summary

Fields inherited from class org.archive.modules.fetcher.FetchHTTP

cookieStorage, digestAlgorithm, HTTP_BIND_ADDRESS, HTTP_SCHEME, HTTPS_SCHEME, RANGE, RANGE_PREFIX, REFERER, serverCache

Fields inherited from class org.archive.modules.Processor

beanName, isRunning, kp, recoveryCheckpoint, uriCount

Constructor Summary

Constructors

Constructor and Description
DeDupFetchHTTP()

Method Summary

Methods inherited from class org.archive.modules.fetcher.FetchHTTP

addResponseContent, checkMidfetchAbort, cleanupHttp, configureHttp, configureHttp, configureMethod, doAbort, getAcceptCompression, getAcceptHeaders, getAttributeEither, getAuthScheme, getCookieStorage, getCredentialStore, getDefaultCharset, getDefaultEncoding, getDigestAlgorithm, getDigestContent, getHttp, getHttpBindAddress, getHttpProxyHost, getHttpProxyPassword, getHttpProxyPort, getHttpProxyUser, getIgnoreCookies, getMaxFetchKBSec, getMaxLengthBytes, getSendConnectionClose, getSendIfModifiedSince, getSendIfNoneMatch, getSendRange, getSendReferer, getServerCache, getShouldFetchBodyRule, getSoTimeoutMs, getSslTrustLevel, getTimeoutSeconds, getUseHTTP11, getUserAgentProvider, handle401, innerProcess, isRunning, process, report, setAcceptCompression, setAcceptHeaders, setConditionalGetHeader, setCookieStorage, setCredentialStore, setDefaultEncoding, setDigestAlgorithm, setDigestContent, setHttpBindAddress, setHttpProxyHost, setHttpProxyPassword, setHttpProxyPort, setHttpProxyUser, setIgnoreCookies, setMaxFetchKBSec, setMaxLengthBytes, setSendConnectionClose, setSendIfModifiedSince, setSendIfNoneMatch, setSendRange, setSendReferer, setServerCache, setShouldFetchBodyRule, setSizes, setSoTimeoutMs, setSslTrustLevel, setTimeoutSeconds, setUseHTTP11, setUserAgentProvider, shouldProcess, start, stop

Methods inherited from class org.archive.modules.Processor

doCheckpoint, finishCheckpoint, flattenVia, fromCheckpointJson, getBeanName, getEnabled, getKeyedProperties, getRecordedSize, getShouldProcessRule, getURICount, hasHttpAuthenticationCredential, innerProcessResult, innerRejectProcess, isSuccess, setBeanName, setEnabled, setRecoveryCheckpoint, setShouldProcessRule, startCheckpoint, toCheckpointJson

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

DeDupFetchHTTP

public DeDupFetchHTTP()