就像我們知道的一樣,nutch是一個架構在lucene之上的網絡爬蟲+搜索引擎.

是由lucene的作者在lucene基礎之上開發,并整合了hadoop,實現在分布式云計算,使用google標準的HFDS文件系統作為存儲結構,是一款高伸縮性能與高效高并發的網絡爬蟲+搜索引擎.

FaceYe在后臺已經整合了nutch,在適當的時候,就可以開始為用戶提供高質量的知識索引服務.順便說一下,nutch在生產環境中,并不能在windows下運行,需要在liux下運行,這其中主要是hadoop采用了一些shello腳本,當然,開發平臺還是可以搭建在window下,但需要安裝cygwin,來模擬shell環境.廢話少說,入nutch正題

正像上面說到的,nutch使用HFDS來存儲索引文件,并沒有將爬取來的數據存儲入數據庫,這是因為HFDS是一種比數據庫更高效,更容易實現負載均衡的結構,對于像搜索引擎這樣的應用,使用數據庫將對嚴重制約性能,所以,使用HFDS再加上倒派索引,會取理滿意的性能,HFDS也是目前搜索巨頭google,以及yahoo所正在使用的文件格式.

雖然有了HFDS,但在進行網絡爬取的時候,我們還是希望,可以將爬取的一些個數據,比如網頁url,比如網頁標題等關鍵信息存儲到數據庫中,但nutch并沒有提供這樣的功能,怎么辦?動手發明輪子~

nutch支持強大 的plugin 機制,這種機制與eclipse中的plugin機制同出一轍,一樣可以方便的進行插拔.

開發將爬取記錄存入數據庫的nutch plugin過程如下.

1.定義這一nutch plugin要實現的主要功能:

在使用nutch爬取網絡資源的同時,將網絡資源的主要信息存儲入數據庫.

2.新建plugin 包:

org.apache.nutch.indexer.store

并開發StoreIndexingFilter工具類如下:

public class StoreIndexingFilter implements IndexingFilter
{
public static final Log LOG = LogFactory.getLog(StoreIndexingFilter.class);

/** A flag that tells if magic resolution must be performed */
private boolean MAGIC;

/** Get the MimeTypes resolver instance. */
private MimeUtil MIME;

public NutchDocument filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) throws IndexingException
{

IResourceEntityService resourceEntityService = (IResourceEntityService) SpringUtil.getInstance().getBean(“resourceEntityService”);

String _url = doc.getFieldValue(“url”);
String _title = doc.getFieldValue(“title”);
if (StringUtils.isNotEmpty(_url))
{
if (!resourceEntityService.isExists(ResourceEntity.class, “url”, _url))
{
ResourceEntity resourceEntity = new ResourceEntity();
resourceEntity.setUrl(_url);
if (StringUtils.isNotEmpty(_title))
{
if (_title.length() > 255)
{
_title = _title.substring(0, 254);
}
}
resourceEntity.setName(_title);
resourceEntityService.saveResourceEntity(resourceEntity);
}
}
return doc;
}
private NutchDocument addTime(NutchDocument doc, ParseData data, String url, CrawlDatum datum)
{
long time = -1;

String lastModified = data.getMeta(Metadata.LAST_MODIFIED);
if (lastModified != null)
{ // try parse last-modified
time = getTime(lastModified, url); // use as time
// store as string
doc.add(“lastModified”, Long.toString(time));
}

if (time == -1)
{ // if no last-modified
time = datum.getFetchTime(); // use fetch time
}

SimpleDateFormat sdf = new SimpleDateFormat(“yyyyMMdd”);
sdf.setTimeZone(TimeZone.getTimeZone(“GMT”));
String dateString = sdf.format(new Date(time));

// un-stored, indexed and un-tokenized
doc.add(“date”, dateString);

return doc;
}

private long getTime(String date, String url)
{
long time = -1;
try
{
time = HttpDateFormat.toLong(date);
} catch (ParseException e)
{
// try to parse it as date in alternative format
try
{
Date parsedDate = DateUtils.parseDate(date, new String[] { “EEE MMM dd HH:mm:ss yyyy”, “EEE MMM dd HH:mm:ss yyyy zzz”,
“EEE, MMM dd HH:mm:ss yyyy zzz”, “EEE, dd MMM yyyy HH:mm:ss zzz”, “EEE,dd MMM yyyy HH:mm:ss zzz”, “EEE, dd MMM yyyy HH:mm:sszzz”,
“EEE, dd MMM yyyy HH:mm:ss”, “EEE, dd-MMM-yy HH:mm:ss zzz”, “yyyy/MM/dd HH:mm:ss.SSS zzz”, “yyyy/MM/dd HH:mm:ss.SSS”,
“yyyy/MM/dd HH:mm:ss zzz”, “yyyy/MM/dd”, “yyyy.MM.dd HH:mm:ss”, “yyyy-MM-dd HH:mm”, “MMM dd yyyy HH:mm:ss. zzz”,
“MMM dd yyyy HH:mm:ss zzz”, “dd.MM.yyyy HH:mm:ss zzz”, “dd MM yyyy HH:mm:ss zzz”, “dd.MM.yyyy; HH:mm:ss”, “dd.MM.yyyy HH:mm:ss”,
“dd.MM.yyyy zzz” });
time = parsedDate.getTime();
// if (LOG.isWarnEnabled()) {
// LOG.warn(url + “: parsed date: ” + date +” to:”+time);
// }
} catch (Exception e2)
{
if (LOG.isWarnEnabled())
{
LOG.warn(url + “: can’t parse erroneous date: ” + date);
}
}
}
return time;
}

// Add Content-Length
private NutchDocument addLength(NutchDocument doc, ParseData data, String url)
{
String contentLength = data.getMeta(Response.CONTENT_LENGTH);

if (contentLength != null)
doc.add(“contentLength”, contentLength);

return doc;
}
private NutchDocument addType(NutchDocument doc, ParseData data, String url)
{
MimeType mimeType = null;
String contentType = data.getMeta(Response.CONTENT_TYPE);
if (contentType == null)
{
mimeType = MIME.getMimeType(url);
} else
{
mimeType = MIME.forName(MimeUtil.cleanMimeType(contentType));
}

// Checks if we solved the content-type.
if (mimeType == null)
{
return doc;
}

contentType = mimeType.getName();

doc.add(“type”, contentType);

String[] parts = getParts(contentType);

for (String part : parts)
{
doc.add(“type”, part);
}

return doc;
}

static String[] getParts(String mimeType)
{
return mimeType.split(“/”);
}

private PatternMatcher matcher = new Perl5Matcher();

private Configuration conf;
static Perl5Pattern patterns[] = { null, null };
static
{
Perl5Compiler compiler = new Perl5Compiler();
try
{
// order here is important
patterns[0] = (Perl5Pattern) compiler.compile(“//bfilename=['/"](.+)['/"]“);
patterns[1] = (Perl5Pattern) compiler.compile(“//bfilename=(//S+)//b”);
} catch (MalformedPatternException e)
{
// just ignore
}
}

private NutchDocument resetTitle(NutchDocument doc, ParseData data, String url)
{
String contentDisposition = data.getMeta(Metadata.CONTENT_DISPOSITION);
if (contentDisposition == null)
return doc;

MatchResult result;
for (int i = 0; i < patterns.length; i++)
{
if (matcher.contains(contentDisposition, patterns[i]))
{
result = matcher.getMatch();
doc.add("title", result.group(1));
break;
}
}

return doc;
}

public void addIndexBackendOptions(Configuration conf)
{

LuceneWriter.addFieldOptions("type", LuceneWriter.STORE.NO, LuceneWriter.INDEX.UNTOKENIZED, conf);

LuceneWriter.addFieldOptions("primaryType", LuceneWriter.STORE.YES, LuceneWriter.INDEX.UNTOKENIZED, conf);
LuceneWriter.addFieldOptions("subType", LuceneWriter.STORE.YES, LuceneWriter.INDEX.UNTOKENIZED, conf);

LuceneWriter.addFieldOptions("contentLength", LuceneWriter.STORE.YES, LuceneWriter.INDEX.NO, conf);

LuceneWriter.addFieldOptions("lastModified", LuceneWriter.STORE.YES, LuceneWriter.INDEX.NO, conf);

// un-stored, indexed and un-tokenized
LuceneWriter.addFieldOptions("date", LuceneWriter.STORE.NO, LuceneWriter.INDEX.UNTOKENIZED, conf);
}

public void setConf(Configuration conf)
{
this.conf = conf;
MIME = new MimeUtil(conf);
}

public Configuration getConf()
{
return this.conf;
}

}

其中最主要的方法為:

public NutchDocument filter(NutchDocument doc, Parse parse, Text url, CrawlDatum datum, Inlinks inlinks) throws IndexingException
{

IResourceEntityService resourceEntityService = (IResourceEntityService) SpringUtil.getInstance().getBean("resourceEntityService");
String _url = doc.getFieldValue("url");
String _title = doc.getFieldValue("title");
if (StringUtils.isNotEmpty(_url))
{
if (!resourceEntityService.isExists(ResourceEntity.class, "url", _url))
{
ResourceEntity resourceEntity = new ResourceEntity();
resourceEntity.setUrl(_url);
if (StringUtils.isNotEmpty(_title))
{
if (_title.length() > 255)
{
_title = _title.substring(0, 254);
}
}
resourceEntity.setName(_title);
resourceEntityService.saveResourceEntity(resourceEntity);
}
}
return doc;
}
也就是說,要在使用nutch構建document文檔的同時,這一資源,存入數據庫.

存入數據庫的代碼resourceEntityService.saveResourceEntity(resourceEntity);不再詳細給出,有興趣的可以查看FaceYe的開源項目相關信息．

接下來需要做的事情是配置本插件的plugin文件，整體配置如下：

provider-name="nutch.org">

?

?

point="org.apache.nutch.indexer.IndexingFilter">
class="org.apache.nutch.indexer.store.StoreIndexingFilter" />

這個xml文件的主要含義是告訴nutch加載哪個jar，使用哪個類．文件中有清晰的描述．

nutch數據存數據庫的插件開發完畢了，接下來要做的是使用ant將本插件編譯為jar文件，為啟用本插件做準備．

編譯nutch源碼及配置文件為jar主要通過修改ant編譯文件來完成．

操作步驟為：打開nutch/src/plugin/文件，找到build.xml中的＂deploy”任務，添加

即可．

到些，將nutch爬取的數據存儲入數據庫的開發工作可以基本完成，接下來是要啟用本插件，這就很簡單了，

打開nutch/conf/nutch-site.xml．

找到plugin-include接點，在value中使用＂index-(basic|anchor|store)＂代替index-(basic|anchor);就完成了將nutch爬取數據存儲入數據庫插件的啟用.

nutch 存儲到數據庫