Marmelator/aho-corasick
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updated aho-corasick site

Browse Source
This commit is contained in:
robert-bor 2014-01-31 20:58:17 +01:00
parent b1ed2144ce
commit 571daa895f
1 changed files with 52 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

89
index.html
View File

@ -44,47 +44,47 @@
<h2>
<a name="introduction" class="anchor" href="#introduction"><span class="octicon octicon-link"></span></a>Introduction</h2>
<p>Nowadays most free-text searching is based on Lucene-like approaches, where the search text is parsed into its
various components. For every keyword a lookup is done to see where it occurs. When looking for a couple of keywords
this approach is great. But what about it if you are not looking for just a couple of keywords, but a 100,000 of
them? Like, for example, checking against a dictionary?</p>
<p>Nowadays most free-text searching is based on Lucene-like approaches, where the search text is parsed into its
various components. For every keyword a lookup is done to see where it occurs. When looking for a couple of keywords
this approach is great. But what about it if you are not looking for just a couple of keywords, but a 100,000 of
them? Like, for example, checking against a dictionary?</p>
<p>This is where the Aho-Corasick algorithm shines. Instead of chopping up the search text, it uses all the keywords
to build up a construct called a <a href="http://en.wikipedia.org/wiki/Trie">Trie</a>. There are three crucial components
to Aho-Corasick:</p>
<p>This is where the Aho-Corasick algorithm shines. Instead of chopping up the search text, it uses all the keywords
to build up a construct called a <a href="http://en.wikipedia.org/wiki/Trie">Trie</a>. There are three crucial components
to Aho-Corasick:</p>
<ul>
<li>goto</li>
<li>fail</li>
<li>output</li>
</ul><p>Every character encountered is presented to a state object within the goto structure. If there is a matching state,
that will be elevated to the new current state.</p>
<ul>
<li>goto</li>
<li>fail</li>
<li>output</li>
</ul><p>Every character encountered is presented to a state object within the <em>goto</em> structure. If there is a matching state,
that will be elevated to the new current state.</p>
<p>However, if there is no matching state, the algorithm will fall back to states with less depth (ie, a match less long)
and proceed from there, until it found a matching state, or it has reached the root state.</p>
<p>However, if there is no matching state, the algorithm will signal a <em>fail</em> and fall back to states with less depth
(ie, a match less long) and proceed from there, until it found a matching state, or it has reached the root state.</p>
<p>Whenever a state is reached that matches an entire keyword, it is emitted to an output set which can be read after the
entire scan has completed.</p>
<p>Whenever a state is reached that matches an entire keyword, it is emitted to an <em>output</em> set which can be read after
the entire scan has completed.</p>
<p>The beauty of the algorithm is that it is O(n). No matter how many keywords you have, or how big the search text is,
the performance will decline in a linear way.</p>
<p>The beauty of the algorithm is that it is O(n). No matter how many keywords you have, or how big the search text is,
the performance will decline in a linear way.</p>
<p>Some examples you could use the Aho-Corasick algorithm for:</p>
<p>Some examples you could use the Aho-Corasick algorithm for:</p>
<ul>
<li>looking for certain words in texts in order to URL link or emphasize them</li>
<li>adding semantics to plain text</li>
<li>checking against a dictionary to see if syntactic errors were made</li>
</ul><p>This library is the Java implementation of the afore-mentioned Aho-Corasick algorithm for efficient string matching.
The algorithm is explained in great detail in the white paper written by
<a>Aho and Corasick</a>.</p>
<ul>
<li>looking for certain words in texts in order to URL link or emphasize them</li>
<li>adding semantics to plain text</li>
<li>checking against a dictionary to see if syntactic errors were made</li>
</ul><p>This library is the Java implementation of the afore-mentioned Aho-Corasick algorithm for efficient string matching.
The algorithm is explained in great detail in the white paper written by
Aho and Corasick: <a>ftp://163.13.200.222/assistant/bearhero/prog/%A8%E4%A5%A6/ac_bm.pdf</a></p>
<h2>
<a name="usage" class="anchor" href="#usage"><span class="octicon octicon-link"></span></a>Usage</h2>
<h2>
<a name="usage" class="anchor" href="#usage"><span class="octicon octicon-link"></span></a>Usage</h2>
<p>Setting up the Trie is a piece of cake:</p>
<p>Setting up the Trie is a piece of cake:</p>
<div class="highlight highlight-java"><pre> <span class="n">Trie</span> <span class="n">trie</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Trie</span><span class="o">();</span>
<div class="highlight highlight-java"><pre> <span class="n">Trie</span> <span class="n">trie</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Trie</span><span class="o">();</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"hers"</span><span class="o">);</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"his"</span><span class="o">);</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"she"</span><span class="o">);</span>
@ -92,13 +92,28 @@ The algorithm is explained in great detail in the white paper written by
<span class="n">Collection</span><span class="o">&lt;</span><span class="n">Emit</span><span class="o">&gt;</span> <span class="n">emits</span> <span class="o">=</span> <span class="n">trie</span><span class="o">.</span><span class="na">parseText</span><span class="o">(</span><span class="s">"ushers"</span><span class="o">);</span>
</pre></div>
<p>You can now read the set. In this case it will find the following:</p>
<p>You can now read the set. In this case it will find the following:</p>
<ul>
<li>"she" at position 3</li>
<li>"he" at position 3</li>
<li>"hers" at position 5</li>
</ul><h2>
<ul>
<li>"she" starting at position 1, ending at position 3</li>
<li>"he" starting at position 2, ending at position 3</li>
<li>"hers" starting at position 2, ending at position 5</li>
</ul><p>In normal situations you probably want to remove overlapping instances, retaining the longest and left-most
matches.</p>
<div class="highlight highlight-java"><pre> <span class="n">Trie</span> <span class="n">trie</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Trie</span><span class="o">().</span><span class="na">removeOverlaps</span><span class="o">();</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"hot"</span><span class="o">);</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"hot chocolate"</span><span class="o">);</span>
<span class="n">Collection</span><span class="o">&lt;</span><span class="n">Emit</span><span class="o">&gt;</span> <span class="n">emits</span> <span class="o">=</span> <span class="n">trie</span><span class="o">.</span><span class="na">parseText</span><span class="o">(</span><span class="s">"hot chocolate"</span><span class="o">);</span>
</pre></div>
<p>The removeOverlaps method tells the Trie to remove all overlapping matches. For this it relies on the following
conflict resolution rules: 1) longer matches prevail over shorter matches, 2) left-most prevails over right-most.
There is only one result now:</p>
<ul>
<li>"hot chocolate" starting at position 0, ending at position 12</li>
</ul><h2>
<a name="license" class="anchor" href="#license"><span class="octicon octicon-link"></span></a>License</h2>
<p>Licensed under the Apache License, Version 2.0 (the "License");