Marmelator/aho-corasick
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

updated aho-corasick site

Browse Source
This commit is contained in:
robert-bor 2014-01-31 20:58:17 +01:00
parent b1ed2144ce
commit 571daa895f
1 changed files with 52 additions and 37 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

89
index.html
View File

@ -44,47 +44,47 @@
<h2> <h2>
<a name="introduction" class="anchor" href="#introduction"><span class="octicon octicon-link"></span></a>Introduction</h2> <a name="introduction" class="anchor" href="#introduction"><span class="octicon octicon-link"></span></a>Introduction</h2>
<p>Nowadays most free-text searching is based on Lucene-like approaches, where the search text is parsed into its <p>Nowadays most free-text searching is based on Lucene-like approaches, where the search text is parsed into its
various components. For every keyword a lookup is done to see where it occurs. When looking for a couple of keywords various components. For every keyword a lookup is done to see where it occurs. When looking for a couple of keywords
this approach is great. But what about it if you are not looking for just a couple of keywords, but a 100,000 of this approach is great. But what about it if you are not looking for just a couple of keywords, but a 100,000 of
them? Like, for example, checking against a dictionary?</p> them? Like, for example, checking against a dictionary?</p>
<p>This is where the Aho-Corasick algorithm shines. Instead of chopping up the search text, it uses all the keywords <p>This is where the Aho-Corasick algorithm shines. Instead of chopping up the search text, it uses all the keywords
to build up a construct called a <a href="http://en.wikipedia.org/wiki/Trie">Trie</a>. There are three crucial components to build up a construct called a <a href="http://en.wikipedia.org/wiki/Trie">Trie</a>. There are three crucial components
to Aho-Corasick:</p> to Aho-Corasick:</p>
<ul> <ul>
<li>goto</li> <li>goto</li>
<li>fail</li> <li>fail</li>
<li>output</li> <li>output</li>
</ul><p>Every character encountered is presented to a state object within the goto structure. If there is a matching state, </ul><p>Every character encountered is presented to a state object within the <em>goto</em> structure. If there is a matching state,
that will be elevated to the new current state.</p> that will be elevated to the new current state.</p>
<p>However, if there is no matching state, the algorithm will fall back to states with less depth (ie, a match less long) <p>However, if there is no matching state, the algorithm will signal a <em>fail</em> and fall back to states with less depth
and proceed from there, until it found a matching state, or it has reached the root state.</p> (ie, a match less long) and proceed from there, until it found a matching state, or it has reached the root state.</p>
<p>Whenever a state is reached that matches an entire keyword, it is emitted to an output set which can be read after the <p>Whenever a state is reached that matches an entire keyword, it is emitted to an <em>output</em> set which can be read after
entire scan has completed.</p> the entire scan has completed.</p>
<p>The beauty of the algorithm is that it is O(n). No matter how many keywords you have, or how big the search text is, <p>The beauty of the algorithm is that it is O(n). No matter how many keywords you have, or how big the search text is,
the performance will decline in a linear way.</p> the performance will decline in a linear way.</p>
<p>Some examples you could use the Aho-Corasick algorithm for:</p> <p>Some examples you could use the Aho-Corasick algorithm for:</p>
<ul> <ul>
<li>looking for certain words in texts in order to URL link or emphasize them</li> <li>looking for certain words in texts in order to URL link or emphasize them</li>
<li>adding semantics to plain text</li> <li>adding semantics to plain text</li>
<li>checking against a dictionary to see if syntactic errors were made</li> <li>checking against a dictionary to see if syntactic errors were made</li>
</ul><p>This library is the Java implementation of the afore-mentioned Aho-Corasick algorithm for efficient string matching. </ul><p>This library is the Java implementation of the afore-mentioned Aho-Corasick algorithm for efficient string matching.
The algorithm is explained in great detail in the white paper written by The algorithm is explained in great detail in the white paper written by
<a>Aho and Corasick</a>.</p> Aho and Corasick: <a>ftp://163.13.200.222/assistant/bearhero/prog/%A8%E4%A5%A6/ac_bm.pdf</a></p>
<h2> <h2>
<a name="usage" class="anchor" href="#usage"><span class="octicon octicon-link"></span></a>Usage</h2> <a name="usage" class="anchor" href="#usage"><span class="octicon octicon-link"></span></a>Usage</h2>
<p>Setting up the Trie is a piece of cake:</p> <p>Setting up the Trie is a piece of cake:</p>
<div class="highlight highlight-java"><pre> <span class="n">Trie</span> <span class="n">trie</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Trie</span><span class="o">();</span> <div class="highlight highlight-java"><pre> <span class="n">Trie</span> <span class="n">trie</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Trie</span><span class="o">();</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"hers"</span><span class="o">);</span> <span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"hers"</span><span class="o">);</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"his"</span><span class="o">);</span> <span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"his"</span><span class="o">);</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"she"</span><span class="o">);</span> <span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"she"</span><span class="o">);</span>
@ -92,13 +92,28 @@ The algorithm is explained in great detail in the white paper written by
<span class="n">Collection</span><span class="o">&lt;</span><span class="n">Emit</span><span class="o">&gt;</span> <span class="n">emits</span> <span class="o">=</span> <span class="n">trie</span><span class="o">.</span><span class="na">parseText</span><span class="o">(</span><span class="s">"ushers"</span><span class="o">);</span> <span class="n">Collection</span><span class="o">&lt;</span><span class="n">Emit</span><span class="o">&gt;</span> <span class="n">emits</span> <span class="o">=</span> <span class="n">trie</span><span class="o">.</span><span class="na">parseText</span><span class="o">(</span><span class="s">"ushers"</span><span class="o">);</span>
</pre></div> </pre></div>
<p>You can now read the set. In this case it will find the following:</p> <p>You can now read the set. In this case it will find the following:</p>
<ul> <ul>
<li>"she" at position 3</li> <li>"she" starting at position 1, ending at position 3</li>
<li>"he" at position 3</li> <li>"he" starting at position 2, ending at position 3</li>
<li>"hers" at position 5</li> <li>"hers" starting at position 2, ending at position 5</li>
</ul><h2> </ul><p>In normal situations you probably want to remove overlapping instances, retaining the longest and left-most
matches.</p>
<div class="highlight highlight-java"><pre> <span class="n">Trie</span> <span class="n">trie</span> <span class="o">=</span> <span class="k">new</span> <span class="n">Trie</span><span class="o">().</span><span class="na">removeOverlaps</span><span class="o">();</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"hot"</span><span class="o">);</span>
<span class="n">trie</span><span class="o">.</span><span class="na">addKeyword</span><span class="o">(</span><span class="s">"hot chocolate"</span><span class="o">);</span>
<span class="n">Collection</span><span class="o">&lt;</span><span class="n">Emit</span><span class="o">&gt;</span> <span class="n">emits</span> <span class="o">=</span> <span class="n">trie</span><span class="o">.</span><span class="na">parseText</span><span class="o">(</span><span class="s">"hot chocolate"</span><span class="o">);</span>
</pre></div>
<p>The removeOverlaps method tells the Trie to remove all overlapping matches. For this it relies on the following
conflict resolution rules: 1) longer matches prevail over shorter matches, 2) left-most prevails over right-most.
There is only one result now:</p>
<ul>
<li>"hot chocolate" starting at position 0, ending at position 12</li>
</ul><h2>
<a name="license" class="anchor" href="#license"><span class="octicon octicon-link"></span></a>License</h2> <a name="license" class="anchor" href="#license"><span class="octicon octicon-link"></span></a>License</h2>
<p>Licensed under the Apache License, Version 2.0 (the "License"); <p>Licensed under the Apache License, Version 2.0 (the "License");