public class SimpleNaiveBayesDocumentClassifier extends SimpleNaiveBayesClassifier implements DocumentClassifier<BytesRef>
http://en.wikipedia.org/wiki/Naive_Bayes_classifier
Modifier and Type | Field and Description |
---|---|
protected java.util.Map<java.lang.String,Analyzer> |
field2analyzer
Analyzer to be used for tokenizing document fields |
analyzer, classFieldName, indexReader, indexSearcher, query, textFieldNames
Constructor and Description |
---|
SimpleNaiveBayesDocumentClassifier(IndexReader indexReader,
Query query,
java.lang.String classFieldName,
java.util.Map<java.lang.String,Analyzer> field2analyzer,
java.lang.String... textFieldNames)
Creates a new NaiveBayes classifier.
|
Modifier and Type | Method and Description |
---|---|
private void |
analyzeSeedDocument(Document inputDocument,
java.util.Map<java.lang.String,java.util.List<java.lang.String[]>> fieldName2tokensArray,
java.util.Map<java.lang.String,java.lang.Float> fieldName2boost)
This methods performs the analysis for the seed document and extract the boosts if present.
|
ClassificationResult<BytesRef> |
assignClass(Document document)
Assign a class (with score) to the given
Document |
private java.util.List<ClassificationResult<BytesRef>> |
assignNormClasses(Document inputDocument) |
private double |
calculateLogLikelihood(java.lang.String[] tokenizedText,
java.lang.String fieldName,
Term term,
int docsWithClass) |
private double |
calculateLogPrior(Term term,
int docsWithClassSize) |
private int |
docCount(Term term) |
java.util.List<ClassificationResult<BytesRef>> |
getClasses(Document document)
Get all the classes (sorted by score, descending) assigned to the given
Document . |
java.util.List<ClassificationResult<BytesRef>> |
getClasses(Document document,
int max)
Get the first
max classes (sorted by score, descending) assigned to the given text String. |
private double |
getTextTermFreqForClass(Term term,
java.lang.String fieldName)
Returns the average number of unique terms times the number of docs belonging to the input class
|
protected java.lang.String[] |
getTokenArray(TokenStream tokenizedText)
Returns a token array from the
TokenStream in input |
private int |
getWordFreqForClass(java.lang.String word,
java.lang.String fieldName,
Term term)
Returns the number of documents of the input class ( from the whole index or from a subset)
that contains the word ( in a specific field or in all the fields if no one selected)
|
assignClass, assignClassNormalizedList, countDocsWithClass, getClasses, getClasses, normClassificationResults, tokenize
public SimpleNaiveBayesDocumentClassifier(IndexReader indexReader, Query query, java.lang.String classFieldName, java.util.Map<java.lang.String,Analyzer> field2analyzer, java.lang.String... textFieldNames)
indexReader
- the reader on the index to be used for classificationquery
- a Query
to eventually filter the docs used for training the classifier, or null
if all the indexed docs should be usedclassFieldName
- the name of the field used as the output for the classifier NOTE: must not be heavely analyzed
as the returned class will be a token indexed for this fieldtextFieldNames
- the name of the fields used as the inputs for the classifier, they can contain boosting indication e.g. title^10public ClassificationResult<BytesRef> assignClass(Document document) throws java.io.IOException
DocumentClassifier
Document
assignClass
in interface DocumentClassifier<BytesRef>
document
- a Document
to be classified. Fields are considered features for the classification.ClassificationResult
holding assigned class of type T
and scorejava.io.IOException
- If there is a low-level I/O error.public java.util.List<ClassificationResult<BytesRef>> getClasses(Document document) throws java.io.IOException
DocumentClassifier
Document
.getClasses
in interface DocumentClassifier<BytesRef>
document
- a Document
to be classified. Fields are considered features for the classification.ClassificationResult
, the classes and scores. Returns null
if the classifier can't make lists.java.io.IOException
- If there is a low-level I/O error.public java.util.List<ClassificationResult<BytesRef>> getClasses(Document document, int max) throws java.io.IOException
DocumentClassifier
max
classes (sorted by score, descending) assigned to the given text String.getClasses
in interface DocumentClassifier<BytesRef>
document
- a Document
to be classified. Fields are considered features for the classification.max
- the number of return list elementsClassificationResult
, the classes and scores. Cut for "max" number of elements. Returns null
if the classifier can't make lists.java.io.IOException
- If there is a low-level I/O error.private java.util.List<ClassificationResult<BytesRef>> assignNormClasses(Document inputDocument) throws java.io.IOException
java.io.IOException
private void analyzeSeedDocument(Document inputDocument, java.util.Map<java.lang.String,java.util.List<java.lang.String[]>> fieldName2tokensArray, java.util.Map<java.lang.String,java.lang.Float> fieldName2boost) throws java.io.IOException
inputDocument
- the seed unseen documentfieldName2tokensArray
- a map that associated to a field name the list of token arrays for all its valuesfieldName2boost
- a map that associates the boost to the fieldjava.io.IOException
- If there is a low-level I/O errorprotected java.lang.String[] getTokenArray(TokenStream tokenizedText) throws java.io.IOException
TokenStream
in inputtokenizedText
- the tokenized content of a fieldString
array of the resulting tokensjava.io.IOException
- If tokenization fails because there is a low-level I/O errorprivate double calculateLogLikelihood(java.lang.String[] tokenizedText, java.lang.String fieldName, Term term, int docsWithClass) throws java.io.IOException
tokenizedText
- the tokenized content of a fieldfieldName
- the input field nameterm
- the Term
referring to the class to calculate the score ofdocsWithClass
- the total number of docs that have a classjava.io.IOException
- If there is a low-level I/O errorprivate double getTextTermFreqForClass(Term term, java.lang.String fieldName) throws java.io.IOException
term
- the class termjava.io.IOException
- If there is a low-level I/O errorprivate int getWordFreqForClass(java.lang.String word, java.lang.String fieldName, Term term) throws java.io.IOException
word
- the token produced by the analyzerfieldName
- the field the word is coming fromterm
- the class termjava.io.IOException
- If there is a low-level I/O errorprivate double calculateLogPrior(Term term, int docsWithClassSize) throws java.io.IOException
java.io.IOException
private int docCount(Term term) throws java.io.IOException
java.io.IOException