rdf_tokenize_literal(+Literal,
-Tokens)Tokenize a literal, returning a list of atoms and integers in the range
-1073741824 ... 1073741823. As tokenization is in general
domain and task-dependent this predicate first calls the hook
rdf_litindex:tokenization(Literal, -Tokens). On failure it
calls tokenize_atom/2
from the NLP package and deletes the following: atoms of length 1,
floats, integers that are out of range and the english words and, an, or, of,
on, in, this and the.
Deletion first calls the hook rdf_litindex:exclude_from_index(token,
X). This hook is called as follows:
no_index_token(X) :-
exclude_from_index(token, X), !.
no_index_token(X) :-
...