The Language module uses heuristic language identifier token table to recognize the most probable language of the linked page content. If no clear language identifier recognized, it uses Hunspell dictionaries for Serbian (Goran Rakić, Igor Nestorović, 2013)⁠ and English (US) (Kevin, 2017)⁠ with TF-IDF term weighting schema to estimate the language.

The Language Module inherits the Frontier Layer Module base class and declares three layers:
• M01.L01: The Primary (the surface layer)
• M01.L02: The Secondary (the 1st layer)
• M01.L03: The Reserve (the 2nd layer)

Execution flow of the Frontier Ranking Algorithm – the crawler design with all four modules SM-LTSD

The Targets are distributed into layers by nine Passive rules followed by two Active, all testing the Target Semantic Terms collection (TST). The collection is created from both URL path (Table 1) and anchor text tokens with several transformation and filtration operations:
• only URL part relative to the domain name is considered
• tokens are extracted by Regex matching alphanumeric chunks
• solely numerical tokens are excluded
• numeric characters in alphanumeric tokens are trimmed
• all tokens are transformed into lower case representation

Table 1: Examples of token extraction from URL path.

The first two rules in the evaluation sequence are checking if identification needle (Table 2) for Serbian and English language is contained in the TST collection and distribute the Targets to the corresponding layer.

Table 2: Needles used to test TST for Serbian and English language identities. For Serbian language additional “yu” identifier (for Yugoslavia, country the Republic of Serbia is successor of) was introduced after observed in crawl logs during debug sessions.

The next seven Passive rules are sending Targets to the Reserve layer if needle of any other language (Table 3) covered by the module is detected.

Table 3: Needles used to test TST for other seven language identities

The last two rules are designed to resolve the Targets without obvious language identification. In context of TF-IDF model all Targets the module received for evaluation (including the ones resolved by first two rules) are treated as documents. Target language weight (T_LW) is calculated as follows:

where Wi is TF-IDF term weight, l has value 1 if the token is recognized by Hunspell dictionary, otherwise the l value is set to -1 and n is number of tokens in TST. If the Target language weight is above 0 the Target is considered positive to the language estimation test. All unassigned targets are distributed to the Reserve layer.