It is designed with premise that URL segments from left to right may be interpreted as parent-child hierarchy structure in order to construct factors of FRA influence based on the following assumptions, given in order of dominance:

1. link-path node closer to the root is more relevant than the one positioned deeper in the graph
2. link-path leaf is more relevant then sibling link path node
3. link-path node with greater number of child nodes is more relevant then the one with less.

where the link-path node represents a path folder, link-path leaf represents actual Target having filename in path or being without any minor URL segments (file path, query pairs) targeting a folder’s default index.

The Structure Module inherits the Frontier Layer Module base class and declares three layers:

• M03.L01: The Primary (the surface layer)
• M03.L02: The Secondary (the 1st layer)
• M03.L03: The Reserve (the 2nd layer)

Quantification is facilitated by construction of link-path structural directed graph from a collection of URLs instances. URL of each Link Instance is broken into segments guiding the tree builder trough existing graph nodes or instructing it to build new node for unmatched segment. The graph nodes are defined by name, score and children collection. The score is risen for one point each time the tree builder walks trough it i.e. the node score represents cumulative frequency of children path match against link detected within the known content. In case of Target having two or more Link Instances the node score is risen only once per unique target.

URL structure link-graph

The module use tree graphs to drive the distribution:

• G_T made of the current Targets
• G_C made of the crawled URLs
  • initial node score set to 2
• G_B is a bonus factor score graph, initially emptyand
• G_D built as union of these two with recalculated node score:
  • G_T score is divided by G_C score of the matching node
  • if intersecting nodes between G_B i G_D exist, G_D score is increased with the score from the matching G_B node

In the special case:

• when the G_C graph is empty G_D is clone of G_T.

The G_T and G_D graphs are temporary instances used in scope of one crawling iteration, while G_B and G_C are preserved in the domain level crawling context.

The Targets are distributed into layers by active rules evaluation:

• G_D node with the highest score (G_D^top) is selected

  • if it has the leafs – all are sent to the Primary

  • if it has the child nodes with leafs – all leafs from children are sent to the Secondary

  • the rest of targets goes into the Reserve

• if G_D^top is without leafs but has children nodes with leafs and/or with grandchildren nodes – the score redistribution is performed:

  • the score of G_D^top is equally propagated among the first line children by writing proper positive score into G_B graph

  • the score of G_D^top is neutralized by writing negative score of the same amount to the matching G_B node

In the Structure _rank implementation the module is extended with another active rule providing the link relevance score adopted from the score of G_D node with matching URL.