Why Two Indexes? Why Not Have One Big Index?
In a perfect world, the web would be an orderly media with a general index and simple and intuitive paths to identify all resources. In this scenario, the web would have an API that enabled developers to build easy-to-use backlink checkers. The backlink checkers would always reflect an accurate snapshot of the whole web. In reality, the web is a decentralised entity with no API; therefore, there is nowhere to plug in an API. Because of this, companies make backlink indexes.
When our crawlers return with their latest packets of raw data, that information needs to be cleaned, separated, and integrated into our previous index. Then, we need to recalculate all metrics across the index and prepare the data for display. We then calculate our quality metrics: Trust Flow and Citation Flow. Trust Flow, like Page Rank, is a proprietary metric calculated at page level. This is also done for each of our Topical Trust Flow categories.
While having one index would make things easier for our marketing team, a single index requires a considerable amount of time to be built: weeks or months rather than hours or days. Reducing this timeframe would require an insane amount of investment with a direct impact on our incredibly competitive prices. There is also a very practical consideration: Most customers need day-to-day fresh data, so this investment would provide no benefit. It would make no sense to provide fresh data every two weeks or so.
In theory, we could cut corners on our index builds. Perhaps we could make the Trust Flow calculation only have one iteration… but that would reduce its accuracy. Or we could stop pre-preparing the data and do it on demand… but that would make our customers endure load times when they visit a screen. These aren’t options that we wanted to take, therefore, a single index is impractical and costly.