The redesign of our database The Backlink Analysis tool is one of the oldest features developed by SEMrush…and it deserved a complete upgrade to keep up with the performance of our SEO toolbox.
A year and a half ago, we set ourselves the goal of setting the bar even higher, to help you optimize your link building in the best possible way.
SEMrush is a complete toolkit dedicated to digital marketing professionals, particularly effective in SEO. Helping professionals generate organic traffic from search engines to their content has always been our main goal. It has led us to develop a world-renowned SEO suite, which wins numerous awards every year.
SEO is a delicate The redesign of our database matter:
the criteria evolve, are complex and often closely linked. To obtain the best rankings, many steps are necessary, both on-site and off-site. It is precisely throughout this winding process that we provide our users with the best solutions.
And to stay on top, we are constantly working on improving our tools. We are proud to share with you our latest success!
Recognizing that The redesign of our database the quality of our
backlink database needed to be improved, we have completely overhauled our data collection process. To achieve this goal, we have suspended the development of all other backlink features, and have compiled a list of all the elements that would improve the backlink database available to our clients.
List in hand, all that was left was to get started, until all the boxes were checked.
No need to get bogged down in the technical details of our backlink database overhaul … So here’s a quick overview of our roadmap:
Crawler
After carefully studying the drawbacks and overseas data limitations of the existing architecture, we decided to completely reprogram our crawler, starting from scratch, using a completely different approach to data collection.
Queue
The first tests of our new crawler robot, and their queues of pending URLs, revealed to us that we were not yet able to properly process the amount of data collected. First solution: simply increase our hardware capacities. But that was not enough. We then developed a more efficient queue.
Seeding
To provide our crawler with a quality initial list, we queued all the URLs in Google’s top 100 for 450 million keywords from our Organic Search tool. A guarantee of relevance of our database.
Storage
Increased data collection necessarily means taiwan lead increased storage space. We had to quadruple the size of our server.
Evaluating our progress
To accurately assess our performance as backlink providers, we decided to measure ourselves against the best: Majestic, Ahrefs and Moz.
Before presenting the methodology applied, here are the results obtained over the last six months. The comparison with the leading SEO tools on the market confirms that we have made a giant step forward.
How did we compare our results?
Finding a methodology that is both clear and objective was not an easy task. You will always find domains that will show your backlink tool in its best light! That is why we decided to use a random set of 100 domains each month (chosen out of 100,000), in order to analyze the performance of our competitors over the last 6 months.
Each time, we looked at the number of referring domains and the total number of backlinks for each competitor, out of the 100 domains. We also compared SEMrush’s results to the others for each domain in our test sample: a ratio lower than 1 means that SEMrush has less information about the site under test; a ratio higher than 1 indicates how much better SEMrush’s result is.
Finally, to obtain a final score, we took the median of all these indicators.
Conclusion
The results obtained by our new database are a real embed and inbound links success for us. But these are just numbers, let’s not forget that quantity of data does not necessarily mean quality. That’s why we are doubling our efforts to keep our database useful and up to date.
Next up: Using this new collection process to design new features and capabilities to make our tools even more powerful. Stay tuned for more!
See for yourself and test