How To Quickly Find Your Competitors’ Most Popular Products

One of the hardest tasks of any ecommerce business owner or specific ‘buyer’ is to locate the products that will sell the most, whilst we’ll explore the wider opportunities in a future post, the one particular method which deserves an in-depth analysis is using your competitors’ own websites and little known-about tools to highlight which are the most popular products that you could either directly source or get inspiration from.

The primary tool we suggest using is URL Profiler. This will allow you to take a competitor’s sitemap load it into the software and obtain an organised list of the pages which either has the most social shares or links into them – or both.

To find a competitor’s xml sitemap the general rule is to visit their website and add /robots.txt on to the end. This will generally show a list of folders the site owner doesn’t want the search engines to visit as well as a list of ‘robots’ that they would like to exclude from visiting the website. For you, however, the important line would that be the one that highlights where the sitemap can be found. If you search the page for ‘sitemap’ you will find it. The reason we suggest you do it this way is that a website may have multiple sitemaps and that the one you want may not actually be at domain.com/sitemap.xml – but the correct one will nearly always be listed in the robots.txt file.

If, for some reason, the website doesn’t have a robots.txt or sitemap.xml then you can get a list of all their pages using Screaming Frog.

When you have the sitemap URL, load up URLProfiler and right-click on the white space to the right and select “Import from XML Sitemap”:

Stage 1

which will then show you the list of URLs it will analyse. If you wanted to filter some out to make it load quicker then you can open the sitemap.xml in a spreadsheet and just copy out the URLs you would like and then paste into this box from your clipboard – this often happens when a website has seperate desktop and mobile sites:

Stage 2

Depending on your requirements you should then select the URL level “social shares” and, if you have access to the Majestic SEO data, select this option to show you which products/categories are most linked to.

If you require proxy IPs to make it run more quickly make sure you have added those to the settings and then select “Run Profiler” and wait:

Stage 3

And then you open up the resulting file in your spreadsheet software and organise the data by your priority metric. Here we’ve organised by “URL Facebook Likes”:

Sorting the Spreadsheet

Data from Hotel Chocolat shows that their “cocoa fudge” information page is very much the most popular page from the point of this metric which may tie in with a particular marketing activity – which is also likely to be the reason for their second most popular page which is a competition which asks people to either share via Facebook, Twitter or Instagram.

Further down you will see pages which have more organically acquired Facebook likes which would give you a better indication of what works for Hotel Chocolat. For them their “fruity chocolate” collection is popular, followed by Easter eggs, information about their ‘milk free’ milk chocolate, and not until their nutty praline page do we see their most popular products from a social point-of-view.

If you reorganise by the “URL Facebook Total” you can see the list organised by the sum of ‘likes’, ‘shares’ and ‘comments’ which gives a more holistic view of the popularity of a page (but you choose).

You can use this data in three ways. You can look to find which are the most popular collections, departments or categories of products and then establish if you also cover those broad topics and then consider what opportunities you have to stock products in those categories such as Hotel Chocolat’s ‘Vegan’ section.

You can look at the product-level data and establish if there are any popular products that could fall into a category you already have and then try and source that exact product or seek an alternative such as ‘caramel chocolate spread’ in Hotel Chocolat’s case.

And thirdly, you should be scanning down the list of URLs for blog or information pages that people find interesting and ‘share-worthy’ such as their use of competitions to build brand awareness and social sharing.

Whist you have the spreadsheet open and if you have the Majestic data then scan the list for the most linked to content – by the metric ;URL Majestic Ref Domains’ to see which pages are linked to from the widest variety of other websites (roughly). This would highlight pages that the website owner may have forgotten to put social sharing buttons on so the social data may be skewed against it, but it still popular in terms of bloggers or newspapers etc. An example of this is the page about their tours in St. Lucia which are linked to, but don’t have explicit social sharing buttons.

If you wanted to take the logic above to extreme lengths then you could use Scrape Box to download all the URLs on Amazon, for example, which contained ‘chocolate’ and then run URL Profiler on those to get their most popular chocolate products.

If you were in content writing mode you could employ the same tactic on Wikipedia and find the most popular pages about your topic and then factor in information from Wikipedia Trends. You could even use the often cited Google Web Trends to find new product niches or even Terapeak to analyse eBay sales.

The issue is that the amount of data available can be overwhelming. Finding competitors that you already know and understand and then seeing what is most popular for them may give you more actionable data sooner than any of the internet-wide tools available.