The best way to see all pages of a website is to combine three sources: a site crawler, the XML sitemap, and a website discovery tool that finds pages through backlinks, search results, and technical signals. No single method sees everything. A crawler finds linked pages, a sitemap lists what the site owner wants indexed, and discovery tools uncover pages that are hidden, orphaned, old, or missed by internal links.
TLDR: Use a crawler first, check the XML sitemap second, then run a discovery tool to find anything both methods missed. For example, a small ecommerce site with 1,200 product URLs might show only 930 pages in a crawl, 1,050 in its sitemap, and 87 extra URLs in Google Search Console or a third party index. That gap often reveals broken templates, orphan product pages, duplicate filters, or old campaign pages still getting traffic. If you care about SEO, audits, migrations, or cleanup, compare all three lists before making decisions.
Why “all pages” is harder than it sounds
A website is not always a neat folder of pages. Some pages are linked in menus. Some sit deep in pagination. Some exist only after a search filter is applied. Others are produced by a CMS, blocked by robots.txt, hidden behind forms, or abandoned years ago.
That is why “show me every URL” becomes messy fast. Honestly, it feels like some sites are built to hide their own content by accident. A blog tag page points to ten posts, a script loads more results, a canonical tag points somewhere else, and the sitemap has not been updated since last quarter.
To get close to a full picture, you need to understand what each method can and cannot see.
Method 1: Site crawlers
A site crawler acts like a search engine bot. You give it a starting URL, usually the homepage, and it follows links across the site. It records status codes, titles, meta descriptions, canonicals, headings, word counts, internal links, redirects, and more.
Popular crawling tools include desktop crawlers, cloud based audit platforms, SEO suites, and command line tools. They are especially useful because they show the site as it is actually linked.
Best for:
- Finding pages reachable through internal links
- Spotting broken links, redirects, and 404 pages
- Checking title tags, meta descriptions, headings, and canonicals
- Finding duplicate content and thin pages
- Auditing internal linking depth
Where crawlers fall short:
- They may miss orphan pages with no internal links
- They may not process JavaScript content correctly unless configured
- They can be blocked by robots.txt or firewall rules
- They may stop early if crawl limits are too low
- They may create huge URL lists from filters, tracking parameters, or calendars
A crawler is your best first tool because it shows what visitors and bots can reach through links. If a page is not found in a crawl, that is a useful warning. It might be orphaned, blocked, or only available through an external source.
Method 2: XML sitemaps
An XML sitemap is a file that lists URLs for search engines. It usually lives at a URL such as /sitemap.xml, though many sites use sitemap indexes with several child sitemaps. Large sites may split them by posts, products, categories, images, videos, or languages.
XML sitemaps are not proof that pages are live, valid, or indexable. They are more like a request: “Here are the URLs we want search engines to know about.” Search engines can ignore them. Site owners can also forget to update them.
Best for:
- Finding URLs not linked from the main site
- Checking what the CMS is submitting to search engines
- Auditing indexable pages during a migration
- Finding old posts, product pages, or category URLs
- Comparing submitted URLs against crawled URLs
Common sitemap problems:
- URLs return 404 or 500 errors
- Redirected URLs are still included
- Noindex pages appear in the sitemap
- Canonicalized URLs point to different pages
- The sitemap contains staging, parameter, or test URLs
It drives me crazy that many CMS plugins keep old URLs in sitemaps long after content is removed. In one audit, a sitemap had 8,400 URLs, but 19% redirected and 6% returned errors. That wasted crawl budget and made reporting look worse than it needed to be.
Method 3: Website discovery tools
Website discovery tools find pages using outside data. They may pull URLs from search engine indexes, backlinks, analytics, server logs, browser data, paid SEO indexes, or public archives. These tools are useful because they can uncover URLs that your crawler and sitemap never show.
Think of them as the cleanup crew. They spot pages that still receive impressions, clicks, links, or bot visits even though the current site structure no longer points to them.
Good sources for discovery include:
- Google Search Console performance reports
- Google Search Console indexed pages reports
- Analytics landing page reports
- Server log files
- Backlink tools
- Search operators such as site:example.com
- Web archive snapshots
- Old sitemap files from previous site versions
Discovery tools are not perfect. Search indexes can be stale. Analytics misses pages with broken tracking. Backlink databases may show URLs that no longer exist. Server logs can be noisy and large. Still, these sources are gold when you need to find forgotten URLs before a redesign, migration, or content purge.
How to compare the three URL lists
The smartest workflow is simple. Export URLs from each source, place them in one spreadsheet, normalize them, then compare them.
- Crawl the site from the homepage and key section pages.
- Export all XML sitemap URLs, including child sitemaps.
- Export discovered URLs from search, analytics, logs, and backlink tools.
- Clean the data by removing URL fragments, tracking parameters, and duplicates.
- Mark each source: crawled, sitemap, search, analytics, logs, backlinks.
- Check status codes for every URL.
- Group URLs by page type, folder, template, language, or business value.
This comparison tells a clear story. URLs found in the crawl and sitemap are usually healthy candidates. URLs in the sitemap but not in the crawl may lack internal links. URLs found in search or analytics but not in either source need review. They might be valuable pages, outdated pages, or cleanup targets.
What each mismatch usually means
Crawled but not in sitemap: The page is linked on the site but not submitted. That may be fine for low value utility pages, but not for key content.
In sitemap but not crawled: The page may be orphaned, blocked, buried too deep, or missing from internal links. Check whether it deserves links.
In search results but not crawled or listed: This can reveal old URLs, duplicate versions, forgotten subfolders, or pages with backlinks.
In analytics but not crawled: Users are landing there from ads, bookmarks, email, social posts, or search. Do not delete these without checking value.
In logs but nowhere else: Bots may be hitting old URLs. Some are harmless. Others expose crawl waste, broken redirects, or security oddities.
When each method is enough
For a small brochure website with 30 pages, a crawler plus a sitemap check may be enough. You can manually review the results in under an hour.
For a blog with 2,000 posts, use all three methods. Tags, author pages, pagination, old drafts, and media URLs create surprises.
For ecommerce, do not skip discovery tools. Product variants, filters, discontinued items, faceted URLs, and seasonal landing pages create thousands of possible URLs. Expect to waste time on parameter cleanup if tracking rules were never controlled.
For a migration, collect every URL you can. Then map old URLs to new pages. Missing just 5% of high traffic URLs can hurt rankings, revenue, and paid campaign performance.
Practical tips for better results
- Use more than one starting URL. Crawl the homepage, blog, store, help center, and language versions.
- Enable JavaScript rendering if the site loads links client side.
- Respect robots.txt, but also run a controlled review to see what is blocked.
- Set crawl limits carefully. Too low misses pages. Too high may trap you in filters.
- Remove junk parameters such as campaign tags before counting URLs.
- Check canonical tags before treating duplicates as unique page problems.
- Review traffic and conversions before deleting or redirecting pages.
The best answer: combine, compare, then decide
If you want to see all pages of a website, do not trust one report. A crawler shows reachable structure. An XML sitemap shows submitted intent. Discovery tools show what the outside world still knows about.
The real value comes from the overlap and the gaps. Pages found everywhere are easy to trust. Pages found in only one source deserve attention. That is where hidden SEO issues, lost traffic, poor internal linking, and messy CMS behavior usually sit.
Start with a crawl. Pull the sitemap. Add search, analytics, logs, and backlink data. Then build one master URL inventory. It is not glamorous work, but it gives you the closest thing to a full map of a website.
