Technical SEO 101: From Robots.txt to Faceted Navigation
Posted: September 16, 2025 to Announcements.
Technical SEO Foundations: Robots.txt, XML Sitemaps, Crawl Budget, Canonical Tags, Internal Linking, and Faceted Navigation
Technical SEO turns a site’s architecture into a search-friendly foundation. Before chasing new keywords, ensure bots can reliably discover, interpret, and prioritize your content. The pillars below focus on controlling access, guiding discovery, conserving crawl resources, and consolidating signals—practical steps that prevent index bloat and strengthen rankings across large catalogs, media libraries, and high-change websites.
Robots.txt: Access Control Without Losing Renderability
Use robots.txt to keep crawlers out of non-content areas while allowing rendering resources. Block login, cart, and internal search URLs; avoid blocking CSS/JS needed to load pages. For example, a retailer might disallow /cart/ and /search?q= while allowing /products/. Test disallows with a robots tester, and verify with server logs that important URLs are still crawled.
XML Sitemaps: Structured Discovery
An XML sitemap lists canonical URLs you want indexed, with lastmod dates to hint freshness. Segment by type (e.g., /blog/, /products/, /videos/) to monitor coverage. A news publisher can ping updated sitemaps after major stories to accelerate discovery. Keep only indexable, non-noindexed URLs; remove 404s, redirects, and parameter variants to avoid wasting crawl.
Crawl Budget: Make Every Fetch Count
Crawl budget is the number of URLs a bot will fetch on your site within a period, shaped by site health and importance. Improve it by speeding up responses, consolidating duplicates, fixing chains/404s, and limiting infinite spaces. One marketplace trimmed 30% duplicative filters and saw Googlebot hits shift from thin variants to revenue pages.
Canonical Tags: Consolidate Signals
Canonical tags signal the preferred URL when duplicates exist. Use a self-referential canonical on each indexable page and point all tracked variants (UTM parameters, case changes, sort orders) to the canonical. Cross-domain canonicals help with syndicated articles. Ensure canonical and internal links align; mixed signals (e.g., canonical to A but linking to B) dilute consolidation.
Internal Linking: Build Topic Hubs and Short Paths
Strategic internal links distribute authority and clarify topical relationships. Use descriptive anchor text and keep important pages within three clicks of the homepage. Build hub pages that link to child topics and vice versa. A SaaS company boosted docs traffic by surfacing “related guides” modules, raising crawl frequency and ranking for long-tail troubleshooting queries.
Faceted Navigation: Taming Combinatorial Explosions
Facets (filters for color, size, price) create combinatorial URLs that can explode index counts. Default to index a clean category page; for selected facets, use canonical to the base, add meta robots noindex on thin combinations, and avoid linking to infinite ranges. Many retailers render secondary facets via AJAX to serve users without exposing crawl traps.