Technical SEO Audit: Crawling, Loading Speed & Core Web Vitals

Technical SEO Audit: Crawling, Loading Speed & Core Web Vitals

Technical SEO forms the foundation of your website's search visibility. Without proper crawlability, fast loading speeds, and solid Core Web Vitals, even the best content struggles to rank. A technical SEO audit identifies the hidden issues preventing search engines from properly accessing, understanding, and ranking your pages.

In this comprehensive guide, we'll walk through every aspect of a technical SEO audit – from ensuring search engines can crawl your site to optimizing Core Web Vitals and implementing structured data.

Why Technical SEO Matters

Technical SEO ensures that search engines can effectively crawl your website and discover all pages, index your content in their search databases, render your pages correctly, and rank your content based on user experience signals. When any of these processes fail, your carefully crafted content never reaches your target audience.

The numbers paint a compelling picture. Pages that load in under 2.5 seconds have significantly higher conversion rates, while 53% of mobile users abandon sites that take longer than 3 seconds to load. Core Web Vitals are confirmed ranking factors for Google Search, and technical issues block an estimated 30% of websites from reaching their ranking potential. A technical SEO audit uncovers these issues before they damage your rankings.

Tools for Your Technical SEO Audit

Essential Free Tools

Several powerful free tools form the backbone of any technical SEO audit. Google Search Console provides invaluable data on indexing status, crawl errors, and Core Web Vitals performance directly from Google's perspective. Google PageSpeed Insights analyzes your pages and offers specific performance metrics alongside optimization suggestions. Rank Chat leverages AI to analyze your Search Console data and surface insights through natural conversation. The Lighthouse tool built into Chrome DevTools runs comprehensive audits covering performance, accessibility, and SEO best practices. Finally, the Google Rich Results Test validates your structured data implementation to ensure you're eligible for enhanced search appearances.

Professional Tools (Optional)

For deeper analysis, professional tools offer additional capabilities. Screaming Frog SEO Spider excels at deep crawling and comprehensive technical analysis of large sites. Sitebulb provides visual crawl reports with prioritized recommendations that help you focus on high-impact fixes. GTmetrix offers detailed performance waterfall analysis showing exactly where time is spent during page loads. WebPageTest enables advanced loading performance testing from multiple locations and connection types.

Phase 1: Crawlability Audit

Understanding Crawlability

Crawlability determines whether search engine bots can access and navigate your website. If Googlebot cannot crawl your pages, they will never appear in search results. This makes crawlability the most fundamental aspect of technical SEO – everything else depends on it.

Check Your Robots.txt File

Your robots.txt file controls crawler access to your site. You can access it by visiting yoursite.com/robots.txt in your browser. A properly configured robots.txt allows crawlers to access important content while blocking administrative areas and private sections.

# Access your robots.txt at: yoursite.com/robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /checkout/
Disallow: /account/

Sitemap: https://yoursite.com/sitemap.xml

When auditing your robots.txt, verify that the file is accessible at your root domain and that important pages are not accidentally blocked. Administrative and private areas should be blocked appropriately, and your sitemap URL should be included using the Sitemap directive. Watch out for accidental wildcard blocks that might exclude more content than intended.

Common robots.txt mistakes can have serious consequences. Blocking CSS and JavaScript files causes rendering issues because search engines cannot see your page as users do – allow the /css/ and /js/ directories. Blocking entire subdirectories prevents those pages from being indexed, so use specific paths instead. Missing a sitemap reference slows down page discovery, while inconsistent trailing slash usage can cause duplicate crawling issues.

Analyze XML Sitemap

Your XML sitemap guides search engines to your important pages. A well-maintained sitemap includes only canonical, indexable URLs and stays under the limits of 50,000 URLs or 50MB per sitemap file. For larger sites, use sitemap index files to organize your sitemaps hierarchically.

Keep your sitemap current by updating lastmod dates when content changes and removing URLs that return non-200 status codes. You can validate your sitemap's accessibility and XML structure using command-line tools:

# Check sitemap accessibility
curl -I https://yoursite.com/sitemap.xml

# Validate XML structure
xmllint --noout sitemap.xml

In Google Search Console, navigate to "Indexing" > "Sitemaps" to submit your sitemap URL. Monitor the report for errors and warnings, and compare the "Discovered URLs" count against your indexed page count to identify potential indexing gaps.

Identify Crawl Errors

Google Search Console provides detailed crawl statistics under "Settings" > "Crawl stats". Review the crawl requests and responses, paying particular attention to any 4xx or 5xx errors. The "By response" view shows the distribution of different response codes across your site.

A 404 Not Found error means the page doesn't exist – fix this by implementing redirects for important URLs or removing internal links pointing to missing pages. A 410 Gone status indicates a page was permanently removed and should be removed from your sitemap. 500 Server Error responses indicate server-side issues that require checking your server logs for the root cause. 503 Unavailable responses signal temporary downtime, so ensure your server stability if these occur frequently.

Check Crawl Budget

For large sites with 100,000+ pages, crawl budget becomes an important consideration. Google allocates a finite amount of resources to crawl each site, so you want to ensure those resources are spent on your most valuable pages.

Optimize your crawl budget by blocking low-value pages such as filtered views, sort variations, and deep pagination. Reduce duplicate content that wastes crawl resources on redundant pages. Improve server response times so crawlers can access more pages in less time. Fix redirect chains that force crawlers to make multiple requests to reach a single page, and remove soft 404 pages that waste crawl budget on effectively empty content.

Phase 2: Indexing Audit

Check Indexing Status

The simplest way to check indexing status is with a site query in Google. Searching for site:yoursite.com shows all indexed pages, which you can compare against your actual indexable page count.

For detailed indexing information, Google Search Console provides the definitive view. Navigate to "Indexing" > "Pages" to review the counts of indexed versus not indexed pages. The report breaks down exclusion reasons, helping you prioritize fixes for high-value pages that should be indexed but aren't.

Common Indexing Issues

Understanding why pages aren't indexed is crucial for fixing the right problems. When Google reports "Noindex detected," your page has a meta robots tag blocking indexing – remove the noindex directive if the page should be indexed. Pages "Blocked by robots.txt" require updating your robots.txt to allow access.

"Duplicate without canonical" issues arise when multiple URLs serve similar content without proper canonical tags pointing to the preferred version. "Soft 404" pages appear to exist but have thin content that Google considers effectively empty – add substantial content or return an actual 404 status. Redirect issues occur when 301 or 302 redirects are in place – verify the target URL is correct and indexable. Perhaps most frustrating, "Discovered - not indexed" means Google found the page but chose not to index it, typically due to low quality signals – improve the content and build internal links to demonstrate value.

Validate Meta Robots Tags

Ensure that pages you want indexed don't contain blocking directives. The following meta tags prevent indexing and should not appear on pages you want in search results:

<!-- Should NOT appear on pages you want indexed -->
<meta name="robots" content="noindex">
<meta name="robots" content="noindex, nofollow">
<meta name="googlebot" content="noindex">

With Rank Chat, you can quickly audit these issues by asking questions like "Which pages have noindex tags?" or "Which important pages are not indexed?" to get AI-powered analysis of your Search Console data.

Canonical Tag Audit

Canonical tags tell search engines which URL version to index when multiple URLs serve similar or identical content. Every page should include a canonical tag pointing to its preferred URL:

<link rel="canonical" href="https://yoursite.com/page/" />

When auditing canonicals, verify that every page has a canonical tag and that each canonical points to the correct URL. Maintain consistency with HTTPS and www/non-www preferences throughout your site. Unique pages should have self-referencing canonicals pointing to themselves. Watch for conflicting signals like a noindex tag combined with a canonical pointing elsewhere, as this sends mixed messages to search engines.

Phase 3: Site Speed Optimization

Why Speed Matters

Loading speed directly impacts every aspect of your site's success. Slow pages frustrate users, leading to higher bounce rates and lower engagement. Core Web Vitals scores suffer on slow pages, and since site speed is a confirmed ranking factor, sluggish performance can hurt your search visibility. Perhaps most importantly, conversion rates drop significantly as load times increase.

Measure Current Performance

Start by measuring your current performance with Google PageSpeed Insights. Enter your URL at pagespeed.web.dev and review scores for both mobile and desktop experiences. Focus on the "Opportunities" section for high-impact improvements and "Diagnostics" for additional issues. Prioritize fixes that offer the largest potential gains.

For local testing, Lighthouse in Chrome DevTools provides comprehensive performance analysis. Run it from the command line or through the Lighthouse tab in DevTools (F12):

# Run from command line
lighthouse https://yoursite.com --only-categories=performance

# Or use Chrome DevTools (F12 > Lighthouse tab)

Server Response Time (TTFB)

Time to First Byte (TTFB) measures how quickly your server responds to requests and should be under 600ms for optimal performance. High TTFB indicates server-side bottlenecks that affect every page load.

Improve TTFB by upgrading your hosting infrastructure to faster servers or managed platforms. Implement server-side caching to serve repeated requests without regenerating content. Use a Content Delivery Network (CDN) to serve content from locations closer to your users. Optimize database queries that slow down dynamic page generation, and enable HTTP/2 or HTTP/3 for more efficient connection handling.

Image Optimization

Images often account for more than half of total page weight, making them a primary optimization target. Modern image formats like WebP and AVIF provide superior compression compared to JPEG and PNG while maintaining visual quality. Responsive images with srcset attributes serve appropriately sized images based on the user's device and viewport.

Lazy loading defers the loading of below-the-fold images until they're needed, speeding up initial page load. Always compress images without introducing visible quality loss, and specify width and height attributes to prevent layout shifts as images load.

<img
  src="image-800.webp"
  srcset="image-400.webp 400w,
          image-800.webp 800w,
          image-1200.webp 1200w"
  sizes="(max-width: 600px) 400px, 800px"
  width="800"
  height="600"
  loading="lazy"
  alt="Descriptive alt text"
/>

JavaScript and CSS Optimization

Render-blocking resources delay the initial paint of your page, frustrating users who see blank screens. Defer non-critical JavaScript so it loads after the main content is visible. Inline critical CSS required for above-the-fold content while loading the rest asynchronously. Remove unused CSS that bloats your stylesheets. Minify and compress all assets to reduce transfer sizes, and use code splitting to break large JavaScript bundles into smaller, on-demand chunks.

<!-- Defer non-critical scripts -->
<script src="analytics.js" defer></script>

<!-- Async for independent scripts -->
<script src="tracking.js" async></script>

Enable Caching

Browser caching allows returning visitors to load resources from their local cache instead of downloading them again. Configure appropriate cache durations based on how frequently different resource types change. Static assets like images and versioned CSS/JavaScript files can be cached for a year since file versioning ensures updates are served when needed. HTML pages should have short cache durations or no caching to ensure users see fresh content. Fonts rarely change and can be cached for a year. API responses vary based on the data freshness requirements of your application.

Cache-Control: max-age=31536000, immutable

Phase 4: Core Web Vitals

Understanding Core Web Vitals

Core Web Vitals measure real-world user experience through three key metrics. These aren't just performance numbers – they represent how users actually perceive your site.

Check Your Core Web Vitals

Google Search Console provides Core Web Vitals data based on real user experiences. Navigate to "Experience" > "Core Web Vitals" to review performance for mobile and desktop separately. Click into problem URLs to see specific issues, and track improvements over time as you implement fixes.

Optimize Largest Contentful Paint (LCP)

LCP measures how quickly the main content of your page becomes visible – typically the largest image or text block in the viewport. Users perceive pages as "loaded" when this element appears, making LCP crucial for perceived performance.

Slow server response times directly delay LCP, so improving TTFB and using a CDN helps significantly. Render-blocking resources prevent the browser from rendering content, so defer JavaScript and inline critical CSS. Slow resource loading delays the LCP element itself – preload key resources and optimize images. Client-side rendering frameworks may defer content rendering to the browser, so consider server-side rendering (SSR) or static site generation (SSG) for faster initial loads. Large hero images are common LCP elements and benefit from optimization and preloading.

<link rel="preload" as="image" href="hero-image.webp">

Optimize Interaction to Next Paint (INP)

INP measures how responsive your page is to user interactions like clicks, taps, and key presses. Poor INP means users click buttons and nothing seems to happen, creating a frustrating experience.

Improve INP by breaking up long JavaScript tasks that block the main thread. Heavy computations should run in web workers instead of the main thread. Optimize event handlers to respond quickly and defer non-critical work. Reduce overall main thread work by removing unnecessary JavaScript, and use requestIdleCallback to schedule non-critical work during browser idle periods.

Fix Cumulative Layout Shift (CLS)

CLS measures visual stability – how much elements move around as the page loads. High CLS creates frustrating experiences where users accidentally click the wrong element because content shifted at the last moment.

Prevent layout shifts by always including width and height attributes on images and videos, allowing the browser to reserve the correct space before assets load. Reserve space for ads, embeds, and other dynamic content with CSS that defines minimum dimensions. Avoid inserting content above existing content, which pushes everything down. Use CSS containment for areas with dynamic content, and preload fonts to prevent Flash of Invisible Text (FOIT) or Flash of Unstyled Text (FOUT) that can cause text-related shifts.

/* Reserve space for dynamic content */
.ad-container {
  min-height: 250px;
}

Phase 5: Mobile Optimization

Mobile-First Indexing

Google primarily uses the mobile version of your site for indexing and ranking, making mobile optimization essential rather than optional. Your mobile site must provide the same content and functionality as your desktop version to rank well.

A mobile-optimized site implements responsive design that adapts to any screen size. Text should be readable without zooming, requiring a base font size of 16px or larger. Tap targets like buttons and links need at least 48x48 pixels of touch area with sufficient spacing between interactive elements to prevent accidental taps. No horizontal scrolling should be required, and content parity with desktop ensures mobile users get the full experience.

Test Mobile-Friendliness

Google's Mobile-Friendly Test provides a quick assessment of your pages. Enter your URL to see any issues Google identifies, and test multiple page types since different templates may have different issues.

Common mobile problems include missing viewport configuration, which causes pages to display zoomed out on mobile devices – add the proper viewport meta tag. Content wider than the screen forces horizontal scrolling, typically caused by CSS width issues that need fixing. Small tap targets frustrate users trying to interact with your site – increase button and link sizes. Blocked CSS or JavaScript resources prevent proper rendering – ensure crawlers can access these files.

Viewport Configuration

The viewport meta tag tells browsers how to scale your page on mobile devices. This single line should appear in the head of every page:

<meta name="viewport" content="width=device-width, initial-scale=1">

Phase 6: HTTPS and Security

SSL/TLS Certificate

HTTPS is both a ranking signal and a user trust factor. Visitors expect secure connections, and browsers prominently warn users about insecure sites. Your SSL certificate must be valid, not expired, and cover all subdomains you use. Strong cipher suites should be enabled with TLS 1.2 or higher required – older protocols have known vulnerabilities.

Redirect HTTP to HTTPS

Ensure all HTTP requests redirect to HTTPS with a permanent 301 redirect. This can be configured in your .htaccess file for Apache servers:

# .htaccess redirect
RewriteEngine On
RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Fix Mixed Content

Mixed content occurs when HTTPS pages load resources over insecure HTTP connections, causing security warnings in browsers. Find mixed content issues by opening Chrome DevTools (F12), going to the Console tab, and looking for "Mixed Content" warnings.

Fix mixed content by updating internal links to use HTTPS, using protocol-relative URLs like //example.com when linking to external resources that support both protocols, and updating any hardcoded HTTP references in your code, database, or CMS.

Security Headers

Security headers provide additional protection against common web vulnerabilities. Implement these headers on your server:

Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Content-Security-Policy: default-src 'self'
X-Content-Type-Options: nosniff
X-Frame-Options: DENY

Phase 7: Structured Data

Why Structured Data Matters

Structured data helps search engines understand your content beyond what's visible on the page, enabling rich results that enhance your appearance in search results. Enhanced SERP appearances with star ratings, prices, FAQ accordions, and other rich elements stand out from plain text results and typically earn higher click-through rates. Structured data also improves voice search compatibility and helps search engines better understand the relationships between concepts on your page.

Common Schema Types

Different content types benefit from different schema markup. Here are examples of the most commonly used types:

Organization/LocalBusiness schema identifies your company or business:

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Your Company",
  "url": "https://yoursite.com",
  "logo": "https://yoursite.com/logo.png"
}

BreadcrumbList schema helps search engines understand your site hierarchy:

{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [{
    "@type": "ListItem",
    "position": 1,
    "name": "Home",
    "item": "https://yoursite.com/"
  }]
}

Article/BlogPosting schema is essential for blog content:

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Article Title",
  "datePublished": "2026-05-11",
  "author": {
    "@type": "Person",
    "name": "Author Name"
  }
}

Validate Structured Data

The Google Rich Results Test validates your structured data implementation. Enter your URL or paste a code snippet to check for errors and warnings. The tool also previews how your rich results may appear in search, helping you verify that your markup produces the desired effect.

Common structured data errors include missing required fields that the schema specification mandates – add all required properties. Invalid date formats cause parsing failures – use ISO 8601 format (YYYY-MM-DD). URL mismatches between the schema and actual page URL confuse search engines – update to the correct URL. Unsupported schema types that Google doesn't process should be replaced with supported alternatives.

Phase 8: URL Structure and Redirects

URL Best Practices

Clean, descriptive URLs help both users and search engines understand your content. The ideal URL structure uses lowercase letters with hyphens between words, includes descriptive and keyword-relevant terms, stays short and memorable, and maintains consistent trailing slash usage:

https://yoursite.com/category/descriptive-page-name/

Avoid underscores between words (use hyphens instead), unnecessary parameters, and excessively long URLs that get truncated in search results.

Audit Redirects

Redirect chains occur when one redirect leads to another, forcing browsers and crawlers to make multiple requests to reach the final destination. Each hop in the chain adds latency and can lose link equity:

Page A → Page B → Page C (bad)
Page A → Page C (good)

Fix redirect chains by updating the original redirect to point directly to the final destination. Remove any intermediate redirects that are no longer necessary, and update internal links to point to final URLs so redirects aren't needed at all.

Handle URL Parameters

URL parameters like sort orders, filters, and tracking codes can create duplicate content issues. In Google Search Console under "Settings" > "URL Parameters," you can specify how parameters should be handled and tell Google which parameters don't change the page content, helping prevent duplicate content issues.

Technical SEO Audit Checklist Summary

A complete technical SEO audit touches many areas. For crawlability, verify your robots.txt is correctly configured, your XML sitemap is valid and submitted, no critical crawl errors exist, and crawl budget is optimized for large sites.

For indexing, ensure important pages are indexed, no accidental noindex tags block content, canonical tags are properly implemented, and no duplicate content issues exist.

Performance requires TTFB under 600ms, optimized images, minimized JavaScript and CSS, and proper caching configuration.

Core Web Vitals targets are LCP under 2.5 seconds, INP under 200ms, and CLS under 0.1.

Mobile optimization means implementing responsive design, passing Google's mobile-friendly test, and maintaining content parity with desktop.

Security demands HTTPS enablement, no mixed content warnings, and properly configured security headers.

Structured data should be implemented correctly, pass the Rich Results Test, and contain no validation errors.

Conclusion

A thorough technical SEO audit reveals the foundation issues holding back your search performance. By systematically checking crawlability, indexing, site speed, Core Web Vitals, mobile optimization, security, and structured data, you create the technical infrastructure that allows your content to reach its ranking potential.

Remember: technical SEO is not a one-time task. Regular audits ensure new issues are caught early and your site maintains optimal performance as it grows.

Ready to streamline your technical SEO audit? Sign up for Rank Chat and leverage AI-powered analysis of your Google Search Console data. Ask questions like "Which pages have crawl errors?" or "Where are my Core Web Vitals failing?" – and get actionable insights instantly!

---

Have questions about technical SEO audits? Reach out at sascha@rank-chat.com