Batch URL Checking: Automating Website Monitoring

When you have a single site with a dozen pages, manually checking HTTP headers and response codes is no problem. But what do you do when you need to check hundreds of URLs after a migration, a new site launch, or a mass content update? массовую проверку URL URL checking lets you automate this process and quickly find issues.

When You Need Batch Checking

Site Migration

When migrating a site to a new CMS, changing domains, or restructuring URLs, it is critical to ensure that all old addresses correctly redirect to the new ones. A single missed page means lost traffic and degraded SEO.

A typical migration scenario:

1. Export the list of all URLs from the old site (from sitemap.xml or a crawler)
2. Check each URL for proper redirect (301 to the new address)
3. Verify all new URLs are accessible (status code 200)
4. Find and fix errors (404, 500, incorrect redirects)

SEO Audit

Regular auditing of all site pages helps identify:

• Pages returning 4xx and 5xx errors
• Unintended redirects
• Pages missing proper caching headers
• Missing security headers
• Slow pages (high TTFB)

Competitor Monitoring

Analyzing competitors' HTTP headers reveals their technology stack: web server, CDN, framework, and caching configuration. Batch checking multiple competitors allows you to gather this information quickly.

Post-Deployment Verification

After every deployment, it's useful to check key pages: homepage, conversion pages, API документацию endpoints. Automated checking in a CI/CD pipeline helps catch issues before users notice them.

Batch Checking Tools

enterno.io Batch URL Checker

Batch URL checking on enterno.io lets you check up to 20 URLs simultaneously right in the browser. For each URL, you'll get:

• HTTP response code
• Server response time (TTFB)
• Response headers (Server, Content-Type, Cache-Control, and more)
• Redirect information
• SSL status

Results can be exported to CSV or JSON for further analysis.

curl and xargs

For checking a large list of URLs from the command line:

# Check a list of URLs from a file
cat urls.txt | xargs -I {} -P 10 curl -sI -o /dev/null -w "%{http_code} %{time_total}s %{url_effective}\n" {}

# With output to a file
cat urls.txt | xargs -I {} -P 10 curl -sI -o /dev/null -w "%{http_code},%{time_total},%{redirect_url},%{url_effective}\n" {} > results.csv

The -P 10 flag runs up to 10 parallel requests, speeding up the check.

Python Scripts

For more complex logic, use Python with aiohttp (async requests) or requests:

import aiohttp
import asyncio

async def check_url(session, url):
    try:
        async with session.head(url, allow_redirects=False, timeout=10) as resp:
            return {
                "url": url,
                "status": resp.status,
                "headers": dict(resp.headers)
            }
    except Exception as e:
        return {"url": url, "error": str(e)}

async def check_all(urls):
    async with aiohttp.ClientSession() as session:
        tasks = [check_url(session, url) for url in urls]
        return await asyncio.gather(*tasks)

# Usage
urls = open("urls.txt").read().splitlines()
results = asyncio.run(check_all(urls))

Screaming Frog SEO Spider

A desktop application for full site crawling. It finds all pages and checks response codes, redirects, meta tags, and headers. The free version is limited to 500 URLs.

API Approach

For integration into CI/CD or automated monitoring systems, use an API. Many services, including enterno.io, provide an API for programmatic URL checking.

What to Check

HTTP Response Code

The basic check — make sure all pages return the expected code:

Caching Headers

Ensure all pages have proper Cache-Control headers. Static resources should be cached for a long time, HTML — with revalidation. For more details, read our article on analyzing server response headers.

Security Headers

Verify the presence of key security headers on all pages:

• Strict-Transport-Security
• X-Content-Type-Options
• X-Frame-Options or CSP frame-ancestors
• Content-Security-Policy

Response Time

Each page's TTFB should be under 200–500 ms. Pages with a TTFB over 1 second need optimization. Use the enterno.io HTTP header checker to measure response time.

Redirect Chains

Every URL should redirect directly to the final destination without intermediate hops. Chains of 2+ redirects slow down page loads and weaken SEO. For more details, see our article on redirect chains.

Automating Checks

CI/CD Integration

Add URL checking to your deployment pipeline:

# Example for GitHub Actions
- name: Check critical URLs
  run: |
    URLS="https://example.com https://example.com/about https://example.com/contact"
    for url in $URLS; do
      STATUS=$(curl -sI -o /dev/null -w "%{http_code}" "$url")
      if [ "$STATUS" != "200" ]; then
        echo "FAIL: $url returned $STATUS"
        exit 1
      fi
    done

Scheduled Regular Checks

Set up a cron job or GitHub Actions schedule for weekly checking of all URLs from sitemap.xml:

1. Download and parse sitemap.xml
2. Check each URL
3. Send a report to Telegram, Slack, or email
4. If errors are found — create a ticket automatically

Post-Migration Monitoring

After a site migration, set up daily checking of the old URL list for one month. Ensure all redirects work correctly and search engines have discovered the new addresses.

Analyzing Results

Tabular Report

Export results to CSV with these columns:

• URL
• HTTP code
• Final URL (after redirects)
• Number of redirects
• TTFB (ms)
• Server
• Content-Type
• Cache-Control

The enterno.io batch checking tool offers one-click CSV and JSON export.

Grouping by Issue

Group results by issue type to prioritize fixes:

1. Critical: 5xx errors — server failures, require immediate attention
2. High priority: 404 on important pages — traffic loss and SEO impact
3. Medium priority: redirect chains — slowdown and link juice loss
4. Low priority: missing security headers — gradual improvement

Limitations and Best Practices

• Rate limiting — don't send too many simultaneous requests to a single server. 5–10 parallel requests is a reasonable limit
• User-Agent — specify a proper User-Agent so your requests aren't blocked by a WAF
• Timeouts — set a reasonable timeout (10–30 seconds) so one slow page doesn't block the entire check
• HEAD vs GET — for status checking, a HEAD request is sufficient; it's faster and doesn't download the response body
• Retries — if a URL returns an error, retry after a minute before considering the issue confirmed

Batch Checking Checklist

1. Gather the complete URL list (sitemap.xml, crawler, database)
2. Check HTTP status codes for all URLs
3. Ensure redirects point to the correct addresses
4. Verify there are no redirect chains
5. Measure TTFB to identify slow pages
6. Check caching and security headers
7. Export results for analysis and reporting
8. Set up regular automated checks