The front-end playbook for discoverability — rendering for crawlers and AI bots, JSON-LD, metadata APIs, AI crawler access, llms.txt, and tracking AI referrals

Implementation & Measurement

This is the hands-on front-end layer: how to actually ship discoverability and prove it's working. Concepts live in the other pages — SEO Foundations, AEO, GEO.

1. Render so machines can read you

The single highest-leverage decision: make important content present in the initial HTML. AI crawlers in particular are often weaker at executing JavaScript than Googlebot.

// Next.js App Router — server components render to HTML by default.
// Generate metadata per route (title, description, OG, canonical).
import type { Metadata } from 'next';

export async function generateMetadata({ params }): Promise<Metadata> {
  const post = await getPost(params.slug);
  return {
    title: post.title,
    description: post.excerpt,
    alternates: { canonical: `https://example.com/blog/${post.slug}` },
    openGraph: {
      title: post.title,
      description: post.excerpt,
      images: [post.ogImage],
      type: 'article',
    },
    twitter: { card: 'summary_large_image' },
  };
}

For SPAs that can't move to SSR/SSG quickly, use prerendering / dynamic rendering to serve a static snapshot to bots as a bridge.

2. Ship JSON-LD structured data

Inject schema as JSON-LD from the front-end. A small reusable component keeps it maintainable:

function JsonLd({ data }: { data: Record<string, unknown> }) {
  return (
    <script
      type="application/ld+json"
      dangerouslySetInnerHTML={{ __html: JSON.stringify(data) }}
    />
  );
}

// Usage on an article page
<JsonLd
  data={{
    '@context': 'https://schema.org',
    '@type': 'Article',
    headline: post.title,
    datePublished: post.date,
    author: { '@type': 'Person', name: post.author },
  }}
/>

Validate with Google's Rich Results Test and Schema Markup Validator. Only mark up content that's actually on the page.

3. Control crawler access

`robots.txt` + sitemap

User-agent: *
Allow: /
Disallow: /admin/

# AI crawlers — allow to be cited, disallow to opt out
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: PerplexityBot
User-agent: Google-Extended
Allow: /

Sitemap: https://example.com/sitemap.xml

In Next.js, generate both from code:

// app/robots.ts
export default function robots() {
  return {
    rules: [{ userAgent: '*', allow: '/', disallow: '/admin/' }],
    sitemap: 'https://example.com/sitemap.xml',
  };
}

// app/sitemap.ts
export default async function sitemap() {
  const posts = await getPosts();
  return posts.map((p) => ({
    url: `https://example.com/blog/${p.slug}`,
    lastModified: p.updatedAt,
  }));
}

`llms.txt` (optional, forward-looking)

Serve a curated Markdown map at /llms.txt (e.g. via a static file or a route handler). See GEO.

4. Make content extractable

For AEO/GEO, structure beats prose:

Question-shaped headings with a direct answer in the first sentences.
Real <ol>/<ul>/<table> for steps and comparisons.
Self-contained passages (no "as mentioned above" dependencies).
Descriptive link text and a clean heading hierarchy.

5. Measure across all three layers

Layer	Tools	What to watch
SEO	Google Search Console, Bing Webmaster	Impressions, clicks, rankings, index coverage, Core Web Vitals
AEO	GSC (snippet/PAA appearance), SERP trackers	Snippet ownership, PAA presence, zero-click queries
GEO	LLM-visibility / GEO tools, manual prompt sampling	Share of voice, citation rate, brand-mention accuracy
AI referrals	Analytics (GA4 / server logs)	Sessions from AI sources

Tracking AI referral traffic

AI engines send referral traffic from recognizable hosts. Segment them in analytics or log analysis:

chat.openai.com / chatgpt.com   → ChatGPT
perplexity.ai                   → Perplexity
gemini.google.com               → Gemini
copilot.microsoft.com           → Copilot
claude.ai                       → Claude

Also analyze server logs for AI crawler hits (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot) to confirm your pages are being fetched in the first place.

End-to-end checklist

Key content in initial HTML (SSR/SSG/prerender), verified with JS disabled.
Per-route metadata: title, description, canonical, Open Graph, Twitter.
JSON-LD for relevant page types, validated.
robots.txt + sitemap correct; AI crawlers intentionally allowed/disallowed.
Content structured as extractable Q→A with lists/tables.
Core Web Vitals in "Good" range.
llms.txt published for docs (optional).
Dashboards for SEO, AEO, GEO, and AI-referral traffic.

Implementation & Measurement

On this page