Next.js Curriculum
MediumMetadata & SEO100% Free25 min read

Dynamic XML Sitemaps and Robots Configuration

The 30-Second Interview Answer

Next.js provides special file-convention endpoints for search engine indexing: sitemap.ts generates standard XML sitemaps dynamically, while robots.ts generates robots.txt rules. For sites with more than 50,000 URLs, Next.js supports generateSitemaps to automatically shard URLs across multiple sitemap indices.

Learning Objectives

Generate dynamic XML sitemaps using the app/sitemap.ts convention.

Configure crawl permissions and sitemap references using app/robots.ts.

Shard large sitemaps (>50,000 URLs) using generateSitemaps.

Assign proper changeFrequency and priority fields to optimize crawler budget.

Prerequisites & Background Knowledge
XML Sitemap standard (sitemaps.org)robots.txt directivesSearch engine crawl budget

Why Interviewers Care

Essential for technical SEO audits and large catalog indexing at tech companies.

The Challenge

Demonstrate how to implement dynamic sitemap.ts and robots.ts in Next.js App Router. How do you shard sitemaps when scaling beyond 50,000 URLs?

Code Architecture & Implementation

1// app/sitemap.ts
2import { MetadataRoute } from 'next';
3
4export default async function sitemap(): Promise<MetadataRoute.Sitemap> {
5 const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://faangprep.com';
6
7 // 1. Static marketing routes
8 const staticRoutes: MetadataRoute.Sitemap = [
9 {
10 url: baseUrl,
11 lastModified: new Date(),
12 changeFrequency: 'daily',
13 priority: 1.0,
14 },
15 {
16 url: `${baseUrl}/nextjs`,
17 lastModified: new Date(),
18 changeFrequency: 'weekly',
19 priority: 0.9,
20 },
21 ];
22
23 // 2. Dynamic content routes fetched from CMS/DB
24 const lessons = await fetch('https://api.cms.com/lessons').then(res => res.json());
25 const lessonRoutes: MetadataRoute.Sitemap = lessons.map((l: any) => ({
26 url: `${baseUrl}/nextjs/${l.category}/${l.slug}`,
27 lastModified: new Date(l.updatedAt),
28 changeFrequency: 'monthly',
29 priority: 0.8,
30 }));
31
32 return [...staticRoutes, ...lessonRoutes];
33}

Architectural Explanation

Rather than manually creating XML strings or maintaining static files, app/sitemap.ts returns a strongly typed array of sitemap objects. Next.js automatically serializes this array into valid XML schema format. For large catalogs, exporting generateSitemaps() instructs Next.js to generate sharded sub-sitemaps (e.g. /sitemap/0.xml, /sitemap/1.xml) and an index sitemap at /sitemap.xml.

Common Mistakes & Anti-Patterns

Hardcoding `localhost:3000` as the domain in sitemap URLs.

Placing confidential administrative endpoints in the sitemap.

Forgetting that `app/robots.ts` automatically outputs to `/robots.txt`.

Interview Follow-Up Traps

The interviewer asks: "What happens if my sitemap has 100,000 URLs?" Export `generateSitemaps = async () => [{ id: 0 }, { id: 1 }]` to partition the sitemap into indexed chunks.

Frequently Asked Questions

Does Google guarantee indexing of all URLs in a sitemap?

No. Sitemaps serve as hints to search engine crawlers, but crawl budget and page quality determine actual indexing.

Yes, you can set `export const revalidate = 86400;` (24 hours) in `sitemap.ts`.

Visit `http://localhost:3000/sitemap.xml` and `http://localhost:3000/robots.txt` in your browser.