- Next.js
- Dynamic Metadata & SEO
- Dynamic XML Sitemaps and Robots Configuration
Dynamic XML Sitemaps and Robots Configuration
Next.js provides special file-convention endpoints for search engine indexing: sitemap.ts generates standard XML sitemaps dynamically, while robots.ts generates robots.txt rules. For sites with more than 50,000 URLs, Next.js supports generateSitemaps to automatically shard URLs across multiple sitemap indices.
Learning Objectives
Generate dynamic XML sitemaps using the app/sitemap.ts convention.
Configure crawl permissions and sitemap references using app/robots.ts.
Shard large sitemaps (>50,000 URLs) using generateSitemaps.
Assign proper changeFrequency and priority fields to optimize crawler budget.
Prerequisites & Background Knowledge
Why Interviewers Care
Essential for technical SEO audits and large catalog indexing at tech companies.
The Challenge
Demonstrate how to implement dynamic sitemap.ts and robots.ts in Next.js App Router. How do you shard sitemaps when scaling beyond 50,000 URLs?
Code Architecture & Implementation
1// app/sitemap.ts2import { MetadataRoute } from 'next';34export default async function sitemap(): Promise<MetadataRoute.Sitemap> {5 const baseUrl = process.env.NEXT_PUBLIC_SITE_URL || 'https://faangprep.com';67 // 1. Static marketing routes8 const staticRoutes: MetadataRoute.Sitemap = [9 {10 url: baseUrl,11 lastModified: new Date(),12 changeFrequency: 'daily',13 priority: 1.0,14 },15 {16 url: `${baseUrl}/nextjs`,17 lastModified: new Date(),18 changeFrequency: 'weekly',19 priority: 0.9,20 },21 ];2223 // 2. Dynamic content routes fetched from CMS/DB24 const lessons = await fetch('https://api.cms.com/lessons').then(res => res.json());25 const lessonRoutes: MetadataRoute.Sitemap = lessons.map((l: any) => ({26 url: `${baseUrl}/nextjs/${l.category}/${l.slug}`,27 lastModified: new Date(l.updatedAt),28 changeFrequency: 'monthly',29 priority: 0.8,30 }));3132 return [...staticRoutes, ...lessonRoutes];33}
Architectural Explanation
Rather than manually creating XML strings or maintaining static files, app/sitemap.ts returns a strongly typed array of sitemap objects. Next.js automatically serializes this array into valid XML schema format. For large catalogs, exporting generateSitemaps() instructs Next.js to generate sharded sub-sitemaps (e.g. /sitemap/0.xml, /sitemap/1.xml) and an index sitemap at /sitemap.xml.
Common Mistakes & Anti-Patterns
Hardcoding `localhost:3000` as the domain in sitemap URLs.
Placing confidential administrative endpoints in the sitemap.
Forgetting that `app/robots.ts` automatically outputs to `/robots.txt`.
Interview Follow-Up Traps
The interviewer asks: "What happens if my sitemap has 100,000 URLs?" Export `generateSitemaps = async () => [{ id: 0 }, { id: 1 }]` to partition the sitemap into indexed chunks.
Frequently Asked Questions
Can sitemaps be cached with revalidate?
Yes, you can set `export const revalidate = 86400;` (24 hours) in `sitemap.ts`.
How do you test sitemaps locally?
Visit `http://localhost:3000/sitemap.xml` and `http://localhost:3000/robots.txt` in your browser.
