Next.js 16+ · Directive

robots

Next.js can serve robots.txt from app/robots.txt or generate it with app/robots.ts or app/robots.js. Generated files return a Robots object describing crawler rules and optional sitemap or host metadata.

Since: 16.3.0 for the other field

How it works

Place robots.ts or robots.js at the app root and return a Robots object. Next.js generates the /robots.txt response from its rules, sitemap, and optional host fields; the route is cached unless request-time APIs or dynamic configuration make it dynamic.

Syntax

export default function robots(): MetadataRoute.Robots

Parameters

Name	Type	Description
`rules`	`RobotsRule \| RobotsRule[]`	A rule object or array of rule objects controlling crawler access; each rule may include userAgent, allow, disallow, crawlDelay, and other.
`sitemap`	`string \| URL \| (string \| URL)[]`	The sitemap URL or URLs advertised to crawlers.
`host`	`string \| URL`	The preferred host URL, when applicable.

Returns

A Robots object that Next.js serializes and serves as /robots.txt.

Key points

Use a root-level app/robots.txt for fixed directives, or app/robots.ts/app/robots.js when you need generated output.
The generated-file convention uses a default export returning MetadataRoute.Robots.
Set rules to an array when different crawlers need different allow or disallow policies.
The other field preserves directive key casing and writes array values as separate lines inside the relevant user-agent block.
The other field was introduced in Next.js 16.3.0 for non-standard per-agent directives.

Common mistakes

The generated route is cached by default, but request-time APIs or dynamic configuration can change that behavior.
Non-standard crawler directives are not universally supported; use the rule's other field only when a target crawler documents the directive.

Example: app/robots.ts

import type { MetadataRoute } from 'next'

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: '/private/',
    },
    sitemap: 'https://acme.com/sitemap.xml',
  }
}

Save this as app/robots.ts; Next.js exposes the generated robots file at /robots.txt.

When to use it

Static crawler policy

A site has stable crawler policy and a fixed sitemap URL.

Per-crawler access rules

A site needs rules that differ by search-engine user agent.

Non-standard directives

A site must add crawler-specific directives supported outside the standard.

Generated metadata

A site derives crawler rules or sitemap values from application configuration.

Interview questions

QHow do you generate robots.txt in the App Router?

Create app/robots.ts or app/robots.js and export a default function returning MetadataRoute.Robots. Next.js turns that object into the site's robots.txt file.

Follow-up: How would you configure different crawlers? Return rules as an array, assigning each entry its own userAgent value and allow or disallow paths.

QWhat can a Robots object contain?

The rules property accepts either one rule object or an array of rule objects. A rule can specify userAgent, allow, disallow, and crawlDelay; sitemap and host are top-level fields.

Follow-up: How do you emit a non-standard directive for one crawler? Put it in that rule's other field; keys retain their casing and array values produce separate lines.

QWhen should you use a static file instead of a generated file?

A static app/robots.txt file can directly contain directives such as User-Agent, Allow, Disallow, and Sitemap. Use robots.ts or robots.js when the response must be generated from code.

Follow-up: What changed in the current API? Next.js 16.3.0 added the other field for non-standard per-agent directives.

MetadataRoute.Robots
MetadataRoute.RobotsRule
app/robots.txt
app/robots.js
app/robots.ts

Source: API Reference for robots.txt file.. Checked 2026-10-15.