Build a robots.txt file with allow/disallow rules per user-agent, crawl-delay, sitemap reference and host directive.
* matches all bots
Optional. Ignored by Googlebot.
Absolute URL to your XML sitemap
Used by Yandex and some other crawlers.
User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /
Sitemap: https://example.com/sitemap.xml