---
title: "Is a static robots.txt file good for performance?"
source: https://4property.com/is-a-static-robots-txt-file-good-for-performance/
updated: 2022-07-18T21:02:29+00:00
---

# Is a static robots.txt file good for performance?

This article is based on a recent tweet by Konstantin Kovshenin:

> I don't usually tweet about [#SEO](https://twitter.com/hashtag/SEO?src=hash&ref_src=twsrc%5Etfw), but this is important. Is a static robots.txt file good for WordPress performance? Let's find out!
>
> To display the default robots.txt, a fresh WordPress install will: 🧵
>
> — Konstantin Kovshenin (@kovshenin) [December 3, 2021](https://twitter.com/kovshenin/status/1466732748517822464?ref_src=twsrc%5Etfw)

Is a static `robots.txt` file good for **WordPress performance**?

To display the default `robots.txt`, a fresh WordPress install will:

Run 15 SQL queries: load all options, `can_compress_scripts`, `WPLANG`, query last 10 posts, query the recent posts widget options, query terms, taxonomies, etc., for found posts, all post metadata for found posts, recent comments widget, recent entries widget and a few others.

![](https://4property.com/wp-content/uploads/2021/12/1.jpg)

Call `file_get_contents()` 61 times to register some core Gutenberg blocks, that’s in addition to `json_decode()` each file, and about 100 calls to `file_exists()` on those files.

![](https://4property.com/wp-content/uploads/2021/12/2.jpg)

Register 22 post types, 24 post statuses, 22 sidebar widgets, 12 taxonomies, 7 block pattern categories, 10 block styles, 3 image sizes.

It will call `gettext` translations: 1917 times for regular strings, and 875 times for strings with context. Using the default locale, there is zero overhead. None of those strings are used in `robots.txt`.

![](https://4property.com/wp-content/uploads/2021/12/3.jpg)

Check whether the front page has been set as a static page, and whether the request `is_front_page()` or `is_home()`. Also, `is_single()`, `is_feed()`, `is_admin()`, `is_category()`, `is_search()`, and the list goes on.

Check whether the user is logged in, 14 times, and whether we need to display an admin bar, also heartbeat settings. It will also attempt to read the user session, and create 3 nonces.
**Reminder:** this is an anonymous request.

Escape some HTML 78 times.
**Reminder:** `robots.txt` is a plain/text file, there’s no HTML. It will check whether the admin needs to be forced SSL. It will also initialise smilies, and Twenty Twenty One “dark mode”.

Run 83 unique actions (one of them is `do_robotstxt`) and apply 530 unique filters (one of them is `robots_txt`).

All combined, that’s > 42,000 function calls at 5.46 megabytes peak memory, about 100 ms wall time. So yes, by all means, please use a static `robots.txt` file.

This behaviour has been previously discussed in WordPress Trac [here](https://core.trac.wordpress.org/ticket/10886) and [here](https://core.trac.wordpress.org/ticket/12256):

> `robots.txt` requests cause WordPress to query the database for all the homepage posts. It’d be good if there was perhaps a query var to short circuit the internal querying and instead perhaps fake a page template… Or probably better to just offer an appropriate hook.

> On a robots request, WordPress appears to make a SQL request for the same as it would on a Home index page. Ideally, WordPress should not query the database for posts on robots.txt requests.

What is the solution?

Currently, on WordPress side, none. However, as part of our [Supernova theme](https://4property.com/4sites/supernova/) package, we create a static `robots.txt` file in order to **bypass all the actions above**.

[Book a consultation now!](https://4property.com/website-design-for-estate-agents/)

**Quick Tip:** Do not use a redirection plugin, set your redirects up directly on the server.