added x-robots-tag header

[r3dg33k]

You're A Rockstar

Thank you for submitting a Pull Request (PR) to the Cheat Sheet Series.

🚩 If your PR is related to grammar/typo mistakes, please double-check the file for other mistakes in order to fix all the issues in the current cheat sheet.

Please make sure that for your contribution:

• In case of a new Cheat Sheet, you have used the Cheat Sheet template.
• All the markdown files do not raise any validation policy violation, see the policy.
• All the markdown files follow these format rules.
• All your assets are stored in the assets folder.
• All the images used are in the PNG format.
• Any references to websites have been formatted as [TEXT](URL)
• You verified/tested the effectiveness of your contribution (e.g., the defensive code proposed is really an effective remediation? Please verify it works!).
• The CI build of your PR pass, see the build status here.

If your PR is related to an issue, please finish your PR text with the following line:

This PR fixes issue #<REPLACE WITH ISSUE NUMBER>.

Thank you again for your contribution 😃

[szh]

Why?

[r3dg33k]

1. The X-Robots-Tag header tells crawlers how to index a URL.
2. awareness

[szh]

Can you provide a reference / source for this recommendation?

[r3dg33k]

https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag

[mackowski]

This recommendation is copied from X-AspNetMvc-Version header is not not valid for X-Robots-Tag right?

[jmanico]

I'm not 100% sure on this one, @mackowski ?

[mackowski]

hey @righettod can you help with this PR? :)

[jmanico]

I tend to be thumbs-up on this PR. It seems reasonable to want to AI crawlers of the day to leave us alone. :)