The playbook
Robots.txt: Template and Field Guide
This robots.txt template turns a broad task into a record another person can use. It captures path, user agent, rule, then connects that information to a decision. Copy the table into your working document or system and create a separate record for each item in scope.
Put the task in context
A disallow rule concerns crawling of matching paths. It is not an access-control mechanism for private data. Test broad rules carefully because a short pattern can affect an entire section. Keep authentication and indexing controls separate from crawler access rules.
Copyable worksheet
| Field | Entry instructions |
|---|---|
| path | Record the path associated with this page or query record. Use a stable reference so later updates can be matched. |
| user agent | Record the user agent associated with this page or query record. Keep the value specific enough for another owner to interpret. |
| rule | Record the rule associated with this page or query record. Keep the value specific enough for another owner to interpret. |
| expected result | Record the expected result associated with this page or query record. Keep the value specific enough for another owner to interpret. |
| Decision field | Entry instructions |
|---|---|
| Observed problem | Describe the actual condition and where it was observed. |
| Recommended action | State the smallest action that resolves the documented problem. |
| Evidence | Link to the record, observation, or test supporting the action. |
| Owner and next review | Name the accountable person and when the decision should be revisited. |
How to complete the worksheet
Begin with path. Use the same reference throughout the record so notes are not attached to the wrong item. Then identify which paths require crawler access restrictions. This establishes the information required before the worksheet becomes a request for action.
Next, check user-agent groups and path matching. Use the observed problem and evidence fields to explain why that step is needed. A useful entry describes an actual condition, such as using a crawl block as a reliable way to remove an indexed page, rather than a broad instruction to “improve performance.”
Finally, test rules against important public URLs. Put choose the appropriate indexing control and preserve crawler access when needed in the action field only when it addresses the observed condition. If the problem is different, change the action. The template is a structure for judgment, not a reason to apply the same fix everywhere.
Illustrative completed decision
| Part | Illustrative entry |
|---|---|
| Problem | Using a crawl block as a reliable way to remove an indexed page |
| Action | Choose the appropriate indexing control and preserve crawler access when needed |
| First check | Identify which paths require crawler access restrictions |
| Follow-up check | Test rules against important public URLs |
| Outcome to define | important URLs accidentally blocked |
Adapt the fields without losing meaning
You can add fields required by your team, but each should change a decision or preserve necessary context. Keep factual observations separate from recommendations. If user agent changes, record the new value and its effective date instead of silently rewriting the context of an earlier decision.
Use the record in a handoff
Ask the receiving owner to explain the next action using only the worksheet. Missing context becomes visible quickly during that exercise. Repair the record before applying the structure to a larger batch. Keep important URLs accidentally blocked connected to the item so the team can later assess whether the action was useful.
Use query and landing-page context when comparing results. A page serving a broad informational query has a different job from a page answering a purchase question. Review whether the next step remains useful to the person arriving.
Skip the manual steps. Let AgentixSEO run this for every client.
Start now