robots.txt with multiple domains

[erikjanwestendorp]

Is there a way to configure robots.txt per domain? Sometimes we run multiple sites in one Umbraco instance and it would be great if we can configure the robots.txt separately. 😄

[patrickdemooij9]

Hi @erikjanwestendorp
It currently isn't possible to configure multiple robots.txt per domain, but it is something that I am planning to add (along with other domain specific settings). I'll let you know when I have more, but you can always give it a start if you want!

[erikjanwestendorp]

@patrickdemooij9 Thanks for your reply! I want to work on this one but before I start I want to discuss a possible solution.

Of coure you don't want to introduce a breaking change (at least not in a minor version), so I was thinking maybe it's a good idea to add a setting, called MultiDomainSupport or so, and if this is set to true than it's possible to configure a robots.txt per domain. If it's false the behaviour will be the same as before.

The RobotsTxtTreeController can be updated like so:

public class RobotsTxtTreeController : TreeController
    {
        public const string TreeGroupAlias = TreeControllerConstants.SeoToolkitTreeGroupAlias;

        private readonly IDomainService _domainService;
        private readonly IUmbracoContextAccessor _umbracoContextAccessor;
        private readonly bool _multiDomainSupport;

        public RobotsTxtTreeController(
            ILocalizedTextService localizedTextService,
            UmbracoApiControllerTypeCollection umbracoApiControllerTypeCollection,
            IEventAggregator eventAggregator,
            IDomainService domainService,
            IUmbracoContextAccessor umbracoContextAccessor,
            IOptionsMonitor<GlobalAppSettingsModel> config)
            : base(localizedTextService, umbracoApiControllerTypeCollection, eventAggregator)
        {
            _domainService = domainService;
            _umbracoContextAccessor = umbracoContextAccessor;
            _multiDomainSupport = config.CurrentValue.MultiDomainSupport;
        }

        protected override ActionResult<TreeNode> CreateRootNode(FormCollection queryStrings)
        {
            var root = base.CreateRootNode(queryStrings);

            root.Value.Icon = "icon-cloud";
            root.Value.HasChildren = _multiDomainSupport;
            root.Value.RoutePath = $"{SectionAlias}/{TreeAlias}/detail";
            root.Value.MenuUrl = null;

            return root.Value;
        }

        protected override ActionResult<MenuItemCollection> GetMenuForNode(string id, FormCollection queryStrings)
        {
            return null;
        }

        protected override ActionResult<TreeNodeCollection> GetTreeNodes(string id, FormCollection queryStrings)
        {
            if (!_multiDomainSupport)
            {
                return null;
            }

            var nodes = new TreeNodeCollection();
            _umbracoContextAccessor.TryGetUmbracoContext(out var context);

            if (id != Constants.System.Root.ToInvariantString())
            {
                return nodes;
            }
            
            foreach (var domain in _domainService.GetAll(false).GroupBy(x => x.RootContentId).Select(x => x.First()))
            {
                var node = context?.Content?.GetById(domain.RootContentId ?? -1);
                if (node is null)
                {
                    continue;
                }
                var newTreeItem = CreateTreeNode(node.Id.ToString(), "-1", queryStrings, node.Name(domain.LanguageIsoCode), "icon-globe", false);
                nodes.Add(newTreeItem);
            }

            if (nodes.Any())
            {
                return nodes;
            }


            var firstRootNode = context?.Content?.GetAtRoot().FirstOrDefault(x => x.TemplateId > 0);
            if (firstRootNode is null)
            {
                return nodes;
            }

            var firstTreeItem = CreateTreeNode(firstRootNode.Id.ToString(), "-1", queryStrings, firstRootNode?.Name, "icon-globe", false);
            nodes.Add(firstTreeItem);

            return nodes;
        }
    }

What do you think about this?

[patrickdemooij9]

Hi @erikjanwestendorp
Sorry for the late reply on this. I had wanted to write my thoughts about this earlier, but then totally forgot about the issue. So let me add them now.

I think what you have here is great, but I think it is currently a bit limited. I see this website-specific functionality being very handy for other functionalities like ScriptManager or future features. Therefore, I think we should have a tree with websites where you can see the website-specific settings like this:

The code for that is quite simple as of now. Probably needs some changes:

namespace SeoToolkit.Umbraco.ScriptManager.Core.Controllers
{
    [Tree("SeoToolkit", "ScriptManager", TreeTitle = "Script Manager", TreeGroup = "SeoToolkit", SortOrder = 2)]
    [PluginController("SeoToolkit")]
    public class ScriptManagerTreeController : TreeController
    {
        private readonly IMenuItemCollectionFactory _menuItemCollectionFactory;

        public ScriptManagerTreeController(
            ILocalizedTextService localizedTextService,
            UmbracoApiControllerTypeCollection umbracoApiControllerTypeCollection,
            IEventAggregator eventAggregator,
            IMenuItemCollectionFactory menuItemCollectionFactory)
            : base(localizedTextService, umbracoApiControllerTypeCollection, eventAggregator)
        {
            _menuItemCollectionFactory = menuItemCollectionFactory;
        }

        protected override ActionResult<TreeNode> CreateRootNode(FormCollection queryStrings)
        {
            var root = base.CreateRootNode(queryStrings);

            root.Value.Icon = "icon-script";
            root.Value.HasChildren = false;
            root.Value.RoutePath = $"{SectionAlias}/{TreeAlias}/list";

            return root.Value;
        }

        protected override ActionResult<MenuItemCollection> GetMenuForNode(string id, FormCollection queryStrings)
        {
            if (id == UmbConstants.System.RootString)
            {
                var menuItemCollection = _menuItemCollectionFactory.Create();

                var item = menuItemCollection.Items.Add<ActionNew>(LocalizedTextService, opensDialog: true, hasSeparator: false);
                item.NavigateToRoute($"{SectionAlias}/{TreeAlias}/edit");

                return menuItemCollection;
            }

            return null;
        }

        protected override ActionResult<TreeNodeCollection> GetTreeNodes(string id, FormCollection queryStrings)
        {
            return null;
        }
    }
}

And I think we can also introduce this without it being a breaking change this way. Instead of the websites being generated by domain, I think they should be added manually by the user. This way, we still keep the old items as a fallback functionality, but it also makes it clearer for users who only have a single website. If you then want specific settings for a website, you can add it to the website list and customize it the way you want to.

The root items could then be used as follows:

• ScriptManager root item could have global scripts. This way, you don't have to go through each website to add a script if you want to do that globally.
• Robots.txt could be the fallback if you don't have the specific site setup yet.

But this is mostly coming from my experiences. I would love to hear what you think of this and if this would also fit your websites. It will be a bit more work, but I could also help work on this one.

[JohanReitsma83]

@patrickdemooij9 is there an update for this? Will it become an option in the next release?