How can I allow SEMrush-Bot-Desktop to scan my website in order to access analytics on SEMrush?
HTTP error 406 usually indicates that the server is refusing to serve the requested resource because it is not acceptable. In this case, your website is likely blocking the SEMrushBot-Desktop user agent.
To allow SEMrushBot-Desktop to scan your website, you need to adjust your website’s settings to accept the user agent. This can be done by modifying your website’s robots.txt file, or by configuring your web server to allow the user agent.
Here are some possible solutions:
- Check your robots.txt file: Ensure your robots.txt file doesn’t explicitly block the SEMrushBot-Desktop user agent. You can add a line to allow the bot:
User-agent: SEMrushBot-Desktop - Configure your web server: If you’re using a web server like Apache or Nginx, you may need to configure it to allow the SEMrushBot-Desktop user agent. Refer to your server’s documentation for instructions.
- Contact your web host: If you’re unsure how to modify your robots.txt file or configure your web server, contact your web host for assistance. They can help you make the necessary adjustments to allow SEMrushBot-Desktop to access your website.
By allowing the SEMrushBot-Desktop to access your website, you’ll be able to utilize SEMrush’s analytics features to gain insights into your website’s performance.