Feed aggregator

Article URL: https://langfield.github.io/ki/

Comments URL: https://news.ycombinator.com/item?id=41960066

Points: 1

# Comments: 0

Article URL: https://www.sanity.io/blog/welcoming-the-begin-team-to-sanity

Comments URL: https://news.ycombinator.com/item?id=41960057

Points: 1

# Comments: 0

Article URL: https://www.youtube.com/watch?v=0Ye9Svizbn8

Comments URL: https://news.ycombinator.com/item?id=41960030

Points: 2

# Comments: 0

Article URL: https://www.denver7.com/news/state-news/should-elephants-have-the-same-rights-as-people-a-colorado-court-may-decide

Comments URL: https://news.ycombinator.com/item?id=41960023

Points: 2

# Comments: 0

Article URL: https://scytheworks.ca/scythe-works-without-borders/

Comments URL: https://news.ycombinator.com/item?id=41960013

Points: 2

# Comments: 0

Article URL: https://codeberg.org/Codeberg/Community/issues/1654

Comments URL: https://news.ycombinator.com/item?id=41960011

Points: 4

# Comments: 0

Article URL: https://typeset.lllllllllllllllll.com/

Comments URL: https://news.ycombinator.com/item?id=41960010

Points: 6

# Comments: 0

This applies to sites indexed on Google that hope to gain organic traffic. As an indie blogger and SEO enthusiast, I foolishly updated my robots.txt file to prevent indexing of certain unwanted parts of my site, leading to subtle repercussions that I couldn't have foreseen.

A few days ago, while reading about SEO, I came across the concept of a "crawl budget." Apparently, Google allocates a specific crawl budget to your indexed site, and the more useless content it has to index and store on its servers, the more it affects your site—resulting in delays for new content indexing, favicon updates, and robots.txt crawling.

Being a minimalist and utilitarian, I decided to prevent indexing of the `/uploads/` directory on my site since it mostly contained images used in my articles. I thought blocking this "useless content" would free up more crawling budget for my primary content, i.e., articles. So, I added this directory to my site's robots.txt:

# Group 1 User-agent: * Disallow: /public/ Disallow: /drafts/ Disallow: /theme/ Disallow: /page* Disallow: /uploads/ Sitemap: https://prahladyeri.github.io/sitemap.xml The way search engines work means there's typically a 5-7 day gap between updating the robots.txt file and crawlers processing it. After about a week, I noticed that my site's favicon disappeared from SERPs on mobile browsers! Instead, there was a bland (empty) icon in its place. That’s when I realized that my favicons also resided in the `/uploads/` directory. After I recently optimized the favicon format by switching from WEBP to PNG, Google was unable to crawl and index the new favicon at all!

Once I realized this mistake, I removed the blocking of `/uploads/` from the robots.txt and requested a recrawl. But who knows how long it will take for Google's systems to sync this change and start showing the site's favicon back in SERPs! Two lessons learned:

1. The robots.txt file is highly sensitive; avoid modifying it if possible. 2. Applying SEO is like steering an extremely large ship or vessel. You pull a lever now, and the ship only moves after several days!

Comments URL: https://news.ycombinator.com/item?id=41960003

Points: 2

# Comments: 0

Article URL: https://github.com/getlago/lago/wiki/Can-Stripe-products-stand-on-their-own%3F

Comments URL: https://news.ycombinator.com/item?id=41959993

Points: 2

# Comments: 0

Article URL: https://directory.fsf.org/wiki/GNU

Comments URL: https://news.ycombinator.com/item?id=41959984

Points: 3

# Comments: 0

Article URL: https://er4hn.info/blog/2024.10.26-gmail-labels/

Comments URL: https://news.ycombinator.com/item?id=41959960

Points: 3

# Comments: 1

Article URL: https://dieter.plaetinck.be/posts/open-source-undefined-part-1-the-alternative-origin-story/

Comments URL: https://news.ycombinator.com/item?id=41959921

Points: 1

# Comments: 0

Article URL: https://salsa.debian.org/deeplearning-team/ml-policy

Comments URL: https://news.ycombinator.com/item?id=41959918

Points: 1

# Comments: 0

Article URL: https://www.thedailybeast.com/elon-musk-was-an-illegal-migrant-worker-who-abused-his-student-visa-when-he-founded-first-company/

Comments URL: https://news.ycombinator.com/item?id=41959905

Points: 2

# Comments: 0

Article URL: https://www.atlasobscura.com/articles/immortal-lichen-growing-in-cemeteries

Comments URL: https://news.ycombinator.com/item?id=41959897

Points: 1

# Comments: 0

Article URL: https://www.theguardian.com/travel/2024/oct/24/rollercoaster-designer-john-burton-thorpe-park-hyperia

Comments URL: https://news.ycombinator.com/item?id=41959893

Points: 1

# Comments: 0

Article URL: https://blog.cloudflare.com/how-we-built-cloudflare-queues/

Comments URL: https://news.ycombinator.com/item?id=41959881

Points: 1

# Comments: 0

Article URL: https://nicole.express/2024/city-disconnection.html

Comments URL: https://news.ycombinator.com/item?id=41959846

Points: 1

# Comments: 0

Article URL: https://www.bloomberg.com/news/articles/2024-10-26/google-to-preview-ai-that-takes-over-computers-information-says

Comments URL: https://news.ycombinator.com/item?id=41959835

Points: 1

# Comments: 0