The past, present, and future of robots.txt

Share
Google Search Central
Published at : 23 Dec 2021
2562 views
85

In this episode of Search Off the Record, Gary, Lizzi, and Martin from the Google Search team are joined by special guest David Price, a Product Counsel for Google. They discuss human initiated crawl requests vs. bot initiated crawl requests, the robots.txt file, and more!



Episode transcript → https://goo.gle/sotr028-transcript



Search Off the Record is a podcast series that takes you behind the scenes of Google Search with the Search Relations team.
developersgaryillyesgoogledevelopers

2ALWAYS ANOTHER OPTION

2ALWAYS ANOTHER OPTION

Shopping Star - 21.10.2021 "Double Date"

Shopping Star - 21.10.2021 "Double Date"

Limited Resources 626 – VOW Draft Walkthrough(s)

Limited Resources 626 – VOW Draft Walkthrough(s)

CAT Percentile & Schools to Apply

CAT Percentile & Schools to Apply

Incredible Drilling Test in COPPER | Testing Speed, Durability, Precision

Incredible Drilling Test in COPPER | Testing Speed, Durability, Precision

Most unearthly video || सबसे अनोखा वीडिय || Unearthly videos ||

Most unearthly video || सबसे अनोखा वीडिय || Unearthly videos ||

VR 360: Horror in the Bath | Video Experience 4K

VR 360: Horror in the Bath | Video Experience 4K

Show A Little Bit Of Love and Kindness | The Collingsworth Family | Official Performance Video

Show A Little Bit Of Love and Kindness | The Collingsworth Family | Official Performance Video

Μεγάλοι στρατηγοί! #1 | Totally Accurate Battle Simulator | Greek

Μεγάλοι στρατηγοί! #1 | Totally Accurate Battle Simulator | Greek

Ferrari 812 GTS Tailor Made | Ferrari Düsseldorf

Ferrari 812 GTS Tailor Made | Ferrari Düsseldorf

M A R I K I T | Justin Vasquez

M A R I K I T | Justin Vasquez

How to set up auto admin assistance on facebook group Pc

How to set up auto admin assistance on facebook group Pc

"THEY'VE GOT AWAY WITH MURDER!"😱 Trevor Sinclair says Steve Bruce was a scapegoat for #NUFC players!

"THEY'VE GOT AWAY WITH MURDER!"😱 Trevor Sinclair says Steve Bruce was a scapegoat for #NUFC players!

ISIS terror warning issued for Virginia malls

ISIS terror warning issued for Virginia malls

Police are searching for a woman after she climbed into the Bronx Zoo lion exhibit a second time

Police are searching for a woman after she climbed into the Bronx Zoo lion exhibit a second time

Best Fails/Wins of 2021 #28

Best Fails/Wins of 2021 #28

Peta: Imagine if you were the one being stuffed for thanksgiving

Peta: Imagine if you were the one being stuffed for thanksgiving

NASA Finally Contacts Voyager 2 Again After A Year of Silence

NASA Finally Contacts Voyager 2 Again After A Year of Silence

Ray Dalio: I’m ‘significantly concerned’ about inflation

Ray Dalio: I’m ‘significantly concerned’ about inflation

Ben Whishaw - A Conversation (From "Mary Poppins Returns"/Audio Only)

Ben Whishaw - A Conversation (From "Mary Poppins Returns"/Audio Only)

JORDAN PETERSON - THE BIGGEST SCAM, MANY People are UNAWARE OF!!!

JORDAN PETERSON - THE BIGGEST SCAM, MANY People are UNAWARE OF!!!

SpacePro Sliding door wardrobes - Install track & doors

SpacePro Sliding door wardrobes - Install track & doors

Kon Banega Bhikhari Final Result 🔥😆

Kon Banega Bhikhari Final Result 🔥😆

ORIGINAL SONGS during the Blind Auditions of The Voice 2021

ORIGINAL SONGS during the Blind Auditions of The Voice 2021

EASYFIX Mats Behind the Milking Parlor

EASYFIX Mats Behind the Milking Parlor

Arabic literature emerged from the 5th century and before that there were only fragmented texts

Arabic literature emerged from the 5th century and before that there were only fragmented texts

Woodworking Business - Here's What Else You'll Discover

Woodworking Business - Here's What Else You'll Discover

Unplanned Six on the street - Funny Prank

Unplanned Six on the street - Funny Prank

TLE 4 | Agriculture: Ornamental Plants and Starting a Home Garden

TLE 4 | Agriculture: Ornamental Plants and Starting a Home Garden

MIRROR OF ERISED- Making a replica

MIRROR OF ERISED- Making a replica

PGA Tour 2k21 #1 - Gob Ross, Golf Ross

PGA Tour 2k21 #1 - Gob Ross, Golf Ross

Josh Duggar trial to resume

Josh Duggar trial to resume

It is necessary to set up a steering committee before activating Rekodak, Maulana Abdul Wasay

It is necessary to set up a steering committee before activating Rekodak, Maulana Abdul Wasay

MY FIDGET COLLECTION!🤩 *SO SATISFYING!*🤩

MY FIDGET COLLECTION!🤩 *SO SATISFYING!*🤩

Typhoon brings treasures, a variety of pearl oysters and delicious seafood

Typhoon brings treasures, a variety of pearl oysters and delicious seafood

When The Quiet Kid Is Asking Too Many Questions in Lockdown #shorts #youtubeshorts

When The Quiet Kid Is Asking Too Many Questions in Lockdown #shorts #youtubeshorts

Sizzla - Dry Cry

Sizzla - Dry Cry

How Strong Should You Be? (Noob To Freak)

How Strong Should You Be? (Noob To Freak)

New Hobbies Have Commenced

New Hobbies Have Commenced

Regina Belle - If I Could (Official Video)

Regina Belle - If I Could (Official Video)

Lene Marlin - You Could Have

Lene Marlin - You Could Have

"YOU WON'T BE SINGLE FOR LONG. PAST PERSON WANTS YOU BACK. EXPECT LOTS OF CHANGES." 20-31 DECEMBER

"YOU WON'T BE SINGLE FOR LONG. PAST PERSON WANTS YOU BACK. EXPECT LOTS OF CHANGES." 20-31 DECEMBER

We Sponsored The Rogue Invitational! | Beyond The B, S1.E14

We Sponsored The Rogue Invitational! | Beyond The B, S1.E14

Stephen Sharer - Home For The Holiday (Official Music Video)

Stephen Sharer - Home For The Holiday (Official Music Video)

Common - When We Move (Live Session) | Vevo Ctrl

Common - When We Move (Live Session) | Vevo Ctrl

What is inventory? | Types of inventory

What is inventory? | Types of inventory

UN Force Disregarded (1964)

UN Force Disregarded (1964)

Home remedy for cold and cough/loose your weight naturally

Home remedy for cold and cough/loose your weight naturally

The Scariest Videos You CANNOT Watch Alone At Home

The Scariest Videos You CANNOT Watch Alone At Home