????????é¢?ï¼???强è???ï¼?ä¸?让ç?ç»?????????害è????

?????©è½¶ ????/P>

??????èµ??¿å?ºã??

????è¿?å¹´æ?¥ï?????移å?¨ä?????人工?ºè?½ã??大æ?°æ????äº?计ç?ç­??°æ??????è¿?????å±?ä¸?????åº????人们??å·¥ä???交å?????æ´»æ?¹å??½å????äº?æ·±å?»å????ï¼??°æ????为驱?¨ç¤¾ä¼???å±????°å????产è?ç´?????è¡???ä¸?å¢?强ç?äº???????è¦?æº?æ³???ä½??¨å¤§?°æ???¶é??è¿?ç¨?ä¸??ä¹?å­??¨ç????ç§?è¿?æ³???½ª?????ç½?ç»??????Web Crawlerï¼?ä½?为ä?ç§??½å¿«??ç²¾å???°è?·å???°æ??¿¡?????ºç??§ç?ç»??????è¿?å¹´æ?¥å??°ä?è¶??¥è?å¤?????????åº??¨ã??å¦?ä½?对å?¶è?è¡?è§??¶ï?驱å?¨å?¶å??好å????ï¼???为å?????è¦???æ³?æ²»è?é¢???

??????å°???????????大è??²â??

????ç½?ç»????????称为??ç½?页è????????ç½?ç»??ºå?¨äºº??ï¼????ç§????§ä?å®???è§???ï¼????¨æ????ç½?ç»?ä¿¡æ????ç¨?åº???????????å®??½å?¨ç?¹å?ç¨?åº???驱å?¨ä?ï¼?模仿人工?¹å?»ä?ç½?ç«??????ºå??¨ã??å°?ç¨?åº?????ç´¢å???ä¸??ç´ ¬????????å­??¨æ?°æ??????ä»???»¥å½¢è±¡?°å?å®?ä»???解为ä¸?ç§?????¨ç?ç»?ä¸???????ï¼?å®?ä»??¹æ???åº?????令ï???常沿??URLï¼?ç½???ï¼?è¿??¹è??ä¸?ï¼??¨ä???ç½?è¿?å¼?大ç?ä¸????¥ç???»ï?寻æ?¾å??å¸þ@??????è¦????°æ???æº?????ä»????¥ç???¾åº¦????????è°·æ?ç­???ç´¢å???ï¼??¶æ?????¸å???ç´?ä¹?ä¸?å°±æ????ç½?ç»?????????ä¾?å¦??¾åº¦????ï¼?å®???è¿?äº???ç½??¥å?£ç????ç½?页ï?å®??¶å??¨å¹¶?´æ?°ç´¢å¼?ï¼??¶å??为ç?¨æ?·æ??ä¾?æ£?ç´¢æ???¡ã??

???????1993å¹´ç?ä¸?ä¸??ç»??????åº???????äº???ç½?æ¼?¸¸????è¢?????ºæ?¥ï?ç½?ç»??????ç§?便æ?·é???????ç´¢æ????¾¿å¾??°ä?ä¸????³æ³¨???¹å?????????°æ???æº??????¸å?å¢??¿ï?ç½?ç»???????åº??¨å?ºæ?????å¾?è¶???广æ?ä¸?å¤???ï¼??¿æ??æ´»è?äº?å¸??ºè??¥ã??产å?????????????????????????§ã??é£??©é?æµ?ç­?é¢?????è¿?å¹´æ?¥ï?ä¸?äº?è¿??¨ç????????¸ºå®¢æ?·æ??ä¾??°æ?????¡ç??äº???ç½?ä¼?ä¸?ï¼?ä¹??·å?äº?å¿?????å±???对个人è??è¨?ï¼???ä»???»¥?©ç?¨ç????½¯ä»¶ç?计ç??ºç?ç¨?è¯??ï¼?让ç?ç»?????¸º??·±???¡ã??ä¾?å¦??³ä¹°?°ç???³ç???¿å?ï¼?å°±å???????¿äº§ä¸???????ä¿¡æ????????¿æ??°æ???è¶??¿ä»¥è¾??©å?³ç???

????ç½?ç»?????´¨ä¸?ä»·å?¼å?¨ä?è¿??¥ï??¸å?????°æ?°æ????æµ??¨ä???äº???ç½?ç»??????为æ?°æ????????????·¥?·ï???è¿????å¢?强ä?ç½?ç»????¹é?´ç????ç»?ï¼?????äº?ç½?ç»????´ä?ä»·å?¼ï?????建ä???ç½?å¼??¾ä??±äº«??念ç????è¦??????ºç?³ã??ä½?ç½?ç»???????è¡?ä¹?å¤?ï¼?并é???»æ????±å????声ï??¹å?????????ç§?é¢??°å??ç½?ç»?å®??¨ä??¶ã????æ­¤ï?为ä?è§????????ç»´æ?¤ç?ç»?秩å?ï¼?1994å¹´è???äº???robots.txt??è®???ï¼?该å??è®??½ç?¶å?????ä¸?????å­???è®???ï¼?ä½?å®???æ¸?è¢??为ç?ç»?空é?´ç??????????¨å?????è¡?ä¸?????ï¼?ä¹???为å?¤å?????è¡?为æ???þA?·å?ç½?ç«?许å????主è?ä¾???????014å¹´ç?¾åº¦???¸è?å¥???360è¿???robots??è®??ä¸??æ³??¢å?¨å?¤å?³æ?¶å°±è®¤ä¸ºï¼?è¢???没æ???µå?????ç½?ç«???robots??è®???¶è?为æ???¾ä?å½?ï¼?åº?å½??¿æ???¸å???ä¸??©å??????

????ç§???å¼???ï¼???好ç??????????????????

????ç§?????º«???ç§?ä¸???§å·¥?·ï????????¶ã????æ³?ä¸???æ³?ï¼??³é???¨ä?使ç?¨è???????©ã????????è¿?robots??è®??ä¸??¹è?·å?äº??°æ???ä¿¡æ???ä¸??¹å???äº?æµ???ä¸??³æ³¨ï¼???大æ?????ä½?robots??è®??½é?²â????å­????´é?¾é?²â??å°?人â??ï¼???äº??????åº?ä¸??µå?è§???ï¼???å¿???欲å?°ç??????ººä¸??¿æ????äº????°æ??¿¡???并大???¤å?????¡å?¨è?æº?ï¼?????äº???ç½?ç»?害è??????ä¾?å¦?è¿?å¹´æ?¥ä?äº?大æ?°æ????§å???¸å?©ç?¨ç??????????¨ç?ç»?ä¸???æ³??¶é???¨æ?·ä¸ªäººä¿¡??¹¶è½???ç»?ç¬???¹ï???以è???ç»?æµ??©ç????

????ç½?ç»??????计ç????è¡·æ????è¿?计ç??ºæ????????¨ä¸ºç½?ç«?ç¼?çº?ç´¢å?ï¼??´æ?°ä¿¡???为ç?¨æ?·æ??ä¾?é«?????æ£?ç´¢æ???¡ã???·æ??访é????ä¸?è½½å??解æ??ï¼?对æ?°æ???è¡?????ä¸?ç­???ï¼?ä¸?ç§??ºæ?????½ã??????ï¼???????¶ä¸»è¦??????两ç????½ç??å¼???ï¼?ä½??°ä¸ºä¸?ç±»ä??¶æ?¹å???

????ä¸????¶æ?????????????°æ????访é???°æ??????ç½?ç»?????????è¡?为起?¹ï?ä¹????¶ä»·?¼æ???ï¼?ä¸?äº?ä¼?ä¸???ä¸?ººä¸ºä?ç»?æµ??©ç?????¶ä??¹æ?????ï¼??³æ?¹è?æ³?让ç?ç»???????´é??å¾·è???ä¸???????ç¢?ï¼?ç½?ç«??????????????½ï?ï¼??????????????è¶?è¿????????´ç???°æ????ä¾?å¦?2018å¹????????´ï?é©?????è¿?ç¼???????????ç¨?åº?ï¼?çª??????ºå??¨å??ç½?ç«????¨æ?·ä¿¡??º¦20ä¸??¡ï??¶å????è¿?å¾?¿¡?ºå???ä»?人ï???æ³??·å?©å?±è?2.4ä¸???ï¼?è¿?ç§?è¡?为就????äº?ä¾µç?????ä¸?ººä¿¡æ??½ª??

????äº????¶æ??ä¾µç???¥è?产æ????ä¸?äº?ç½?ç»??????¨æ?°æ??????è¿?ç¨?ä¸??????????¹ç??许å???å°±ä?è½½ã??å¤??¶æ??ä¼?????å­????¾ç????è§?é¢?ç­???容ï?并以此è?·å?©ã????常形å¼??????????·æ??????å½?å±?????å­þ=??å½±è?ä½???ç­?设置??æ¡??¾æ?¥ï??¸å??¨æ?·ç?¹å?»ï??¶å????è¿?????广å??ç­?å½¢å????©ã??ä¾?å¦?段æ??建ç?????çª?çª??µå½±??ç½?ç«?ï¼??©ç?¨ç?????????å¤?家è?é¢?ç½?ç«???500ä½??¨å½±è§?ä½???设置äº???æ¡??¾æ?¥ï??¸å??¨æ?·ç?¹å?»æ???¾ï?并è???广å??è´?00ä½?ä¸???ï¼?è¿?ç§?è¡?为就è§þ>?????ä½???æ³???

????ä¸????¶æ???°ä¹±ç³»ç?è¿?è¡???ç½?ç»????????³»ç»??¶ï?è·?人工?¹å?»å????ä¸??·ï???æ­¤å?å·¨é?????????¶è?????ä¸?ç½?ç«??¶ï?å°±ä????¨å¤§????å¸þ@?èµ?æº?ï¼????????¡å?¨ä?????è´????³å´©æº???ä¸???ä¸?家表示ï?2019å¹´æ??è¿?40%??ç½?ç»?æµ????¥è????????¶ä¸­20%?¥è???¶æ?????????ä¸?æ¯?ä¾?è¿?å°?ä¸?????????ºè?è¡?ä¸??³æ?????¾å?ºã??以中?½é??è·???12306ç½?ç«???为ä?ï¼???011å¹´ä?线è??¥ä»¥?¥ï??¥è????´å?次å?ºç?°å???????主è????¢ç¥¨è½?»¶ï¼??»å?»è????æ³?访é???????µã?????计ï?2018å¹´æ?¥è????´ï?ç½?ç«???é«?å³°æ??天å??页é?¢æ?è§???è¾?13.4亿æ?ï¼??¶ä¸­è¿?90%????????????为ã??

????????è§??¶ï?让â??????????????ä¸?

???????计ï?????020å¹?2??ï¼????½ç?æ°?è§?模达9.89亿ã??å¢?强ç?ç»?空é?´æ²»??ï¼?ä¿???ä¸?ººä¿¡æ?????°æ??????å·²æ??为é??大ç???¶ä»£?½é???

???????????½å?ç½?ç»???????è§??¶è?没æ??ä¸??¨ç??æ³?å¾?æ³?è§?ï¼?主è?ä¾???æ°?æ³??¸ã????ä¸?æ­£å?ç«?äº?æ³?????ä½???æ³???ç½?ç»?å®??¨æ?以å????æ³?ä¸????¸å?³æ?å¾??¡æ???ä½?ä»?æ³?å¾?è§??¶æ?????¥ç??ï¼????µä?容ä?è§???ä¸??¹é?¢æ???±ä?äº???ç½???????å¿?????å±?ï¼?以å??ç½?ç»????·æ????å¼??¾æ?§ã???????§å??æµ??¨æ?§ç??¹å?ï¼?让å?è¿?æ³???½ªè¡?为ç????管å??è¯???????è¾?为å?°é?¾ï??¹å?????????°æ??????ä»?ç½?页æ??å±??³æ???ºå??¨ç?åº???å°?ç¨?åº?ç­?移å?¨ç?å¹³å?°ï???????段就?´å??å¤???ä¸????½ï??þ???¹é?¢æ???ç»?ä¸??????°å????¡å??å±?è¿???ï¼?æ³?å¾?æ³?è§?è¿??·æ??è¾?大ç??æ»????§ï??¸æ?å®?è·µå?ç½?ç»?????????»¥?¨å?????æ³?å¾????缺ä?足å????¸æ?ç»?éª?ï¼????¸å?³ç???¸æ?解é??äºþ=???½ä?以足å¤??³æ³¨ï¼??¾ä»¥å½¢æ??????è§??¶ã??ä¾?å¦?2017å¹´ä??¥å¤´?¡ä??°æµªå¾???ä¹??´å????å¾?¤´?¡â??ä¸??¡å??æ­¥â??å¾???????è¨???容ç??çº?纷æ?ä¾?ï¼??¶å?è´¨æ???°æ????å±??¾ä»¥ç¡?????????ä½??±ä?缺ä???ç¡????¸æ?解é????å¼?ï¼?å¼???äº?å®?践中??????äº?è®???

????2019å¹???ï¼??½å?äº???ç½?ä¿¡æ??????????äº????°æ????¨ç?????æ³?ï¼?å¾?æ±???è§?稿ï???ï¼?该å??æ³?ä¸??¨é??对å?©ç?¨ç?ç»?å¼?å±??°æ???¶é????å­??¨ã??ä¼?è¾???å¤?????使ç?¨ç?æ´»å?¨è?è¡?äº?è§??¶ï?????对ç?ç»???????????为è?è¡?????æ²»ç????该å??æ³?强è?ï¼??½å?????ä¿????°æ????¨ä???å±?并é??ï¼?é¼??±ç?????°æ????¨ä??¤æ?????ç§????¨è??°æ???æº?å¼????©ç???ä¿????°æ???æ³???åº????±æ??¨ã??è¿?ä¹???????ä»??ç½?ç»????????´¨ä¸???¡ºåº??°å?ç»?æµ???å±???æ±?????ï¼??¢è?ä»?æ³?å¾?å±??¢é?åºþB??è§?ï¼?ä¹?è¦????¡å¥½æ³?å¾?è§??¶ç??è¾¹ç??ï¼???æ±?è¾¾å?°æ?°æ???©ç?¨ä?????è§??¶ä??´ç?????§å¹³è¡¡ï?以便让ç?ç»??????´å¥½?°æ???¡ç?æµ?社ä?ä¸?人æ???æ´»ï??©æ?¨æ?°å?ä¸??½å»ºè®¾ã??

????ï¼?ä½???ï¼??©è½¶ ???¶ï?????³»ä¸?¤®æ°???大å?æ³?å­þB?¢é?¢é?¿ã??????ï¼?ä¸?¤®æ°???大å?æ³?å­þB?¢ç??½ª?²æ?§ç??究中å¿???究å??ï¼?