Results 1 to 19 of 19

Thread: Robots.txt

  1. #1
    Join Date
    Dec 2006
    Location
    Sweden
    Posts
    10,812
    Mentioned
    3 Post(s)
    Quoted
    16 Post(s)

    Default Robots.txt

    I though I read somewhere SRL wasn't supposed to be on google?

    Well I was talking to Ron from SRL (He is uber smart) about my server being on Google (people could google for my caver and leech it there :O) and he linked me here http://www.robotstxt.org/robotstxt.html
    Why doesn't SRL do this? You can Google for SRL and get results from it.


    If you add the robots.txt, results from Google go away the next time Google indexes the site.

    Or is it because of the Google ads here? :/


    Send SMS messages using Simba
    Please do not send me a PM asking for help; I will not be able to help you! Post in a relevant thread or make your own! And always remember to search first!

  2. #2
    Join Date
    Jun 2006
    Posts
    3,861
    Mentioned
    3 Post(s)
    Quoted
    1 Post(s)

    Default

    I thought I heard that it did? Anyways, if it doesn't, I think it should be added.

  3. #3
    Join Date
    Dec 2006
    Location
    Sweden
    Posts
    10,812
    Mentioned
    3 Post(s)
    Quoted
    16 Post(s)

    Default

    Quote Originally Posted by bullzeye95 View Post
    I thought I heard that it did? Anyways, if it doesn't, I think it should be added.
    Explain please ...

    http://www.srl-forums.com/robots.txt 404 error They don't have it AFAIK


    Send SMS messages using Simba
    Please do not send me a PM asking for help; I will not be able to help you! Post in a relevant thread or make your own! And always remember to search first!

  4. #4
    Join Date
    Jun 2006
    Posts
    3,861
    Mentioned
    3 Post(s)
    Quoted
    1 Post(s)

    Default

    I meant that I thought SRL used it, and that if it doesn't, then it should be added

    EDIT: Does robots.txt have to be at www.srl-forums.com/robots.txt, or can it be located elsewhere on the site?

  5. #5
    Join Date
    Feb 2007
    Location
    Het ademt zwaar en moedeloos vannacht.
    Posts
    7,211
    Mentioned
    26 Post(s)
    Quoted
    72 Post(s)

    Default

    http://www.srl-forums.com/forum/robots.txt
    They do have it, but they only disable a few pages.
    I made a new script, check it out!.

  6. #6
    Join Date
    Dec 2006
    Location
    Sweden
    Posts
    10,812
    Mentioned
    3 Post(s)
    Quoted
    16 Post(s)

    Default

    @bullzeye: Thought you said it shouldn't be added.


    Send SMS messages using Simba
    Please do not send me a PM asking for help; I will not be able to help you! Post in a relevant thread or make your own! And always remember to search first!

  7. #7
    Join Date
    Nov 2007
    Location
    The Netherlands
    Posts
    1,490
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    Quote Originally Posted by Markus View Post
    http://www.srl-forums.com/forum/robots.txt
    They do have it, but they only disable a few pages.
    why don't they add it to al pages?
    because leeching is a bad thing

  8. #8
    Join Date
    Dec 2006
    Location
    Copy pastin to my C#
    Posts
    3,788
    Mentioned
    8 Post(s)
    Quoted
    29 Post(s)

    Default

    I still don't understand why do people think that SRL shouldn't be found on Google or Yahoo etc, people know about us, well maybe not the people that think Macro = EvilHackphr0mGoogleAutoFighter.exe, but for example when SMART was released to public - what happened to the Message of the Week?

    "Do not download programs that help you play RuneScape - they might steal your password!".


    Harry, I think it's stupid to keep your scripts on your server, why not to attach them straight on like regular people, it's some trouble to run a script that downloads a script to your files somewhere, and then you have to open it again. If I want to see your script, I just leech the URL from the updater script and copy paste it.

    If you would attach it, you also wouldn't have to be afraid of leeching...

    Haha - someone has even pastebinned it -
    http://www.google.fi/search?q=Hy7119...ient=firefox-a

  9. #9
    Join Date
    Nov 2007
    Location
    Chile
    Posts
    1,901
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    Haha - someone has even pastebinned it -
    http://www.google.fi/search?q=Hy7119...nt=firefo x-a
    Harry, your script is getting leeched i think.

    I think there is another posibility to leech, new users try to download this:
    http://www.srl-forums.com/forum/atta...0&d=1205861097


  10. #10
    Join Date
    Jun 2007
    Location
    Wednesday
    Posts
    2,446
    Mentioned
    3 Post(s)
    Quoted
    1 Post(s)

    Default

    I'm 100% sure it won't let them as it runs the same check as if they just clicked on it. I'm sure vBulletin has it built in the attatchments php file.
    Anyway, keeping SRL off google won't do anything. I'm pretty sure even now it's very hard to find a common way of putting runescape hack into google and getting SRL anywhere near the front, so there's not much point in using a Robots.txt to make them avoid half the forum. Anyway, I'm pretty sure the bots only get access to the un-registered users stuff, so not much to loose from there.
    By reading this signature you agree that mixster is superior to you in each and every way except the bad ways but including the really bad ways.

  11. #11
    Join Date
    Jun 2006
    Posts
    75
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    Quote Originally Posted by Cazax View Post
    Harry, your script is getting leeched i think.

    I think there is another posibility to leech, new users try to download this:
    http://www.srl-forums.com/forum/atta...0&d=1205861097
    I was able to download that script, should I be able to?

  12. #12
    Join Date
    Nov 2007
    Location
    Chile
    Posts
    1,901
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    Quote Originally Posted by Mistran View Post
    I was able to download that script, should I be able to?
    yes you should i think.


  13. #13
    Join Date
    Mar 2008
    Posts
    10
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    People cant leech using google since the pages for the downloads you need an auth which google-crawler doesnt has. robots.txt is used when you want to block a crawler/bot/indexer to stop indexing your whole site. But it also means that if you google this site you wont find it.

  14. #14
    Join Date
    Jan 2008
    Location
    UK
    Posts
    500
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    This forum does not use robots.txt... you can simply tell by trying to visit http://www.srl-forums.com/robots.txt.... you get page not found, because it doesn't exist!

    I see no problem with this site being listed in google. At the end of the day you need new members and new blood to keep this community going.. becoming too elitist will just kill it..

    Encourage Loyalty, encourage n00bs, encourage inovation, we all have to start somehwere!

    Personally I don't feel this community is at all welcoming and I have no loyalty to it! I'm a member and that's as far as it goes!
    For the Ultimate Monk Fisher: Ultra Monkfish n Bank Click Here


  15. #15
    Join Date
    Jan 2007
    Location
    Kansas
    Posts
    3,760
    Mentioned
    1 Post(s)
    Quoted
    3 Post(s)

    Default

    As already stated, it does http://www.srl-forums.com/forum/robots.txt


  16. #16
    Join Date
    Jan 2008
    Location
    UK
    Posts
    500
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    robots.txt should be in the root of the domain. anywhere else and under the specifications it ain't binding it's still just proposed...
    For the Ultimate Monk Fisher: Ultra Monkfish n Bank Click Here


  17. #17
    Join Date
    Feb 2007
    Posts
    433
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    >_< THE REAL ISSUE HERE IS THAT HY IS GROUNDED FOR A MONTH >_<

    umm.. anyways

    it doesn't really matter if SRL is on google because Jagex knows about SRL... they just can't do anything about it really ^^ just don't put your username/password/skills in anything !

  18. #18
    Join Date
    Jun 2007
    Location
    Hell Stream
    Posts
    584
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    robots.txt file only will be function unless they have been declare on every each page with the html code implement like

    <meta name="robots" content="noindex,nofollow">

  19. #19
    Join Date
    Jan 2008
    Location
    UK
    Posts
    500
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    Quote Originally Posted by HellBoyz View Post
    robots.txt file only will be function unless they have been declare on every each page with the html code implement like
    Wrong.. That meta is if you want to do it on a page by page basis and was implimented for users that don't have enogh server access to add a robots.txt file. The robots.txt File allows you to instruct a bot on a site-wide basis to exclude/include an entire site, entire paths or extention of a particular type etc etc.
    For the Ultimate Monk Fisher: Ultra Monkfish n Bank Click Here


Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •