Page 1 of 2 12 LastLast
Results 1 to 25 of 47

Thread: PyBot [Deep learning to object detection]

  1. #1
    Join Date
    Dec 2008
    Posts
    135
    Mentioned
    0 Post(s)
    Quoted
    44 Post(s)

    Default PyBot [Deep learning to object detection]

    update: 8/28/2017:
    Finished amethyst bot, Added support for finite state machine, (generate pretty graphs dynamically):

    Update 8/05/2017
    Improved rock detection, needs a little bit more work (more data) to improve detecting the different types of rocks.

    Update 8/01/2017
    working minimap

    Updated: 6/07/2017
    -Been working on a way to recognize coordinates on a map. I been testing a convolutional network that outputs just an x coordinate. I started out with a map picture of the varrock area, the scale is approximately the same as the real minimap on a client. I designated a custom coordinate system for each pixel.
    varrock_map.jpg

    After briefly training the network, I was able to get some impressive results.
    test_1.jpgtest_2.jpgtest_3.jpgtest_4.jpg
    [302.53319, 358.72432, 420.84034, 372.00929]
    Just from training observations, I notice I can get the loss very low, it's very possible its overfitting. The future plan is to include an output node for y coordinate, and an output node for angle.


    What is pybot:
    Pybot is an api that does pure color detection using deep learning, and the opencv library. The technique for object detection is SSD-multibox detector. It is implemented in tensorflow. The actual clicking is implemented through uinput, so it doesn't work on windows. I may make a windows version that controls an arduino acting as a hardware mouse. (depends on interest)

    Here is a little preview:



    Once this is more fleshed out, I plan to open source it. The tentative date for this is the end of June. I need to work out some logistic on how to release it.

    It's not perfect yet, I trained it on very little data, alot of objects are missing. In the future I will expand the model to detect more objects.
    Last edited by Grunt; 08-28-2017 at 07:23 AM.

  2. #2
    Join Date
    Oct 2012
    Posts
    1,258
    Mentioned
    40 Post(s)
    Quoted
    588 Post(s)

    Default

    That's pretty neat. How's the regular arduino botting been for ya?

  3. #3
    Join Date
    Dec 2008
    Posts
    135
    Mentioned
    0 Post(s)
    Quoted
    44 Post(s)

    Default

    Right now, I am working on creating a way to detect x,y coordinates based on the image of the minimap. Since detecting location accurately is probably the next obstacle. I am thinking of some kind convnet architecture that outputs a combination of linear regression and classification nodes. The problem I have is collecting data. It would involve the picture of the minimap + coordinates output. Do you guys have a recommendation for a reflection client that have updated reflection hooks for coordinates? Or perhaps an even better way is to use a custom client that can teleport around glienor.


    And the Arduino bot has not gotten me banned.

  4. #4
    Join Date
    Mar 2016
    Location
    Canada
    Posts
    32
    Mentioned
    0 Post(s)
    Quoted
    3 Post(s)

    Default

    Damn, seeing stuff like this gives me the motivation to learn how to program, but anyways nice job!

  5. #5
    Join Date
    Jun 2017
    Posts
    13
    Mentioned
    0 Post(s)
    Quoted
    2 Post(s)

    Default

    Your strategy for detecting world coordinates from image position will be inaccurate to say the least. Can you be more specific about a 'custom client' to teleport around glienor?

  6. #6
    Join Date
    Dec 2010
    Posts
    483
    Mentioned
    30 Post(s)
    Quoted
    328 Post(s)

    Default

    Quote Originally Posted by Grunt View Post
    Right now, I am working on creating a way to detect x,y coordinates based on the image of the minimap.
    https://villavu.com/forum/showthread.php?p=829125


    I gave a presentation on TensorFlow awhile back as part of a Machine Learning course. Though admittedly, very cool, don't you think this will be a tad too "gimicky" to actually work as a focused botting architecture?

  7. #7
    Join Date
    Dec 2008
    Posts
    135
    Mentioned
    0 Post(s)
    Quoted
    44 Post(s)

    Default

    That thread hasn't been updated in a while. Plus features on my world map localization is not handcrafted, and has the potential to work better.

    too "gimicky"
    I might have agreed with you if this was caffe or theano. But tensorflow is specifically designed for production ready environments. Tensorflow is probably the most popular deep learning library. From everyday tasks to even Alpha-go, runs on tensorflow. I predict its gonna be very popular in mobile development. In fact, snapdragon is releasing a processor optimized to run tensorflow.

  8. #8
    Join Date
    Dec 2007
    Posts
    2,112
    Mentioned
    71 Post(s)
    Quoted
    580 Post(s)

    Default

    That last point seems to be a little off?

    Anyhow, nice to see someone exploring some other angles with rs botting. I suspect that this only works at north. and also doesn't work as well in areas with less color variation.

  9. #9
    Join Date
    Feb 2007
    Location
    Colorado, USA
    Posts
    3,716
    Mentioned
    51 Post(s)
    Quoted
    624 Post(s)

    Default

    I wouldn't make a windows version, it's a useless operating system.

    We actually used to have a full coordinates system with auto walking (anywhere) with x,y for the entire RS map hehehe

    good job though, that's hilarious and truly undetectable.
    In the future our robot house cleaners can game for us. I currently unemploy people with raspberry Pis so it'll happen

    is tensorflow better than torch for you?
    The only true authority stems from knowledge, not from position.

    You can contact me via matrix protocol: @grats:grats.win or you can email me at the same domain, any user/email address.

  10. #10
    Join Date
    Dec 2008
    Posts
    135
    Mentioned
    0 Post(s)
    Quoted
    44 Post(s)

    Default

    Currently testing power mining iron using pybot? 24/7 running no breaks and script have no randoms whatsoever. Now if the theory about using hardware mouse is right?I should not get banned

  11. #11
    Join Date
    Aug 2007
    Location
    Colorado
    Posts
    7,421
    Mentioned
    268 Post(s)
    Quoted
    1442 Post(s)

    Default

    Impressive indeed. Between this and the other Python-related project, it's possibly a new realm of RS-macroing. I'll be following this though I fear any contributions I could make personally wouldn't go further than scripts, if these projects ever reach that point.

    Current projects:
    [ AeroGuardians (GotR minigame), Motherlode Miner, Blast furnace ]

    "I won't fall in your gravity. Open your eyes,
    you're the Earth and I'm the sky..."


  12. #12
    Join Date
    Dec 2008
    Posts
    135
    Mentioned
    0 Post(s)
    Quoted
    44 Post(s)

    Default

    Implemented OCR with tesseract-ocr from source (4.0.0), significantly better than the released 3.5 version
    Mine Rocks / 3 more options
    Mine Rocks / 3 more options
    Walk here
    Build Inconspicuous rocks (easy)
    Walk here

    Walk here
    Walk here
    Chop down Tree I 2 more options
    Chop down Tree I 2 more options
    Chop down Tree I 2 more options
    Walk here

  13. #13
    Join Date
    Jul 2017
    Posts
    3
    Mentioned
    0 Post(s)
    Quoted
    3 Post(s)

    Default

    Hey! I just made an account on this website to respond to you.

    I'm also working on something like this.

    As it is right now, my bot is able to detect other players, I already have OCR with tesseract and I can mimic human mouse movement!

    We should collaborate if you're down.

    I couldn't get SSD to work so I'm using tiny-yolo instead.

    Would you be down to share your image dataset for the iron rocks?

  14. #14
    Join Date
    Dec 2008
    Posts
    135
    Mentioned
    0 Post(s)
    Quoted
    44 Post(s)

    Default

    Here are the rocks dataset.
    https://www.dropbox.com/s/ifpj2wz2wg...ks.tar.gz?dl=0
    I am still not done with making the dataset for rocks. So I am always looking for more annotations and especially diverse annotations.

    As far as coding goes, pybot is still very much in experimental phase. Currently I wrote a couple of rudimentary scripts that is working very well. Once I successfully a achieve a few 99 skills using those script. I will be confident to release the entire thing. To me writing the script itself is not how I plan to contribute to this open source project. Rather I see my role as implementing deep learning framework as well as some general computer vision technique to solve automation tasks.

    The most important and difficult problem to solve is the rs map. I am working on detecting the coordinate from rs map. Tbh I don't think any botting framework is worth relesing without such system.

  15. #15
    Join Date
    Mar 2017
    Posts
    32
    Mentioned
    1 Post(s)
    Quoted
    10 Post(s)

    Default

    what kind of progress have you had w/ the powermining acct? still no ban?

  16. #16
    Join Date
    Dec 2008
    Posts
    135
    Mentioned
    0 Post(s)
    Quoted
    44 Post(s)

    Default

    Quote Originally Posted by ccesssu View Post
    what kind of progress have you had w/ the powermining acct? still no ban?
    No bans, no random whatsoever. Clicking on the rocks varies sometimes since it detects rocks which move with screen shift, other than that no random whatsoever. Script logs me back in after 6 hours log. I bot it about 18-20 hours out of 24 hours. Since I use the computer when I get home.

  17. #17
    Join Date
    Mar 2012
    Posts
    107
    Mentioned
    2 Post(s)
    Quoted
    49 Post(s)

    Default

    Are you gonna post a source? I love this project and would like to develop for it and learn from it in some way.

  18. #18
    Join Date
    Dec 2008
    Posts
    135
    Mentioned
    0 Post(s)
    Quoted
    44 Post(s)

    Default

    Quote Originally Posted by One Kid View Post
    Are you gonna post a source? I love this project and would like to develop for it and learn from it in some way.
    Yes I will

  19. #19
    Join Date
    Dec 2007
    Posts
    2,112
    Mentioned
    71 Post(s)
    Quoted
    580 Post(s)

    Default

    Quote Originally Posted by Grunt View Post
    The most important and difficult problem to solve is the rs map. I am working on detecting the coordinate from rs map. Tbh I don't think any botting framework is worth relesing without such system.
    I suspect this should be a lot easier considering the map update.

  20. #20
    Join Date
    Feb 2007
    Location
    Het ademt zwaar en moedeloos vannacht.
    Posts
    7,211
    Mentioned
    26 Post(s)
    Quoted
    72 Post(s)

    Default

    Could you share some info on your CNN architecture? Which hyperparameters etc, and how did you train it with what kind of training data?

    My (professional) experience with ConvNets is that they can be a real PITA to train without too much overfitting, and can get too big quickly (as in, multiple gigabytes of weights). Just wondering how you dealw ith those problems.
    I made a new script, check it out!.

  21. #21
    Join Date
    Dec 2008
    Posts
    135
    Mentioned
    0 Post(s)
    Quoted
    44 Post(s)

    Default

    Quote Originally Posted by Markus View Post
    Could you share some info on your CNN architecture? Which hyperparameters etc, and how did you train it with what kind of training data?

    My (professional) experience with ConvNets is that they can be a real PITA to train without too much overfitting, and can get too big quickly (as in, multiple gigabytes of weights). Just wondering how you dealw ith those problems.
    I used a very simple alexnet architecture on data that I collected myself(manually) But from my experience training from scratch will always get you over-fitting even with simple convnets. There just isn't enough balanced data. So it's almost always better to use per-trained weights. However, in the case of detecting locations on a maps, overfitting is not a big deal since your test data is almost the same as your training, so even if your net memorize the datasets it doesn't matter.

    For detecting objects, I am using the pretrained weights from VGG-512. I am on discord very often, we can discuss more.

  22. #22
    Join Date
    Jul 2017
    Posts
    1
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    This is really cool!
    I'm very new to RS botting, and pretty new to tensorflow.. and am having trouble wrapping my head around how you are managing the bot's state.

    Specifically, how does it know to go to the bank vs return to rocks vs mine?

    Are you using parent TF model using the outputs of minimap + rock CNN to make a decision?
    or do you have a script managing most of the process and are using the model for the mining part?

  23. #23
    Join Date
    Mar 2016
    Posts
    9
    Mentioned
    0 Post(s)
    Quoted
    2 Post(s)

    Default

    Quote Originally Posted by PshYouLost View Post
    Hey! I just made an account on this website to respond to you.

    I'm also working on something like this.

    As it is right now, my bot is able to detect other players, I already have OCR with tesseract and I can mimic human mouse movement!

    We should collaborate if you're down.

    I couldn't get SSD to work so I'm using tiny-yolo instead.

    Would you be down to share your image dataset for the iron rocks?

    Hey, how did you manage to mimic human mouse movements?

  24. #24
    Join Date
    Mar 2016
    Posts
    9
    Mentioned
    0 Post(s)
    Quoted
    2 Post(s)

    Default

    Is it possible for you to PM your discord to me? I have a few questions about the arduino mouse, and this project in general Thanks

  25. #25
    Join Date
    Jul 2017
    Posts
    3
    Mentioned
    0 Post(s)
    Quoted
    3 Post(s)

    Default

    Quote Originally Posted by legitmail View Post
    Hey, how did you manage to mimic human mouse movements?
    I used a sort of LSTM and just made predictions based on single time intervals until the correct x and y coordinate

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •