Page 1 of 2 12 LastLast
Results 1 to 25 of 26

Thread: May 19th, Stats updates

  1. #1
    Join Date
    Feb 2006
    Location
    Amsterdam
    Posts
    13,691
    Mentioned
    146 Post(s)
    Quoted
    130 Post(s)

    Default May 19th, Stats updates

    I've taken some time to update the SRL Stats caching code. We'll have to see how it goes, I'm not extremely happy about the code right now, but I wasn't before either.

    It also seems we've reached the limit of PostgreSQL integers... So I might consider dropping all commits before say, 2012. (And safely store them somewhere else)

    We'll see how it goes. It shouldn't block anymore. So no more connection time-outs.

    E: Oh, and all cache is now real-time.
    Last edited by Wizzup?; 05-19-2012 at 05:46 PM.



    The best way to contact me is by email, which you can find on my website: http://wizzup.org
    I also get email notifications of private messages, though.

    Simba (on Twitter | Group on Villavu | Website | Stable/Unstable releases
    Documentation | Source | Simba Bug Tracker on Github and Villavu )


    My (Blog | Website)

  2. #2
    Join Date
    Mar 2012
    Location
    127.0.0.1
    Posts
    3,383
    Mentioned
    95 Post(s)
    Quoted
    717 Post(s)

    Default

    Awesome. Nice job, Wizzup. xD

  3. #3
    Join Date
    Oct 2008
    Location
    C:\Simba\Includes\
    Posts
    7,566
    Mentioned
    19 Post(s)
    Quoted
    180 Post(s)

    Default

    Woot! Nice job! Thanks a bunch! <3
    Away for awhile, life is keeping me busy. | Want to get my attention in a thread? @Kyle Undefined; me.
    { MSI Phoenix || SRL Stats Sigs || Paste || Scripts || Quotes || Graphics }

    When posting a bug, please post debug! Help us, help you!

    I would love to change the world, but they won't give me the source code. || To be the best, you've got to beat the rest. || Logic never changes, just the syntax.
    If you PM me with a stupid question or one listed in FAQ, or about a script that is not mine, I will NOT respond.


    SRL is a Library of routines made by the SRL community written for the Program Simba. We produce Scripts for the game Runescape.


  4. #4
    Join Date
    Feb 2006
    Location
    Amsterdam
    Posts
    13,691
    Mentioned
    146 Post(s)
    Quoted
    130 Post(s)

    Default

    I'm also seriously considering dropping all the Exp gained variables. They're exceeding the maximum value of an integer...:

    Code:
    >>> 10705011677 
    10705011677
    >>> 2**32
    4294967296
    I never expected stats to grow this large, you see.
    Last edited by Wizzup?; 05-19-2012 at 05:58 PM.



    The best way to contact me is by email, which you can find on my website: http://wizzup.org
    I also get email notifications of private messages, though.

    Simba (on Twitter | Group on Villavu | Website | Stable/Unstable releases
    Documentation | Source | Simba Bug Tracker on Github and Villavu )


    My (Blog | Website)

  5. #5
    Join Date
    Oct 2008
    Location
    C:\Simba\Includes\
    Posts
    7,566
    Mentioned
    19 Post(s)
    Quoted
    180 Post(s)

    Default

    The vars still aren't showing up for my Shilo script :/

    http://stats.villavu.com/api/script/841

    That's the API and as you can see, the vars are empty. The latest commit does infact submit data for the vars, as you can see here.

    http://stats.villavu.com/commit/16275569
    Away for awhile, life is keeping me busy. | Want to get my attention in a thread? @Kyle Undefined; me.
    { MSI Phoenix || SRL Stats Sigs || Paste || Scripts || Quotes || Graphics }

    When posting a bug, please post debug! Help us, help you!

    I would love to change the world, but they won't give me the source code. || To be the best, you've got to beat the rest. || Logic never changes, just the syntax.
    If you PM me with a stupid question or one listed in FAQ, or about a script that is not mine, I will NOT respond.


    SRL is a Library of routines made by the SRL community written for the Program Simba. We produce Scripts for the game Runescape.


  6. #6
    Join Date
    Feb 2012
    Location
    Minneapolis
    Posts
    117
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    You sound really illigeticalant.
    Progress Report:
    2/3

  7. #7
    Join Date
    Feb 2006
    Location
    Amsterdam
    Posts
    13,691
    Mentioned
    146 Post(s)
    Quoted
    130 Post(s)

    Default

    Quote Originally Posted by Kyle Undefined View Post
    The vars still aren't showing up for my Shilo script :/

    http://stats.villavu.com/api/script/841

    That's the API and as you can see, the vars are empty. The latest commit does infact submit data for the vars, as you can see here.

    http://stats.villavu.com/commit/16275569
    Bah, the current cache mechanism indeed doesn't account for new variables... I'd have to extend the table. Not that hard; but won't do it right away. Again, no worries - no information will be lost.

    That said, it may take a few days before I fix this. I really wanted to move on to the new code. Sorry for the issue. It's better than not updating at all though, right?



    The best way to contact me is by email, which you can find on my website: http://wizzup.org
    I also get email notifications of private messages, though.

    Simba (on Twitter | Group on Villavu | Website | Stable/Unstable releases
    Documentation | Source | Simba Bug Tracker on Github and Villavu )


    My (Blog | Website)

  8. #8
    Join Date
    Oct 2010
    Posts
    1,255
    Mentioned
    0 Post(s)
    Quoted
    15 Post(s)

    Default

    Quote Originally Posted by Adamcoled22 View Post
    You sound really illigeticalant.
    You now hold the sole google result for "illigeticalant", how do you feel?

    I googled it cause I thought it was a meme like "moran" or something.
    I'm back

  9. #9
    Join Date
    Feb 2011
    Location
    The Future.
    Posts
    5,600
    Mentioned
    396 Post(s)
    Quoted
    1598 Post(s)

    Default

    Hmm why not use Int64 aka unsigned ints? It goes from 0 to 9.2+ billion :S Since stats don't use negative values, we can exclude integers and use only the unsigned versions of them? Does this language not have them?
    I am Ggzz..
    Hackintosher

  10. #10
    Join Date
    Feb 2006
    Location
    Amsterdam
    Posts
    13,691
    Mentioned
    146 Post(s)
    Quoted
    130 Post(s)

    Default

    Quote Originally Posted by Brandon View Post
    Hmm why not use Int64 aka unsigned ints? It goes from 0 to 9.2+ billion :S Since stats don't use negative values, we can exclude integers and use only the unsigned versions of them? Does this language not have them?
    I think it already uses 64 bit integers and just gave a warning. I'll need to investigate.



    The best way to contact me is by email, which you can find on my website: http://wizzup.org
    I also get email notifications of private messages, though.

    Simba (on Twitter | Group on Villavu | Website | Stable/Unstable releases
    Documentation | Source | Simba Bug Tracker on Github and Villavu )


    My (Blog | Website)

  11. #11
    Join Date
    Oct 2008
    Location
    C:\Simba\Includes\
    Posts
    7,566
    Mentioned
    19 Post(s)
    Quoted
    180 Post(s)

    Default

    Quote Originally Posted by Wizzup? View Post
    Bah, the current cache mechanism indeed doesn't account for new variables... I'd have to extend the table. Not that hard; but won't do it right away. Again, no worries - no information will be lost.

    That said, it may take a few days before I fix this. I really wanted to move on to the new code. Sorry for the issue. It's better than not updating at all though, right?
    No worries

    Just a heads up, that is all. Yes, I agree. I'd rather it update than not.

    Thanks for the work!
    Away for awhile, life is keeping me busy. | Want to get my attention in a thread? @Kyle Undefined; me.
    { MSI Phoenix || SRL Stats Sigs || Paste || Scripts || Quotes || Graphics }

    When posting a bug, please post debug! Help us, help you!

    I would love to change the world, but they won't give me the source code. || To be the best, you've got to beat the rest. || Logic never changes, just the syntax.
    If you PM me with a stupid question or one listed in FAQ, or about a script that is not mine, I will NOT respond.


    SRL is a Library of routines made by the SRL community written for the Program Simba. We produce Scripts for the game Runescape.


  12. #12
    Join Date
    Jan 2012
    Posts
    2,568
    Mentioned
    35 Post(s)
    Quoted
    356 Post(s)

    Default

    why not reduce xp/gp gained to "in thousands", so any commit make to gp/xp will be automatically be divided by 1000 and rounded (i.e. Round(XpGained/1000), then just add headings to indicate that the values are in thousands. This would make xp/gp gains less accurate but since the amount gained is hugh, and over time it will get balanced and wont deviate from the true value too much.

    Things like monsters killed or loads/trips done would remain as it is now though, only for variables that have large values being committed.

  13. #13
    Join Date
    Oct 2009
    Location
    Stockton, CA
    Posts
    2,040
    Mentioned
    0 Post(s)
    Quoted
    1 Post(s)

    Default

    Quote Originally Posted by riwu View Post
    why not reduce xp/gp gained to "in thousands", so any commit make to gp/xp will be automatically be divided by 1000 and rounded (i.e. Round(XpGained/1000), then just add headings to indicate that the values are in thousands. This would make xp/gp gains less accurate but since the amount gained is hugh, and over time it will get balanced and wont deviate from the true value too much.

    Things like monsters killed or loads/trips done would remain as it is now though, only for variables that have large values being committed.
    Exactly my thoughts.
    Join the IRC! irc.rizon.net:6667/srl | SQLite (0.99rc3+) | SRL Doc | Simba Doc | Extra Simba Libraries (openSSL & sqlite3)
    Quote Originally Posted by #srl
    10:45 < Toter> daphil when can get sex anyday I want
    10:45 < Toter> he is always on #SRL
    "A programmer is just a tool which converts caffeine into code"

  14. #14
    Join Date
    Feb 2011
    Location
    The Future.
    Posts
    5,600
    Mentioned
    396 Post(s)
    Quoted
    1598 Post(s)

    Default

    Quote Originally Posted by Wizzup? View Post
    I think it already uses 64 bit integers and just gave a warning. I'll need to investigate.
    Well according to the values you gave, matching it up with the ranges of variable data types, that's still 32 bit (4bytes).. Just the unsigned versions which made me assume that it's overflowing rather than warning. Still not sure if the language stats is written in supports 64 though. I don't even know the language so yeah.

    int: –2,147,483,648 to 2,147,483,647
    uint32: 0 to 4,294,967,295 (Current range that the stats system uses..)
    int64: –9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
    uint64: 0 to 18,446,744,073,709,551,615


    So that's why I assumed that it was still using the 4 byte values instead of the 8 byte longs. Also the guy above me has a great suggestion about dividing them by the thousands though I say the change in datatypes are probably easiest to implement (could be wrong though). =]

    Goodluck though with whatever decision you have to make. As long as it works.
    Last edited by Brandon; 05-20-2012 at 05:25 AM.
    I am Ggzz..
    Hackintosher

  15. #15
    Join Date
    Mar 2012
    Posts
    22
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    Not really sure what you did but thanks for your hard work!

  16. #16
    Join Date
    Feb 2006
    Location
    Amsterdam
    Posts
    13,691
    Mentioned
    146 Post(s)
    Quoted
    130 Post(s)

    Default

    Quote Originally Posted by riwu View Post
    why not reduce xp/gp gained to "in thousands", so any commit make to gp/xp will be automatically be divided by 1000 and rounded (i.e. Round(XpGained/1000), then just add headings to indicate that the values are in thousands. This would make xp/gp gains less accurate but since the amount gained is hugh, and over time it will get balanced and wont deviate from the true value too much.

    Things like monsters killed or loads/trips done would remain as it is now though, only for variables that have large values being committed.
    I don't think it works that way. Say I commit 450xp every commit, then that'll all be lost. Plus, XPGained can be calculated from all the other variables.

    I'll keep it for now and see how long it holds.



    The best way to contact me is by email, which you can find on my website: http://wizzup.org
    I also get email notifications of private messages, though.

    Simba (on Twitter | Group on Villavu | Website | Stable/Unstable releases
    Documentation | Source | Simba Bug Tracker on Github and Villavu )


    My (Blog | Website)

  17. #17
    Join Date
    Jan 2012
    Posts
    2,568
    Mentioned
    35 Post(s)
    Quoted
    356 Post(s)

    Default

    Quote Originally Posted by Wizzup? View Post
    I don't think it works that way. Say I commit 450xp every commit, then that'll all be lost. Plus, XPGained can be calculated from all the other variables.

    I'll keep it for now and see how long it holds.
    Well then maybe change it to "in hundreds"? If a script commits even lower xp then it can always be set to commit once every 30min so that the cumulative xp is in the hundreds/thousands.

    Or just retain the current variable and add the thousands version, so scripts that commit low values will continue using the old variables, and scripts that commit very high values can use the new "in thousands" variable.

    If you have other ways to fix this limit it would be better though, but i think dropping all commits before a date would only be a short-term plan, as regularly dropping commits would no longer make a script's stat reflect its real longevity and total usage.

  18. #18
    Join Date
    Jan 2012
    Posts
    273
    Mentioned
    7 Post(s)
    Quoted
    6 Post(s)

    Default

    Quote Originally Posted by Wizzup? View Post
    I've taken some time to update the SRL Stats caching code. We'll have to see how it goes, I'm not extremely happy about the code right now, but I wasn't before either.

    It also seems we've reached the limit of PostgreSQL integers... So I might consider dropping all commits before say, 2012. (And safely store them somewhere else)

    We'll see how it goes. It shouldn't block anymore. So no more connection time-outs.

    E: Oh, and all cache is now real-time.
    just fyi... not all stats are being shown, they may be accounted for but when you look at stats in the commits page, you will see many gaps in numbers

    now, i don't know how many scripts are affected, but as for my script (1037), only commits without variables are showing up - which normally is about 5% of commits and now of the visible is 100%, but if you look back 1 1/2 days you would have hard time finding one of those

    it broke sometime in last 1 1/2 days

    here is an example:

    Commit ID Script Name User Name Time (Minutes) Submitted
    18444102 MethDwarfMiner Anonymous 5 Sun May 20 17:08:51 2012
    18444101 Heavy-rodder Anonymous 5 Sun May 20 17:08:50 2012
    18444100 Siel's Karamja Fisher Anonymous 7 Sun May 20 17:08:50 2012
    18444099 Narcle's AK Smelter syg44ry 5 Sun May 20 17:08:49 2012
    18444097 Er1k/Harry's SWFletch Anonymous 6 Sun May 20 17:08:49 2012
    18444096 Narcle's AK Smelter Anonymous 6 Sun May 20 17:08:46 20


    so if you look at it, where is commit 18444098?

    EDIT- if it helps this is last commit with variables:

    18286496 5 Anonymous (Show commits only from Anonymous) Sat May 19 19:24:01 2012 (http://stats.villavu.com/commit/18286496)

    currently on page 22 (http://stats.villavu.com/script/1037/commits/22)
    Last edited by zmon; 05-20-2012 at 03:36 PM. Reason: last variable commit
    Perfect script? There is no such thing as "perfect", only "better than you expect".

  19. #19
    Join Date
    Apr 2012
    Posts
    19
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default error_game_jsconnect

    unavable to update connect server.any one here experience this error_game_jsconnect

  20. #20
    Join Date
    Jan 2012
    Posts
    273
    Mentioned
    7 Post(s)
    Quoted
    6 Post(s)

    Default

    this may be of help too:

    Progress Report:
    formWriteln: SRL_Stats: No POST return
    Perfect script? There is no such thing as "perfect", only "better than you expect".

  21. #21
    Join Date
    Jan 2012
    Posts
    273
    Mentioned
    7 Post(s)
    Quoted
    6 Post(s)

    Default

    Quote Originally Posted by zmon View Post
    this may be of help too:

    Progress Report:
    formWriteln: SRL_Stats: No POST return
    EDIT: trapped the response for you:

    Progress Report:
    Traceback (most recent call last):
      File "/usr/lib64/python2.7/site-packages/flup/server/fcgi_base.py", line 558,
    in run
        protocolStatus, appStatus = self.server.handler(self)
      File "/usr/lib64/python2.7/site-packages/flup/server/fcgi_base.py", line 1118,
     in handler
        result = self.application(environ, start_response)
      File "/usr/lib64/python2.7/site-packages/beaker/middleware.py", line 152, in _
    _call__
        return self.wrap_app(environ, session_start_response)
      File "/srv/srlstats/SRL-Stats/stats.py", line 208, in __call__
        ret = self.app(env, start_response)
      File "/srv/srlstats/SRL-Stats/stats.py", line 253, in __call__
        return self.app(env, start_response)
      File "/srv/srlstats/SRL-Stats/stats.py", line 165, in stats
        r = wt.apply_rule(env['PATH_INFO'], env)
      File "/srv/srlstats/SRL-Stats/webtool.py", line 47, in apply_rule
        return fv['func'](env=env, **dict(zip(fv['vars'], l)))
      File "/srv/srlstats/SRL-Stats/stats.py", line 737, in api_commit
        res = ct.add(user, script, time, vars)
      File "/srv/srlstats/SRL-Stats/query.py", line 305, in add
        v.amount += y
    AttributeError: 'NoneType' object has no attribute 'amount'


    so it actually looks like the stats are being lost because query.py blows up before it tries to save it to db, so that's why they're not showing up


    if you need to debug, here is one that i tried to commit when i was trapping the response:


    /srv/srlstats/SRL-Stats/query.py in add(self=<query.CommitTool object>, user=User(zdenek), script=Script(Preemptible RuneSpan : User(zdenek)), time=5, vars={Variable(Total EXP Gained): 3022, Variable(Earth Runes (Crafted)): 25, Variable(Runecrafting EXP (Gained)): 3022, Variable(Rune Essence (Mined)): 271, Variable(Cosmic Runes (Crafted)): 52, Variable(Water Runes (Crafted)): 25})

    Edit #2 i missed the line where you log requests so i suppose you can pull it up and test a few more scenarios, if you are logging info level
    Last edited by zmon; 05-20-2012 at 11:33 PM.
    Perfect script? There is no such thing as "perfect", only "better than you expect".

  22. #22
    Join Date
    Dec 2011
    Location
    Los Angeles, California
    Posts
    151
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    EDIT: Never mind.
    Last edited by Green Bay Packers; 05-23-2012 at 12:00 AM.

  23. #23
    Join Date
    Oct 2006
    Posts
    22
    Mentioned
    0 Post(s)
    Quoted
    0 Post(s)

    Default

    Awesome, thanks wizzup?!

  24. #24
    Join Date
    Jan 2012
    Posts
    273
    Mentioned
    7 Post(s)
    Quoted
    6 Post(s)

    Default

    still getting same error on updates, it would be helpful if i would know whether i should disable all variables in scripts that i'm running (aka, are all the submissions being lost?)

    i have not had one successful submit on my account since 19th and i had run several scripts that report variables

    EDIT:

    I am now working on script that does not even have any variables and it still fails to post with stats server kinda blowing up (script 1056) and this is what i'm seeing:

    Progress Report:
    formWriteln: SRL_Stats: No POST return
    formWriteln: SRL_Stats: No POST return
    formWriteln: SRL_Stats: No POST return
    SMART Disabled
    Disabeling Events
    Starting Image Transfer
    SMART Enabled
    SMART Disabled
    Disabeling Events
    Starting Image Transfer
    SMART Enabled
    formWriteln: SRL_Stats: No POST return
    Current Simba version: 984
    Latest Simba Version: 984
    formWriteln: SRL_Stats: No POST return
    Last edited by zmon; 05-25-2012 at 09:57 PM.
    Perfect script? There is no such thing as "perfect", only "better than you expect".

  25. #25
    Join Date
    Nov 2011
    Location
    England
    Posts
    3,072
    Mentioned
    296 Post(s)
    Quoted
    1094 Post(s)

    Default

    Quote Originally Posted by zmon View Post
    still getting same error on updates, it would be helpful if i would know whether i should disable all variables in scripts that i'm running (aka, are all the submissions being lost?)

    i have not had one successful submit on my account since 19th and i had run several scripts that report variables

    EDIT:

    I am now working on script that does not even have any variables and it still fails to post with stats server kinda blowing up (script 1056) and this is what i'm seeing:

    Progress Report:
    formWriteln: SRL_Stats: No POST return
    formWriteln: SRL_Stats: No POST return
    formWriteln: SRL_Stats: No POST return
    SMART Disabled
    Disabeling Events
    Starting Image Transfer
    SMART Enabled
    SMART Disabled
    Disabeling Events
    Starting Image Transfer
    SMART Enabled
    formWriteln: SRL_Stats: No POST return
    Current Simba version: 984
    Latest Simba Version: 984
    formWriteln: SRL_Stats: No POST return
    Same problem here :S

Page 1 of 2 12 LastLast

Thread Information

Users Browsing this Thread

There are currently 1 users browsing this thread. (0 members and 1 guests)

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •