PDA

View Full Version : May 19th, Stats updates



Wizzup?
05-19-2012, 05:41 PM
I've taken some time to update the SRL Stats caching code. We'll have to see how it goes, I'm not extremely happy about the code right now, but I wasn't before either. ;)

It also seems we've reached the limit of PostgreSQL integers... So I might consider dropping all commits before say, 2012. (And safely store them somewhere else)

We'll see how it goes. It shouldn't block anymore. So no more connection time-outs.

E: Oh, and all cache is now real-time.

NKN
05-19-2012, 05:45 PM
Awesome. Nice job, Wizzup. xD

Kyle Undefined
05-19-2012, 05:53 PM
Woot! Nice job! Thanks a bunch! <3

Wizzup?
05-19-2012, 05:56 PM
I'm also seriously considering dropping all the Exp gained variables. They're exceeding the maximum value of an integer...:


>>> 10705011677
10705011677
>>> 2**32
4294967296


I never expected stats to grow this large, you see.

Kyle Undefined
05-19-2012, 07:12 PM
The vars still aren't showing up for my Shilo script :/

http://stats.villavu.com/api/script/841

That's the API and as you can see, the vars are empty. The latest commit does infact submit data for the vars, as you can see here.

http://stats.villavu.com/commit/16275569

Adamcoled
05-19-2012, 11:32 PM
You sound really illigeticalant.

Wizzup?
05-20-2012, 12:36 AM
The vars still aren't showing up for my Shilo script :/

http://stats.villavu.com/api/script/841

That's the API and as you can see, the vars are empty. The latest commit does infact submit data for the vars, as you can see here.

http://stats.villavu.com/commit/16275569

Bah, the current cache mechanism indeed doesn't account for new variables... I'd have to extend the table. Not that hard; but won't do it right away. Again, no worries - no information will be lost.

That said, it may take a few days before I fix this. I really wanted to move on to the new code. Sorry for the issue. It's better than not updating at all though, right? ;)

smurg
05-20-2012, 01:05 AM
You sound really illigeticalant.

You now hold the sole google result for "illigeticalant", how do you feel?

I googled it cause I thought it was a meme like "moran" or something.

Brandon
05-20-2012, 01:13 AM
Hmm why not use Int64 aka unsigned ints? It goes from 0 to 9.2+ billion :S Since stats don't use negative values, we can exclude integers and use only the unsigned versions of them? Does this language not have them?

Wizzup?
05-20-2012, 01:42 AM
Hmm why not use Int64 aka unsigned ints? It goes from 0 to 9.2+ billion :S Since stats don't use negative values, we can exclude integers and use only the unsigned versions of them? Does this language not have them?

I think it already uses 64 bit integers and just gave a warning. I'll need to investigate.

Kyle Undefined
05-20-2012, 02:29 AM
Bah, the current cache mechanism indeed doesn't account for new variables... I'd have to extend the table. Not that hard; but won't do it right away. Again, no worries - no information will be lost.

That said, it may take a few days before I fix this. I really wanted to move on to the new code. Sorry for the issue. It's better than not updating at all though, right? ;)

No worries :)

Just a heads up, that is all. Yes, I agree. I'd rather it update than not.

Thanks for the work! :D

riwu
05-20-2012, 04:44 AM
why not reduce xp/gp gained to "in thousands", so any commit make to gp/xp will be automatically be divided by 1000 and rounded (i.e. Round(XpGained/1000), then just add headings to indicate that the values are in thousands. This would make xp/gp gains less accurate but since the amount gained is hugh, and over time it will get balanced and wont deviate from the true value too much.

Things like monsters killed or loads/trips done would remain as it is now though, only for variables that have large values being committed.

Sex
05-20-2012, 05:17 AM
why not reduce xp/gp gained to "in thousands", so any commit make to gp/xp will be automatically be divided by 1000 and rounded (i.e. Round(XpGained/1000), then just add headings to indicate that the values are in thousands. This would make xp/gp gains less accurate but since the amount gained is hugh, and over time it will get balanced and wont deviate from the true value too much.

Things like monsters killed or loads/trips done would remain as it is now though, only for variables that have large values being committed.

Exactly my thoughts.

Brandon
05-20-2012, 05:22 AM
I think it already uses 64 bit integers and just gave a warning. I'll need to investigate.

Well according to the values you gave, matching it up with the ranges of variable data types, that's still 32 bit (4bytes).. Just the unsigned versions which made me assume that it's overflowing rather than warning. Still not sure if the language stats is written in supports 64 though. I don't even know the language so yeah.

int: –2,147,483,648 to 2,147,483,647
uint32: 0 to 4,294,967,295 (Current range that the stats system uses..)
int64: –9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
uint64: 0 to 18,446,744,073,709,551,615


So that's why I assumed that it was still using the 4 byte values instead of the 8 byte longs. Also the guy above me has a great suggestion about dividing them by the thousands though I say the change in datatypes are probably easiest to implement (could be wrong though). =]

Goodluck though with whatever decision you have to make. As long as it works.

Eluvies
05-20-2012, 10:15 AM
Not really sure what you did but thanks for your hard work! :D

Wizzup?
05-20-2012, 10:44 AM
why not reduce xp/gp gained to "in thousands", so any commit make to gp/xp will be automatically be divided by 1000 and rounded (i.e. Round(XpGained/1000), then just add headings to indicate that the values are in thousands. This would make xp/gp gains less accurate but since the amount gained is hugh, and over time it will get balanced and wont deviate from the true value too much.

Things like monsters killed or loads/trips done would remain as it is now though, only for variables that have large values being committed.

I don't think it works that way. Say I commit 450xp every commit, then that'll all be lost. Plus, XPGained can be calculated from all the other variables. ;)

I'll keep it for now and see how long it holds. ;)

riwu
05-20-2012, 12:38 PM
I don't think it works that way. Say I commit 450xp every commit, then that'll all be lost. Plus, XPGained can be calculated from all the other variables. ;)

I'll keep it for now and see how long it holds. ;)
Well then maybe change it to "in hundreds"? If a script commits even lower xp then it can always be set to commit once every 30min so that the cumulative xp is in the hundreds/thousands.

Or just retain the current variable and add the thousands version, so scripts that commit low values will continue using the old variables, and scripts that commit very high values can use the new "in thousands" variable.

If you have other ways to fix this limit it would be better though, but i think dropping all commits before a date would only be a short-term plan, as regularly dropping commits would no longer make a script's stat reflect its real longevity and total usage.

zmon
05-20-2012, 02:54 PM
I've taken some time to update the SRL Stats caching code. We'll have to see how it goes, I'm not extremely happy about the code right now, but I wasn't before either. ;)

It also seems we've reached the limit of PostgreSQL integers... So I might consider dropping all commits before say, 2012. (And safely store them somewhere else)

We'll see how it goes. It shouldn't block anymore. So no more connection time-outs.

E: Oh, and all cache is now real-time.

just fyi... not all stats are being shown, they may be accounted for but when you look at stats in the commits page, you will see many gaps in numbers

now, i don't know how many scripts are affected, but as for my script (1037), only commits without variables are showing up - which normally is about 5% of commits and now of the visible is 100%, but if you look back 1 1/2 days you would have hard time finding one of those

it broke sometime in last 1 1/2 days

here is an example:

Commit ID Script Name User Name Time (Minutes) Submitted
18444102 MethDwarfMiner Anonymous 5 Sun May 20 17:08:51 2012
18444101 Heavy-rodder Anonymous 5 Sun May 20 17:08:50 2012
18444100 Siel's Karamja Fisher Anonymous 7 Sun May 20 17:08:50 2012
18444099 Narcle's AK Smelter syg44ry 5 Sun May 20 17:08:49 2012
18444097 Er1k/Harry's SWFletch Anonymous 6 Sun May 20 17:08:49 2012
18444096 Narcle's AK Smelter Anonymous 6 Sun May 20 17:08:46 20


so if you look at it, where is commit 18444098?

EDIT- if it helps this is last commit with variables:

18286496 5 Anonymous (Show commits only from Anonymous) Sat May 19 19:24:01 2012 (http://stats.villavu.com/commit/18286496)

currently on page 22 (http://stats.villavu.com/script/1037/commits/22)

nyaks
05-20-2012, 03:34 PM
unavable to update connect server.any one here experience this error_game_jsconnect

zmon
05-20-2012, 03:59 PM
this may be of help too:

formWriteln: SRL_Stats: No POST return

zmon
05-20-2012, 11:07 PM
this may be of help too:

formWriteln: SRL_Stats: No POST return

EDIT: trapped the response for you:

Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/flup/server/fcgi_base.py", line 558,
in run
protocolStatus, appStatus = self.server.handler(self)
File "/usr/lib64/python2.7/site-packages/flup/server/fcgi_base.py", line 1118,
in handler
result = self.application(environ, start_response)
File "/usr/lib64/python2.7/site-packages/beaker/middleware.py", line 152, in _
_call__
return self.wrap_app(environ, session_start_response)
File "/srv/srlstats/SRL-Stats/stats.py", line 208, in __call__
ret = self.app(env, start_response)
File "/srv/srlstats/SRL-Stats/stats.py", line 253, in __call__
return self.app(env, start_response)
File "/srv/srlstats/SRL-Stats/stats.py", line 165, in stats
r = wt.apply_rule(env['PATH_INFO'], env)
File "/srv/srlstats/SRL-Stats/webtool.py", line 47, in apply_rule
return fv['func'](env=env, **dict(zip(fv['vars'], l)))
File "/srv/srlstats/SRL-Stats/stats.py", line 737, in api_commit
res = ct.add(user, script, time, vars)
File "/srv/srlstats/SRL-Stats/query.py", line 305, in add
v.amount += y
AttributeError: 'NoneType' object has no attribute 'amount'

so it actually looks like the stats are being lost because query.py blows up before it tries to save it to db, so that's why they're not showing up


if you need to debug, here is one that i tried to commit when i was trapping the response:


/srv/srlstats/SRL-Stats/query.py in add(self=<query.CommitTool object>, user=User(zdenek), script=Script(Preemptible RuneSpan : User(zdenek)), time=5, vars={Variable(Total EXP Gained): 3022, Variable(Earth Runes (Crafted)): 25, Variable(Runecrafting EXP (Gained)): 3022, Variable(Rune Essence (Mined)): 271, Variable(Cosmic Runes (Crafted)): 52, Variable(Water Runes (Crafted)): 25})

Edit #2 i missed the line where you log requests so i suppose you can pull it up and test a few more scenarios, if you are logging info level

Green Bay Packers
05-21-2012, 10:46 PM
EDIT: Never mind.

Surfer
05-22-2012, 11:36 PM
Awesome, thanks wizzup?!

zmon
05-24-2012, 01:52 AM
still getting same error on updates, it would be helpful if i would know whether i should disable all variables in scripts that i'm running (aka, are all the submissions being lost?)

i have not had one successful submit on my account since 19th and i had run several scripts that report variables

EDIT:

I am now working on script that does not even have any variables and it still fails to post with stats server kinda blowing up (script 1056) and this is what i'm seeing:

formWriteln: SRL_Stats: No POST return
formWriteln: SRL_Stats: No POST return
formWriteln: SRL_Stats: No POST return
SMART Disabled
Disabeling Events
Starting Image Transfer
SMART Enabled
SMART Disabled
Disabeling Events
Starting Image Transfer
SMART Enabled
formWriteln: SRL_Stats: No POST return
Current Simba version: 984
Latest Simba Version: 984
formWriteln: SRL_Stats: No POST return

Olly
06-01-2012, 12:20 AM
still getting same error on updates, it would be helpful if i would know whether i should disable all variables in scripts that i'm running (aka, are all the submissions being lost?)

i have not had one successful submit on my account since 19th and i had run several scripts that report variables

EDIT:

I am now working on script that does not even have any variables and it still fails to post with stats server kinda blowing up (script 1056) and this is what i'm seeing:

formWriteln: SRL_Stats: No POST return
formWriteln: SRL_Stats: No POST return
formWriteln: SRL_Stats: No POST return
SMART Disabled
Disabeling Events
Starting Image Transfer
SMART Enabled
SMART Disabled
Disabeling Events
Starting Image Transfer
SMART Enabled
formWriteln: SRL_Stats: No POST return
Current Simba version: 984
Latest Simba Version: 984
formWriteln: SRL_Stats: No POST return

Same problem here :S

Wizzup?
06-03-2012, 11:17 AM
EDIT: trapped the response for you:

Traceback (most recent call last):
File "/usr/lib64/python2.7/site-packages/flup/server/fcgi_base.py", line 558,
in run
protocolStatus, appStatus = self.server.handler(self)
File "/usr/lib64/python2.7/site-packages/flup/server/fcgi_base.py", line 1118,
in handler
result = self.application(environ, start_response)
File "/usr/lib64/python2.7/site-packages/beaker/middleware.py", line 152, in _
_call__
return self.wrap_app(environ, session_start_response)
File "/srv/srlstats/SRL-Stats/stats.py", line 208, in __call__
ret = self.app(env, start_response)
File "/srv/srlstats/SRL-Stats/stats.py", line 253, in __call__
return self.app(env, start_response)
File "/srv/srlstats/SRL-Stats/stats.py", line 165, in stats
r = wt.apply_rule(env['PATH_INFO'], env)
File "/srv/srlstats/SRL-Stats/webtool.py", line 47, in apply_rule
return fv['func'](env=env, **dict(zip(fv['vars'], l)))
File "/srv/srlstats/SRL-Stats/stats.py", line 737, in api_commit
res = ct.add(user, script, time, vars)
File "/srv/srlstats/SRL-Stats/query.py", line 305, in add
v.amount += y
AttributeError: 'NoneType' object has no attribute 'amount'

so it actually looks like the stats are being lost because query.py blows up before it tries to save it to db, so that's why they're not showing up


if you need to debug, here is one that i tried to commit when i was trapping the response:


/srv/srlstats/SRL-Stats/query.py in add(self=<query.CommitTool object>, user=User(zdenek), script=Script(Preemptible RuneSpan : User(zdenek)), time=5, vars={Variable(Total EXP Gained): 3022, Variable(Earth Runes (Crafted)): 25, Variable(Runecrafting EXP (Gained)): 3022, Variable(Rune Essence (Mined)): 271, Variable(Cosmic Runes (Crafted)): 52, Variable(Water Runes (Crafted)): 25})

Edit #2 i missed the line where you log requests so i suppose you can pull it up and test a few more scenarios, if you are logging info level

Okay. This is a bug and I've tried to fix it. Give it a go?