PDA

View Full Version : What is the future of srl stats?



bevardis
12-18-2013, 04:02 PM
Hello.

So what exactly is the reason srl stats don't currently work? I've heard that it's server is full of data?
And what is the future of the srl stats?

Coh3n
12-18-2013, 05:29 PM
As far as I know, it's because the server is full. Unless we reset everything (which I'm not sure @Wizzup? wants to do), we need a new server.

Kevin
12-18-2013, 06:01 PM
From what I understood, it's because so many scripts hit Integer.MaxValue and we needed to change our stored values to Longs instead. However, if we could get someone to graciously host it, we could write a new SRL stats.

Hmm... To shoot a few names out that I think have servers that may or may not be available...
J J; I think you even wanted to do a .NET web project, and this is the perfect chance that I could even help with.
The Killer;
BMWxi;
Zyt3x; Ditto on what I said to JJ.

The Killer
12-18-2013, 06:24 PM
From what I understood, it's because so many scripts hit Integer.MaxValue and we needed to change our stored values to Longs instead. However, if we could get someone to graciously host it, we could write a new SRL stats.

Hmm... To shoot a few names out that I think have servers that may or may not be available...
J J; I think you even wanted to do a .NET web project, and this is the perfect chance that I could even help with.
The Killer;
BMWxi;
Zyt3x; Ditto on what I said to JJ.

I have two dedi's which I use for botting. If you want to use mine I'm happy to let you as long as it doesn't use much cpu :/ (everything else has a fair amount of space for you)

http://www.ovh.com/us/dedicated-servers/ks2.xml <- the dedi, it has 90-100mb/s up and down
Running debian wheezy I think. Coh3n;

Ian
12-18-2013, 06:26 PM
From what I understood, it's because so many scripts hit Integer.MaxValue and we needed to change our stored values to Longs instead. However, if we could get someone to graciously host it, we could write a new SRL stats.

Hmm... To shoot a few names out that I think have servers that may or may not be available...
J J; I think you even wanted to do a .NET web project, and this is the perfect chance that I could even help with.
The Killer;
BMWxi;
Zyt3x; Ditto on what I said to JJ.

Although I have srlbmwxi.tk, I don't know if it would be sufficient to host SRL stats. I also can't make any guarantees about it's reliability either unfortunately. :p

J J
12-18-2013, 07:13 PM
From what I understood, it's because so many scripts hit Integer.MaxValue and we needed to change our stored values to Longs instead. However, if we could get someone to graciously host it, we could write a new SRL stats.

Hmm... To shoot a few names out that I think have servers that may or may not be available...
J J; I think you even wanted to do a .NET web project, and this is the perfect chance that I could even help with.
The Killer;
BMWxi;
Zyt3x; Ditto on what I said to JJ.
Yeah I think we hit the limit on a few things. I learned some PHP a while ago to keep track of my script usage: http://obduro.org/scriptstats.html

I wouldn't make it in .NET but PHP or even Java Servlets as we learned about those :P PHP would probably be the fastest to make.

I can definately help but I'm pretty busy. Xmas holiday is coming up though but I got a lot planned there already :P

Kevin
12-18-2013, 07:18 PM
Yeah I think we hit the limit on a few things. I learned some PHP a while ago to keep track of my script usage: http://obduro.org/scriptstats.html

I wouldn't make it in .NET but PHP or even Java Servlets as we learned about those :P PHP would probably be the fastest to make.

I can definately help but I'm pretty busy. Xmas holiday is coming up though but I got a lot planned there already :P

I can't say I'm familiar with web development outside .NET, sooooo... :p

Coh3n
12-18-2013, 08:49 PM
I have two dedi's which I use for botting. If you want to use mine I'm happy to let you as long as it doesn't use much cpu :/ (everything else has a fair amount of space for you)

http://www.ovh.com/us/dedicated-servers/ks2.xml <- the dedi, it has 90-100mb/s up and down
Running debian wheezy I think. Coh3n;I don't think we'll need to pay for a new one. I'll talk to Wizzup? a bit and see what he thinks.

J J
12-18-2013, 09:01 PM
I don't think we'll need to pay for a new one. I'll talk to Wizzup? a bit and see what he thinks.
Think there was some spare money according to Wizzup?; that could be used towards a fast server.

Coh3n
12-18-2013, 09:43 PM
Think there was some spare money according to Wizzup?; that could be used towards a fast server.The problem is integer size, just got confirmation from Wizzup?.

Kevin
12-18-2013, 09:52 PM
The problem is integer size, just got confirmation from Wizzup?.

Which can be fixed by using Longs or Doubles. :)

The Killer
12-18-2013, 10:33 PM
The problem is integer size, just got confirmation from Wizzup?.

Ahh ok, but if its a case of needing a extra server, you can use mine aslong as it isn't cpu heavy...
(I should keep mine for the next 3+ months :))

Flight
12-19-2013, 12:14 AM
grats?

Olly
12-19-2013, 03:35 AM
Which can be fixed by using Longs or Doubles. :)

https://github.com/MerlijnWajer/SRL-Stats

il leave it with you. :)

Coh3n
12-19-2013, 03:45 AM
Which can be fixed by using Longs or Doubles. :)Let's hope it's that simple. I'll try to talk to Wizzup? more about it, see if it can be done easily.

Kevin
12-19-2013, 04:51 AM
https://github.com/MerlijnWajer/SRL-Stats

il leave it with you. :)

I don't know python, but I'll legitimately start looking into it starting tomorrow.

Justin
12-19-2013, 04:59 AM
I think that if the SRL stats were to be hosted, it should be hosted on one of the vservers inside nala, that way we know it's going to be secure and stable (Wizzup?'s server)

Wizzup?
12-22-2013, 01:15 PM
I don't know python, but I'll legitimately start looking into it starting tomorrow.

I would have to reinvestigate. But there were a few issues. Maybe I already fixed some. In no particular order:

Updating the cache with a full table scan became infeasible due to the size. So now it's done incrementally, this may be buggy.
The sum of commits/minutes is too large to fit in the cache table.
...?


I would prefer to reset the DB and start from scratch. Makes it more fun, too.

Justin, they are already in a vserver @ nala. :)

Coh3n
12-22-2013, 04:40 PM
A reset would be better than nothing. I don't think it helped that we had an Anonymous user that was used by virtually every user.

Ashaman88
12-22-2013, 05:31 PM
A reset would be better than nothing. I don't think it helped that we had an Anonymous user that was used by virtually every user.

Maybe have the historical data saved somewhere though? Be a shame to lose it, tells a story!

The Mayor
12-22-2013, 07:00 PM
A reset would be better than nothing. I don't think it helped that we had an Anonymous user that was used by virtually every user.

Yes, the anonymous user was quite annoying. Although I feel forcing every single new user to create a stats account is somewhat of a burden. Wouldn't it be much easier if you could somehow use your SRL un/pw? Or is this too risky/hard?

Ian
12-22-2013, 07:05 PM
Yes, the anonymous user was quite annoying. Although I feel forcing every single new user to create a stats account is somewhat of a burden. Wouldn't it be much easier if you could somehow use your SRL un/pw? Or is this too risky/hard?

I also would like it if srl stats were linked to the forum account, but I'm not sure if it could be done safely.

What do people who are more experienced in this think?

Kevin
12-22-2013, 07:36 PM
I would have to reinvestigate. But there were a few issues. Maybe I already fixed some. In no particular order:

Updating the cache with a full table scan became infeasible due to the size. So now it's done incrementally, this may be buggy.
The sum of commits/minutes is too large to fit in the cache table.
...?


I would prefer to reset the DB and start from scratch. Makes it more fun, too.

Justin, they are already in a vserver @ nala. :)

I've already started rewriting Stats as a .NET web-app as that should increase ease of feature extension and maintainability in the future. I've ran into snags as there will always be, however, it's definitely coming along. If you'd be willing to host it on the same old vserver of old, that would be great. If not, I'm sure I can find somewhere else to host it and get everything up and running again.

Hopefully with the .NET direction, I'll be able to add some better GUI functionality and graphs for scripts/users.

And since I'm doing this... Are there any bits of functionality anyone ever wanted in Stats?

Ashaman88
12-23-2013, 12:11 AM
I've already started rewriting Stats as a .NET web-app as that should increase ease of feature extension and maintainability in the future. I've ran into snags as there will always be, however, it's definitely coming along. If you'd be willing to host it on the same old vserver of old, that would be great. If not, I'm sure I can find somewhere else to host it and get everything up and running again.

Hopefully with the .NET direction, I'll be able to add some better GUI functionality and graphs for scripts/users.

And since I'm doing this... Are there any bits of functionality anyone ever wanted in Stats?


Usage graphs would be great... hourly averages, monthly and such...umm I need to think on this!

Justin
12-23-2013, 12:21 AM
J J; Do you have anything on this? I know you have stats with graphs for your scripts

J J
12-23-2013, 01:17 AM
I also would like it if srl stats were linked to the forum account, but I'm not sure if it could be done safely.

What do people who are more experienced in this think?
Perhaps the forum software could be extended with a request token functionality. Generate a token that you can put in the script. You send the token with the request, server knows which token belongs to which user -> insert data.


J J; Do you have anything on this? I know you have stats with graphs for your scripts
Just make sure you put a timestamp and runtime for every commit. Using those two you can generate every graph you want :) Can even make a search function or plot the average xp/h. That's what I've done with my graphs.

JJ's Willow Chopper

CREATE TABLE jjswillowchopper (
id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
time BIGINT UNSIGNED,
runtime SMALLINT UNSIGNED,
xp SMALLINT UNSIGNED,
level TINYINT UNSIGNED
);

Result: http://obduro.org/scriptstats/jjswillowchopper.php
Can do quite a bit with just 6 columns.

Or JJ's Fighter.

CREATE TABLE jjsfighter (
id INT UNSIGNED NOT NULL AUTO_INCREMENT PRIMARY KEY,
time BIGINT UNSIGNED,
runtime SMALLINT UNSIGNED,
attack_xp SMALLINT UNSIGNED,
attack_lvl TINYINT UNSIGNED,
strength_xp SMALLINT UNSIGNED,
strength_lvl TINYINT UNSIGNED,
defence_xp SMALLINT UNSIGNED,
defence_lvl TINYINT UNSIGNED,
ranged_xp SMALLINT UNSIGNED,
ranged_lvl TINYINT UNSIGNED,
magic_xp SMALLINT UNSIGNED,
magic_lvl TINYINT UNSIGNED,
hitpoints_xp SMALLINT UNSIGNED,
hitpoints_lvl TINYINT UNSIGNED
);

It now has a total runtime of 49242066 seconds which is 1 years, 6 months, 29 days, 22 hours, 21 minutes, 6 seconds.
I let the script commit every minute so that is a total of 820701 rows or so.
Warning: long loading time.. http://obduro.org/scriptstats/jjsfighter.php

Just make sure you have at very least a timestamp and runtime for every table.
Hardest part is to make sure it is fast. Eventually you are going to have loads of commits which will make it slow like my jjs fighter graph. So you will have to cache stuff, I'm not sure how to do that.

Another thing is security. All scripts are open source so you can't really do much in there. Ideally you would put the token in the table so you could easily remove all data from that user if they are abusing the commit method.

So I would have the following fields AT LEAST for every table:


id
timestamp
runtime
token

id: auto increment
timestamp: grab it during the commit, server sided
runtime: send it from the script (in seconds I'd say)
token: send it from the script, user has to generate it and can be identified with it.

EDIT: For the token perhaps do something smart so it stays the same. Base it on username + join date or something and encrypt it. That way you can actually just make a nice php file or something that does this for you. No need to extend the forum software. Make it look like it is a random token. In reality it is based on constant data so you can always identify the user ;)

Just make sure users won't know what encryption and what data is used. Otherwise they could grab that data from anyone like my username and joindate, encrypt it and commit stats on my name.. Although that wouldn't really be a problem.. But they could commit false stuff and get me banned from SRL Stats :P

Coh3n
12-23-2013, 01:31 AM
Yes, the anonymous user was quite annoying. Although I feel forcing every single new user to create a stats account is somewhat of a burden. Wouldn't it be much easier if you could somehow use your SRL un/pw? Or is this too risky/hard?I think the very first stats system was linked to the forum accounts. Not sure why we moved away from that.


I've already started rewriting Stats as a .NET web-app as that should increase ease of feature extension and maintainability in the future. I've ran into snags as there will always be, however, it's definitely coming along. If you'd be willing to host it on the same old vserver of old, that would be great. If not, I'm sure I can find somewhere else to host it and get everything up and running again.

Hopefully with the .NET direction, I'll be able to add some better GUI functionality and graphs for scripts/users.

And since I'm doing this... Are there any bits of functionality anyone ever wanted in Stats?That's awesome! :)

One thing that annoys me about the current stats is you can't remove scripts or script variables.

Graphs would be awesome though, for sure.

Kevin
12-23-2013, 02:08 AM
WORDS....
I would love a Villavu-based secure request token. If I make my own, as I would be an open source project, it would be impossible to create using public information and not still be fakeable.

I've been debating use of a database versus using custom made local xml files. Databases would be so much easier. but I don't know if the server I'll be able to put it on will also allow for use of a database. I suppose I could always do it the easy way and then do it the hard way later if need be...


I think the very first stats system was linked to the forum accounts. Not sure why we moved away from that.

That's awesome! :)

One thing that annoys me about the current stats is you can't remove scripts or script variables.

Graphs would be awesome though, for sure.
My every intention was that I would just use the user ID to match to stats (in your case, #45543 (http://villavu.com/forum/usertag.php?do=list&action=hash&hash=45543) ), or the actual Villavu account name and just translate it to the id. I mean, why not have it be linked and super convenient? I absolutely want the script creator to have the rights to remove their own scripts and/or variables. Although... having something in place to verify that the user is who they say they are would be important, and have to be verified somehow through villavu.

Coh3n
12-23-2013, 02:53 AM
My every intention was that I would just use the user ID to match to stats (in your case, #45543 (http://villavu.com/forum/usertag.php?do=list&action=hash&hash=45543) ), or the actual Villavu account name and just translate it to the id. I mean, why not have it be linked and super convenient? I absolutely want the script creator to have the rights to remove their own scripts and/or variables. Although... having something in place to verify that the user is who they say they are would be important, and have to be verified somehow through villavu.User ID would be better. People change their username all the time.

I'm sure we could somehow link the forum accounts, but I'm not sure if Wizzup?; would want to unless there were extra security measures. You might somehow be able to embed the system in the forums as a plugin, but I really have no idea what's involved with that.

Wizzup?
02-28-2014, 12:42 PM
I think the very first stats system was linked to the forum accounts. Not sure why we moved away from that.

That's awesome! :)

One thing that annoys me about the current stats is you can't remove scripts or script variables.

Graphs would be awesome though, for sure.

The current system has graphs! See http://wizzup.org/stats/

http://stats.villavu.com/graph/commits/month/1/year/2012

It was also quite secure. But I never dared to integrate it with the forum because of grave security concerns. It is much nicer to isolate everything.

If you want me to host this alternative stats project, then it should run on a normal linux system. ;)

sarmetto
02-28-2014, 11:52 PM
The current system has graphs! See http://wizzup.org/stats/

http://stats.villavu.com/graph/commits/month/1/year/2012

It was also quite secure. But I never dared to integrate it with the forum because of grave security concerns. It is much nicer to isolate everything.

If you want me to host this alternative stats project, then it should run on a normal linux system. ;)

Hey, I can't PM / create new threads for some reason. No idea what I'm suppose to do, can you help?