PDA

View Full Version : Stats broken?



Bobzilla69
01-11-2011, 12:19 PM
when i try and get to the stats site i get
<class 'sqlalchemy.exc.InvalidRequestError'> Python 2.6.6: /usr/bin/python
Tue Jan 11 13:18:49 2011

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. and a load of other stuff after about python.

Sgt Soul
01-11-2011, 12:23 PM
when i try and get to the stats site i get
<class 'sqlalchemy.exc.InvalidRequestError'> Python 2.6.6: /usr/bin/python
Tue Jan 11 13:18:49 2011

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred. and a load of other stuff after about python.

I have the same problem, I am guessing the website crashed :/

Bobzilla69
01-11-2011, 12:26 PM
hope its not the fact that they are getting to many post's from scripts, possibly mine, lol

Sgt Soul
01-11-2011, 12:27 PM
haha yes that could be bad :p I contributed to that also ;) (by running your script that is)

Bobzilla69
01-11-2011, 12:30 PM
its a conspiracy to stop my script beating MSI in the stats

:) hopefully its a quick fix, i need to work on a script stats image

Dgby714
01-11-2011, 12:54 PM
Guys it looks like the sql server is down.

Wizzup?
01-11-2011, 02:46 PM
Restarted. I logged to stdout/err, so I missed the actual error. Hopefully I'll find it again soon.

Sorry for the inconvenience.

Wizzup?
01-11-2011, 02:53 PM
It seems some scripts are committing after one minute. That didn't cause the error but it's a bit silly nevertheless. Should commit every 5 minutes max, IMO. Perhaps I'll have to enforce that server side. Also, there are lots of variables being sent with the value '0'. Why send them in the first place?

I'll probably disallow sending empty variables at server side too. So we'll have to change stats.simba as well once I implement the features server side.

lordsaturn
01-11-2011, 03:08 PM
It seems some scripts are committing after one minute. That didn't cause the error but it's a bit silly nevertheless. Should commit every 5 minutes max, IMO.
Probably starting-restarting scripts.

Wizzup?
01-11-2011, 03:22 PM
Probably starting-restarting scripts.

No; nearly all commits from script 21. :d

E: Seems it went down again. Will look at it this evening.

Sgt Soul
01-13-2011, 05:08 AM
it has gone down once more :(
do you know any reasons for this to occur wizzup??

Bobzilla69
01-13-2011, 08:43 AM
here is the whole page this time

<class 'sqlalchemy.exc.InvalidRequestError'> Python 2.6.6: /usr/bin/python
Thu Jan 13 09:43:27 2011

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.
/usr/lib/pymodules/python2.6/flup/server/fcgi_base.py in run(self=<flup.server.fcgi_base.Request object>)
556 """Runs the handler, flushes the streams, and ends the request."""
557 try:
558 protocolStatus, appStatus = self.server.handler(self)
559 except:
560 traceback.print_exc(file=self.stderr)
protocolStatus undefined, appStatus undefined, self = <flup.server.fcgi_base.Request object>, self.server = <flup.server.fcgi.WSGIServer object>, self.server.handler = <bound method WSGIServer.handler of <flup.server.fcgi.WSGIServer object>>
/usr/lib/pymodules/python2.6/flup/server/fcgi_base.py in handler(self=<flup.server.fcgi.WSGIServer object>, req=<flup.server.fcgi_base.Request object>)
1116 try:
1117 try:
1118 result = self.application(environ, start_response)
1119 try:
1120 for data in result:
result = None, self = <flup.server.fcgi.WSGIServer object>, self.application = <beaker.middleware.SessionMiddleware object>, environ = {'DOCUMENT_ROOT': '/srv/srlstats/srlstats-new', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_HOST': 'stats.villavu.com', 'HTTP_KEEP_ALIVE': '115', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13', ...}, start_response = <function start_response>
/usr/lib/pymodules/python2.6/beaker/middleware.py in __call__(self=<beaker.middleware.SessionMiddleware object>, environ={'DOCUMENT_ROOT': '/srv/srlstats/srlstats-new', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_HOST': 'stats.villavu.com', 'HTTP_KEEP_ALIVE': '115', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13', ...}, start_response=<function start_response>)
150 headers.append(('Set-cookie', cookie))
151 return start_response(status, headers, exc_info)
152 return self.wrap_app(environ, session_start_response)
153
154 def _get_session(self):
self = <beaker.middleware.SessionMiddleware object>, self.wrap_app = <function stats>, environ = {'DOCUMENT_ROOT': '/srv/srlstats/srlstats-new', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_HOST': 'stats.villavu.com', 'HTTP_KEEP_ALIVE': '115', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13', ...}, session_start_response = <function session_start_response>
/srv/srlstats/srlstats-new/stats.py in stats(env={'DOCUMENT_ROOT': '/srv/srlstats/srlstats-new', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_HOST': 'stats.villavu.com', 'HTTP_KEEP_ALIVE': '115', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13', ...}, start_response=<function session_start_response>)
146 # Search in the known ``rules'' (see rules.py) and call function if
147 # existant.
148 r = wt.apply_rule(env['REQUEST_URI'], env)
149
150 # Result 'None' means 404.
r undefined, global wt = <webtool.WebTool object>, wt.apply_rule = <bound method WebTool.apply_rule of <webtool.WebTool object>>, env = {'DOCUMENT_ROOT': '/srv/srlstats/srlstats-new', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_HOST': 'stats.villavu.com', 'HTTP_KEEP_ALIVE': '115', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13', ...}
/srv/srlstats/srlstats-new/webtool.py in apply_rule(self=<webtool.WebTool object>, url='/', env={'DOCUMENT_ROOT': '/srv/srlstats/srlstats-new', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_HOST': 'stats.villavu.com', 'HTTP_KEEP_ALIVE': '115', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13', ...})
45 raise WebToolException('Matches does not equal variable \
46 amount')
47 return fv['func'](env=env, **dict(zip(fv['vars'], l)))
48 return None
49
fv = {'func': <function general>, 'vars': []}, env = {'DOCUMENT_ROOT': '/srv/srlstats/srlstats-new', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_HOST': 'stats.villavu.com', 'HTTP_KEEP_ALIVE': '115', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13', ...}, builtin dict = <type 'dict'>, builtin zip = <built-in function zip>, l = []
/srv/srlstats/srlstats-new/stats.py in general(env={'DOCUMENT_ROOT': '/srv/srlstats/srlstats-new', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_HOST': 'stats.villavu.com', 'HTTP_KEEP_ALIVE': '115', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13', ...})
185 tmpl = jinjaenv.get_template('base.html')
186
187 return template_render(tmpl, {'session' : env['beaker.session']} )
188
189 def user(env, userid=None):
global template_render = <function template_render>, tmpl = <Template 'base.html'>, env = {'DOCUMENT_ROOT': '/srv/srlstats/srlstats-new', 'GATEWAY_INTERFACE': 'CGI/1.1', 'HTTP_ACCEPT': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'HTTP_ACCEPT_CHARSET': 'ISO-8859-1,utf-8;q=0.7,*;q=0.7', 'HTTP_ACCEPT_ENCODING': 'gzip,deflate', 'HTTP_ACCEPT_LANGUAGE': 'en-us,en;q=0.5', 'HTTP_CONNECTION': 'keep-alive', 'HTTP_HOST': 'stats.villavu.com', 'HTTP_KEEP_ALIVE': '115', 'HTTP_USER_AGENT': 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.13) Gecko/20101203 Firefox/3.6.13', ...}
/srv/srlstats/srlstats-new/stats.py in template_render(template=<Template 'base.html'>, vars={'_import': {'datetime': <module 'datetime' from '/usr/lib/python2.6/lib-dynload/datetime.so'>}, 'baseurl': '', 'session': {'_accessed_time': 1294908207.3039141, '_creation_time': 1294908207.3039141}}, default_page=True)
79
80 if default_page:
81 vars['topusers'] = ut.top(_limit=4)
82 vars['topscripts'] = st.top(_limit=4)
83 vars['lastcommits'] = ct.top(_limit=4)
vars = {'_import': {'datetime': <module 'datetime' from '/usr/lib/python2.6/lib-dynload/datetime.so'>}, 'baseurl': '', 'session': {'_accessed_time': 1294908207.3039141, '_creation_time': 1294908207.3039141}}, global ut = <query.UserTool object>, ut.top = <bound method UserTool.top of <query.UserTool object>>, _limit undefined
/srv/srlstats/srlstats-new/query.py in top(self=<query.UserTool object>, _offset=0, _limit=4)
30 (Commit, Commit.user_id == User.id)).group_by(
31 User).order_by(desc(coales),asc(User.id)).offset(
32 _offset).limit(_limit).all()
33
34 def info(self, uid):
_offset = 0, ).limit undefined, _limit = 4, ).all undefined
/usr/lib/python2.6/dist-packages/sqlalchemy/orm/query.py in all(self=<sqlalchemy.orm.query.Query object>)
1451
1452 """
1453 return list(self)
1454
1455 @_generative(_no_clauseelement_condition)
builtin list = <type 'list'>, self = <sqlalchemy.orm.query.Query object>
/usr/lib/python2.6/dist-packages/sqlalchemy/orm/query.py in __iter__(self=<sqlalchemy.orm.query.Query object>)
1562 context.statement.use_labels = True
1563 if self._autoflush and not self._populate_existing:
1564 self.session._autoflush()
1565 return self._execute_and_instances(context)
1566
self = <sqlalchemy.orm.query.Query object>, self.session = <sqlalchemy.orm.session.Session object>, self.session._autoflush = <bound method Session._autoflush of <sqlalchemy.orm.session.Session object>>
/usr/lib/python2.6/dist-packages/sqlalchemy/orm/session.py in _autoflush(self=<sqlalchemy.orm.session.Session object>)
875 def _autoflush(self):
876 if self.autoflush and not self._flushing:
877 self.flush()
878
879 def _finalize_loaded(self, states):
self = <sqlalchemy.orm.session.Session object>, self.flush = <bound method Session.flush of <sqlalchemy.orm.session.Session object>>
/usr/lib/python2.6/dist-packages/sqlalchemy/orm/session.py in flush(self=<sqlalchemy.orm.session.Session object>, objects=None)
1344 try:
1345 self._flushing = True
1346 self._flush(objects)
1347 finally:
1348 self._flushing = False
self = <sqlalchemy.orm.session.Session object>, self._flush = <bound method Session._flush of <sqlalchemy.orm.session.Session object>>, objects = None
/usr/lib/python2.6/dist-packages/sqlalchemy/orm/session.py in _flush(self=<sqlalchemy.orm.session.Session object>, objects=None)
1423
1424 flush_context.transaction = transaction = self.begin(
1425 subtransactions=True)
1426 try:
1427 flush_context.execute()
subtransactions undefined, builtin True = True
/usr/lib/python2.6/dist-packages/sqlalchemy/orm/session.py in begin(self=<sqlalchemy.orm.session.Session object>, subtransactions=True, nested=False)
589 if subtransactions or nested:
590 self.transaction = self.transaction._begin(
591 nested=nested)
592 else:
593 raise sa_exc.InvalidRequestError(
nested = False
/usr/lib/python2.6/dist-packages/sqlalchemy/orm/session.py in _begin(self=<sqlalchemy.orm.session.SessionTransaction object>, nested=False)
248
249 def _begin(self, nested=False):
250 self._assert_is_active()
251 return SessionTransaction(
252 self.session, self, nested=nested)
self = <sqlalchemy.orm.session.SessionTransaction object>, self._assert_is_active = <bound method SessionTransaction._assert_is_acti...qlalchemy.orm .session.SessionTransaction object>>
/usr/lib/python2.6/dist-packages/sqlalchemy/orm/session.py in _assert_is_active(self=<sqlalchemy.orm.session.SessionTransaction object>)
231 if not self._active:
232 raise sa_exc.InvalidRequestError(
233 "The transaction is inactive due to a rollback in a "
234 "subtransaction. Issue rollback() to cancel the transaction.")
235

<class 'sqlalchemy.exc.InvalidRequestError'>: The transaction is inactive due to a rollback in a subtransaction. Issue rollback() to cancel the transaction.
args = ('The transaction is inactive due to a rollback in...ion. Issue rollback() to cancel the transaction.',)
message = 'The transaction is inactive due to a rollback in...ion. Issue rollback() to cancel the transaction.'

Wizzup?
01-13-2011, 09:19 AM
Noticed. Restarted it for now. I think it has to with the way I use SQLAlchemy and threads. Possibly two queries at almost exactly the same time can cause this. I'll need to debug this further. May have to migrate to the Pylons framework instead.

mastaraymond
01-13-2011, 10:09 AM
So, after years of work it still fails!? ;).

Frement
01-13-2011, 10:09 AM
So, after years of work it still fails!? ;).

Its the way python coders do their stuff.

Nava2
01-13-2011, 05:09 PM
Wizzup, advice, sounds like over-kill, but might be worth it:

Create queue object for sql changes -- thread
Add items to the queue internally, as well as commits and other external calls -- thread
Process SQL requests -- Thread


It would remove the issue of simultaneous hits on the sql database. I can write this up if you want.

Wizzup?
01-13-2011, 06:01 PM
I have switched from the threaded server to the forking server. While this may (hopefully) solve the problem temporarily; it will not solve the issue with the threaded server. I've done some more investigation and I know how to solve it now; I think. Expect a real fix in the upcoming weekend. (Although the forking server should work fine)