Hooking openGL functions for new style reflection

Hi,

This is just an idea I've been thinking about for a while, since Jagex started using the jOGL library to do its graphics.

I'm not sure if any of you are aware of the FPS (e.g. counterstrike, CoD etc) cheating scene. There are two style of hacks - one hooks the game functions directly (similar to the way RS client hacks work) and the others hook at the openGL/D3D level.

As RS can now use openGL to render (only in high gfx settings?) it it now trivial to hook openGL and perform standard model detection. (see http://forum.gamedeception.net/forumdisplay.php?f=261 for some general examples of what can be done in other games).

As a proof of concept I used GLIntercept (http://glintercept.nutty.org/) to log OGL calls and make sure this is plausible.

Here's an example of what I believe is a 3D model being rendered:

Code:

---

glGenBuffersARB(1,0x2320add4)
glBindBufferARB(GL_ARRAY_BUFFER,1028)
glBufferDataARB(GL_ARRAY_BUFFER,3168,0x23783ffc,GL_STATIC_DRAW) //0x23783ffc points to the vertex buffer... read this for model recognition
glEnableClientState(GL_VERTEX_ARRAY)
glVertexPointer(3,GL_FLOAT,24,0x0000)
glEnableClientState(GL_NORMAL_ARRAY)
glNormalPointer(GL_FLOAT,24,0x000c)
glEnableClientState(GL_COLOR_ARRAY)
glBindBufferARB(GL_ARRAY_BUFFER,171)
glColorPointer(4,GL_UNSIGNED_BYTE,28,0x000c)
glEnableClientState(GL_TEXTURE_COORD_ARRAY)
glBindBufferARB(GL_ARRAY_BUFFER,135)
glTexCoordPointer(2,GL_FLOAT,36,0x001c)
glBindTexture(GL_TEXTURE_2D,145)
glBindBufferARB(GL_ELEMENT_ARRAY_BUFFER,136)
glDrawElements(GL_TRIANGLES,168,GL_UNSIGNED_SHORT,0x0000) Textures[ (0,145) ]
glBindTexture(GL_TEXTURE_2D,144)
glDrawElements(GL_TRIANGLES,138,GL_UNSIGNED_SHORT,0x0150) Textures[ (0,144) ]
glPopMatrix()
glMatrixMode(GL_PROJECTION)
glLoadMatrixf([1.183333,0.000000,0.000000,0.000000,0.000000,2.453564,0.000000,0.000000,0.000000,0.000000,-1.027477,-1.000000,0.000000,0.000000,-96.387375,0.000000])
glMatrixMode(GL_MODELVIEW)
glMatrixMode(GL_PROJECTION)
glLoadMatrixf([1.183333,0.000000,0.000000,0.000000,0.000000,2.453564,0.000000,0.000000,0.000000,0.000000,-1.026092,-1.000000,0.000000,0.000000,-87.635582,0.000000])
glMatrixMode(GL_MODELVIEW)
glPushMatrix()
glMultMatrixf([1.000000,0.000000,0.000000,0.000000,0.000000,1.000000,0.000000,0.000000,0.000000,0.000000,1.000000,0.000000,7360.000000,-320.000000,5952.000000,1.000000])

---

Once you can copy out the vertex array, you can generate a unique ID for each model (e.g. devise an algorithm to generate a unique id based on vertex positions) and do a WorldToScreen conversion to get the 2D coords of the model. This can then be passed to scar.

Similar methods would allow things such as map recognition etc.

The advantage over the SMART reflection is that it doesn't need to be updated every time RS is updated, only if they change the models.

Thoughs, ideas, suggestions? I could code up a small proof of concept if people are interested.

Yeah, I also thought of something like this. We should also be able to get direct access to their data, which would allow us to perform fast (search) operations on the client, possibly with shaders.

Good idea, I'd like to see a PoC. :)

Yup, I think 3D model rec and 2D rec such as inventory items should be fairly easy to do.

You mentioning shaders made me take a quick look at the logs - here's a dump:

Code:

---

!!ARBvp1.0
ATTRIB  iPos        = vertex.position;
ATTRIB  iColour      = vertex.color;
OUTPUT  oPos        = result.position;
OUTPUT  oColour      = result.color;
OUTPUT  oTexCoord0  = result.texcoord[0];
OUTPUT  oTexCoord1  = result.texcoord[1];
OUTPUT  oFogCoord    = result.fogcoord;
PARAM  time        = program.local[65];
PARAM  turbulence  = program.local[64];
PARAM  lightAmbient = program.local[66];
PARAM  pMatrix[4]  = { state.matrix.projection };
PARAM  mvMatrix[4]  = { state.matrix.modelview };
PARAM  ivMatrix[4]  = { state.matrix.texture[1] };
PARAM  fNoise[64]  = { program.local[0..63] };
TEMP    noise, clipPos, viewPos, worldPos;
ADDRESS noiseAddr;
DP4  viewPos.x, mvMatrix[0], iPos;
DP4  viewPos.y, mvMatrix[1], iPos;
DP4  viewPos.z, mvMatrix[2], iPos;
DP4  viewPos.w, mvMatrix[3], iPos;
MOV  oFogCoord.x, -viewPos.z;
DP4  worldPos.x, ivMatrix[0], viewPos;
DP4  worldPos.y, ivMatrix[1], viewPos;
DP4  worldPos.z, ivMatrix[2], viewPos;
DP4  worldPos.w, ivMatrix[3], viewPos;
ADD  noise.x, worldPos.x, worldPos.z;SUB  noise.y, worldPos.z, worldPos.x;MUL  noise, noise, 0.0001220703125;
FRC  noise, noise;
MUL  noise, noise, 64;
ARL  noiseAddr.x, noise.x;
MOV  noise.x, fNoise[noiseAddr.x].x;
ARL  noiseAddr.x, noise.y;
MOV  noise.y, fNoise[noiseAddr.x].y;
MUL  noise, noise, turbulence.x;
MAD  oTexCoord0, worldPos.xzww, 0.0078125, noise;
MOV  oTexCoord0.w, 1;
MUL  oTexCoord1.xy, worldPos.xzww, 0.0009765625;
MOV  oTexCoord1.zw, time.xxxw;
DP4  clipPos.x, pMatrix[0], viewPos;
DP4  clipPos.y, pMatrix[1], viewPos;
DP4  clipPos.z, pMatrix[2], viewPos;
DP4  clipPos.w, pMatrix[3], viewPos;
MUL  oColour.xyz, iColour, lightAmbient;
MOV  oColour.w, iColour.w;
MOV  oPos, clipPos;
END

---

These lines are their 'anti random move stuff a bit'?:
ADD noise.x, worldPos.x, worldPos.z;SUB noise.y, worldPos.z, worldPos.x;MUL noise, noise, 0.0001220703125;
MAD oTexCoord0, worldPos.xzww, 0.0078125, noise;
MUL oTexCoord1.xy, worldPos.xzww, 0.0009765625;

My OGL is rusty (I made an OGL aimbot/wh etc for CS1.5 back in the day) but I might enjoy getting back into it :)

http://www.moparisthebest.com/smf/in...,420642.0.html

I made this thread 4 days ago, did you hack into my mind or something :p But yea it'd be cool for something to come out of it...

Well I've put a bit of time into this... the use of VBO's is annoying :P

I have some code to extract models from RS as they're rendered and then a standalone program to render them. So far the results aren't great - it's all about figuring out the data structure RS uses for vertices, colours, textures and how they're interleaved. Just figured out what's been going wrong, but I'm too tired to continue... will have another look later. Hopefully by the end I'll be able to render coloured models directly extracted from the OGL hook :) Once I know I'm getting valid model data I can work on model rec and worldToScreen.

Right, well I've figured out how RS draws its models and below is a render of the first model I've extracted (and I believe to have rendered correctly). Can anyone recognise what it is? I looks kind of like the side of a building or something... I will render some more dumps later when I have some spare time.

What I've done so far is hooked the required OGL functions, dumped appropriate buffers (vertices and indices) and then written a standalone program to render the dumps, to show that it is working correctly (and because it is cool :)). I could incorporate colour, texture and bump-map data at a later date if desired.

Anyway, this is proof of concept to show that you can capture model data being sent to the GPU. The next step is to write a WorldToScreen function which can give you an x,y coord of where each model is in the 3D world. It is then trivial to search for a 3D model you are interested in (an NPC, a rock, etc) and return the x,y coords, in much the same way as reflection works.

http://i47.tinypic.com/2m4dn28.png

Very interesting.
And that image looks like a curtain or something..

Keep us updated :)
And texture would help a lot

The other one looked a bit dubious, so I just wanted to check and render something else... here it is:
http://i46.tinypic.com/jkul2h.png

It's definitely something :)

Very cool. I don't really recognize it, but it is indeed probably something. ;)

Fun fact - vertex data for NPCs are unqiue in that they are packed with stride 12. Here's proof with wireframe for just NPCs. I will render some stride 12 buffer dumps and we should get some more interesting models :)

http://i47.tinypic.com/dltb9g.png

Oh, and by coincidence the two rendered dumps clearly fit snuggly side by side if you look at the outline of the models :)

Ah, looks cool

I noticed that picture is the high def version though, will this work with normal rs client in low def?

Quote:

---

Originally Posted by IceFire908

Ah, looks cool

I noticed that picture is the high def version though, will this work with normal rs client in low def?

---

OpenGL is only HD

Quote:

---

Originally Posted by Zyt3x

OpenGL is only HD

---

Yup. Would that have a great effect on most scripts?

Quote:

---

Originally Posted by silentwolf

Yup. Would that have a great effect on most scripts?

---

Well SRL today is designed to run on low detail, but if this turns out cool I would use it.

Yeah, none would work (pretty much). All scripts/SRL is all made for the lowest possible detail, in fact LoginPlayer contains a function that makes all of the graphics on lowest settings before/after you login. But we could always mess around with some stuff and just alter code to make it work for both versions, I think mainly just different colors (if that) would be the only major difference between low and high def shouldn't be too big of a change?

For the fixed HD client everything would be the same except the MainScreen and possibly minor changes in the minimap.

Quote:

---

Originally Posted by YoHoJo

Yeah, none would work (pretty much). All scripts/SRL is all made for the lowest possible detail, in fact LoginPlayer contains a function that makes all of the graphics on lowest settings before/after you login. But we could always mess around with some stuff and just alter code to make it work for both versions, I think mainly just different colors (if that) would be the only major difference between low and high def shouldn't be too big of a change?

---

Yeah an easy {.DEFINE OpenGLRef} would solve that :)

Quote:

---

Originally Posted by Zyt3x

Yeah an easy {.DEFINE OpenGLRef} would solve that :)

---

Not exactly, it would effectively require all minimap and mainscreen functions to be written twice... random solvers, object finders, bank openers ect..

Quote:

---

Originally Posted by IceFire908

Not exactly, it would effectively require all minimap and mainscreen functions to be written twice... random solvers, object finders, bank openers ect..

---

Wouldn't reflection cover all those functions anyways? R_OpenBank, R_SolveMime blah blah

Can't you use Hardware Acceleration (OpenGL) and then turn all the stuff on low/off? In theory it should be the same.

Quote:

---

Originally Posted by Wizzup?

Can't you use Hardware Acceleration (OpenGL) and then turn all the stuff on low/off? In theory it should be the same.

---

Yeah, I'd imagine most of the stuff will be almost identical.

Here's a render of an 'NPC' - really anything that is animated like things drawn in wireframe in the image above. This is actually passed to the gfx card in a different way to everything else (e.g. scenary etc, anything static) so I was just testing I could grab vertex data for animated things as well.

http://i47.tinypic.com/op1wky.png

Again, not really sure what it is, but it's clearly something :)

When I get some more time I will work on model rec and WorldToScreen. I can then overlay some text on the x,y coords of each model so you can see what is positioned where in the RS world. Hopefully given time I will also be able to get some kind of animation rec as I believe this will be useful too?

These kinda look like pieces of something bigger, to me.

Quote:

---

Originally Posted by senrath

These kinda look like pieces of something bigger, to me.

---

Yes, you're right. Each piece is scaled and rotated to fit into the RS world. Multiple bits will fit together to form everything, but things such as NPC's and monsters should be one big mesh. Sadly there are a huge number of meshes being drawn to form the RS world, so the chances of me picking out at random something interesting is low. As soon as I get worldToScreen working I can see what models correspond to interesting things and then we should have something useful :)

Good luck with that. It's nice to see someone new to the community contributing so much.

Quote:

---

Originally Posted by senrath

It's nice to see someone new to the community contributing so much.

---

It indeed is! This is great, silentwolf :)
You'll be accepted and respected in no-time with this great knowledge and attitude ;)

Also, the two first images looks like a wall or something, maybe similar to the walls inside Sos (http://runescape.wikia.com/wiki/Stronghold_of_Security) Level 2 and the most recent one looks like a vase

This could be the next step in macroing - it all appears unencrypted and not "messed" with yet like hooks and colours are - it seems unlikely there'd be a way of jagex encrypting the data sent to openGL and making it inaccessible to us, as they are standalone programs etc.

Some questions though:

- Do you think it will be possible to search through the parsed models with tolerance, or will something like a 1 pixel change on the model mess it up?
- Is there an "ID" system like with reflection or do we have to qualitatively analyse each model we grab and then compare with a database we've collected to identify it?

This is extremely interesting, always wanted to know how this stuff worked in aimbots etc, and now there is a relation to runescape ^_^. I really hope to see more of this, keep it rollin'!

Quote:

---

Originally Posted by The Claw

it seems unlikely there'd be a way of jagex encrypting the data sent to openGL and making it inaccessible to us, as they are standalone programs etc.

---

This is impossible :)

They could potentially find a way of detecting the OGL hook though. There are ways around this though.

Quote:

---

Originally Posted by The Claw

- Do you think it will be possible to search through the parsed models with tolerance, or will something like a 1 pixel change on the model mess it up?

---

I think we'd have to cross this hurdle when we come to it. I don't think I'll be using exact pixel location for model recognition anyway.

Quote:

---

Originally Posted by The Claw

- Is there an "ID" system like with reflection or do we have to qualitatively analyse each model we grab and then compare with a database we've collected to identify it?

---

It will be an ID system. I probably wont be able to give model names though, e.g. 'King Scorpion', just an id.

Hopefully I'll have a small proof of concept to release in a few days :)

Sounds pretty cool man, definately has potential. Any chance you could write up some basic tutorials explaining the basics and neccessary info so once it gets off the ground people here could learn and contribute? Or link us to some things we should read, assuming no prior knowledge in this area?

Quote:

---

Originally Posted by The Claw

Sounds pretty cool man, definately has potential. Any chance you could write up some basic tutorials explaining the basics and neccessary info so once it gets off the ground people here could learn and contribute? Or link us to some things we should read, assuming no prior knowledge in this area?

---

I would want to learn this too :)

Seeing you managed to wireframe RS, do you think you'd be able to choose a specific NPC or an Object to render? e.g otherwise black main screen but a trees are showing or something...

Quote:

---

Originally Posted by n3ss3s

Seeing you managed to wireframe RS, do you think you'd be able to choose a specific NPC or an Object to render? e.g otherwise black main screen but a trees are showing or something...

---

Yeah, it would ease the scripting alot.

Quote:

---

Originally Posted by n3ss3s

Seeing you managed to wireframe RS, do you think you'd be able to choose a specific NPC or an Object to render? e.g otherwise black main screen but a trees are showing or something...

---

Wow that would make everything MUCH easier.. O_o
This has indeed a lot of potential!

This looks amazing. Very interested as all.

~RM

By the way, do not stop posting pictures! They look awesome!

Quote:

---

Originally Posted by Zyt3x

I would want to learn this too :)

---

Which language is it written in?

Another thing, you want to use the grapfic cards cpu instead of the real CPU (considering your using OpenGL) how would that effect the amount of SMARTs we can use?

To answer a few Qs: I will probably open source this once it is in a suitable state. There are plenty of openGL tutorials for legit game programming and cheating out there :) I will provide links later. It is written in C.

Quote:

---

Originally Posted by n3ss3s

Seeing you managed to wireframe RS, do you think you'd be able to choose a specific NPC or an Object to render? e.g otherwise black main screen but a trees are showing or something...

---

Although a nice idea and easy to implement, wouldn't you just prefer GetCoords(int NPCid)? Instead of just rendered the tree, wouldn't giving you the x,y coords of the tree be better? I would like to avoid overlays if possible, as I believe it would be possible for jagex to take screenshots of the current scene. This would = instant ban.

Having said that, exciting news: my first overlay!
http://i49.tinypic.com/1676yjs.png

That small black x is drawn on the first vertex of the player model (don't worry, any GetCoords would return the centre :P). It follows the player no matter the camera position, etc etc. So I can now get 2D coords of the 3D world :)

There is plenty of debugging to do, but soon I will start working on generating NPC ids based on model data (or something else) and drawing ID's over each model. This could then be used by the scripter (in a dummy acount they don't mind getting banned) to find ID's for every model they want.

Looks freaking fantastic!