Pretty sure they don't.. it's not even a good idea to ban someone by IP. Imagine you have a brother using the same computer as you. He doesn't bot, but you do.
Your IP gets flagged, is it fair that he gets banned too? No.. When last have you played counter-strike or any steam game and get banned via IP?
Secondly, if your router uses DHCP and you have a PPPOE connection like my neighbour, and your IP changes everytime you do a dns flush and renew your IP via:
Progress Report:
ipconfig /release
ipconfig /flushdns
ipconfig /renew
and it actually changes, Jagex would be in for one.. Also connecting through a VPN or a Proxy upon every login would completely overcome an IP flag. I've never actually had to physically reset or use the admin login and reset my neighbour's router to change my IP when using it. My own router doesn't do this though. I have to physically push reset or login and reset it. It's not your Mac address either because that's easier to spoof than your IP.
It is your account that actually gets flagged. Not your IP. Pretty sure Jagex also said they don't flag IP's.
How jagex actually bans or at least how we think they ban is really all speculation (maybe some theories and maybe some small facts). No one actually knows 100% but from patterns and experience over the years, many have come up with ideas as to how it is done. There isn't any "foolproof" way of knowing without actually working for them or hacking them.