I tried running it again today & found out that the actual problem might just be:
After a few times of going in/out of the portal, the portal color changes randomly from yellow to white, making my char stuck and unable to get out of the mine.
Add: Actually there seems to be 3 types of portals. 1 that is white. 1 that is a different type of yellow, almost gold. Too bad I can't upload SS.