I just checked out my office. It looks like a dungeon. There are no windows. It is kind of cramped. Everything looks kind of industrial. I don't think I would like to study/work there. There are other places in the math building that look like much less depressing places to work such as the math library. Also coffee shops, at home, etc.
I have to have some number of office hours which I will gladly have. But for studying for my classes, working on research, etc. I would much prefer other places. There is no formal rules on where you have to be. I don't have an adviser yet and when I do, he won't be watching over me just looking for results. So there is no formal reason why I have to be in my office other than office hours.
If I simply don't use my office will I look bad to other grad students or professors? Will I maybe miss out on productive discussions? I can see if my office mates are in the same classes; we can work on the problem sets together.
What are the potential pitfalls of not being in my office other than for office hours?
Never is an absolute. Having said that, you have outlined reasons why you should use your office (office hours, spending time between department events and classes etc.).
Unless your adviser makes it mandatory (highly unusual), I don't see why you shouldn't work wherever you feel most comfortable in. I was very fortunate, in graduate school, to have a spacious, very well lit and clean office. In spite of these structural advantages, I chose to work from coffee shops most of the time. I had no issues with the administration, faculty or my peers.
To summarize, there are some (usually official) reasons why you would need to go to your office but there is usually no compulsion to have to continue using the office at all other times.