Last year, I wrote a paper and it got accepted and was presented at a conference.
This year, I am writing a more in-depth paper about a similar subject; but since one of the figures in my first paper is adequate for my purpose, I want to reuse that one (of course citing it adequately).
Am I OK in doing that, or is that seen as self-plagiarism or unprofessional?
(EDT: "in this case.." ) There's no such thing as self-plagiarism. It's your figure. you drew it, you have the right to use it anywhere you want any time you wish and as much as possible. I don't even see a need to cite previous appearances of the same figure (unless this figure is the main theorem/claim/result rather than an explanatory tool.)
The only thing to check is that the conf that published your paper doesn't hold some rights on it, due to editing it, improving it or that you gave up your rights when you signed a copyright-transfer form (as the Anonymous Mathematician mentioned in his comment)