I've been collaborating on a project with a researcher in another field (I am being vague on purpose here in case they read this), which so far has pretty much gone nowhere. I've done a few preliminary calculations, though, and they want to publish them - I suspect they are under pressure to up their publication count.
I don't think that what we've done so far is worth publishing; there is nothing actually wrong, but I don't think there is anything new or interesting. I could imagine developing this into something useful, but it is not a high priority for me, and this collaboration does not seem to be working out, so I don't think pushing to hold off on publication would do more than draw things out.
The possible paper is in a field which I haven't worked in previously, but which I could see myself coming back to in the future (it is interesting to me, it is related to my main field, and I've picked up a fair amount about it by osmosis). I'm worried that if I approach other people in that field in the future (and I would need collaborators to get started effectively) the one thing of mine that they will see will be this paper, which will reflect poorly on me (in a way I would like to think is not accurate).
Should I decline authorship of this paper?
So long as there's nothing wrong with the quality of the work, I'd let the person in the field make the decision about it's worthiness for publication. Often, what's humdrum in one field is new and exciting in another. I've seen people making livings moving routine info from one area to another.
When seeking collaborators in the future, my opinion is that having published a solid, if not exciting, piece of work in the field will be more advantageous than having published nothing.