I seem to have reached the phase of my career where I receive regular requests to externally evaluate faculty: this means that I am evaluating the research program of tenure-track faculty (so far always at US institutions, so let's concentrate on this case) and writing a letter to be used in some ways for the pre-tenure / tenure / promotion processes. If it makes any difference, I would be more interested in answers that apply to the field of mathematics.
Well, sometimes I have reasons that I might not want to do this. Here is a list of reasons that either have occurred to me or seem plausible that they might occur with others:
I feel that I am too busy: either too busy to do a good job in the time allotted, or just so busy that it would make my life easier to decline.
I don't have much insight into the candidate's work, and I feel that many other people could do a better job.
In order to do a good job I would have to investigate certain things, e.g. whether and why a certain paper has not yet been published. It's hard to investigate things in academia completely anonymously, and whether this type of investigation would be well-received or even appropriate is not completely clear.
I feel that the candidate's work is not very strong. (Note: not very strong compared to what is a key question here, but a sticky one. I have found that institutions which are more teaching focused often ask their candidates to be evaluated by standards which sound very rigorous and exacting to me, a faculty member at a major research university.)
Especially in the last case, it's not so clear "what's in it for me" to write an evaluation that says that a candidate's work is not as good as that of many other people I know in the field. They are still working in my field, so I would rather have them there than not. I have no idea what the chance is that my letter would be taken seriously in a failure to hire/promote them. Either way, there are reasons for concern on my end.
My main question is: if I decline to write such an evaluation, is the act of declining likely to have implications for the candidate? (E.g. is it likely that the declination would appear on the candidate's dossier?) The subsidiary question is: if I have reasons like the above that would make me prefer not to write an evaluation, is it nevertheless important to write one? Is it "the right thing to do"? (Of course one wants to do a certain amount of service to the academic community. On the other hand, many/most academics get offered so many service obligations that they have to turn some down. This question should be understood as relative to doing other service tasks, not doing less overall.)
Chairs / deans / provosts will tell you that declining will have no effect. This is the official and morally as well as legally correct answer.
However, I've had the great pleasure (not) of serving on a committee (at a private, elite R1) where the promotion review chair did not like the candidate and raised the issue that there were a certain percentage of declines to strengthen their tenuous case against the scholar ("no one likes/respects this scholar thus they are declining to serve as reviewers).
This was counter to policy and in my mind immoral and possibly illegal depending on the circumstances.
If you do decline, please include a phrase that you are declining due to personal / workload reasons and that the committee must absolutely not use your declination to in anyway prejudice the candidate's case.
Now as to your subsidiary question as to whether you should feel obliged to write, this is more of a moral question. As a tenured full professor, I do feel an obligation to my discipline to do these reviews.
However, I will only do a review if I can do it well. If a candidate's work is out of my field of expertise or if my workload prevents me from doing a good job, I won't write. My first obligation is, after all, to my research agenda and my students.
Furthermore, there have been cases where I know too much about a candidate to write an objective review. In those cases, I ask to be relieved due to a conflict of interest.
There are varying opinions on whether one should write negative letters and this should really be a separate question. My own philosophy is that my letter is a third-person's neutral observation of the strength of the candidate's application for tenure. I thus write about the strengths that I see for this case, but whether the strengths are sufficient for tenure is ultimately a decision that should be made by the tenuring committees.