Scientific publishing absorbs a lot of money from the budgets of scientific organisations, either by publication fees or subscription fees, and many feel that the value added by the publisher is questionable (source). That money comes from funders such as NIH, NSF, the EC Framework Programme, national research councils and large charities (Wellcome, Gates). Why do such funders not close the circle and offer a not-for-profit mechanism for publication, either individually or collectively?
Many (most, I suspect) scientific societies operate journals, and some of these offer travel grants or small project grants for research with no restrictions on the submission of resulting papers to their own journals, so I don't see how it could be a conflict-of-interest thing. Some universities do the same. I also see sites like PubMed Central and EuropePMC which are funded by research funders and act as repositories for full-text publications.
There are some suggestions of a move in this direction from the mathematics and physical sciences (link), although this would be run by the academic community, not the funders.
 I forgot about this when writing the question, but PLoS Neglected Tropical Diseases was started using a $1.1 million grant from the Gates Foundation, so this is a relatively high-profile example of a journal that was (at launch) directly funded by a grant-funding body. And I'd also forgotten that PLoS is explicitly non-profit, although they do use the income from some journals to help support the others financially so some individual journals might run at a profit.
[edit2] I'd also somehow forgotten this announcement back in July that Wellcome was launching a journal called "Wellcome Open Access". This one will only accept submissions from Wellcome-funded researchers, but will also accept negative results, null results and data-only papers, so will arguably overlap less with traditional publishers.
As you note, in a major sense this is actually already done on a large scale by the NIH in the US. The NIH requires that all publications resulting from its funding be made freely available on PubMed under an open access license within 12 months. While PubMed Central does not do peer-review, it does provide the other half of the functions of an archival publication, in providing hosting and curation.
By doing this, the NIH is accomplishing most of the goals you mention, because there is now much less pressure to subscribe to lots of high-cost journals. At the same time, however, this is better for the NIH than running journals itself for at least the following reasons:
- Running journals does have a significant cost, even if that cost is often hidden in free society journals and overcompensated for in for-profit journals. Forcing compliance as a byproduct of funding is probably much cheaper.
- If the NIH started running journals, it would be competing with traditional journals, and thus would obtain much less coverage than it does by just forcing public deposit of their articles.
- Scientific publication is highly international, and even the biggest countries are a minority of the researchers in any significant field. There may thus be significant perceived or real conflicts of interest if a major government funder is running the peer review process for other nations.
In short, it appears to be currently much more efficient to accomplish such goals by using the compulsory power of contract clauses than by investing in competition with the existing publication ecosystem.