For a long time, I had been using open source software for my work to boost "reproducible research". I believed that if I made my codes open source and the softwares to run those codes in were open source too (or at least free), my research would be utmost reproducible. However, recently, in a discussion, it came through that research is more reproducible if one uses "popular" softwares instead of "unpopular" free ones.
I had been using Scilab (Free) for a lot of my work and distributed my files to others. But I was surprised that more people had MATLAB ($$) and preferred if I sent them MATLAB files instead (little modifications).
My question is :
Assuming I'm starting a new project and I wish to make it as reproducible as possible. Should I be using relatively unpopular free software or extremely popular proprietary ones?
You can do whatever you damn well please, but there are a few considerations:
1) You might have an obligation to disseminate your work. If you are supported by an external funding agency such as the NSF or NIH then dissemination is an obligation. Many private foundations and other funding sources also provide support with the intent of dissemination, whether it's stated explicitly or not.
If you have such an obligation, then absent any other consideration this would suggest you use the most accessible software possible, regardless of whether that is proprietary or free.
2) Open source-ness is important for some kinds of research, but for most research it's irrelevant. Unless you're doing computer systems research, where it really matters how the computer arrives at an answer, the method does not matter so much as correctness. Sometimes it can matter (e.g. if your work is heavily dependent on numerical methods), but it probably doesn't.
3) Communities establish standards of validity and integrity. If everyone else uses MATLAB, then the community has deemed it accurate. Absent evidence to the contrary, using open-source software does not make your results seem more correct or more verifiable in anyone's eyes.
As as side note, Mathworks has a strong reputation for working with researchers. If you really felt that MATLAB was giving you incorrect results, and you had examples to show it, they would be knocking down your door to fix the issue. I've had Mathworks issue me a custom support patch the very same day I called about an obscure hardware incompatibility that was causing incorrect behavior.