I am an undergraduate student with no experience and no guidance from any teacher whatsoever.
I have done research on Recreational Mathematics (Maths tricks). I have a lot of new tricks to introduce, in a completely new format.
But the problem is that there doesn't seems to been any research paper on this topic before, but there are many books. Can I quote books as a source of reference instead of research papers published? (I am quoting research papers related to the theorems and concepts I have used - but there is no reference to the trick on which I have worked upon.) Also the trick seems to be age-old and extremely simple, and I think that the inventor of it is not known to the world at all.
Can I cite the book? (I know that the author of the book didn't invent the concept behind the trick.)
You can and you should.
Sure, peer-review papers are preferred over regular (not peer-reviewed books) but if the information you want to cite only is written down in a book (peer reviewed or not) rather than in a research paper, of course you should reference to the former.
Technically, you can refer to whatever you want (a painting, a tv show, a private discussion with a friend of yours or The Journal of Ghosts and Humbug). However, if you want to write a believable paper, you should strive towards citing as trusted and/or high quality sources as possible.