A "Russian style" seminar, as in the Israel Gelfand form, breaks from the traditional format of talk followed by Q&A and just opens the floor to questions at any time. I've seen a few professors who follow this format, so I definitely see its value.
What is the best way to start something like this, and how do you get the most out of the seminar?
I have seen this format work well in colloquia.
The speaker generally says at the start that they are happy to answer any questions during the talk. If needed at various points the speaker can also ask if anyone has any questions. Ideally, after asking the speaker pauses for a couple of seconds to give people a chance to ask before moving on to the next section of the talk.
It can also sometimes be necessary to close off a particular discussion if it is going on for too long and is not of particular interest to the general audience.
In general, I think the format works best when the presenter knows the material well, and the audience is reasonably well informed.