I'm unashamedly fond of the stereotypical fantasy setting: a pseudo middle-earth/medieval land ruled by kings and fought over by powerful magics. It just works for me. It's where I live inside my own head, and it's where I want my characters to live (I may be a bit guilty of reverse Mary-Sue-ism). A lot of people moan and complain about it, that it's overused, boring, or an unimaginative rip-off of Tolkien. Some of this may be true, but I would argue that there is also plenty of great fantasy out there that doesn't follow this formula. So much of it in fact, that I am surprised these complainers haven't tripped on it on their way to the forums (lame joke, I know). This is a problem to me: a bunch of loud-mouthed minority opinions that step on aspiring writer's dreams of writing the fiction they love.
Why should you ignore it? Because you are free to do as you wish, and there will always be enough readers to go around. But this isn't what I want to discuss. I want to explore the many options we have as writers in creating the world our characters live in. After all, you are the god of your story. Your word is law. Just make sure it's good stuff.
A lot of people start with setting, before anything else. And there's nothing wrong with that. Some major-hitting authors (I can't remember which at the moment, it might have been Orson Scott Card) have gone on record as saying they start with nothing but a map, then build everything else from there. There is no right or wrong (I personally start with a character first). Culture, politics, magic systems, and geography all have a part in Setting, so be thinking about all those things as you read on.
So you have this vague idea. Maybe its an image that struck you of a misty bridge built between clouds, or an Inn's warm kitchen set in a forest. Perhaps it's a smell that exists only in your head: the salty smell of a calm beach or the char-choked air of a burned-out city. The smallest things can spark a setting, you just have to build on that spark... or maybe a better analogy would be: fan that spark into roaring flames. Whatever.
It's sometimes difficult to explain the process of world building from a tiny idea, it happens extremely fast in my own head, and I'm sure most of you know what I mean. It's a process of exploring the surroundings of that calm beach from above, asking where it exists, why is it there. Are there people about? What do they do? The key is to ask questions of yourself. Then it's your job to make up the answers. To me, this process is incredibly enjoyable. It's pure make-believe and creation. Allow yourself to go wild, no holds barred. If you start out big and crazy, it's easy to reel yourself back in if need be, however, starting out too small and conservative can be troublesome. So explore those dusty corners of your brain, think back to when you were a kid and that one book about talking animals made you crazy (or whatever did the same for you). Why did it stick with you for so long? Use that feeling it gave you and run with it, making your own world in the process.
A word of caution that I had to learn the hard way: make sure everything makes sense. Your readers are willing to suspend their disbelief, but only so far. If you have floating pink mush balls for characters that live in someone's lower intestine, make sure every other detail fits with that. Don't break the illusion. It's hard to coordinate all the facts, so make a list, or write a small history book entry for your world. Use time lines or cork boards and note cards, whatever you are comfortable with. Don't be afraid to write pages of world-building.
On the other hand, don't. It works both ways. On my second book (after giving up the first one as a bad job) I went into it with the express intent to not do any world building. Any details were to come about in the actual pages. My goal was to put all my creative effort into the book itself, and not draw maps or make index entries for characters. And you know, it worked pretty well. I actually finished the first draft this time, in less than a month. This may not work for you, but don't be afraid to experiment.
In conclusion, there is no right or wrong way, just results. Find what works best for you, and do it. And have FUN.