When I start a game design, I typically start with a giant box of junk. All the things I want, all the pieces, all the story parts and characters, all the systems, and everything gets thrown into the box in a huge, happy, messy pile of junk.
And it sucks as a game.
A lot of designers make the mistake of stopping here and shipping the game. I have shelves full of games that feel they stopped design at this point. They are huge boxes of a thousand different things, as if variety alone made up for a lack of cohesive design. Ironically, this is how Pathfinder ended for us, with dozens of books from several publishers and no real central concept or theme to anything. Sticking to the core book, there was a focused game there, but the curse of the splatbook and chase for financial sustainability ruined what was a clean, interesting take on 3.5 and bloated the game into something that tried to do everything - and many of the classes and powers started to repeat each other and be shuffles and multiclass classes mixers in one class.
But back to designing and my big box of junk. So I throw away the design, keep the pieces, and redesign the whole thing. Maybe I could focus this better on just the best parts. So I have a box full of the best junk only this time!
And it still sucks.
So I begin another process of refolding and refactoring again and again, taking one aspect of the game and aligning all the pieces along that idea - rotating the matrix of pieces to focus on the one of the central ideas. If you have a game "with" monster capturing and training, what happens when you make the game "all about" monster capturing and training? Let's take the pieces, make monster training the central core, and see how everything else lines up.
Supporting Concepts Matter
You may discover entire parts of your game become meaningless and can be discarded. A crafting system? Well, why force players to puts lots of time into crafting - away from their monsters - when we could focus that time on bonding and training the player's monsters? Wouldn't that time be better spent closer to the game's focus and fun? The answer is almost always yes, and you begin cutting features or aligning others to support the main idea.
Crafting pet equipment? A better use for a crafting system in a monster training game because you are still spending that time for your monsters instead of yourself, if you need crafting at all. It is so easy to drop in systems that end up weakening your concept, but at first glance, they seem great to toss into the box.
Please note, don't get too attached to your design - it still may suck.
Your Premise May Be Bad
You may discover that your central premise sucks, let's say your heart isn't into monster training, and you toss out that entire part of the game, shift everything again, and align the game along a new set of core fun features and concepts. Everything else shifts and aligns around the new core, and you begin throwing out and strengthening your core idea. Instead of monster training, let's focus on putting dragon souls into weapons and armor. Everything then aligns to this new idea, does it support the concept or take away from it?
I have had designs get to this phase, a focused, unified whole, yet they still weren't fun. It takes a while of working through things, throwing things away, and really getting to know what you are trying to do and also why you are designing a game. After a while that becomes a feeling, why am I doing this?
Stick with it, there is fun in there somewhere, and it will take a lot of experimenting, redesign, and patience to find.
And then you start playtesting, and everything still sucks.
But you are in a better place now, because you now have a focused design and play experience with something that has a fun core, but broken mechanics. Mechanics can be fixed and tweaked to align along the fun parts of the game.
I wrote a very cool mini rules set, found out it did not focus on the parts I really needed it to focus on, most of the rules were never used, and the parts I did think were going to be fun and work beautifully - sucked. It was okay though, I learned a lot from the playtest and this is all good data. When you playtest, expect to be blindsided about your initial assumptions and how wrong and unfun they are.
Playtesting is the point in the process when you discover how utterly wrong you are. I laugh a lot at myself during this time and ask myself what was I thinking. You do have to approach this process with humor, because being too hard on yourself leads to frustration.
This is why I print all my rules out and scribble all over them with data, cross out things that suck, and make notes like the printout is note paper. Things will get better, but the more you playtest the better they get.
You may need to refocus your design at some point, so there will be back to the drawing board moments. Do not be afraid to toss things away and start over. This is your best skill as a game designer.
My Designs Suck - At First
I did a design recently that went through these steps, starting off with a big box of junk that sucked. It took a couple transformations to get it to a point where it was less "everything cool" and into "a cool something special." I had to rotate the feature matrix and transform the game several times, with plenty of thrown away work, before I could "see" the game hiding in there.
Sometimes it takes time and repeated tries. Even your highly refined and focused ideas, the ones you get most excited about, can be completely wrong. They can blow up in your face. You will sit there holding a pile of broken bits of rules and concepts and wonder, what happened?
But once you see the game and it captures your imagination, you can rebuild and realign your feature set to support the fun, and toss out rules, patches, and features that distract from your idea.
The game is there, trust me.
Just keep at it, and keep refolding and rebuilding your idea until you find it.