How does one choose a move in the maze of so many possible moves and variations?
I find that nowadays, when I look at the position, I first look for something to do (where my position is still safe and the kings on both sides are safe) is to in order:
a) look for loose pieces,
b) pieces that can be attacked (eg. backward pawns)
c) look for undefended squares
d) look for squares which can be used to improve my position
Once I found a move, I look for problems should I make that move.
a) Am I vacating a square which can be used by my opponent?
b) Have I left a piece en prise which the opponent can attack/gain a tempo with?
c) Have my defensive position weakened as a result?
d) If it is a pawn move, what has happened to the squares with which it used to defend?
I find that by constantly asking these questions, it becomes easier to make a good move and the number of blunders I make during the game as a consequence fall.
I played a game on Tuesday night and my opponent made a surprising move that I had not figured in my calculations. It was slightly counter-intuitive in that he moved his bishop to target a pawn chain formation directly along the same diagonal as the bishop. It was then that I realised that because the base of my pawn chain is only guarded by a rook and could be pinned, did I discover his idea and thankfully was able to repel his attack in time. I was silently cursing myself in that I had missed his move completely and I chalk it up as a move I should watch out for in the future.
I still find that the moves that are hardest to detect OTB are what I called "counter-intuitive" moves and non-forcing moves and it is from there I usually lose or proceeded to lose my advantage.
Okay, you may ask, having these set of rules is fine but how do I improve on learning how to make good moves during training?
The way I do is this (your mileage may vary) and I find it works for me:
Take any modern tournament book/autobiography that is deeply annotated and well explained by the players themselves. Now go to a random middlegame position of any game (anything from move 18 onwards is probably the best). Set up the board and start analysing. Now test yourself on how far you are able to calculate as many variations as possible (make sure that that the move has been annotated) within a 20 minute time frame. This is to simulate as close as possible to an actual match situation.
Start writing down as much as you can. Once the time is up, check if the variations you've made are the same as what the players have made. If the variation you've chosen is not explained, load the position in Fritz and see if there are any problems with the variations themselves.
At my level, I find that at times, I can find the GM moves but at times, my moves are weak and Fritz/Stockfish suggest alternative moves which are just as good.
It is not just you must know how to calculate but also what to calculate that is equally important.
So what kind of books would I recommend?
Being a patzer, I can only speak from my own experience and would not proclaim that the following list is the ONE but I find the following books have been fantastic when used. I'm sure you will have your favourite books as well.
If you're starting out, Bobby Fischer's My 60 Most Memorable Games is one of the best there is.
Mikhail Tal's Life And Games Of Mikhail Tal is another book I would gladly recommend for use in training.
Vishy Anand's My Best Game Of Chess is an excellent read where Anand annotates the moves and the variations of his games in clear detail (explaining the ideas/motifs and plans)
San Luis 2005 is a very intense book. The annotations are very nicely done, highly detailed and well explained. If you want to truly test yourself, this book will not disappoint when used for such training.
Note that I'm positive that there are other more effective training methods but I find that this training method I used has been effective for me. It may or may not work for you and I would not dare suggest anything to the contrary given my rating. I find that by making known what I use in an effort to play better chess and sharing this information, I hope you may find it useful.
As always, YMMV. Cheers!