How to fix paint after patching holes in the wall?

My wife and I are first time homeowners and are doing a lot of stuff for the first time. We removed some shelves from a wall that left some larger holes, so we got spackle and sealed them up. We painted once over the spots, then again to try to get a consistent color, but even after that, we have two problems:

1) I don't think we sanded enough spackle down, leaving bumps on the wall.

2) The color of the repainting does not match the rest of the wall, even though it's the exact same paint as when the wall was painted 4 years ago.

So… – Should we sand down again to try to get rid of excess spackle, or is this too late now that we've repainted? – How do we get a color match here? Should we try to repaint a third time? Or should we repaint the whole wall? How do we ensure when we repaint it's all a consistent color (do we need to paint white or prime first)?

See pics: