It is always suggested to keep citrus trees at a height for easy harvesting. Overall size is up to you. Container trees are going to be smaller and pruning is done to shape and keep the tree in a visually presentable form.
For Container Citrus Trees
Pruning tree branches from container trees is a visual approach. Tree branches sometimes have a mind of their own and can run wild if you don't help guide them where you might want them to be. A suggestion is to prune in a fashion that keeps the canopy of the tree in a consistent shape. Don't worry if larger branches have to be removed... Just remove one and let growth fill in before removing another. Over the course of time you will be able to shape your tree just like you want.
Remove sprouts at the base of the tree. Sprouts, sprigs or suckers should generally be removed from the trunk of the tree. These generally happen below the union where a specific variety of citrus is placed on a rootstock at the nursery. Removing these sprigs assures that your tree stays the variety it was budded to be. In addition there is no need to grow something you don't want. Most suckers are of a sour variety and not desirable. In some cases though, rooted cuttings are of the same variety and sprouts from the base are desirable when it comes to bushing citrus plants.
For Large Dooryard Trees
You can trim and prune trees any time of the year. Heavy cutting back is best done in February right before the spring flush and bloom. When you do heavy trimming this time of year look to remove damaged and broken branches as they will not be able to hold fruit over the extended amount of time for ripening. You are better off to remove the branch and help the tree focus on sizing fruit that will hold and ripen on the tree.
Another thing to look for are crossing branches on the inside of the canopy. These crossing branches can be removed to allow for fruiting limbs to get the amount of ventilation and light needed for vigorous growth. Sparingly is the suggestion if you are concerned with removing to many branches. One here and there is the best bet.
Diseased branches should be removed as disease can spread to other parts of the tree if not removed. A good way to look at removing diseased branches is that you are taking a preventative approach to helping your tree live healthy.
Skirting a tree is common. What this means is that you are trimming the bottom branches to create a specific height from the ground. It also removes branches that may break due to weakness under heavy load, and keeps your fruit off the ground and away from critters.