This would be how my garrison is currently built. If it matters, I am currently level 100 and fairly casual. Still working on obtaining the garrison resources and achievements to upgrade everything to level 3.
Barracks (Large): I dropped the Dwarven Bunker in favor of picking up the Barracks once again. The plethora of easy missions in combination with the Level 3 Salvage yard yields a LOT of salvage and follower upgrades; Way more than the Dwarven Bunker ever did. Plus, Level 2 Barracks = can assign a bodyguard to follow you around, and Level 3 barracks = 5 more followers.
Stables (Large): Not having to dismount while poking at treasures and what not while in zones that aren’t Nagrand is pretty spiffy. Also: Mounts and easy quests that yield garrison resources.
Lunarfall Inn (Medium): Once a week I can recruit a follower with an ability or trait of my choosing. Super awesome for grabbing followers with rare traits such as Epic Mount and Scavenger. Also, a source of dungeon daily quests; Some of which have pretty nifty rewards: Aviana’s Feather, Everbloom Peachick, Sentinel’s Companion, Ever-Blooming Frond, Oralius’ Whispering Crystal.
Lumber Mill (Medium): At some point I would like to replace the Lumber Mill with something else, probably after obtaining enough resources to upgrade all garrison buildings to level 3. The Barn would be my first replacement choice since at Level 3 it has a chance at allowing you to harvest Savage Blood, which is needed for gear upgrades.
Salvage Yard (Small): The Level 3 Salvage Yard crates pop pretty often and usually yield quite a bit of stuff. Mostly crap greens, but occasionally follower upgrade tokens. If you do not do so already, always check the 3D models on the BoE greens. If they look unique or nice, odds are that they will auction pretty well for people seeking transmog gear. I’ve made several thousand gold just by doing that and selling off extra garrison mats on my phone…
2x Profession buildings (Small): The Blacksmithing building is nice to have around for the sake of having a forge + the follower buff + extra mats. Aside from that, I don’t really feel the need to upgrade either of these to level 3 until the other buildings are done.