Poor nutrition and a lack of exercise are also related to excess body fat, the second contributor to back fat. The combination of both of these contribute to excess fat tissue accumulation around the back and promotes the appearance of back flab around your upper and lower back as well as around your sides.