What Causes Smaller Breasts?

Hi, I am 21 years old and recently visited a gyn to rule out a hormone imbalance because of hair on certain areas of my body and low libido. The gyn told me I was normal based on an anatomical evaluation. I have been having periods every month since my menarche at age 14. However, I believe my breasts never completely developed. My breasts seem to have remained in the second stage of development - "breast budding". I was an athlete in middle school and throughout high school and I remain active today. I eat a healthy diet and weigh 140 lbs at 5'5". When I do gain weight, it goes to my stomach and thighs, but never to my breasts. I have very minimal fat tissue in my breasts, with the outer areas having the fat while the inner areas of my breasts have none at all. Two of my aunts have small breasts but even so are much fuller than mine; my mother is at a 36C. Is this delayed puberty? Are my hormones working properly?