What Severe Constipation While On Alfuzosin?

I was prescribed Alfuzosin. I took it for two days, during which I had no bowel movements. After the second day, I stopped the medication and started taking Miralax and magnesium citrate. It took an additional three days before I had a bowel movement and three days after that I am still having difficulty moving my bowels. My urologist wants me to restart the medication, but I feel I should at least wait until I am having regular bowel movements before attempting to restart the medication. What do you recommend?