The concept of cancer cells being hierarchically organized and arising from their own progenitor stem cells will have important implications on cancer therapy. If this hypothesis were to be true then the paucity of estrogen receptors in stem cells as well as their inherent drug resistance mechanisms pose a challenge to current targeted therapies. In this study, we sought to examine the prognostic relevance of ALDH1, a putative cancer stem cell marker, by immunohistochemistry. The four cohorts analyzed included an adjuvantly treated series of 245 invasive cancers, a neoadjuvantly treated series of 34 cases, and two series of 58 and 40 triple negative cases, respectively. Both tumor cell and stromal expression for ALDH1 was evaluated, where possible. Tumor cell ALDH1 expression significantly correlated only with basal-like and HER2 tumor types in the adjuvant series and tumor grade in the neoadjuvant cohort. No significant enrichment for ALDH1 positive cells was observed in the postneoadjuvant therapy specimens compared to pretreatment samples. On the other hand, high degree of stromal expression was significantly associated with best disease-free survival as well as a trend for overall survival. The association of stromal expression was confirmed in an independent cohort of triple negative cases. The novel finding is that tumor microenvironment may play a significant role in determining the prognostic impact of stem/progenitor cells in human breast tumors.