The microenvironment of a tumor has emerged recently as a critical contributor to the development of cancer. Within this environment, fibroblasts and immune cells are the cell lineages that seem to be active mediators of tumour development. The activated fibroblasts that are also present during wound healing and chronic inflammation have been studied extensively. Their activation leads to altered gene expression profiles that markedly increase growth factor and cytokine secretion, leading to major alterations in the immune cell microenvironment. To better understand normal tissue development, wound healing and the chronic inflammation that leads to cancer, we review here information available on the role of fibroblasts and immune cells in normal breast development and in cancer. We also discuss the immunogenicity of breast cancer compared to other cancers and the contribution of the immune microenvironment to the initiation, progression and metastasis of tumors. Also reviewed is the limited knowledge on the role of immune cells and fibroblasts in normal development and whether the risk of cancer increases when their control is not tightly regulated.