The natural history of a tumor includes phases of 'in situ' growth, invasion, extravasation and metastasis. During these phases, tumor cells interact with their microenvironment and are influenced by signals coming from stromal, endothelial, inflammatory and immune cells. Indeed, tumors are often infiltrated by various numbers of lymphocytes, macrophages or mast cells. It is generally believed that the latter produce factors that maintain chronic inflammation and promote tumor growth, whereas lymphocytes may control cancer outcome, as evidenced in mouse models. In this study, we analyze data from large cohorts of human tumors, clearly establishing that infiltration of the primary tumor by memory T cells, particularly of the Th1 and cytotoxic types, is the strongest prognostic factor in terms of freedom from disease and overall survival at all stages of clinical disease. We review data suggesting that tertiary lymphoid structures adjacent to tumors and composed of mature dendritic cells (T and B cells organized as germinal centers) may be the site of an antitumor reaction. We propose an immune scoring based on the type, density and location of lymphocyte infiltrates as a novel prognostic factor for use in addition to tumor node metastasis staging to predict disease-free survival and to aid in decisions regarding adjuvant therapies in early stage human cancers.