Apoptosis in ovarian surface epithelial (OSE) cells is induced by transforming growth factor-beta (TGF-β). However, high-grade serous ovarian carcinomas (HGC) are refractory to the inhibitory functions of TGF-β; their invasiveness is up-regulated by TGF-β through epithelial-mesenchymal transition (EMT) activation. Serous borderline ovarian tumors (SBOT) have been recognized as distinct entities that give rise to invasive low-grade serous carcinomas (LGC), which have a relatively poor prognosis and are unrelated to HGC. While it is not fully understood how TGF-β plays disparate roles in OSE cells and its malignant derivative HGC, its role in SBOT and LGC remains unknown. Here we demonstrate the effects of TGF-β on cultured SBOT3.1 and LGC-derived MPSC1 cells, which express TGF-β type I and type II receptors. TGF-β treatment induced the invasiveness of SBOT3.1 cells but reduced the invasiveness of MPSC1 cells. The analysis of apoptosis, which was assessed by cleaved caspase-3 and trypan blue exclusion assay, revealed TGF-β-induced apoptosis in MPSC1, but not SBOT3.1 cells. The pro-apoptotic effect of TGF-β on LGC cells was confirmed in another immortalized LGC cell line ILGC. TGF-β treatment led to the activation of Smad3 but not Smad2. The specific TβRI inhibitor SB431542 and TβRI siRNA abolished the SBOT3.1 invasion induced by TGF-β, and it prevented TGF-β-induced apoptosis in MPSC1 cells. In SBOT3.1 cells, TGF-β down-regulated E-cadherin and concurrently up-regulated N-cadherin. TGF-β up-regulated the expression of the transcriptional repressors of E-cadherin, Snail, Slug, Twist and ZEB1. In contrast, co-treatment with SB431542 and TβRI depletion by siRNA abolished the effects of TGF-β on the relative cadherin expression levels and that of Snail, Slug, Twist and ZEB1 as well. This study demonstrates dual TGF-β functions: the induction of SBOT cell invasion by EMT activation and apoptosis promotion in LGC cells.