We use data on Mexican and Chinese immigrants in the US to calculate the average marginal effects of residential and occupational segregation on immigrants' ability to speak English, and similarly the effects of English fluency of family members. Our results confirm that residential segregation is generally inversely related to English language proficiency of immigrants, except for skilled Chinese immigrants. Allowing for occupational fixed effects, the minority population share at the place of work is relevant for proficiency in English among skilled Chinese, but not for Mexicans and unskilled Chinese. We also find that the presence of English-speaking adults in the household increases the probability of immigrants' proficiency in English.