Although most sex offenses are committed by relatives or acquaintances of the victims, our public policy approach has been to focus on the stranger sex offender and punish sex offenders through residency restrictions. These residency restrictions effectively banish these locally undesirable and dangerous individuals from our communities in fear that they may reoffend in our neighborhoods. Rather than being thrust into some wilderness, sex offenders are “banished” to neighboring counties or states and into poor, minority neighborhoods where they often live in boarding houses with other sex offenders. Banishing sex offenders through these residential restrictions impacts individual liberty, our national structure, and social policy considerations. This Article offers a legal analysis of the adverse impacts these restrictions impose on the constitutional rights of both sex offenders and our communities, which for economic or political limitations do not have the appropriate representation to mitigate these consequences. This Article also examines what methods from the environmental justice movement might be available to deal with the “social justice” issue of sex offenders disproportionately burdening poor, minority communities. Finally, because there is not yet evidence to support the efficacy of residency restrictions on sex offender recidivism, this Article concludes that legislators should reexamine the current trend of using residency restrictions to address concerns about sex offender recidivism. Instead, public policy decision makers should look toward alternatives, such as individualized risk assessment and management of these individuals, so that public resources can be properly directed to confine, monitor, and treat those sex offenders most likely to commit serious reoffenses.