Lee grew up in Fort Greene, Brooklyn. He made his comments during a Black History Month speech at Pratt Institute in Brooklyn on Wednesday night.
He says gentrification had little regard for people who "have a culture that's been laid down for generations."
While gentrification brought improvements to neighborhoods such as Fort Greene, he demanded to know, "Why did it take this great influx of white people" to make things better?
Some audience members said they understand why some of their neighbors couldn't pass up high offers for their homes.
Lee lives on Manhattan's Upper East Side and recently listed his town house for $32 million.