I've had 6, in groups of 1 or 2. There are a lot of kids in the neighborhood, but many parents will only let their kids go to houses where they know the people; since I don't have kids, I don't know many of the parents, and smaller kids (and their parents) tend to pass by my house.
When I was a kid in Wisconsin (near Milwaukee), trick-or-treating was only legal from 1-5 on the Saturday or Sunday closest to Halloween. I was 14 before I went trick-or-treating after dark. I've always been somewhat surprised that I've never seen that anywhere else. It had some disadvantages - mostly, we rarely got to go out on Halloween - but it was warmer (important in Wisconsin), and if you got to someone's house near 5, they usually dumped most of what they had left in your bowl.