Four year olds are toddlers really, much too young for formal classes in anything.
I think it depends on how, well, formal the class is. Our Little Dragons class is 4-6s, and we've found over time that taking more a of a structured play approach and teaching them basic skills as games, obstacle courses and so forth works better for that age than a more traditional martial arts class. Half of what they're learning at that age is just, like, how to take turns and listen to directions and have confidence and develop their basic motor skills. The martial arts stuff is mostly just a way to learn that.