Charles Widmore is banished from the island, Ben sees him off

Presumably sometime after 1992 (which is when the purge may have happened) Ben is at the barracks pushing young Alex on the swing set. Richard comes over to tell Ben that the submarine is about to leave. Richard tells Ben that he doesn't have to see him (Charles Widmore) off, but Ben says that he does.

At the pier, Charles Widmore is being walked to the submarine by some armed others in black vests. Ben calls Widmore's name and says that he's come to tell him goodbye. Widmore says that Ben is actually there to gloat. Ben says "Don't act as if I wanted this. You brought this on yourself." Widmore asks Ben if he's certain he wants to do this. Ben says that Widmore left the island regularly and he had a daughter with an outsider. He continues saying "You broke the rules Charles." Widmore asks "What makes you think you deserve to take what's mine?" Ben says that he won't be selfish and he'll sacrifice anything to protect the island. Charles says "You wouldn't sacrifice Alex." Ben says that he's the one who wanted her dead, not the island. Charles says "I hope you're right Benjamin, because if you aren't, and it is the island that wants her dead, she'll be dead. You'll be standing where I"m standing now. You'll be the one being banished and you'll finally realize that you can't fight the inevitable." Before being taken away Widmore says "I'll be seeing you boy."

So it seems that Widmore was being banished for breaking some of the same rules that Ben has broken. From the passports in Ben's secret room, it seems he left the island regularly. It seems possible that Widmore would have built that room in the first place. It seems that over the years, Ben became more and more like Widmore. We'll have to wait to find out more.