The address book contains all users (current and past) who have archived data.
As long as users have archived messages, they cannot be removed from the address book.