How to remove "Enterprise Team Members" who are NOT in the "User Management" section of the Enterprise Portal?

When I pick "Roles and Permissions" from the Enterprise Portal, the list of users that appears includes people who are NOT in my "User Management" section of the Portal. How do I remove them? I see only the option to "Modify Role(s)". I want to delete them, not modify their role(s).

Bruce