You can invite users to become collaborators to your personal repository.
When you remove a collaborator from your project, they lose read/write access to your repository. If the repository is private and the person has created a fork, then that fork is also deleted.
If you no longer want to be a collaborator on someone else's repository, you can remove yourself.
You can invite someone to manage your user owned repositories if you are not able to.