My shells were starting very slow which is really bad as a dev/admin. I found out that all those version managers I installed were to blame for that. So I decided to get rid of nvm and rvm by simply removing the directories and cleaning up my .bashrc and .profile in $HOME.
#nvm
rm -rf ~/.nvm
rm -rf ~/.npm
rm -rf ~/.bower
rm .zshrc # i am using bash
#rvm
rm -rf .rvm
# cleanup configs
vim ~/.bashrc ~/.profile
After those simple steps my shell is up within an instand instead of needing some thinking time.
sdkman and brew seem to be ok and are allowed to stay installed for now.