install custom ms connect not working

It appears from your description that you need to install MicroStation CONNECT Edition so the workspaces, ie. the located under appdata\ folder, are in a different location. This would be similar to setting up a networked workspace, just that the files are local.

It would always be a suggestion here that the installation be performed in all of the default locations. Then you can copy the required files and folder into the location of choice and configure any future installations. The following Wiki article should assist with any settings that you might require.

https://communities.bentley.com/products/microstation/w/microstation__wiki/23984.setting-up-the-microstation-connect-workspace-on-a-network

AH ! This a bit confusing. During install with the older version it allowed to control the location of your workspace. This is a new beast and I need to wrap my head around it. If you are installing for the 1st time how do you have a existing Organization directory and Workspace directory?
"In the Installation Wizard, in the “Custom Configuration” dialog box, the settings “Select Organization Directory” and “Select WorkSpaces Directory” are intended to be used to select an existing Organization directory and WorkSpaces directory. They are not to be used to redirect where you want files to be installed."

AH ! This a bit confusing. During install with the older version it allowed to control the location of your workspace. This is a new beast and I need to wrap my head around it. If you are installing for the 1st time how do you have a existing Organization directory and Workspace directory?
"In the Installation Wizard, in the “Custom Configuration” dialog box, the settings “Select Organization Directory” and “Select WorkSpaces Directory” are intended to be used to select an existing Organization directory and WorkSpaces directory. They are not to be used to redirect where you want files to be installed."

Yes - custom configuration is intended to point to an already existing configuration. It does not create a folder structure as the previous versions did.