This is something i picked up during my initial configuration of vCloud director and stumped me for a many days until i was finally able to figure out what was going on.
I created various isolated-backed networks, however during the course of the configuration changes were required and I delete ALL of the network pools to start over. When I then recreated the network pools with the same VLAN. When I deployed the vApps with local vApp network the VM’s did not want to communicate with each other at all.
Debugging the problem:
I tried recreating all of the network pools again, and deployed the vApps multiple times with different configuration. Tried to manually setup IP address and multiple network cards. However when i change the VM network to a portgroup that was not part of vcloud director I was able to communicate with the VM which lead me to believe the issue was not with the VM itself but with the vCloud network component. Tested the vShield Manager which seems to be working correctly.
VMware support case opened:
VMware was unable to figure out the problem at the time…
vCloud director 126.96.36.1998405
vShield manager 5.1.1-848085
Did some research to see if any other users were experiencing this problem and ran into the a community post which describe the problem to the teeth exactly like mine. Link below.
Solution we both came up with was to re-prepare the physical ESXi host within vCloud director, without needing to do anything else.
Another workaround to this was to always leave one network pool, and this problem only appears when you delete all the isolated-backed network pools.