During the installation process of MDM 11.5 we often forget to change soap request timeout setting and jvm heap size in WAS 8.5.5.x. This should be done before MDM installation starts in Installation manager.
Open <soap.client.props> file from following location:
C:\IBM\WebSphere\AppServer\profiles\profile1\properties (can vary according in your installation)
Find the following variable and change the value to a high number:
com.ibm.SOAP.requestTimeout=18000 (default is 180 seconds)
Next, login to WAS admin site. Go to [Application servers > server1 > Process definition > Java Virtual Machine]
Change heap size for both initial and max heap. Better to keep both same size. A sample screen shot is below: