Drivers Window Loses Power?

The driver side window works one time but then once you have moved it up or down and leave it for 5 seconds, it doesn't move. My guess is that it loses power because when you turn off the car and leave it for a while. It will start working again. Also if it is not fully down, when the window motor inside the door is disconnected then reconnected it works again. But not if it is fully down.
Has anybody had a similar problem, How was it fixed ?
Cheers

Drivers Window Loses Power?