How do you tell a neighbor he can't use your phone. Without being disrespectful.?

In my building we have a neighbor who is about 60 years old. When we first moved in he'd ask my husband to fix his tv for him. Then he asked for a stick of butter, and then he started asking to use our phone. He has a bit of a gambling problem and his car got repossessed, and he asked my husband to drive him to go get his car from the impound. We are very generous and giving people, and don't mind doing favors, but its now gotten to the point where he asks to use our phone almost everyday. I have two small children who demand my attention, and he has even banged on our door and woke them up from their nap...just so he could ask to use the phone. He brought us a ham for easter (even though we don't really celebrate easter or even eat ham) and asked us to cook it for him, and just give him a plate of it. We did so because we felt bad that he didn't have anyone to celebrate with, but I just feel like we are being used. He has also borrowed money (30 bucks) from us, which he payed back about a month later. I have started not answering the door, but he will wait till my husband comes home to then ask to use the phone. He does not get along with his children, and we feel sorry for him and we even joke we adopted a senior citizen.... My question is.. How do I get him to stop asking us to use the phone without being rude or disrespectful? I have missed a couple important phone calls while he was borrowing it. I want to stop this behavior.