Home inspection gone wrong?

My husband and I recently came to an agreement on an offer on a home. The house is 5 years old and we decided to do a home inspection for our safety. My husband was unable to attend the home inspection but I was able. When I arrived at the home the sellers were still inside and I felt very uncomfortable that I immediately called the realtor and told her. She called the sellers agent and told them they need to be gone from the home because its an awkward situation and when purchasing a home you need to be able to ask questions freely without offending the sellers. They said they would be gone by a certain time and when I returned the inspection was already over and the sellers were still inside. When talking to the inspector he said when they asked if they had to leave he told them no. He said the house was cluttered and he couldn't see everything. The inspector didn't give me a walk through and in the pictures he sent me there are the sellers kids in them. I don't feel a proper inspection was done and I don't know if I even want to go through with the house anymore. I just have a bad feeling. If we backed out now would we loose our escrow money? Am I overreacting?