Not Your Pool
We’re sure most people dream of owning a house with a swimming pool. Just because you don’t have one of your own, doesn’t mean you can use your neighbor’s and that too without asking. This OP realized that his neighbor was doing exactly this when they were at work during the day.
We’re sure it was awkward to confront him about it but when they did, he just claimed that he had an agreement with the previous tenant that he could use the pool whenever he felt like it. That sounds like a likely story but even if it was true, wouldn’t you need a new agreement with the current tenant?