Secondhand smoke is the reason that has been cited for outlawing smoking in public places. But it appears that when people do drugs in private, trace amounts of those drugs end up in the water supply. Air is a gas and water is a liguid, but they are alike in that it is impossible to remove all contaminant. No amount of filtration will remove all smoke particles, and no amount of filtration can remove all antibiotics, antibiotics, and caffiene.
So the question is, does my recreational use of caffiene infringe on the rights of those who must abstain from the drug? Just like secondhand smoke, caffiene can cause an increase in blood pressure.
The answer is no. Trace amounts of a substance do not cause any harm. Smoke from any fire anywhere in the world will spread throughout the entire Earth's atmosphere. Sewage discharged into any river in the world will spread throughout every ocean. While it is true that some of the molecules in your drinking water may have spend time in Robert Downey Jr's bloodstream, the amount is so tiny that it is not worth getting upset about it. In fact, getting upset will do more to increase your blood pressure than other people's tobacco and caffiene usage.