Yes it matters. Final rinse water must be taken from the final rinse equipment: this will determine if the rinse equipment is clean (unless a hose is used to rinse product; then the sample may be taken from the water source) when testing for potability.
Additionally, some other activities like post-harvest agricultural chemical application, and humidity and misting require that water tests are taken from the equipment to ensure that the equipment is not a source of contamination, Treated water needs to be taken from where it is being treated to ensure it is being treated appropriately. Water being stored (e.g., in a container, tank, cistern) needs to be taken from where it is being stored to ensure that it is clean. Water used for other uses (e.g., to fill ponds, dump tanks, handwashing, etc.) needs to be taken from the appropriate location (e.g., equipment, tap, storage cistern/tank/container, etc.).
IN ALL CASES operations should refer to the manual to ensure all relevant factors are being considered. There are many different water sources (e.g., municipal, well, surface), uses (e.g., washing, fluming, hydrocooling, etc.) methods (e.g., submersion, spray nozzles, etc.) and water variations (e.g., closed circuit pipes, storage in tanks, treatments, etc.) that need to be assessed. The requirements/procedures will be specific to each unique situation.