Yes, that appears to be a concern which is why a few users have advised to let the shower run for a few minutes in recirculated n mode after use to allow a good sanitization of the system/piping before shutting it down.You've got a good start but the inside of the system will be a breeding ground if there isn't something to sanitize the inside of the system.
Which is why I like @CrappyJax approach to allow the system to bypass the shower head and continue to circulate/sanitize the water without the shower flowing.
This fellow in the UK has a good run down on his system and points out the importance of hot water to free some of the soap residue.
AdvancedRV has also developed a system that is dual mode (fresh water/recirculating) which is pretty clever.
Sent from my iPhone using Tapatalk Pro