@PerfectHorse, welcome to the forum!
We have tried it both ways: like you and with just the iPhone. We find that, for us (iPhone SE, several of them over the past 2 years), if we restart the phone religiously every evening, with no other Bluetooth device on, we never get the dreaded “signal loss” error anymore. But it seems we do get the signal loss error occasionally, if we restart once a day, if we use both the receiver and the iPhone simultaneously.
You may want to introduce yourself on the Welcome thread if you wish.