Bob is having issues with Google Assistant and the GPS in the Google Pixel 5. The accuracy is terrible. You can ask GA where you are, and it'll tell you your home address. Leo says it sounds like the phone isn't getting a GPS lock, and as such, the phone's GPS isn't able to get a track. GPS needs three satellites to triangulate your location. It could be the GPS radio isn't that good, or it's lagging. There is a switch for accuracy in the location services menu of the phone. You can make sure the slider is reflecting high accuracy as well. Turning on WIFI can also help because it can triangulate using hotspots. Also, try restarting your mobile phone or recalibrate your compass. Or maybe even doing a factory reset.
But it sounds like the GPS radio just isn't working right and could be defective. One other option is to get a GPS Dongle. You can get them on Amazon.
From the chatroom - here's an article from GadgetFeatures.com on how to fix the Pixel 5 GPS issue. There are other articles here and here.