It seems this is buggy on the macOS part, the solutions is not to open Bluetooth Prefernces window while streaming audio to your Mac. Hopefully this should be fixed on the next macOS updates.