Fair enough, though there is a group out there that would like your dark mode and their system's light mode (or the other way around), or you have your system set up to switch between light and dark depending on the time and they always want the dark version of your site. A toggle would help in that case.
True! But I've checked myself and some colleagues: none uses the toggle for readability/accessibility reasons.
They toggle to check the dark theme or just play around.
For sure that doesn't mean that there's no one using a toggle for not only fun reasons. But also for accessibility: a lot if not most dark themes are poorly done. Not enough contrast, only swapped background and primary text color, don't darken images/videos and so on.
That's why I've decided to support dark theme, but only by system preference. Because this way I don't need any JS and don't have to double my CSS/selectors to work with media query and body class or anything. Sure CSS variables could help a bit. And possibly even could work without any JS by doing some crazy radio button check styling.
But all of this wasn't worth it for me. I think instead of putting time and energy into a toggle the time is better spent into creating a good dark theme. Styling the whole page including buttons, links, images, forms and so on.
Doing some accessibility checks/work with AXE.js or similar.
Because creating the toggle in an accessible way is also a bit harder.
But if the dark theme itself is good already and you have free time: for sure a toggle doesn't decrease UX!