Unable to turn off DirSync

I went through the Hybrid Setup configuration, unaware that this would enable DirSync. I now want to disable AD synchronization again.

When I go to the Azure AD Admin Center and disable directory synchronization for our domain, the switch will go back to Activated right after it was deactivated.

I also tried disabling DirSync through PowerShell:

Set-MsolDirSyncEnabled -EnableDirSync $false -Verbose

This only results in the unhelpful error message:

Set-MsolDirSyncEnabled : You cannot turn off Active Directory synchronization.