The Detailing Symbol Style in the Place Callout tools dialog is grayed out and cannot be changed to a custom detailing symbol style, or if changed it reverts to default.
Go to your desired dataset location similar to this: C:\ProgramData\Bentley\Configuration\Datasets\Dataset_NM\DgnLib\