I have an DGN with a relativily large 3MX file attached. Now the display in LumenRT designer is not displayed correct. The output when I make a film isn't good also.
How can I correct this behavor?
The 3mx file is converted to a fixed resolution mesh at time of export based upon the setting for it's Export Resolution. To change this setting, select the 3MX and view it's Properties dialog. There you will find a field for Export Resolution.
Adjust this to a smaller value to export a higher resolution mesh, but remember that this will also increase the file size of the LumenRT livecube created accordingly.
A better solution would be to use a 3SM rather than a 3MX. A 3SM is not converted to a fixed resolution mesh when exported to LumenRT. It maintains its high quality and remains fully scalable as you zoom in and out!
You can use ContextCapture to create a 3SM rather than a 3MX, and you can easily convert an existing 3MX to 3SM via a pull-down menu in ContextCapture.