Boundaries showing for non-display references?

Is this expected behaviour?

When I turn on Boundary Display in some views I get

  

I have 30 or so references, all of which have display turned off.

What can I do to stop these from displaying?