When I run the RAM frame model and check the report for time periods, I noticed that the mass participation is as low as 40% for 12 modes. Is there some way to check if this is caused due to the instability in the model ?
It usually due to a lot of semi-rigid diaphragm flapping. Use the process - results - mode shapes to check it. See: https://communities.bentley.com/products/ram-staad/w/structural_analysis_and_design__wiki/3348/ram-frame---semirigid-diaphragms-faq