Failed during Photos Pairwise matching.

Hello,

I am rendering out a 77 tile 3D mesh using aerial shots only and the same tile keeps failing and the error message is "Failed: Failed during Photos Pairwise matching.  I have tried making different reconstructions and just tried rendering out that tile so I can just replace it into the one where the other tiles finished but it keeps failing.  I even tried making a totally new block, still Tile_13 keeps failing.  

I am also getting an error message that reads "Failed: Failed during 3D mesh texturing. This is for a different block that includes aerial and ground photos. I have attached both error logs.

  • Oops, doesn't look like it attached. Here is for Tile_13

    task_log_0.log

    <?xml version="1.0" encoding="utf-8"?>
    <Job>
    	<Version>1.3</Version>
    	<JobName>job_20190115T211857_25_Santa_Monica_LADWP_B345R1P1_Tile_13</JobName>
    	<TaskGraph>
    		<JobInfo>
    			<Descr/>
    			<ProjectPath>//storage-a/main-storage-1/CC_Projects/Santa_Monica_LADWP\Santa_Monica_LADWP.ccm</ProjectPath>
    			<ItemPath>Block_345\Reconstruction_1\Production_1\Tile_13</ItemPath>
    		</JobInfo>
    		<RunInfo>
    			<SubmitHostName>AlchemyWorkstation1</SubmitHostName>
    			<SubmitUser>workstation2</SubmitUser>
    			<SubmitTime>20190115T211857</SubmitTime>
    			<Run>
    				<RunHostName>Node2</RunHostName>
    				<RunUserName>Node2</RunUserName>
    				<StartTime>20190115T212121</StartTime>
    				<EndTime>20190115T230243</EndTime>
    			</Run>
    		</RunInfo>
    		<JobFeedback>
    			<Version>1.3</Version>
    			<Status>32</Status>
    			<Percent>12</Percent>
    			<Msg>Failed during Photos pairwise matching.</Msg>
    			<PT/>
    			<Tasks/>
    		</JobFeedback>
    		<Task>
    			<Type>4</Type>
    			<Workload>100</Workload>
    			<ProjectPath>//storage-a/main-storage-1/CC_Projects/Santa_Monica_LADWP\Santa_Monica_LADWP.ccm</ProjectPath>
    			<ItemPath>Block_345\Reconstruction_1\Production_1\Tile_13</ItemPath>
    			<Run>
    				<RunHostName>Node2</RunHostName>
    				<RunUserName>Node2</RunUserName>
    				<StartTime>20190115T212121</StartTime>
    				<EndTime>20190115T230245</EndTime>
    			</Run>
    			<Id>0</Id>
    			<TaskMsg>
    				<Time>20190115T230243</Time>
    				<Code>280248320</Code>
    				<Msg>Failed during Photos pairwise matching.</Msg>
    			</TaskMsg>
    			<Percent>12</Percent>
    			<Status>307</Status>
    			<RetVal>280248320</RetVal>
    			<DefinitionJson>false</DefinitionJson>
    		</Task>
    	</TaskGraph>
    </Job>
    

  • Here is for the other error message of "Failed during 3D mesh texturing"

    A3DF
    �`p���m�	�QR̵���A�Z�=�[��q7�4Z�,�&���Y��	o�g�莭>�P}eȥ|���+W-�녃�@mS���T..��X�zW�s׼��(�az�]	��������U�al���	mIe1��s��6>��P��F�i%=��yI[��z�U��5~d���r��[T���v�1�a�� �сE3�Fx�*W�\ �E��D|���$G�B�!���[��ZG!�<�q
    ��B��)B'�:����q���;�;F^2m���d׵P��"2��p�l��xkv�,�2�ɵ�{�Uf�6�bX,��EX��W2e��᜻��O�q���H�[��S���f�����WXl�4�e<� K����KL�ɐ嚀W���6~�v�"O#�;�[�q�t��U�YY��HqB����Zxng�=UN#��r]���v@�H�U.[]��o?e���ˋ/�hg{J���ʬe�QKL���p詗�;�ӹD��W��J�V62l���g"�ڝ�]�#�I�„��� �Xw��I���=,<�<�R�����G��IC�՟�>�+}6��LX�f,�Z�X�uq���,M'8�=��I��'�)�(G��K�+�F����9w|�d���Ѭ��uh.�D��/z��>X5&b�m��Ew�Ԛ��W���'t~��t
    տWM���V	r�q�x!�R�1"V���0�I������ZRP�"�v���"���[z�7�'pW4!������.����mN,ǪZu_�O
    �b߮(f\�X+#(���V�BY�ҝo�ZQ�	���C9��j�غ�c����:��
    �!�ۑb-Q �.<����}᪘�!|l@�l�BCZ�n!��a�,vMѬ_a5~��㛵���5�%��l|Ta|����>|f7=�Q"�k��pN� $�[a�Z��^��}��NH�1�p&,:���
    F-J���	�)��2u�m⫼p#b��|E�xĥ�d
    f���kA�xj1J��g���U��ܙ�-�M]�8��gPDg�x��q�ُޔ��8഍;L��ܗSX=�w1C*�,�MOE�?p��g����\U��lz6#u�_�m���i�������#`r�KFR+7~������P��
    �1h(7�1�Rw��j(N����7��]L���'F����WN? ?��dW-Km�a������8QJ��K���o���.ܔ��D������!i�^�!Mdn�ih�L!rt)�)����\�d�h��w}�9��jt/� p|��;�-20w�R��#ї
    �]�fe̍�Q��"�x��a.�Ĩ~�0���+�[LU�amH�ղ@��Ӡ�#�F+oT�]�$�K6�v����ݿ#�F^MU����D?�d0�Ҙ)�2r��(���^$F�h� ة����Ba'�u�����&�
    �x �1��pB!��Cy�8��8�
    V��M(��q�#�^U�=q�S�F6�����H�x��,-=E^۠ 1 ���Eq�"�Vnz%��F���8��ȗ�6�n�]�ѤK�<�YcT�5Z�� �B.�_�3Ǭ�����_pC~�n���яꝂ���u��w3�2ݢ���Ú�x��8C8%�i�'��0r��w���E-+��e���kSO9Z��lTE��h�	�������%eB��c���q�ݙ#��K�9j���
    2�����(a%po���'���a]M3W���_퇆�Ƚ|FJ�~��w�ږ��[-��Y*qk����|.Q5��x��.���Ud��D'ol<�p�q*�Q������M�E�~$�N>�7O�.C�R��o�}�4׬�E�S�*	o/#i�R���w��;uy�w~z7�����%)�&eݭ��WW.<e�#D����1	��r��0��
    �C��)�l!�2���<��b�dK��b	���}�s��V��KǨ�"������uF�}4��%%��&y��}�Xc��Ĺ;�5UV�8--_^K�S��ӞR�m�	��7�&F�zn-h�i�é�l��d��TY��V�.�Sn ������W����h����#�������8�Z@�#��]����p�V;�E"������Ꮽ�t�K�H��/V�G�A�(�9A�Eޕ���{>l�Ub�fƆ���
    �((��Q��/6<8�s� �X�8i󀳌�W�u����~c���j�K�v_���"�p������0�m�����{o������ʝuo.E��X�p��~F�0�.��m�ñ��-G��o��Y�e������3���&�8!�f>)j�۠AD��?�v���yTF�`�<Dd���D�
    ���on�Cd~Z�#	� ���e�)"�d��;ϸ:`���'�yC�\�I�fH���$�y�J�i�%CYo��k�A����W��u�83����]ȍb�S��^8��U�X�V|}k�e�{e�e��C����d�
    !�)�a��6:i%�FĨdv�)\d�۱PT��{�6󋄮-�XAQ<1���6��s?W��'G��n<��*�OH������Ske��4p�Ҥ�t�MO�g����22d���a�/��c�[�Mc�'S�څ�׷0B���&P�`���4��H��٧�ߒk��}\��n*��F���0cɃ`�j��b����ڲ����\��9J���a�]��҄|A�P�V:��6s��USe���4�c�Ѻ,���*>
    ����&�5�t�+��yw������w��#���߀F>�%���U��#բ|���E:�I����!�+��0��l(��wU'q��.ԩJV='�{Do�U��f�YBb�WF�[7��B,�<Tz��3]<w��� .2ʲ�&��(���#�@&`j�o��RE㺏U2H�sw�����U]#+��.�F̻�(�m{�t�4(����3�ޕta�zG���2a_{��&�#�I�E����t��9��J�/ڡ�E^��n1;��j����(�-�^�8�
    ��x��R���Zc�,ÿ�q*�^�F-���vYP(7brN�P�
    ?���N�iԉiU��?�C�u�O�P�Sq�3�:�%��7q-*y{��3��}Ղ�o��t<O�x�x�p�S���Z���r�Y�Xp/.�}k]j�f����B�+��%�z�s�U���'�-�c�HV�_��K=O=��qJ����ȍ�ų�]!Y�<�oƯ
    ��}�3:m���B�Eʺ
    �l�w(1)�0,�����H���o&���p�o7������F�˅�nY�Y��;?�g9��w�xѺ:���f����'���f�Ĭr�%~��+(��x�3Ʒ�����X #��p�j�+=�t�f.�J7�y���c��_�#`S��f|�K�de��)wM13�eK̘���Z6NMAY>ɗт=+�1���L�І~�����4'����������ޫ�4 WbQ@5�^���I
    �I��J�&h�`�IvӮ����F�����׿��Z���;��-<m�)3�焏��֎�K�Gg}-�і�s����d�3����ʩ���26�w1~�I�R㮃����%V��	��G��Eg���l
    ħ��[����ښ����d��*�:���&/Wt���ݳY��Խ�Z6{��t���7��}���@�`Wy`{�v��E�Q=��BP���:��X�T�w���{��o�/��4�X�~]���e�����(�17��+}�*�ׂ���``נ���`#��P*��=[�Vj��t�|��J�5��j�:0�F�x@)�T(s9L�X�՛��"�Q�&���*S����g1a�Tl�y@��G�9��DD	
    �I
    �5[!��[_�)yP����j��U���[h��b�W]��N��@iG��Z}B���x�a�J�2#�u�c��ج�_>Р�^�9r�Q�I@%�N������(�U���D!��/����
    ���.m�?4S��*�Q�b����~yr���5������@��V��_@e�((�%�������(C��Y���6Km���(�c��u���h�2	�1Q4�����Rv�ñ%c�����U{�`��V��jKW�lw�����1O,`���M��K�Ùљ�˹?�ZQt�]^ZSc��l�@�l
    ��1�<5�Ё`����>�8C|�ڋKeH�g�H���6�e��Q+D�\Q�]C(x�p��zy/��$��19�v�o�	���ʞ�����W�-CD����U�gP�wG��jW�s����yo�~2
    �������n{ɞ��� �Z��Z�y��3��'�ǜ��Ӊы\�
    �֜�m�ӭ�m������Y�pZg*��;㮱Zc� �y7B�a��B�T�|P� dkb��@��'ap��dfr�oD�f�O{<S[��S4р)Eް�mnK
    �B8P�C6��`�.�/ʤ�+���t�wF�2����(®g��H��b
    F��5/��?�"@>��y�\�n�<C�M��15-¹p��L-d����<ۊ���j@�����v���|����x�C�,-DŽK�ǔ�]�0��cp�̣���Vk���a��N���3�FR���v�����%�z�f�~�	��"�ن%Y �۹��L��bv��M
    (*����U���y�vUB����}��̚C�7�iz��S~����,��8�
    �er�{��{F���	�(��	3s�5)�p�O%�><MY�۳��a��^�"<�e%6\�=�o���0��oّ��z`�>����z��? ���/�.
    
    ��u쌛&w�[,�HTH��������n��%��hvr�c����\l�e	���%S��v���w��C�:�C�g���5�DO�w�7)%:!Ѳ;�.�g^uZ!��Ɓ� :,��3{1���D���WH�/���6��ϔ�Ӽп�k��E�u�D��y��y���5�#��2��;���M��l�d�o�ڭx�{�6�n�E�X�I���G�q��h`�J�=U�����o���K�ԑC�}p��Mh7!��,�����(X�J;�S�PٔE�<��X?j�-!A.T^������a�lU��k�����M}�h�j���`�̭�>�7En˺�&�U�B[t��t���>(���u����$۔�oX�^�"��F��G�q!v�����d7P�V�@;�I�����<"̉����W���{�{8r��H�J01u}��尥e�I����=(���j�@d�⭑p�9���K�l�nm�����k)��$
    )�(�P1����9�ӱy�R���.="->.���/��_?
    ����Lʬ9�lV��<�7F,��sR(ڃ���vN�A��m��V��v�LO�����g����U��F�'�6��5���Rh[Ǿ^D�7����Q��0��X��Y��M-�b�~����B�u/$7#H����X�Gw�]��f��J�ɷu�_��&��V��Eu��[����`��b���
    $Dg6�j�Ĕ�F�;*UH|�E�Q�����h�1,�H��I�����ٷ�*�Gj��qD�h�lh��!R�A��?����RuȄ���C�����jZR=x�n�E߽%hD���E�Pu����`�m�q/�+�n�%��3�9�o؎�H�,(usQ��UV�;[ǯ��ͣ$�s��Ve1 f�p6
    3�c8�KW�[suu��F�Gl\Z�Ǖ�����y�Ϗ�،�z�+>	FY<0��y	*u����&SI���Ѳ��kʾ=07�8��kd�k�鹫�eչ#�M��-h���SD���W|�o�n���4
    ��9�lH��*tۗ1�lF�x�$�ϥ_ǘTז)��
    6t�3$Z��:u|,�1�����>��s$	vYje��&	b�~����0�#�>
    ����.�Eanζ���WO��s�`�&���o���\�������u͍���:x����eЗ"Y�$W��tXq��"g'�2I�w���0��)�R�bK_�[����&��~�{7�(u�"G���I��f�Q�N�(�}).Ʈ֔
    ��:�1�
    V��\7}S�$������pr���=t݁�?���jq���q���y\	�_G�Z�(@�8���0��d��<��~���o�P ��~`��X�����\��l��c��4�gv1�����/�	"hu���r=�Sה�����]L%(�I��`K�dWΗ>����A�g4/�^����18/��3|>�C.�^�7�d���j+��_�&�\����w��#
    $w�>I�;�񘐀>.�y�5A��ǣ��H�d7���ݢ&�8�5�9EA�{��e@�T���
    ��>D��g��T��^�zG�Z~������&	�b�dب�3����dRU���:�yޒ�"n,���Ϫ06�$��.�C�����›�����o*����-�>��c[�y9!����;<�{\Đq���mג���\��ZnݱP(���ͫ�)ת�z�K��(1�{ǜى3��M_r��t����9�b��B�;WDž9�״�MM��\�[1��W��{`*�P�N��Nq̦�K��iV4d
    �A�=�/��K�5�H]�����C���6�]Y7#����PAvq;B�%i�?��VoEu��I�����f@_�&˕�|j�͕�x�c6���2	*�1M�q�t}H�f�\�`��ও�z�#�:�Y0E���g[���.⺵�b�
    j6^q�O��˔���XC�ٹ�� BEtp�SF@Og��~ �uX.h�Ĩ�CC�
    ��c�<���$ǒ<�AIӡ��E��:��g�z}�JW����l����7��[��2�$N�Na|n���Y$��5��HleF�oJ�@|C.����dhP��}�(f%U�Lm}�ux��y�7�z١-�5���mg�g�"�de�]~i����[.�z�H��8IL�T�������0�042t�:�ɉ&A�
    ޘ����?T��{�tePm
    ������1�79��z$`�c�>��$K��N>�@����^�RP�?�����A}ĹS?uJuWz�8��tЋc?��#������de��XT�ֻO8
    ���5�G������?B���D
    �-����#F@��x��쭆*�s�U��H������; C���r캩2FưMJ�(��R�F� _$�G��#%�6��:���c�a����'�Q*
    �b��an}�2s䠅��$�3�̭�G���\Ƒ����ݢ��0��:���ú�_HdO��F;��<o���H���,t?h�Ai��fg*{�xi�W���}�d������O�%ޛ�Y��#��E��t����/Ct�Y?�*���G�s�F�Y�՘�������!2Y@;pK9)�����[߃x`O����Z�����H�����4i��~�S��Z��c]�������K]Қ����W�[y�X��ly��,N���f]\�
    ��7ajD�X�D�������f��n�%�V��OJ1�w�3��%��6}�`V�sv�Y�v:c��(g	��ˎΨ��U#�?���^��S�7n͡�9�Z��[���`�{2�91Հ����NH�#����K[�{H�,�0e�wT"�8��0��ɮz��;�j����YxQ��"=��Q�;O����oN6���J��܈De����V:���}�=AMh�Zc,9i�A�@64L�r��m$խ�z�%��K�0�ԥ�	B���e�Qq�?&CmxƎ���ij��>\��F�lr�)@�">jG�=VT�2�V�ۊejc4UmIW:� �����u�Qt����'a��a���;��,�o�J����ۘHU�xJ�W�ݥ( $}(�%��a��ڗ�R���n�\�5�*��A�@Q:��ȟ���<���?[+��X�#���"�[<К�����qv�
    ed��J���@k��`KY�+�CF��<=˜���&l�oR1���Iy}p�@^�~"hظ�k�,�<��I���k��|�p:z�g��U��Dz(�>FL+��H�9Z��I�����i�Y(�uuomc�:�w`�q��s�c�S�F��R���2:{ǹ�-��.,���7�9��� �_u3iG��A2�B_���������}��i�ƶ��u�8G��d}��܇A�%�	���-lR�����^�Ar
    ��;�#cK�(�ZĥO�&�����Cv���	�74�s���Uש��0��,��fy^B�v'���v��ڿ*���%� e�U���_a�EK6ԻN��o�~G�%�OE�@���)'Iz��\$�-����5��?����
    �j�#7��[����c&o�`�����z[�0g���-�,%��T��/{�+�����	����I��/����r��F���8�|aa���yQ»Xq��g5�3��#�!�Ť����q�Yv6�5����9�F���+��5�x�;)N���q#q��Y�[%}L簠A���J� V{j�����,�c��*�z�
    +�l�	�yXٓL�e;�\<~�WΦ�`Apw�LNxq|l@9�h�����XԔz���i��`��O�|���#�P���#�u���0{[~�c����F��¿\��*y�o0��8��@�tz���{c���;��$+m��o�����̫9'�
    u	��Dž�v-=�����ѯ�ղ�ֈ�a,�t��e��:5?��Z&(�ք��k@�{Y�k���DF56��
    z��	s�q� ��FÍ�w'����$�DmE<���J(��Tq[Q-��[[a?�߀#zt����o�R��Mz�5��M�%���;�u����.V�F5҂n�k Vѧ����/٤����
    ��U�ޗ���2������0.&јHIS�;�nV�xb�^z5M�K>׺ލ��^P�HtQ�F��q�<���ӻW?c��Ş]�s{q!�V����*k2M�vM-	���$�8.Fj�oN2~kQ�pT^ub��u��i�Q@�^���Z�8%{�����"������:`�F��8!�l��i#/�H����05"��2���Dw�1L���sM�%9��(ˠBĩ�NIh�v��NE0��a�n�ĉ��-N�H��\��e	•���k�dV"5;U$B�)Op:٣��c�ҍnM��ۜ�y[��������4�N h����(��3�x��;���YEvEಃ��2�5O�~�;n	�䜼L�[[�e¸����Ŵ��o�7���^��g�� �"a���U0��~l��
    L�!ht��qE�pS�O��i���?�!��9X�Ŵ�`$����(��~�R��R�o�m�5��6h�D��?�R}���Xĕ8���l�`|)5Ӷt��
    ]�prZ��
    �j���GT�Y�F�o֦}D��(f�1�356��[�$t~�}Eoi��%������l�yj=��"��@�A
    z2-H��	+��r���9��+YJ3�^V�aN�3,���t�	��{�!	����U�H����%���nl"���y|�����o�ov��ahMʣ_K��Ɏ�z�=M����f�HgQd^�f�u��;�zYY�r!w`���b�ܫeE�_�#'/՞4�:"���Ϝ�Ь1D�J�rE�o�����@$Z���F�!�I���h]�
    ��-�_�x��X6i,t��l��/a��)~p�םW��zYs�3���Q�'�f5����M��#�3F�)O�G�#	\VG'�H׍P&25 k��*�V�r�0V#�+=~Tܧ���;�D��5T�6��=Zt-&��4~S̸`~��������d30V\q4G����00w�[m���w94��G㽁����yN��6�_�\����e��.`��3��������pW�8�
    �!�.e�ք�.r`W)!JA��Z���!��R���`��<@����?޶���@�k���]�6!��\m�-=B�B��d���>⥋�i����VId-O�E$�J���5(B�
    Юw��}��!v�|r�Q�;�W"�/�'x��h2�-���r�����(��>�q����M���`.�e
     ȩ�v}"!��sXMo�.�(�}��jj��ɡ��o������x��	;k�m�p��ة��{���5��Ø?��A����@�Tw�I�ɟ�0z$a�A�"'-����eK�������d�S�:�`=|q���v����̥����6�w~�g��bP����L(�8���b,t��ÆM!H�|�[�e����={Jq
    �^��8�]?&_�%?���r�{��a��<�>
    ‹�@�Dd��go��:<��q)�l3M�t�y{��F�I�O
    j��.���r���z�[M���3�jmЉ�|�
    <?xml version="1.0" encoding="utf-8"?>
    <Job>
    	<Version>1.3</Version>
    	<JobName>job_20190115T175127_06_Santa_Monica_LADWP_B293R1P1_Tile_10</JobName>
    	<TaskGraph>
    		<JobInfo>
    			<Descr/>
    			<ProjectPath>//storage-a/main-storage-1/CC_Projects/Santa_Monica_LADWP\Santa_Monica_LADWP.ccm</ProjectPath>
    			<ItemPath>Block_293\Reconstruction_1\Production_1\Tile_10</ItemPath>
    		</JobInfo>
    		<RunInfo>
    			<SubmitHostName>AlchemyWorkstation1</SubmitHostName>
    			<SubmitUser>workstation2</SubmitUser>
    			<SubmitTime>20190115T175127</SubmitTime>
    			<Run>
    				<RunHostName>Node1</RunHostName>
    				<RunUserName>Node1</RunUserName>
    				<StartTime>20190115T175128</StartTime>
    				<EndTime>20190115T190946</EndTime>
    			</Run>
    		</RunInfo>
    		<JobFeedback>
    			<Version>1.3</Version>
    			<Status>32</Status>
    			<Percent>66</Percent>
    			<Msg>Failed during 3D mesh texturing.</Msg>
    			<PT/>
    			<Tasks/>
    		</JobFeedback>
    		<Task>
    			<Type>4</Type>
    			<Workload>100</Workload>
    			<ProjectPath>//storage-a/main-storage-1/CC_Projects/Santa_Monica_LADWP\Santa_Monica_LADWP.ccm</ProjectPath>
    			<ItemPath>Block_293\Reconstruction_1\Production_1\Tile_10</ItemPath>
    			<Run>
    				<RunHostName>Node1</RunHostName>
    				<RunUserName>Node1</RunUserName>
    				<StartTime>20190115T175128</StartTime>
    				<EndTime>20190115T190950</EndTime>
    			</Run>
    			<Id>0</Id>
    			<TaskMsg>
    				<Time>20190115T190946</Time>
    				<Code>280256512</Code>
    				<Msg>Failed during 3D mesh texturing.</Msg>
    			</TaskMsg>
    			<Percent>66</Percent>
    			<Status>307</Status>
    			<RetVal>280256512</RetVal>
    			<DefinitionJson>false</DefinitionJson>
    		</Task>
    	</TaskGraph>
    </Job>
    

  • Hi Julianna,

    Log file shows that there is not enough memory left! You may have to reduce the tile size and then resubmit. Also make sure that you have enough disk space.

    Also, keep in mind that when you set the tile size in the Spatial Framework, do not set it to close to the RAM usage. Otherwise it could happened that one tile contains more details which need more RAM to calculate it. What was your settings in 'Spatial Framework'?

    Regards,

    Bruneau  

    Answer Verified By: Julianna DaSilva 

  • Hi Bruneau,

    That does make sense! I had my target RAM usage to 27 gb with a total of 32 gb of RAM in that particular node.  I have a threadripper with 64 gb, but that just runs an AT engine, but the other 3 nodes are only 32 gb.  All other tiles rendered out fine except for that one.  I will lower the RAM usage and see if that works or could I just set that particular tile to render out on the threadripper which has 64 gb of RAM with a regular engine.

  • Hi Julianna,

    Let me know then if this solve your issue.

    Regards,

    Bruneau