SDK: AAHOOK_EXEC_MENU_COMMAND throws blank dialog

I'm having an issue with a Pre hook I've written for AAHOOK_EXEC_MENU_COMMAND. Specifically, I'm hooking on a native PW command (command id of IDMD_Redline). The hook works, but has the undesired effect of throwing a blank information dialog when the hook returns AAHOOK_ERROR (you hear the windows error chime, the information button shows in PWE bottom right, and clicking it shows a completely blank dialog titled "Information").

I built a clean version of the DLL with no functions other than the hook, thinking maybe another custom module was to blame, but the problem still persists. I've checked the Last error, and it is always 0. I've tried removing last errors, and still same problem. I have also tried using an Action hook and get the same result. Setting *pResult = TRUE didn't help either. At this point, I'm stumped.

Below is a simplified version built for testing that should demonstrate the problem. We're on PW V8i.

LONG AAAPIHOOK HookMenuCommand_Pre
(
LONG hookId, // i Hook Identifier
LONG hookType, // i Hook Function Type
AAPARAM aParam1, // i Parameter for Hook Function
AAPARAM aParam2, // i Parameter for Hook Function
AARESULT* pResult // o Return code of the hook function
)
{
AFX_MANAGE_STATE(AfxGetStaticModuleState());
TRACE (L"\nHookMenuCommand_Pre called...");
switch(aParam2)
{
case AAOPER_EXEC_MENUCMD:
TRACE (L"AAOPER_EXEC_MENUCMD...\n");
break;
default:
return AAHOOK_SUCCESS;
}
HDOCCMDPARAM hDocParam;
LPAADOC_ITEM lpDocItem;
#pragma warning( push )
#pragma warning( disable : 4312 )
LPAACMDPARAM pParam = (LPAACMDPARAM)aParam1;
#pragma warning( pop )
if (aParam2==AAOPER_EXEC_MENUCMD)
{
if (pParam->lCommandId == IDMD_REDLINE)
{
hDocParam = *((HDOCCMDPARAM*)pParam->lpParam);
lpDocItem = (LPAADOC_ITEM)aaApi_GetDocCmdParamElements(AADOCCMDPT_DOCUMENT1, hDocParam);
if (lpDocItem)
{
if(AfxMessageBox(L"Redline command intercepted.\n\nContinue?", MB_OKCANCEL)!=IDOK)
{
return AAHOOK_ERROR;
}
else return AAHOOK_SUCCESS;
}
}
}
TRACE (L"HookMenuCommand_Pre() finished...\n");
return AAHOOK_SUCCESS;
}
  • Action hook was the trick. Although I had tried that, I was still returning AAHOOK_ERROR instead of AAHOOK_SUCCESS to kill it.

    I knew that one, I really did...  *heading back to the dunce's corner*

    Please note that I post here on a voluntary basis and am not a Bentley employee. 

  • It appears that for a prehook for AAHOOK_EXEC_MENU_COMMAND, you need to set the value of pResult as well as setting the last error if you return AAHOOK_ERROR.  Something like this:

    *pResult = AAERR_USERERROR_FIRST+3;

    aaApi_SetLastError (*pResult, L"Your error message.", L"Details about the error.");

    return AAHOOK_ERROR;

    This doesn't appear to be true for Prehooks for most AAHOOK* types.  In fact, at least from the ones I've tried, setting the value of pResult and setting the last error is just ignored, but the action is prevented.  So, it appears that sometimes setting pResult is required (along with the last error), and sometimes it isn't.

    I'll see if I can find out more about the details as the SDK help isn't clear on this point.

    In the mean time, I think the solution offered and the one used - i.e. use an action hook instead, appears to be a good solution.