On Matlab’s loadlibrary, proto file and pcwin64 thunk

Today we’ll try to shed some light on dark undocumented corners of Matlab’s external interfaces.

Matlab provides several ways to call into external native code – if you have just the binaries for this code, the way is loadlibrary.  To help parse the external dll contents you’d need to initially provide loadlibrary a C/C++ header file, but you can then tell loadlibrary to transform this header info into a Matlab-native representation typically named a ***proto.m file.  When the proper loadlibrary call is made, a proto.m file is generated along with something named ***_pcwin64_thunk.dll (well, on x64 pc’s, obviously).  The documentation says nearly nothing about either proto files or thunk files:

A prototype file is a file of MATLAB commands which you can modify and use in place of a header file. …

A thunk file is a compatibility layer to a 64-bit library generated by MATLAB.

One could do with this terse phrasing until something goes wrong – as it inevitably does.  Googling shows only that this seems to be an open question online as well. Time to peek inside.

Peeking inside

Take a toy C++ dll:


// ToyDLL.h
#ifdef TOYDLL_EXPORT
#define TOYDLL_API __declspec(dllexport)
#else
#define TOYDLL_API __declspec(dllimport)
#endif

extern "C" { TOYDLL_API bool ToyFunc(int a, int b, double c); }

// ToyDLL.cpp: build with /D TOYDLL_EXPORT

#include "ToyDLL.h"
#include <stdio.h>
#include <tchar.h>

extern "C" {
	TOYDLL_API bool ToyFunc(int a, int b, double c)
	{
		_tprintf(_T("%d, %d, %f"), a, b, c);
		return true;
	}
}

Build it, try to loadlibrary it in Matlab, and get:

Error using loadlibrary
Call to Perl failed.  Possible error processing header file.
Output of Perl command:
Working string is 'extern " C " {  bool ToyFunc ( int a , int b , double c ); }'.
at C:\Program Files\MATLAB\R2016a\toolbox\matlab\general\private\prototypes.pl line 1099
main::DumpError('extern "C" { found in file. C++ files are not supported.  Use...') called at C:\Program Files\MATLAB\R2016a\toolbox\matlab\general\private\prototypes.pl line 312
ERROR: extern "C" { found in file. C++ files are not supported.  Use #ifdef __cplusplus to protect.

Found on line 13 of input from line 12 of file ToyDLL.h

Hmmmm. The perl script prototypes.pl is shipped with Matlab (the error message above gives its full path), and the first few of its 1000 lines are:

# Parse a C/C++ header file and build up three data structures: the first
# is a list of the prototypes defined in the header file; the second is
# a list of the structures used in those prototypes.  The third is a list of the
# typedef statements that are defined in the file

It should be noted already that the C – C++ boundary is extremely fuzzy as far as Matlab is concerned. Not only is this script declared to ‘Parse C/C++ headers’ only to complain later that ‘C++ is not supported’, loadlibrary itself is advertised to ‘Load C/C++ shared library into MATLAB’ only to disclaim elsewhere that ‘The MATLAB® shared library interface supports C library routines only’ and offer various workarounds for C++.  More details later, but for now let’s humor the grumpy perl script and modify the header (it never touches the cpp) into:

// ToyDLL.h
#ifdef TOYDLL_EXPORT
#define TOYDLL_API __declspec(dllexport)
#else
#define TOYDLL_API __declspec(dllimport)
#endif

#ifdef __cplusplus
extern "C" {
#endif

TOYDLL_API bool ToyFunc(int a, int b, double c);

#ifdef __cplusplus
}
#endif

And now loadlibrary quietly succeeds.

Peeking deeper inside

As the perl source is available you could in principle study it to understand what it does – but I certainly couldn’t, in principle or not (it’s horrible even as far as perl scripts go). With some semi-hacking, we can just inspect its output. First, fire up Process Monitor and filter to process ‘perl.exe’ to observe the exact files it receives and generates:

ToyDLL

Observe that the perl script operates on the VC-preprocessed file ToyDLL.i   This could also be seen by examining earlier portions of the ProcMon trace or by noting that prototypes.pl itself declares internally its usage as –

# prototypes [options] [-outfile=name] input.i  [optional headers to find prototypes in]

Next, observe that it outputs the source file ToyDLL_thunk_pcwin64.c.  Alas, trying to open it teaches that it is very short lived.  The final hack is to copy this temp file somewhere upon its creation. I considered coding such a tool (shouldn’t be too much trouble) but thought I’d google for one first and luckily did come across the free edition of Limagito. After some tweaking (*Edit: more details below) I got a nice persistent copy of the thunk.c source, which is essentially:

…
#include <tmwtypes.h>

/* use BUILDING_THUNKFILE to protect parts of your header if needed when building the thunkfile */

#define BUILDING_THUNKFILE

#include "ToyDLL.h"

/*  bool ToyFunc ( int a , int b , double c ); */
EXPORT_EXTERN_C bool boolint32int32doubleThunk(void fcn(),const char *callstack,int stacksize)
{
int32_T p0;
int32_T p1;
double p2;
p0=*(int32_T const *)callstack;
callstack+=sizeof(p0) % sizeof(<em>size_t</em>) ? ((sizeof(p0) / sizeof(<em>size_t</em>)) + 1) * sizeof(<em>size_t</em>):sizeof(p0);
p1=*(int32_T const *)callstack;
callstack+=sizeof(p1) % sizeof(<em>size_t</em>) ? ((sizeof(p1) / sizeof(<em>size_t</em>)) + 1) * sizeof(<em>size_t</em>):sizeof(p1);
p2=*(double const *)callstack;
callstack+=sizeof(p2) % sizeof(<em>size_t</em>) ? ((sizeof(p2) / sizeof(<em>size_t</em>)) + 1) * sizeof(<em>size_t</em>):sizeof(p2);
return ((bool (*)(int32_T , int32_T , double ))fcn)(p0 , p1 , p2);
}

For completion, here is the generated ToyDLL_proto.m file:

function [methodinfo,structs,enuminfo,ThunkLibName]=ToyDLL_proto
%TOYDLL_PROTO Create structures to define interfaces found in 'ToyDLL'.
%This function was generated by loadlibrary.m parser version  on Fri Jul 15 17:50:20 2016
%perl options:'ToyDLL.i -outfile=ToyDLL_proto.m -thunkfile=ToyDLL_thunk_pcwin64.c -header=ToyDLL.h'
ival={cell(1,0)}; % change 0 to the actual number of functions to preallocate the data.
structs=[];enuminfo=[];fcnNum=1;
fcns=struct('name',ival,'calltype',ival,'LHS',ival,'RHS',ival,'alias',ival,'thunkname', ival);
MfilePath=fileparts(mfilename('fullpath'));
ThunkLibName=fullfile(MfilePath,'ToyDLL_thunk_pcwin64');
%  bool ToyFunc ( int a , int b , double c );
fcns.thunkname{fcnNum}='boolint32int32doubleThunk';fcns.name{fcnNum}='ToyFunc'; fcns.calltype{fcnNum}='Thunk'; fcns.LHS{fcnNum}='bool'; fcns.RHS{fcnNum}={'int32', 'int32', 'double'};fcnNum=fcnNum+1;
methodinfo=fcns;

Putting it all together

Now that all the raw material is at hand, we can gain some insight on what these components do and how.

The proto file is about calling the thunk

It contains the path to the thunk dll, and a ‘dictionary’ that tells Matlab what thunk function needs to be called en route to each dll-exported function. For our toy case, to get to ToyFunc Matlab calls into ‘boolint32int32doubleThunk’ – a name encoding the output and all input types, but in principle every other name could be used.

The thunk DLL is about adjusting calling conventions

The thunk function boolint32int32doubleThunk receives its arguments in the Matlab calling convention: all arguments are passed consecutively on the stack, untyped and aligned on sizeof(size_t) (64 bytes in x64) boundaries.  It also receives a function pointer to the actual DLL export, and after copying the arguments to local typed variables – calls this function with its native calling convention.  It never uses the ‘stacksize’ argument.

In real life cases the headers and generated thunks can get considerably more complicated – one notable omission so far is how structs and other compound types are handled (as far as I can tell this is the sole reason for inclusion of the library header in the thunk.c). The same technique laid out above can be used to investigate these cases, but we can already put all this newfound knowledge to good use.

Troubleshooting

Take our ToyDLL.h above and add to it the seemingly benign lines:

class ToyClass
{
ToyClass() {};
~ToyClass() {};
};

Build and try to load the resulting DLL into Matlab, only to get:

Error using loadlibrary
Building ToyDLL_thunk_pcwin64 failed.  Compiler output is:
cl -I"C:\Program Files\MATLAB\R2016a\extern\include" /Zp8  /W3  /nologo
-I"[…]ToyDLL" "ToyDLL_thunk_pcwin64.c" -LD -Fe"ToyDLL_thunk_pcwin64.dll"
ToyDLL_thunk_pcwin64.c
[…]\ToyDLL\ToyDLL.h(10): error C2061: syntax error: identifier 'ToyClass'
[…]ToyDLL\ToyDLL.h(10): error C2059: syntax error: ';'
[…]ToyDLL\ToyDLL.h (11): error C2449: found '{' at file scope (missing function header?)
[…]ToyDLL\ToyDLL.h (14): error C2059: syntax error: '}'

We didn’t violate any of loadlibrary’s documented limitations but we already have enough visibility into the process to understand what’s going on.  The root issue is that for whatever reason the perl script generates a c file, not a cpp one.

The workaround I used, both in this toy scenario and in the real life DLLs I wanted to load into Matlab, was – grab the perl-generated sources, rename them to cpp and build your own thunk.dll .

In the immortal words of Todd Howard, It just works.   Solves plenty of other C/C++ idiosyncrasies too.

Now I have exactly zero insight into Mathworks considerations and decisions, but I have this vague suspicion the only reason for these half-hearted ‘C-only’ limitations is that they’re stuck with this essentially black-box perl script that parses headers.  Most of the time it succeeds (on C++ headers too), sometimes it doesn’t. When it doesn’t, they sometimes document the root cause as unsupported.

The question remains, who in their right mind would try to parse a header with a perl script. Based on my own experience, I can suggest sort of an answer.

Bonus

This discussion can lead to all sorts of other workarounds and solutions. Here’s, briefly, another one that was useful to us.

When the Matlab component consuming the native DLL is deployed, the loadlibrary call is made from wherever the CTF archive was extracted to – and it occasionally fails to find the thunk dll. Our solution was to intervene in the proto.m file, and have it take the thunk path from the registry.

Perhaps more on CTF archives one day.


Edit June 2017: Per @AlishaMenon’s request, here are some more details about the Limagito usage.

First, under Scan Setup/Change Notify/WIN source:

limagito1

Second, set the directory where compilation takes place (where the temp files are generated) under Source/WIN:

limagito2

Finally set the destination folder where you want the copy:

And you should be good to go.

Advertisements
This entry was posted in Matlab, VC++. Bookmark the permalink.

12 Responses to On Matlab’s loadlibrary, proto file and pcwin64 thunk

  1. Aviv Hurvitz says:

    Thanks for sharing that.

  2. Alisha Menon says:

    Would it be possible for you to elaborate on the following workaround you used: “grab the perl-generated sources, rename them to cpp and build your own thunk.dll”?
    I am currently running into this issue while trying to upload a commercialized dll into MATLAB using loadlibrary and through googling have discovered that the solution is to either:
    1. Try again in 32-bit MATLAB which I haven’t yet tried but will if this doesn’t work out
    2. Follow the workaround you suggested above.
    However I am not quite sure how to go about doing what you have stated. I hope this isn’t too much trouble but would it be possible for your to provide a step-by-step process? I am not too experienced with this sort of code so it would be super helpful if you would describe exactly what you did.
    Thank you so much for this post and for going through all the trouble to share what you have already found!

    • Ofek Shilon says:

      @Alisha thanks. This was quite a while ago but i can try and add details. But first please explain the exact scenario – is the dll 32 bit or 64? What is the bitness of your matlab? Does the header that accompanies the dll indicates C or C++?

      • Alisha Menon says:

        I am currently actually have both versions of the dll, 32 bit and 64 bit, and neither of them are working. I am using 64-bit MATLAB and the header that accompanies the dll is I think C++ because it has an extern C over the majority of the code, however the top part of the header file isn’t in that loop and that is where errors are popping up. Here is the error I am receiving:

        >> loadlibrary(‘SLAB_USB_SPI’,’SLAB_USB_SPI.h’)
        Warning: Warnings messages were produced while parsing. Check the functions you intend to use for correctness. Warning text can
        be viewed using:
        [notfound,warnings]=loadlibrary(…)
        > In loadlibrary
        Error loading library intermediate output follows.
        The actual error is at the end of this output.
        *********

        Type ‘char_Null_terminated_Ptr’ was not found. Defaulting to type voidPtr.

        Found on line 53 of input from line 52 of file C:\\SiliconLabs\\MCU\\CP2130_SDK\\Software\\Library\\SLAB_USB_SPI.h

        Type ‘char_Null_terminated_Ptr’ was not found. Defaulting to type voidPtr.

        Found on line 54 of input from line 53 of file C:\\SiliconLabs\\MCU\\CP2130_SDK\\Software\\Library\\SLAB_USB_SPI.h

        Type ‘int_Out_LPGUID’ was not found. Defaulting to type error.

        Found on line 418 of input from line 417 of file C:\\SiliconLabs\\MCU\\CP2130_SDK\\Software\\Library\\SLAB_USB_SPI.h

        Type ‘LPGUID’ was not found. Defaulting to type error.

        Found on line 421 of input from line 420 of file C:\\SiliconLabs\\MCU\\CP2130_SDK\\Software\\Library\\SLAB_USB_SPI.h
        *********
        Error using loadlibrary
        Building SLAB_USB_SPI_thunk_pcwin64 failed. Compiler output is:
        cl -I”C:\Program Files\MATLAB\R2017a\extern\include” /W3 /nologo -I”C:\Program Files\MATLAB\R2017a\extern\include”
        -I”C:\Program Files\MATLAB\R2017a\simulink\include” /DTARGET_API_VERSION=700 /D_CRT_SECURE_NO_DEPRECATE
        /D_SCL_SECURE_NO_DEPRECATE /D_SECURE_SCL=0 -I”C:\\” -I”C:\SiliconLabs\MCU\CP2130_SDK\Software\Library”
        “SLAB_USB_SPI_thunk_pcwin64.c” -LD -Fe”SLAB_USB_SPI_thunk_pcwin64.dll”
        SLAB_USB_SPI_thunk_pcwin64.c
        C:\SiliconLabs\MCU\CP2130_SDK\Software\Library\SLAB_USB_SPI.h(52) : error C2054: expected ‘(‘ to follow ‘_Null_terminated_’
        C:\SiliconLabs\MCU\CP2130_SDK\Software\Library\SLAB_USB_SPI.h(52) : error C2085: ‘LPSTR’ : not in formal parameter list
        C:\SiliconLabs\MCU\CP2130_SDK\Software\Library\SLAB_USB_SPI.h(53) : error C2628: ‘_Null_terminated_’ followed by ‘char’ is illegal
        (did you forget a ‘;’?)
        C:\SiliconLabs\MCU\CP2130_SDK\Software\Library\SLAB_USB_SPI.h(53) : error C2085: ‘LPCSTR’ : not in formal parameter list
        C:\SiliconLabs\MCU\CP2130_SDK\Software\Library\SLAB_USB_SPI.h(73) : error C2085: ‘CP213x_DEVICE’ : not in formal parameter list
        C:\SiliconLabs\MCU\CP2130_SDK\Software\Library\SLAB_USB_SPI.h(74) : error C2085: ‘USB_SPI_STATUS’ : not in formal parameter list
        C:\SiliconLabs\MCU\CP2130_SDK\Software\Library\SLAB_USB_SPI.h(146) : error C2085: ‘inline’ : not in formal parameter list

    • Ofek Shilon says:

      I added some details about Limagito. HTH

      • Alisha Menon says:

        Thank you so much for your help! Again sorry for the many posts but I am trying the Limagito method and had a few more questions:
        Firstly, how do I find where the “compilation takes place (where the temp files are generated)”? I am having some difficulty locating that directory.
        Secondly, you mentioned that after you grab the perl generated source files which, to my understanding are the temp files that are being relocated through limagito, the next step is to rename them to cpp and build your own thunk.dll. If it’s not too much trouble would it be possible for you to elaborate on those steps as well?
        Thank you so much for all of your help, the information you posted has been incredibly helpful!

      • Alisha Menon says:

        Just wanted to let anyone else that runs into this problem know that I managed to get the dll uploaded into MATLAB but I had to use 32-bit version of MATLAB (the last version of MATLAB for which the 32-bit version was also released was 2015b). It seems to be running ok so far and all the functions are in MATLAB. I don’t think the 32-bit version generates a thunk.c file so there is no issue with the C++ header file. Thank you for your help Ofek and hopefully my post helps anyone else who runs into a similar issue!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s