Codementor Events

Making/compiling C functions for use in Julia on Windows

Published Dec 30, 2017Last updated Jun 27, 2018
Making/compiling C functions for use in Julia on Windows

I often find it harder to do development work on Windows as it doesn't come with essential developer tools as out-of-box standard (an example is R's excellent Rtools bundle which helps alleviate the problem in the R world). In fact, I struggled with getting Julia's ccall to work on Windows for a while, so I have decided to write the steps that I followed to make it work.

This post is meant to be a guide for complete beginners like I was at the beginning of this process.

Calling C is supposed to be easy

Julia was designed to be able to call C function easily. Indeed the official manual states that

To allow easy use of this existing code, Julia makes it simple and efficient to call C and Fortran functions. Julia has a "no boilerplate" philosophy: functions can be called directly from Julia without any "glue" code, code generation, or compilation – even from the interactive prompt. This is accomplished just by making an appropriate call with ccall syntax, which looks like an ordinary function call.
I found that this is true: but only after jumping through a few undocumented hoops in Windows. To be fair, this is not something that Julia should document, as it's probably "common knowledge" to Windows software developers and is not specific to Julia.

Choice of compiler

Julia's official manual mentions gcc as the C compiler to use and that's what I have set up. The compiler gcc is an open source C compiler (it can compile other languages as well). The Microsoft VC compiler is also available. However, I am not familiar with any of these tools, so I will detail how I made it work for me, and it's by no means the only way.

What you need to do to compile C functions on Windows

  1. Set up environment variable LD_LIBRARY_PATH (google: how to set up environment variable) and set it to a path that you have access to, e.g. d:/c_lib.
    • This is where Julia will look for compiled C libraries (clibname).
    • As a number of Julians have pointed out, the LD_LIBRARY_PATH may contain multiple paths, so joinpath(env_path, clibname) may not work; and you may have to manually extract the appropriate path you need, e.g. use regex.
    • The other way is to have the .dll file in the same folder as julia.exe or provide the full path to the variable in the form of a const string in place of clibname in the ccall call
  2. Install MinGW e.g from this link
    • I have tried some other MinGW installations and some of them didn't work, the above link contains a MinGW distribution that I have tested to work on Windows 10 Pro 64 bit machine
  3. In the folder of where MinGW is installed you will find a bin folder. Add the path to the bin folder to your PATH environment variable

Now you are ready to write some C code and compile it in Julia, see example below. The trick is something I learned from Julia Discourse member @Liso

# write the C code inside a raw string
C_code = raw"""
#include <stdlib.h> //size_t is defined in stdlib.h now

int add2(size_t a, size_t b){
    return a+b;
}
"""

# obtain the environment variable
env_path = ENV["LD_LIBRARY_PATH"]
# your compiled C code must have a library name
# it MUST be declared as a `const` for `ccall` to work
const clibname = "addc"
# the path to store the C library file
const Clib = joinpath(env_path,clibname) 

# compile the C code into a shared library
open(`gcc -fPIC -O3 -msse3 -xc -shared -o $(Clib * "." * Libdl.dlext) -`, "w") do f
  print(f, C_code) 
end

# define a Julia function to call the C library 
add2jl(x, y) = ccall(
  (:add2, clibname), # ("function_name", library)
    Int, # return type from the C library
    (Cint, Cint), # input parameter type to the C function
    x, y # actual values to pass to C function
    )

# call C function via Julia
add2jl(UInt(80), UInt(8))

add2jl(80,8)

Things to watch out for

  • The Clib must be declared as a constant or it won't work, see this SO post
  • Once you have used the function add2jl once, the addc.dll file that contains the compiled C library is now "locked" so you can't recompile or delete it. The only way I found so far to "free" the file is to close Julia.

Side note: why do you need C? Doesn't Julia solve the two language problem?

Sometimes you want to benchmark a C implementation vs. a pure Julia one.

For my use case, I found the C implementation of string sort to be more performant and so I can call a quick C function instead as it has efficient sorting algorithms already coded up. In this case, Julia didn't solve the two-language problem as I wanted more performance than what I can get out of Julia at the moment, and there is an efficient C implementation already. I guess that my use-case is one of the reasons why the Julia devs made calling C functions is so painless in Julia.

Discover and read more posts from ZJ
get started
post commentsBe the first to share your opinion
Tamas Gal (Tom)
6 years ago

Be aware that the LD_LIBRARY_PATH is almost always a list of directories (on Linux), separated by :.

ZJ
6 years ago

Really hoping that those that has multiple paths in LD_LIBRARY_PATH don’t need to read this. It’s more for people where it is empty, like it was for me.

Chris Rackauckas
6 years ago

I found the C implementation of string sort to be more performant and so I can call a quick C function instead as it has efficient sorting algorithms already coded up. In this case, Julia didn’t solve the two-language problem as I wanted more performance than what I can get out of Julia at the moment, and there is an efficient C implementation already.

I thought from the thread (https://discourse.julialang.org/t/how-to-make-a-c-function-compiled-by-myself-available-to-ccall/7972/39) your Julia radixsort! was eventually the most efficient?

ZJ
6 years ago

only for strings of fixed size 8 or less. i will post my findings soon.

Chris Rackauckas
6 years ago

I see. Have you done a conversion of the code to Julia yet? Or is it just a complicated function?

ZJ
6 years ago

“simple” function. just a quicksort actually. i think Julia has slowish access to underlying bytes and slow string reaasignments. Radixsort based method in Julia is already faster than data.table for integers. String-sort is the hard case.

Chris Rackauckas
6 years ago

I see. That’s one of the things Scott has been asking about with the memory buffer implementation. I know nothing about strings so I’ll just say I hope you work it out.

Show more replies