jhbuild on windows

Last update: 4th January 2012

It's now possible to use jhbuild to manage application dependencies on MS Windows using mingw and MSYS. I'm maintaining a fork of jhbuild at: http://www.gitorious.org/jhbuild which provides windows-devel and windows-stable modulesets. One builds from git, the other prefers binary packages where available and otherwise builds from tarballs.

Feel free to add more useful things to the modulesets or start tidying up the many, many hacks. Patches, comments and hate mail all welcome to Sam Thursfield <ssssam at gmail.com>.

Note: I am still interested in this, but it seems about time to work a bit more inside the Gnome infrastructure. See: https://live.gnome.org/Windows/MingwNativeBuild for further progress.

Pre-requisites:

(the paths are just what I suggest if you're not sure what you're doing; the versions are what I know to work. MSYS is a lot more stable than it used to be, so feel free to use the latest versions of things.)

mingw (automated installer) [2011-05-30, latest]. Install to: c:\tools\mingw. Make sure you include at least the C++ and MSYS components.
Start the MSYS shell and install the following extra packages using mingw-get:
- mingw32-autotools
- msys-cvs (seriously, autopoint still requires CVS)
- msys-patch
- msys-vim
- msys-wget
Python [2.7.2]. Install to: c:\tools\python
MSYS git [1.7.6-preview20110708]. Install to: c:\tools\git. When asked, choose the 'Use Git Bash only' option (avoid adding anything to the PATH). In the line ending conversions choice, pick "Checkout as-is, commit Unix-style line endings" (core.autocrlf = false).
Bazaar [2.3.3-3]. Install to: c:\tools\bazaar (keep checked the option to add bzr to the PATH variable)
XML::Simple [2.18]. Install this Perl module manually into the / prefix (unless you really want to try getting CPAN to work under MSYS. Hint: it doesn't.). Note that Perl is installed as part of MSYS.

A note on what follows. For extra confusion, MSYS-git ships with its own MSYS package, which is inside the c:/tools/git/ tree. When I say an MSYS shell, this should be your main installation of MSYS, which can be started by running c:\tools\mingw\msys\1.0\msys.bat or from Start Menu > Mingw > MinGW Shell. When I say a git shell, this is c:/tools/git/bin/sh.exe, which is probably in the start menu as Git > Git BASH. Now we will set everything up!

MSYS:

Start up a normal MSYS shell, and edit your .profile (type: vim ~/.profile, or outside of MSYS you can find the file at c:\tools\mingw\msys\1.0\home\[your username]\.profile). Paste in the following contents:
```
export PATH="/c/build/bin:/c/tools/bazaar:/c/tools/python:$PATH"
```
You can use Notepad to edit the file if you like, but beware that it might add a .txt extension that you need to remove by hand. c:/build is the directory where we will put all of the code jhbuild compiles. Restart the MSYS shell so it loads the new .profile.

Git:

Create a file called /bin/git.bat (c:\tools\mingw\bin\git.bat outside of MSYS) which should contain exactly this:
```
@c:\\tools\\git\\bin\\sh.exe -c "/c/tools/git/bin/git %*"
```
jhbuild runs this file when it needs to execute git.

jhbuild:

Now run a git shell, and create a directory for the build environment (I use c:/build), and a place to download source code to (c:/build/src). Don't use a path with any spaces in, especially if you suffer from high blood pressure. In that directory, run:
```
git clone git://gitorious.org/jhbuild/jhbuild.git
```

Now, inside a normal MSYS shell:

cd c:/build/src/jhbuild
PREFIX=c:/build make -f Makefile.windows install

Next, we can create a simple .jhbuildrc. Copy this to ~/.jhbuildrc (c:\tools\mingw\home\[your username]\.jhbuildrc outside the MSYS shell).

moduleset = 'windows-devel'
modules = ['gtk+-3']

# available modulesets:
#   windows-stable - build from mainly binary packages and some tarballs
#   windows-devel  - build everything from git

# prefix: target path to install the compiled binaries
prefix = 'c:/build/'

# checkoutroot: where to place downloaded/checked out source code
checkoutroot = 'c:/build/src'

# You will need to comment this out if you edit one of the deps and want a
# recompile ... but then your tarball builds will take a long long time as they
# will make install each time.
build_policy = 'updated'

# Our modulesets aren't on git.gnome.org
use_local_modulesets = True

# Maintainer mode (automatically rerunning autoconf, automake etc. when the
# build scripts are changed) is broken on Windows
autogenargs = '--disable-maintainer-mode'

# -fno-unit-at-a-time to work around apparent bug in latest mingw gcc.
os.environ['CFLAGS'] = "-O3 -fno-unit-at-a-time -mms-bitfields"

# This is very important! Some versions of gcc4 require this flag or will
# produce an error saying "The application could not be started (0xc00000005)"
os.environ['LDFLAGS'] = "-Wl,--enable-auto-import"

module_autogenargs['zlib'] = ""
module_autogenargs['gdk-pixbuf'] = "--without-libtiff --without-libjpeg " + autogenargs

Now:
```
jhbuild bootstrap
```
This installs a few build tools that are not distributed as part of MSYS. They are installed into the build prefix (c:/build).
That's it! Type in jhbuild build gtk+ and go to the pub.

Did you really think it was that easy

Of course it's not that easy. Here are some issues you will probably come up against.

libtool 2.4 generates some bad code, for at least gdk-pixbuf-query-loaders and gtk-query-immodules. This causes a build failure which you will need to fix manually for now: in the libtool wrapper script (for example, gdk-pixbuf/.libs/gdk-pixbuf-query-loaders_ltshwrapper) search for the PATH entries and change all of the c:/build/foo format paths to /c/build/foo. The problem here is that on Unix, PATH entries a separated with a ':' so we need to use a special MSYS-specific format for the Windows paths which do not contain a colon so they don't confuse the shell.
libxml2 bug #590302: patch here (they merged a fix which doesn't seem to have actually fixed anything, another fix which I know works is to simply remove the defined(IN_LIBXML) that you find in the area the patch touches).
Line ending problems: Pixman, glib, libpng, gdk-pixbuf, atk, gtk+ and libpng have some files with CRLF line endings in their git repos. For you, this means 'git pull --rebase' will fail, complaining that you have unstaged changes (because git has implicitly converted these files to LF line endings). When you get this error, the solution is to do (in a git shell):
```
git commit -a -m "Fix line endings"
git pull --rebase
git rebase --skip
```
Then tell jhbuild to ignore the error and continue to configure. Or if you're not bothered about the changes, it's fine to just ignore the error and continue. A more long-term solution is to get these packages to put a .gitattributes file in their repo, which has the -crlf attribute for each file with a CRLF ending (which is kind of weird, and in fact there is a better way to do it by setting eol=crlf, but this doesn't yet work with MSYSgit so is kind of futile right now)
jhbuild isn't really written to be transactional. This means if a download or checkout interrupts halfway through, you will probably get some weird error. You should delete the file or directory in question and run jhbuild again .. and if it keeps not working, try downloading or checking out manually. (jhbuild will be able to handle that).

Available modulesets

My current approach towards the modulesets is to have two; windows-stable and windows-devel. The former downloads and installs from binary packages wherever possible and is good for when you want to compile a Windows build of an application for distribution. windows-devel builds everything from git, so you can get to work fixing all of the recent regressions in win32 Gtk+.

Errors you can ignore

I: some bootstrap modules have been updated, perhaps you should update them: autoconf, m4, libtool. I'm not sure why this happens but it's not important.

Future plans:

Fix all of the above issues.
Do away with -windows modulesets, and make all of the stock jhbuild modulesets work on Windows.
Speed up the build process, and stop wasting so much effort ..

It would be great if Git wasn't so awkward, and didn't take up > 100MB with its own MSYS install. They do this because it's so sensitive to the underlying tools, but with good versioned dependency-tracking package manager, it should one day be possible to use the system's MSYS.
Fit the shared GTK+ runtime into this somehow. This needs some serious thinking and discussion.
Merge as much as possible into mainline jhbuild.
Fix either rxvt or jhbuild to work together. Or port a better terminal to windows.
Module state tracking - at the moment, if a tarball extract or download is interrupted then the build is left broken. jhbuild on the next run will see the partially-extracted file and think it has been fully downloaded. It should store in the packagedb once a download or untar completes successfully, and only assume that the state finished successfully if it is logged. Essentially, the operations needs to become atomic.
Add unit tests for all the new features.
Update GNOME live on this subject.
Close #502970
Do away with jhbuild and mingw-get; just use Nix

Relevant open bugs

jhbuild: git.bat support.
libxml2: exports don't work with mingw gcc 4.4.0. (fixed but untested by me so far).
pkg-config: smiley face in output ; this was actually a bash issue and the workaround now causes pkg-config 0.25 to be broken (the irony). Using 0.23 is the current workaround.

Thanks

People who have helped in this endeavour: John Stowers (who started the whole jhbuild on windows thing), Tor Lillqvist (started the whole gtk+ on windows thing) and Frédéric Péters (who has merged a number of my poor-quality patches :) .. and thanks to everyone who has fixed my bugs or done work porting things to Windows that were never meant to run there.