My Azure job manager

I've spent the last week or so investigatingĀ  exactly what I can do with my Microsoft Azure grant. As I listed in my last blog post, I want to be able to split my job up into a bunch of small chunks, which I can then distribute out to some humble linux virtual machines, and then collect the results back again.

I started by investigating Azure-Batch, which is Azure's built in way of mimicking a gigantic cluster. However, after spending a couple of days wrestling with their broken tutorial, I discovered that they don't support the service for their simplest VMs, which means I would have to use a more expensive one, which would effectively half my CPU hours.

I then spent some time trying to work out how to deploy a virtual machine usig their python API. Again, the tutorials are largely broken or incomprehensible, and in the end I ended up coming across the Azure-CLI (command line interface). This was a lot more intuitive (or at least, one of their CLIs is, they appear to have two distinct ones), so I set about writing some code which will use the CLI to launch virtual machines.

The code took a couple of days to write and debug, but it's in a more or less complete state. You can find it here.

Alas, it turns out I now rail against limits imposed by Microsoft, stopping me from launching more than 20 VMs concurrently. The website explaining the limits is pretty difficult to follow, so I've raised a service desk ticket to ask them to help me out.

In other news, we locked in the first "stable" version of our population synthesis code - COMPAS, which is the version we used to write the first COMPAS paper (watch this space).

Leave a Reply

Your email address will not be published. Required fields are marked *