An unexpected piece of cleverness

Sketching out a piece of python, I realised that I could replace some of the calls to subprocess.Popen() with a wrapper object I wrote a couple of weeks ago, so I took the object, dropped it into one of my existing library modules and tried it out. It all worked as expected so I pulled in an option parser derived from argpase.ArgumentParser and got on with the rest of the code.

With the program complete, I added something something to vary the level of the logging library according to the command line switches and got on with testing. At which point I noticed something unexpected: I didn't get any log output from my process object until I increased the verbose level to 3, even though I'd set the the log level to debug at verbose = 2, and I didn't start getting debug information from my object until I went up to 4. Surprised, I went back and started taking the modules to pieces until I eventually realised that I'd fallen victim to my own cleverness.

Examining the modules, I realised that my custom ArgumentParser included a custom __init__() method that added some extra standard options to each of my programs, including a verbose flag and noticed that it also overrode parse_args() to increase the log level via logging.basicConfig(). The override also contained a bit of code that, either dubiously or cleverly, called logging.getLogger() using the name of the library containing the command object — I'd named my logging hierarchy after the package — and set it to logging.ERROR until the verbose level was at 3 or 4, dropping it to logging.INFO or logging.DEBUG.

I'm not entirely sure whether this is entirely sensible, but I think it is acceptable in this context. Both the argument parser and the command object are part of the same package hierarchy and are intended to play closely together. By putting them together, it makes it trivially easy to separate out the debug stream from the core script from the debug output from the commands; something that is especially important when troubleshooting scripts that wrap around a child command, where, as a last resource, it is often useful to able to be able see the output from the underlying commands but which, if provided routinely, tend to swamp useful output from the surrounding script.

NWP and computing power

Via insideHPC, a Washington Post piece on the US National Weather Service's plans to increase their supercomputing capacity. But what I really found interesting was a link to a 2012 blog posting by Cliff Mass criticising the performance of NWS' model when comparied with ECMWF and the Met Office.

Mass' comments remind me of a discussion I had with someone from another met service — not the NWS &mdahs; who was talking earnestly & excitedly about their forthcoming procurement and what a huge difference it was going to make to their supercomputing capacity. The scientist I was with breezily dismissed this, saying "You don't need more computer capacity. It's not going to help because all your models are at least 20 years out of date..." It may have been true, but it left the other completely crushed.