Spyder v2.1 is available

After a year of hard work on stability, performance and new features, Spyder v2.1 is now available for Windows XP/Vista/7, GNU/Linux and MacOS X.

This release introduces major enhancements and new features:

• Large performance and stability improvements
• PySide support (PyQt is no longer exclusively required)
• PyQt API v1 and v2 support
• New profiler plugin (thanks to Santiago Jaramillo, a new contributor)
• Experimental support for IPython v0.11+
• And many other changes

We [the Spyder development team] are all very excited by this new release: Spyder is now becoming the [GUI-based] scientific development environment. Its future seems very promising. First there is the PySide support and the PyQt API v1/v2 support which were strategically essential (this will also help us supporting Python 3 in time). Second there is a cleaner and optimized code: this was critical for Spyder's behavior to be the closest possible to a standard Python interpreter. Third there is a very stable application: a lot of crashes and segmentation faults were fixed during the last year. Last, but not least, the future looks bright thanks to the new project members: we are now 5 committers and 6 contributors.

Stand-alone executable on Windows platforms
Note that, on Windows platforms, Spyder is also available as a stand-alone executable (don't forget to disable UAC on Vista/7). This all-in-one portable version is still experimental (for example, it does not embed sphinx -- meaning no rich text mode for the object inspector) but it should provide a working version of Spyder for Windows platforms without having to install anything else (except Python 2.x itself, of course).

Help needed
More than ever, we welcome any contribution that helps making Spyder an efficient scientific development/computing environment. Join us to help creating your favourite environment!

New enhanced scientific Python interpreter

As I mentionned in my previous post, the standard Python interpreter provided by Spyder's console has been considerably enhanced. Moreover, despite these interesting enhancements, this is based on a real Python interpreter, not an emulation.

The scientific startup script mentionned in my previous post was integrated in Spyder's development version (under the name scientific_startup.py).

The goal of all these recent changes was to provide a MATLAB-like experience with a minimum of requirements: NumPy, SciPy, Matplotlib and Spyder, that's all.

And here are two screenshots of the result (Spyder was executed in `light` mode: `spyder --light`):

No IPython v0.11 support in Spyder's console, but standard Python interpreter enhancements

It's been a month since IPython v0.11 public release and I've been trying to solve issues related to IPython support in Spyder's console, all in vain. At first, I thought that it was only a problem on Windows platforms, but apparently it is not. The more I think about it (and the more I spend time on it), the more I doubt that there will ever be a decent support of IPython >=v0.11 in Spyder's console. In the meantime, I have fixed a couple of bugs related to the new IPython plugin: this new plugin will probably be the only way to run IPython >=v0.11 within Spyder.

So, to compensate this lack of IPython support in Spyder's console, I have implemented the following enhancements for the standard Python interpreter (this is not much, but it provides all the IPython features I am using personnally):

1. I have fixed the PyQt input hook issue on Windows platforms. Before this revision, the console was crashing (non-responsive actually) on Windows platforms after trying to manipulate Qt objects interactively, like interactive plotting with Matplotlib for example. Now, when enabling the "Replace PyQt input hook by Spyder's" option (which is enabled by default on Windows platforms), the standard Python interpreter may be used to do interactive plotting, even on Windows.
2. Plus, I have implemented some basic special commands in the standard Python interpreter (see here more details), adding support for %pwd, %ls, %clear or !dir (or !ls on UNIX platforms), etc. (I haven't implemented the %edit, %run and other similar commands which have really no use within Spyder).
3. And I have fixed a bug with PYTHONSTARTUP substitution option (the packages imported in the startup script were not available in the Python interpreter).

These three changesets were intended to facilitate the use of standard Python interpreter as an interactive computing shell. I've also created two PYTHONSTARTUP scripts (on my machine), one to support Matplotlib's pylab interface and the other to support guiqwt's interactive plotting mode:

• Matplotlib's startup script:

print "Running pylab startup script...",

# Import modules following official guidelines:

import numpy as np

import scipy as sp

import matplotlib as mpl

import matplotlib.pyplot as plt

# Pollute the namespace but also provide MATLAB-like experience:

from pylab import *

# Enable Matplotlib's pylab mode:

ion()

print "done."

• guiqwt startup script:

print "Running guiqwt startup script...",

# Import modules following official guidelines:

import numpy as np

import scipy as sp

import guiqwt.pyplot as plt

import guiqwt.io as io

# Pollute the namespace but also provide MATLAB-like experience:

from numpy import *

from guiqwt.pyplot import *

# Enable guiqwt's interactive mode:

ion()

print "done."

With these startup scripts, I have my own IPython shell (not as powerful as IPython, of course) but it suits my needs... why not yours?

To select you own startup script, go to Tools > Preferences > Console > Advanced settings:

I think I'll add these startup scripts to Spyder v2.1 but I don't know how exactly. Maybe the simplest way would be to change the settings GUI above and add a third choice: "Build-in startup script: " with a combo box allowing to choose a script between the two scripts above (and others).

Preview of the new IPython plugin for Spyder

Here is a first preview of the new IPython plugin which will be available in Spyder v2.1 (to be released in September 2011, but the first beta will hopefully be available in a few days):

Important notes :

• This feature is still experimental and will remain so until Spyder v2.2 (scheduled for December 2011).
• It requires IPython v0.12dev (i.e. the current IPython development version, available through the github repository).

The idea is to start a new IPython kernel within the "Console" plugin (bottom left of the screenshot above), then a new IPython frontend dockwidget is created automatically (top left of the screenshot above). This first implementation already supports the Variable Explorer (top right of the screenshot above) through the IPython kernel associated to the frontend but nothing else is supported (link with the Editor, the Object Inspector, and so on).

In the "Console" plugin preference page, the user may change the IPython kernel command line options (defaults are: "python --pylab=inline").

IPython has taken a huge step forward!

Two years of efforts were necessary for the IPython development team to achieve their goal and take a huge step forward with IPython v0.11, providing a powerful interactive shell with outstanding new features, a cleaner source code... and a brand new great-looking website! (at last... the old one was really minimalist)

To my opinion, the most impressing new feature is the standalone Qt console (see the screenshot below): combined with the brand new "inline pylab mode", it allows plotting figures directly in the console itself, between lines of code (like in a notebook).

But there are other interesting new features such as high-level interactive parallel computing features, an improved vim integration, an integration in Python Tools for Visual Studio and support for Python 3 (the last major scientific library which does not support Python 3 is now matplotlib... hurry up guys!).

New IPython v0.11 breaks Spyder's IPython support
Unfortunately for Spyder users, this new version will break the IPython support within the "Console" plugin. We were not expecting this release until a few weeks from now, so we were not able to prepare Spyder for this.
However, some preliminary changes were made to both v2.0 and v2.1 branches to make them compatible with IPython v0.11:

• For v2.0.12 users, simply copying and replacing the original files by the following should be sufficient to add support for IPython v0.11 (at least on Linux and MacOS X):
• monitor.py
• startup.py
• If you are using the development version (v2.1.0), you may simply update your local repository.

Support for IPython v0.11 in Spyder's Console
Anyway, the most disappointing thing about IPython v0.11 for Spyder users is that it's now using Python's PyOS_InputHook mechanism to support GUI event loops. This is, from a general point of view, a good thing because this is much cleaner than the old thread-based implementation. But, on Windows platforms, PyQt's input hook is implemented in such a way that, when enabled, it's blocking the standard output redirection pipe with the remote process (I don't know why), thus blocking the whole shell (that's why an option to disable the PyQt's input hook was added a long time ago to Spyder's console preferences and this option is enable by default on Windows platforms). In other words, when running IPython in pylab mode (i.e. with matplotlib's interactive mode turned on), matplotlib's figures will freeze. That's why there is a warning message in Spyder v2.1.0 when it's running with IPython v0.11 on Windows: "Spyder does not support GUI interactions with IPython >=v0.11 on Windows platforms (only IPython v0.10 is fully supported).".

The future of IPython support in Spyder
Currently, in Spyder v2.0.12 and v2.1.0, IPython is supported simply by running an IPython shell in Spyder's console: a Python script running an IPython shell is executed in a separate process (QProcess). This was a very efficient way to support IPython, as there was a lot of code refactoring between the pure Python interpreter and the IPython interpreter (code completion, variable explorer, ...).
Now that IPython v0.11 is using a two-process model like Spyder, they were able to implement a GUI-based console (the "standalone Qt console" mentioned above) which could be integrated directly into Spyder.
What needs to be done:

1. The first step to make it possible is to add support for PyQt's API #2 in Spyder: this is now done (see previous post)
2. Implement a specific "IPython plugin" that will encapsulate the IPython Qt console widget
3. Add options support to customize IPython
4. Add links with the Editor (run scripts, run selection, debug, breakpoints support, ...) and the Object inspector
5. Add support for the Variable explorer (after some tests, I think this is a hard one and will probably require a lot of interactions with the IPython development team)

This is a lot of work but it's worth the effort.

PyQt API #1, PyQt API #2 and PySide compatibility

One year after its creation, the new Python library for Qt named PySide has reached version 1.0 for a while now. Even if Python developers are still hesitating to use intensively this new library despite its permissive license (LGPL), the fact is that PySide has brought a constructive competition in a field which was exclusively dominated by PyQt, the historical Python-Qt library which is still distributed under non-permissive licensing terms (GPL or commercial license). So, a lot of projects have already chosen to make the necessary changes to be able to migrate from PyQt to PySide when it's time, i.e. when PySide is mature enough (version 1.0 is still moving fast: very nasty bugs have been fixed between v1.0.0 and the latest v1.0.3 release). For example, in scientific Python ecosystem, IPython (with its new Qt frontend which will be integrated in a future Spyder version), Matplotlib and Enthought Tool Suite have made the choice to move forward.

PySide is intended to be compatible with PyQt's API #2 which was introduced with PyQt v4.6 (this is the default API for Python 3). So the first step to migrate from PyQt to PySide is to switch from API #1 to API #2, which means no more QString, QStringList or QVariant objects (replaced by Python objects):

http://developer.qt.nokia.com/wiki/Differences_Between_PySide_and_PyQt

For Spyder, I chose to deal with this transition period with the following solution. The idea is to be able to test alternatively (and without changing too much code) the three cases (PyQt API #1, PyQt API #2 and PySide) by implementing a transitional package that your code will import instead of importing directly PyQt or PySide. See for example the Spyder's Qt transitional package :

http://code.google.com/p/spyderlib/source/browse/#hg%2Fspyderlib%2Fqt

(in the rest of the code, all import PyQt4.QtGui statements are for example replaced by import spyderlib.qt.QtGui)

At first, I thought that the best solution was to migrate directly to PyQt API #2, hence breaking the compatibility with PyQt API #1 (it's hard to resist to the temptation of getting rid of QString and QVariant objects!). But then I realized that other projects using the Spyder's widgets (source code editor, array editor or dictionnary editor widgets) like guidata would be forced to perform this migration at the same time: this was not acceptable and would lead to maintain two different versions of Spyder, one compatible with API #1 (Spyder v2.1) and the other with API #2 (Spyder v2.2). So I'm now convinced that the best solution is to change the code to make it compatible with both APIs and this is quite easy to do (it's now done in v2.1): QString objects can be used implicitely and QVariant objects and methods may be replaced by simple conversion functions (see 'from_qvariant' and 'to_qvariant' in spyderlib's code):

http://code.google.com/p/spyderlib/source/browse/spyderlib/qt/compat.py

Here is the example of the NumPy arrays editor widget which is compatible with PyQt API #1, API #2 and PySide:

http://code.google.com/p/spyderlib/source/browse/spyderlib/widgets/arrayeditor.py

Another difference between PyQt and PySide is the way the QFileDialog static methods are wrapped. In PyQt, getOpenFileName returns only the filename. With PySide, the same function returns a tuple containing the filename and the selected filter. So we have to deal with it and the best way is to write wrappers that ensure full compatibility with PyQt (API #1 and #2) and PySide. This is done here:

http://code.google.com/p/spyderlib/source/browse/spyderlib/qt/compat.py#77

There are still a lot of bug to fix for Spyder to be fully compatible with PySide (only Spyder's light mode is running with PySide, with very minor bugs) but that's a great start!

Welcome to Spyder's blog

This is the Spyder's official blog.

Of course this is the right place to keep you informed about the new features, the philosophy and the future directions of the project. But Spyder is more than a development environment: it's a realistic and very efficient replacement for MATLAB or IDL. In other words, Spyder is intimately related to the scientific Python ecosystem (the Python language and all the Python packages available out there): this is a global solution for your scientific development projects as well as for your data analysis/visualization work, the perfect companion for all your numerical computing needs. So from this perspective, this is also the right place for presenting our subjective thoughts on this scientific Python world.

Note that there are a lot of interesting online resources on scientific Python but the best starting point would probably be the SciPy website: home page of the SciPy conference and the SciPy Python library (see the mailing lists, the blogs, the cookbook and the brand new "Ask SciPy" Q&A forum).