Featured

Start of the GSoC 2016 Project

This is the post excerpt.

This is my blog detailing the progress on the GSoC project – ‘Octave Symbolic Package‘… The results were out on 23rd April 00:30 a.m. (IST – GMT+5.30)

The official octsympy project can be found on github:
https://github.com/cbm755/octsympy
My fork of the project can be found on my profile:
https://github.com/genuinelucifer/octsympy

After the commencement of GSoC 2016, the following PR has been merged already:
https://github.com/cbm755/octsympy/pull/432
which deals with some tests failing on windows due to new scripts not being loaded into octave cache.

We are currently working on deciding the upcoming goals. The main goal would be to get PyTave working with octsympy and replace the existing approach of using popen2 to communicate with python.

This project has a long way to go before completing its goals…

I look forward to the exciting journey ahead🙂

Summary of work done during GSoC

During Google Summer of Code 2016, I worked on the Symbolic Project under the organisation Octave. The project was directed towards adding a new IPC mechanism into symbolic – Pytave. During the course of GSoC work was done in 2 repositories (octsympy on github & pytave on bitbucket).

Code Changes:

All of the changes done by me can be viewed at the following links:

Summary of changes:

As we were to use pytave in Symbolic we worked on it to make it more mature. Add more functionalities to provide a better interface for Symbolic to deal with. Also many new features for conversion to and from python objects were introduced. The most important addition would be @pyobject and its functionalities.
On symbolic side, we used the features of pytave as much to our benefits as possible. We currently only have extra conversion for @sym objects which are specific to Symbolic. For rest of the octave types we use pytave‘s internal conversion.

Detailed PR wise changes:

To know about changes done before August, please read my other posts on this blog in the sequence: post-1, post-2, post-3 and post-4. Here I will summarise PR wise work done in August only.

In pytave, the PRs #33, #31 and #32 were merged which were under review in july. The PR 33 did some cleanup and added tests. PR 31 adapted existing tests since after inclusion of @pyobject we were now converting  python – lists, dicts and tuples; to @pyobject in octave. PR 32 fixed some doctests which were either not working in Python 3 or gave errors due to the latest @pyobject changes.
PR #24 added a new functionality to specify a namespace to be passed in py* functions to ensure that any changes in symbols/objects happens only in that namespace. This required some refactoring which was done in the same PR in a separate commit.
Then PR #40 added a method to allow comparison of 2 pyobjects. This would be useful in writing tests that need a desired output.

On the Symbolic side, PR #512 is under review which takes care of updating out existing methods to use the new pytave features. We have had a good amount of changes in pytave which now allow us to do most of the things without the need to execute and python code on our own (using pyexec). We now do it using py.* calls.

Goals not met:

Almost all the goals were met during the project. Only 1 major goal was not met which was building pytave on windows. Efforts were made towards that but later on it was more fruitfull to add new functionalities that would make the projects more mature. I would definetly try to work on this afterwards and hopefully in near future we will have fully working pytave on windows too.

 

Project Report – 1st August, 2016

Work in the past month has been distributed over symbolic and pytave side.

The work that was merged to symbolic in this time was mostly to convert objects from python to octave types. PR #474 and #491 worked in this regard.  PR #490 helped to make the tests compatible with Pytave IPC. Then we also added Pytave IPC to the build system (travis) with the PR #489.
Also, we worked to improve the passing of variables to python by using pytave. PR #465 has mostly achieved the goal. But it is failing on travis. The cause was finally detected in PR #502 to be the passing of struct to pytave. I couldn’t get much time to work on this as I was working on pytave. And hopefully when pytave matures then this error will be resolved in pytave itself.

In pytave, there were some bug fixes which caused octave to segfault. In the PR #12, the issue was addressed when an octave value could not be converted to python which threw an exception and crashed octave. In the PR #15, the issue where the exception text from Python could not be properly extracted was handled. This was later improved with the PR #30 for the cases of crash with Python 3. In PR #29, the issue was handled when a large number (integer) was being converted from python to octave then boost threw positive_overflow exception.

Also, there were some feature improvements too. PR #13, added the feature that checks for a function called via pycall into main namespace and builtins namespace too for existence. After pyobject was merged into pytave, most of the work was related to pyobject. PR #17 completed the functionality of returning pyobject from py* functions by adding the same mechanism to pycall which was implemented in pyeval. PR #16 added the feature to check length of a pyobject. PR #25 dropped the existing conversion mechanism which converted python list, tuples and dicts to octave matrices and structs. Now they return a pyobject instead.

There are a few PRs currently under review. PR #33 removes some of the unused code from pycall and adds a few BIST tests which showcase changes due to PR #25. PR #32 fixes some doctests which are also affected by the changes in the same PR. PR #31 aims at adapting all the methods of pyobject to the changes which dropped the conversion of lists, dicts and tuples. In PR #24, we are trying to add a new feature to allow users to specify the namespace in which pyexec/pyeval would run their code. Hence allowing isolation of different codes to have the same names/functions by running in different namespaces. Currently it is buggy and doesn’t actually work as expected. The symbols in a local namespace are also accessible from the global namespace. Moreover any 2 local namespace are sharing the same symbols. Currently it just works in the case where 1 global and 1 local namespaces are required. In such case, they are having separate symbol tables (dictionary actually as it is python).

During this time we had one setback. We tried a lot to build pytave on windows but failed. It has been stopped currently but we will resume it after GSoC ends. Tatsuro helped a lot in this. Thanks to him we have gotten really close to successfully building pytave on windows. The complete details about what was tried and which errors occurred can be found on my wiki page where I described Building Pytave on windows.

The project has been going smoothly because of a great participation from the mentors and octave community. Hopefully we will be able to have a working release of symbolic with pytave IPC (atleast for linux) before the end of GSoC.

Summary of work done till 17th June – GSoC 2016

The mid term evaluations of GSoC 2016 are close. This blog post is to summarize the complete work done under the ‘Symbolic Project’ under the organization ‘Octave’.

I will mention each of the goals that were set for the mid term and then give an overview about what was done for that:

1a) Octave, Symbolic, and PyTave dev versions installed.
I built Octave from source successfully on ubuntu 16.04 then used symbolic with it. Also built pytave from source using the dev version of Octave.
My Symbolic fork on github can be found at:
https://github.com/genuinelucifer/octsympy
My main Pytave fork can be found at:
https://bitbucket.org/genuinelucifer/pytave_main
I also forked pytave from Colin’s fork (to work on experimental features) which can be found at:
https://bitbucket.org/genuinelucifer/pytave

1b) Some basic communication working between PyTave and Symbolic. This might use some existing implementations in a non-optimal way (e.g., its ok if Symbolic returns objects as xml strings).
PR #452 added the functionality to convert python types into Octave types including @sym objects using pytave. Then PR #460 fixed the communication when lists/tuples were passed from python to Octave.
Then we also added proper conversion of tuples and booleans from python to Octave types in pytave repo in the PRs #9 and #11 respectively.

1c) Most of Symbolic’s tests and doctests continue passing, although some failures ok at this point.
Tested on Ubuntu 16.04. Most of the tests pass on Linux. There are still many tests which are failing (with the new IPC) but we will work on them once we have a stable pytave IPC mechanism.
With use of pytave, the errors have now been converted to python exceptions which seems to be the main reason of many tests failing. Also, sometimes it seems that the way we chose to return the outputs is conflicting with how the existing IPC mechanisms were doing it. But, these are minor failures and can be taken care of later as the new IPC matures.

1d) The above works on at least one OS (probably GNU/Linux).
It works good on Linux (our local machines).
We are currently trying to integrate use of pytave on the build-bot. The work can be tracked on the PRs #477 & #478 on github.

2a) PyTave converts various Python objects into appropriate Octave objects. PyTave needs to be extended to return proper @pyobj when it cannot convert the object. Also, symbolic must be able to convert such objects to @sym objects by calling proper python functions via PyTave (if they are indeed @sym). That is, bypass the current generating of xml strings.
Instead of editing pytave to convert @sym objects before returning, we incorporated a different mechanism in Symbolic only which converts the objects it gets into @sym objects if necessary using py* functions. The major work was on the PR #452
Groundwork has been laid for storing objects persistently on python side using @pyobject from pytave. We will also work on improving @pyobject to allow for calling all the attributes of a python object from Octave.

2b) Improve the BIST test set coverage of both PyTave and Symbolic for any new features added.
Added some tests to pytave with the PRs #10 and #11 on bitbucket.

2c) Improve doctest coverage of PyTave.
Not many doctests were needed so this part is low priority for now. Might still do it this week.

We also had some Stretch Goals’ which were planned in case we get time and following is the progress for them:

3) Improve on 2a) with some TBD mechanism: perhaps a pure m-file callback feature in the PyTave export code.
Since the workaround for @sym conversion was already used in symbolic so this was not required. Moreover, we are working on adding @pyobject support to pytave on Colin’s fork:
https://bitbucket.org/macdonald/pytave

4) The above works on both GNU/Linux and MS Windows.
Pytave needs to be built on Windows which might take a while. On Linux, we have it working. For building on windows, we are having an extensive and comprehensive discussion and are getting closer to the goal everyday.
First ‘cygwin’ was tried to build pytave on windows but many tools were not found on cygwin, moreover Octave already has a MSYS environment so we shifted to using something like that. Currently we were trying MSYS2 environment for building pytave, all the libraries were available but we got stuck at a point when trying to link with octave libraries. Octave was build with GCC 4 while MSYS2 has GCC 5 and due to some ABI incompatibility of GCC between the two versions, it was not possible to continue the build.
So, now we will try to build it from within octave, using the the MSYS bash shell. But, now we need to (probably) build libboost-dev and python-dev using that shell and then move to building pytave with it.

5) Objects passed from Symbolic back to Python should use PyTave mechanisms.
The work can be seen in the PR #465 which tries to call py* functions to store the variables into python. We have written a function which stores the required variables to a python list. Currently, it is not very stable and causes octave to crash in some tests but we are working on improving it. It now supports passing lists and 2-D matrices. But, it needs to be extended to support other types like cell-arrays and n-D matrices.

In the start of the project we had some minor enhancements to Symbolic. In the mid way we also had some enhancements on the pytave side which were needed to move ahead with the aforementioned goals.
All of my contributions to the repos can be tracked at following links:
My contributions to octsympy‘s main repo on github
My contributions to pytave‘s main repo on bitbucket
My Contribution to Colin’s pytave fork

This was the summary as per the goals set. Now, a quick summary as per the timeline we decided just to showcase how closely we are following the path we decided at the start of the project:

30th May — Work on adding the sym conversion to PyTave and cleaning up the conversion mechanism in Symbolic.
Till 30th May we had already added the mechanism to convert sym objects that come from pytave. We still have to work on it for some cases, as we have many failing tests. But, we have a basic structure that doesn’t rely on any XML stuff for object conversion.

15th June — Improve tests and doctests. Work on building PyTave and testing on Windows. No more crufty XML!
Till 15th June we had improved BISTs of pytave. The building of pytave has been underway and we have got rid of the XML stuff in pytave IPC.

27th June —  Try to get working PyTave with Symbolic on Windows (if needed, use cygwin) 
We have been following the timeline from the start. We also have other smaller goals in progress aside to this (including build pytave on travis and using pytave to pass variables to python). We have used MSYS2 to build pytave on windows. But that led to some conflict in gcc versions of MSYS2 and the ones present in Octave. Finally it seems that boost and python-dev have to be build using the MSYS environment of Octave and then only pytave can be built using those tools.

 

All of the aforementioned goals and timeline can be found at my wiki page:
http://wiki.octave.org/User_talk:Genuinelucifer

 

For any feedback/comment, feel free to post a comment here…

Progress Report: 31st May 2016

So, according to the timeline that we decided, the first phase has been complete. The timeline stated:

30th May — Work on adding the sym conversion to PyTave and cleaning up the conversion mechanism in Symbolic.

I think we have done some good work to fulfill the objective. We have had many changes to add proper @sym conversion on Symbolic side. The following 2 PRs take care of that:

https://github.com/cbm755/octsympy/pull/460
https://github.com/cbm755/octsympy/pull/452

Also, we worked on the PyTave side to fix conversion of tuples and booleans properly from python to octave. It can bee seen in the following PRs:

https://bitbucket.org/mtmiller/pytave/pull-requests/8
https://bitbucket.org/macdonald/pytave/pull-requests/2

.

As far as the midterm goals are concerned. We have completed 1a), 1b) 1d) & 2a).

Most of 1c) is also done. For now it is acceptable is my view. We now move to 2b) and 2c).

The next goal (as per timeline is)-
15th June — Improve tests and doctests. Work on building PyTave and testing on Windows. No more crufty XML!

We already got rid of the XML stuff in the new PyTave IPC. We will now be working on adding and improving test coverage of  PyTave (both BISTs and doctests). In approximately 10 days,  we will move on to start trying to build PyTave on windows.

Timeline and Goals for the project

We (me and both mentors) decided upon the following timeline to be followed during the course of this project:

30th May — Work on adding the sym conversion to PyTave and cleaning up the conversion mechanism in Symbolic.
15th June — Improve tests and doctests. Work on building PyTave and testing on Windows. No more crufty XML!
27th June — Try to get working PyTave with Symbolic on Windows (if needed, use cygwin) [Mid Term Evaluations]

5th July — Get a successfully working Symbolic with PyTave (all the tests and features working). Continue work on Goal 3.
20th July — Finalize implementation for Goal 3.
31st July — Work on improvements to PyTave such as adding support for Python objects from within Octave (printing, calling methods, saving etc…).
10th August — Recode @sym methods as required to take benefit of PyTave. Also, add some other methods into Symbolic from #215
15th August — “Blue-skies” stuff. Try to fix the unicode utf8 rendering on Windows. Explore possibility of incorporating PyTave into Octave. [gsoc final code submission begins]
23rd August — Finish all the code, test on buildbots and Windows. Submit to Google. [Final deadline for code submission]
Afterwards — Complete any goals which are left. Continue contributing to Octave…

———————————————————————-

Following goals were set for the mid term evaluations :

1a) Octave, Symbolic, and PyTave dev versions installed.

1b) Some basic communication working between PyTave and Symbolic. This might use some existing implementations in a non-optimal way (e.g., its ok if Symbolic returns objects as xml strings).

1c) Most of Symbolic’s tests and doctests continue passing, although some failures ok at this point.

1d) The above works on at least one OS (probably GNU/Linux).

2a) PyTave converts various Python objects into appropriate Octave objects. PyTave needs to be extended to return proper @pyobj when it cannot convert the object. Also, symbolic must be able to convert such objects to @sym objects by calling proper python functions via PyTave (if they are indeed @sym). That is, bypass the current generating of xml strings.

2b) Improve the BIST test set coverage of both PyTave and Symbolic for any new features added.

2c) Improve doctest coverage of PyTave.

Stretch Goals: 

3) Improve on 2a) with some TBD mechanism: perhaps a pure m-file callback feature in the PyTave export code.

4) The above works on both GNU/Linux and MS Windows.

5) Objects passed from Symbolic back to Python should use PyTave mechanisms.