12 Replies Latest reply on Feb 12, 2019 4:38 AM by ilg

    Modular software development and the xPack project

    ilg

      For those following this project for some time, they might have noticed my references to xPacks, to the package manager, the package builder, etc.

       

      My 2019 new year's resolution was to complete these tools, and any help will be highly appreciated.

       

      Goal

       

      In short, the idea is to design and implement a set of tools allowing to build modular software, where the modularity unit is the 'xPack', seen as one or more closely related files, usually source files and includes, plus a minimum of metadata, to help the tools automate the process.

       

      The design is greatly inspired by Node.js & npm, xPacks being actually specialised node modules, sharing the same package.json description and possibly being stored on the same npmjs.com public server.

       

      An example of an experimental such project is the SiFive template (https://www.npmjs.com/package/@sifive/templates), which can generate simple blinky projects for some of the SiFive RISC-V demo boards. Don't be scared by RISC-V, the tools have nothing specific to RISC-V, the same approach can be used for Arm or any other architecture.

       

      I invite those willing to experiment to try this project generator, and proceed with building the project.

       

      The plan is to write similar project templates, first for STM32 devices, later possibly for other families.

       

      xPack tools

       

      Crucial for this project are several tools:

       

      - xpm - the xPack Project Manager (https://www.npmjs.com/package/xpm)

      - xbld - the xPack Builder

      - xmk - the xPack Make

      - xsvd - the xPack SVD

       

      xpm - the xPack Project Manager

       

      The current xpm already implements the basic functionality; its main goal is to manage dependencies, and bring together the needed xPacks, either with source code, or with binary tools.

       

      I expect your reaction to be: Yet another package manager? Aren't the Linux package managers, or existing package managers (like Microsoft vcpkg, Arm yotta, Arm CMSIS packs, etc) enough?

       

      Well, most existing package managers are not designed to handle very well multiple versions of the same package at the same time; xpm uses the same concept as npm, and allows each project to refer to specific versions of the existing xPacks, and specific versions of the tools used to build the project.

       

      Thus, once the project dependencies are defined and validated, the project should build properly at any time, and in any environment, even on CI servers.

       

      Another advantage od xpm is that it is highly portable, it works on most relevant platforms.

       

      xbld - the xPack Builder

       

      xbld is the successor of the initial xmake, with functionality redefined. (plus that it avoids the name clash with another Lua make tool).

       

      This tool is intended to be functionally equivalent to the Eclipse CDT builder, i.e. to build a given artefact (executable or library), based on a set of simple rules, without the need to manually write make files.

       

      The initial xmake used in the SiFive template was reworked, and now can build both regular projects and associated tests.

       

      Currently it generates either make or ninja configuration files, and runs the build. It can also export Eclipse projects.

       

      The next step is to implement the internal builder, and be able to run the build directly, without the need of external tools.

       

      The latest version is not ready, I plan to update the GitHub project soon.

       

      xmk - the xPack Make

       

      This tool is currently in design stage.

       

      The plan is to implement a similar functionality as ninja, i.e. a very simple tool that takes a graph of target dependencies, a list of command lines, and runs them in order to satisfy the dependencies.

       

      The input file is a relatively simple json, with a content inspired from the ninja.build;  the json format is simple enough to be generated by upper tools, and also provides a minimum validation when written by hand.

       

      The tool will be available with both a CLI and an internal API, allowing easy integration in other tools, like xbld, which will use it directly.

       

      I expect xmk to raise the same questions: Yet another make tool? Aren't make/ninja/etc good enough?

       

      Well, make is definitely not good enough; it is old, slow, it poorly processes names with spaces, on Windows it is not easy to find a properly functioning one, etc.

       

      ninja is generally better from many points of view, and some newer tools like meson do not even use make, supporting only ninja. However, ninja is a binary tool, that needs to be compiled and installed on each machine using it.

       

      xmk is planned to be a replacement of ninja, but using more modern technologies, like node.js, making it automatically portable for all platforms supported by node.js (all relevant ones). As a node module, it can be automatically installed as a dependency by npm/xpm, greatly automating builds and tests, even on CI servers.

       

      xbld may raise similar questions, like Why not continue to use the Eclipse CDT builder?

       

      Well, first that Eclipse is no longer on the wave; I would say it lived its life and now slightly goes down the slope.

       

      One of the main design goals of the xPack tools is to fully provide the required functionality from console, on all relevant platforms. This not only helps running tests on CI servers, but will also provide a good starting point to integrate them in any of the new graphical tools, like Visual Studio Code, Atom, etc.

       

      Schedule

       

      The immediate entries on my schedule are:

       

      - complete the refurbish of cli-start-options node module (https://github.com/xpack/cli-start-options-js ), to be used for all xPack CLI tools

      - implement the xmk tool (https://github.com/xpack/xmk-js , currently an empty placeholder)

      - rename xmake as xbld and integrate the internal builder, using xmk

       

      Once the build tools are fully functional (hopefully in 6-8 weeks), work on the STM32 template will start. I'll take from where the SiFive template left it, integrate CubeMX and add support for an several RTOSes (FreeRTOS, CMSIS RTOS and µOS++).

       

      The main components will be based on the existing code in µOS++, which will be split into several xPacks.

       

      xPack & µOS++ Forum

       

      Given the planned migration away from Eclipse, I created a new site to host discussions about xPacks and µOS++:

       

      The xPack & µOS++ Projects

       

      Apart from initial comments on this post, I invite all those interested in contributing to the xPack and µOS++ projects to use the new forums, which are planned to replace this element14 community.

      • Reply
        • Re: Modular software development and the xPack project
          bluenow

          I don’t know how to comment on this. Certainly Liviu has all the right to do whatever he want to his open sourced project.
           

          First I don’t like the idea of migration away from Eclipse. It is not perfect but certainly a good framework and lot of useful plugins available the Eclipse CDT.

          For the build procedure starting from command line, the Eclipse provides its native method, as something like “eclipse -build {projectName[/configName] | all}” or “ eclipsec.exe -build {projectName[/configName] | all}” on Windows platform. IF you have all the configuration done properly and can make/build within Eclipse GUI, the command line will work the same way.

           

          To xPack I don’t have much to say, because I have no idea to Node.js, but it seems a good tool. If I can find spare time, I may willing to learn it.

          One thought is, as I had mentioned before, I like a simplified installation which avoid manually separate installation. When I look the Code Composer Studio from Ti, it also use Node.js, but install its own version of Node.js redistribution within the off-line installer. BTW the Ti’s CCS is eclipse CDT based tool, integrated with GNU-ARM-EABI toolchain and GDB server with hardware debugger driver. The installation procedure is as simple as a click.  I wish GNU MCU Eclipse can do the same.

            • Re: Modular software development and the xPack project
              ilg

              >  Eclipse provides its native method, as something like “eclipse -build

               

              sure. did you ever use it?

               

              here is a functional sub-script used on Travis: https://github.com/micro-os-plus/eclipse-test-projects/blob/master/f4discovery-tests-micro-os-plus/scripts/travis.sh

               

              compare it with the example in the previous comment.

               

              not to mention that headless Eclipse support has several bugs, and the builds are not always identical to those running from the graphical interface, and sometimes do not even build the expected artefact. not good.

              • Re: Modular software development and the xPack project
                ilg

                > I don’t like the idea of migration away from Eclipse. It is not perfect but certainly a good framework and lot of useful plugins available the Eclipse CDT.

                 

                I don't like it either, since I put lots of work into it, but if you analyse the evolution of the number of active Eclipse maintainers over time, especially the number of Eclipse CDT maintainers, and the evolution of the number of unfixed bugs listed in bugzilla, you'll see that on middle/long term things are not that optimistic.

                 

                on the other side, advanced editors like Visual Studio Code are making important progress towards including IDEs functionality. if I'd have to bet, I'd put my money on VSC. I successfully use it for all other non-Eclipse specific development. highly recommended.

                  • Re: Modular software development and the xPack project
                    bluenow

                    I had heard Visual Studio Code and sounds it good reputation. I never have a chance use it though. I will try it when have time.

                    If you can integrate ARM development tools with VSC, I will be glad to try it.

                      • Re: Modular software development and the xPack project
                        ilg

                        > Visual Studio Code ...  I will try it when have time

                         

                        highly recommended. I first evaluated Atom, but finally decided for VSC.

                         

                        > If you can integrate ARM development tools with VSC, I will be glad to try it.

                         

                        yes, that's the point with the xPack tools, they are portable on all platforms and can be used with any editor, even vi/emacs.

                         

                        but if you want this to happen sooner, you are welcomed to contribute feedback to the xPack project.

                         

                        >  just wondering why xPack is better than makefile to handle all those things.

                         

                        with xPacks you don't have to write make files to build regular projects, you only define where the sources are, what are the compile options (include files, preprocessor defines, etc) and the build is performed automatically. same for unit tests.

                         

                        you also do not have to manually handle dependencies, xpm automatically brings all xPacks mentioned in the dependencies, source packages and/or tools, like toolchains.

                         

                        I invite you to take a look at https://www.npmjs.com/package/@sifive/templates  and try to understand what happens, possibly try yourself the examples; even if you do not have the RISC-V board to run the resulting binary, the journey is more important than the destination, since a similar procedure will be used for Cortex-M projects.

                  • Re: Modular software development and the xPack project
                    ilg

                    To help you get a better picture of xPacks, imagine you have several large monolythical projects that share lots of pieces of code. Sooner or later you reach a point where things get so complicated, with so many interdependencies between components, that you feel no longer confident to make changes without fearing subtle crashes.

                     

                    One solution is to split the code into small independent libraries, that can be tested separately as much as possible, ideally automatically, with each push to the repository.

                     

                    The xPack project plans to address exactly this: xpm manages dependencies and brings all needed packages; xbld runs the automated builds for multiple configurations and tests.

                     

                    For example, with npm, the Travis workflow is very simple, after cloning the repo, you need to satisfy the dependencies and then run the tests:

                     

                    install:
                          npm install
                    
                    script:
                         npm test

                     

                    Similarly, with xpm, the Travis workplow is also simple, just that xpm needs to be installed explictly:

                     

                    before_install:
                         npm install xpm
                    
                    install:
                          xpm install
                    
                    script:
                         xpm test

                     

                    xbld is not seen here, since it is used internally by xpm; this is not mandatory, the action behind 'xpm test' is a script, thus it can do anything and in any way, just that xbld is probably the simplest solution.

                     

                    {
                       "name": "myPackage",
                       "version": "1.2.3",
                       "scripts": {
                         "test": "xbld test --all"
                       },
                       "devDependencies": {
                         "xbld": "~1.2.3"
                       }
                       "xpack": {}
                    }

                     

                    The 'devDependencies' must list all tools required, like toolchains, qemu, etc. in order for the test to be compiled and eventually executed.

                     

                    I think that the npm mechanism for javaScript projects, and in extension, the xpm mechanism for C/C++ projects, is very powerful, and worth considering.

                    • Re: Modular software development and the xPack project
                      al!

                      Looking forward to this. Thank you so much for such great work. When is VSC setup expected ?

                        • Re: Modular software development and the xPack project
                          ilg

                          > VSC setup

                           

                          I don't have a date yet, the initial plan was to do the Eclipse plug-ins first, since I'm already familiar with the IDE, and then, based on the experience gained, do the VSC plug-ins; but priorities may change.

                           

                          however, the first step is to finish the CLI tools, but for this I would need help to review, and possibly improve, the design and implementation.