30 Replies Latest reply on Sep 14, 2011 5:31 AM by blalor

    CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!

    ptorrone

      hello makers -

       

      this is my first post here, i'm the senior editor at MAKE magazine and also the creative director at adafruit. i've been part of a growing movement called "open-source hardware" for the last 5 years or so. today we just got some great news from the CadSoft EAGLE development team about the their long term commitment to text based file formats in the world of electronics development. with their permission i am posting this here and on MAKE (as well as adafruit). it's extremely exciting for those of us who wanted an XML format for files and we're thrilled CadSoft EAGLE has said they're committed to it!

       

      here's the overview of what we learned:

       

      • CadSoft EAGLE (our preferred schematic/layout software) has told us they are committed to a text based format in the long term.
      • The CadSoft EAGLE development team is developing a new format for schematic, layout and parts libraries that is XML, this means every object and line will be written out in text description.
      • You won't have to worry about binary file corruption, you will be able to hand-edit or generate schematics and layouts and of course the magic of version-control (such as github) will be easier than ever with real text 'diff's!
      • This will be a fully documented format and also of course a converter for old CadSoft EAGLE binary-type formats.

       

       

      this is a great direction and i encourage you to support EAGLE as you decide which tools to use now and in the future. but that's not all, i've chatted with the element-14 folks and they're all set up for your feedback, if they're very interested in Maker voices their product development. so here's your chance folks!

       

      cheers,

      pt

        • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
          debreuil

          Personally I don't think that is a good idea. Xml itself introduces ambiguitiy, as does text. It is also language based (and large, slow to parse and all the regular issues). I would way rather they open the binary formats if they haven't already. The existing text formats (like the spice format) work fine and are way more readable than xml as is.

           

          I understand I may be a minority in this view : ).

            • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
              ptorrone

              it doesnt take longer to parse text than to parse binary.  XML is much less ambiguous than text. also, XML -is- text

                • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                  debreuil

                  Well seeing as binary can just be a cast I'd say it is slower - I agree it doesn't have to be 'too slow', but it often ends up that way.

                   

                  XML is more ambigous than a good documented text format imo - eg like spice, PS etc. For example you have tags vs attributes, or unicode vs ascii, etc. That can be doc'd, but as a user you have to remember if it is one, the other, or both. And that is 'if' it is doc'd properly. That said, it is binary now, not text - I think they use text for some of the config stuff (like hotkeys iirc) though, which is why I mentioned it...

                   

                  Most tools for xml formats have problems accepting files that are 'legal' as far as the schema once the format becomes a bit large. I've had lots of issues with that in both SVG and XAML, and of course HTML seems to be hard for everyone to get right on the edges (even harder for editors). With a binary format it tends to blow up or work, which imo is more desirable.

                   

                  As far as viewing, you can view it in Eagle. I think the format is mostly interesting to those who want to tool against it.

                   

                  The one major plus of a text format is version control. You could have a tool that converts to and from open binary <> test/xml for that, but I agree that is less optimal. So one large point for text formats there : ).

                   

                  Ah well, I'll take open xml is over closed binary any day, so I'm with you for sure... Just feel 'open binary' is never given fair consideration. Ironically I think its main strength is portability - not of the parser tool, but of the exact meaning intended.

                    • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                      clothbot

                      Well, I for one will be very glad to see an XML file format implemented.  Back in my Cadence days I wrote a Java-based netlist parser app to convert between languages (mostly SPICE, Spectre, Verilog-AMS and VHDL-AMS) using XML as an intermediary format and XSL Transforms to interact with the source and generate the destination formats.

                       

                      Aside: I hate SPICE precisely because it is totally ambiguous, poorly documented and incompatible between all the permutations and combinations out there between EDA platforms.  At least the hierarchical nature of properly formatted XML is parsable independent of the content or even a formally defined schema.

                       

                      Aside 2: IMHO, if you have to use regular expressions to get at the atomic structures contained in an "XML" format like you do in SVG, it's not proper XML.  Things like coordinates should be accessible as discrete XML elements or attributes accessible through simple XSLT code.

                       

                      If size is a concern (which it was when dealing with GB-size netlists) you can use a generic zip algorithm to compress the text and serialize the I/O.  As a bonus you get encryption, authentication, error correction and all that other good serialization infrastructure for free.  It'll add a layer of I/O complexity for revision control, but if the RC supports generic zip-diffs, all text formats will benefit, not just XML.  You also get ascii vs unicode vs localization controls for free with most XML-based APIs, independent of whether the tools (don't) have it.

                       

                      Aside 3: Avoid the DOM-style XML readers that suck in the whole XML content before you can do anything with the content.  You're much better off using a SAX approach to create your own (sparse) data structures on the fly, with the added benefit that when data's corrupt you can more easily identify and work around where things are broken in the stream.

                      • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                        ptorrone

                        the file only has to be parsed on opening and closing. it doesn't affect anything else. "badly documented XML is as bad as badly documented text. this is something that doesn't even exist so no one can say it's ambiguous, yet we aren't talking about HTML. we are talking about the file format Eagle CAD, which will be used by Eagle CAD for saving and opening files.

                         

                        Robin Debreuil wrote:


                        -The one major plus of a text format is version control. You could have a tool that converts to and from open binary <> test/xml for that, but I agree that is less optimal. So one large point for text formats there : ).

                         

                        -I'll take open xml is over closed binary any day, so I'm with you for sure...

                         

                        yay!

                         

                        cheers,

                        pt

                          • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                            debreuil

                            I'm super happy it is opening, and I think there is a real chance I will use it for something that will be useful to me... so double yaay! I should have led off with that, sorry : ).

                             

                            For the record, I'm not saying use text instead of xml - I'm saying binary has a lot of advantages you lose and xml has potential pitfalls. If you do everything right xml can be reasonable, but if most major uses of xml are 'doing it wrong' then maybe it needs to be a bit more explicit and enforceable. I can deal with it though, minor quibble. Eagle is an awesome dev tool - regardless of how it is serialized : ).

                              • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                                clothbot

                                I'm cautiously optimistic that they'll do the right thing.  The major (ab)uses of XML can get away with the mutated forms they do precisely because they are major.  EDA doesn't have that luxury.

                                 

                                I've found that XML works best for separating form (e.g. SPICE vs CDL vs Verilog language quirks) from function (i.e. instantiation, connectivity, parametrization, etc). Its strengths really shine when you need to do complex scripted interaction with the data.  Pick a language (aside from Cadence's SKILL ) and you'll find at least basic XML parsing hooks.  No need for custom regexp, antlr, javacc, binary-reading parsers for the N+1th platform variation.  You can even test XSLT works-in-progress via your browser of choice!

                      • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                        yzf600

                        I'm all for openness and transparency in data files. If the data in your text file is large, though, then it can get pretty nasty for a human to parse and edit. I consitently deal with verilog netlists that are on the order of 100Mb in size for our ASICs at work. In fact, lots of files for my job are text (SPEF, netlist, DEF, LEF, LIB). We don't edit netlists by had because they are too large and complicated. We use EDA tools to manually manipulate netlists with languages like TCL (edit netlist disconnect, edit netlist connect, edit netlist create and_gate). I would recommend supporting the reading and writing of compressed XML files out of the gate. Why waste 100MB of disk space when gzip will compress it down to 2MB? Most linux distros command line utils/editors support auto decompress of gzipped files. But I guess that doesn't apply to windows very well. Since all my EDA work is on linux (none of the asic tools support windows), I tend to foget it exists for other eda industries like PCB.

                         

                        I don't think you can claim checking them into a revision control system as a feature/positive. The text in a file can look completely different to "diff", but still represent the exact same design data. The same net coordinates can live on line 5 or line 5,000,000 and still represent the same thing. We check in our text verilog netlists to RCS at work, but the diffs usually end up being 95% different despite only having 3 new logic gates added. Might as well been a binary file at those diff rates. Now if you can guarentee exact text formatting and order from output to output, then this diff issue obviously goes away.

                         

                        That being said, it is great when I run across an EDA vendor using a straight forward text file for a input or output file. Did you board dimentions change, but you want to keep all the x,y coordinates of your components? Start up a new board and draw the dimenstions, then save to a text file. Rip out the old componet coordinate section from the old text file and and place it in the new one and vola! Instant component placements! ( this is a somewhat bad example since you can just take your board file and edit the dimensions with components in place - but I think you get my point).

                         

                        BTW - I love eagle cad. Any chance you guys could get a graphics designer to re-work your icons? Nothing says polished tool like professional and modern icons.

                        • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                          Folknology

                          I think having a text based format would be a real win for Cadsoft, it would bring greatly desired version control into our process scope.Being able to edit the files also adds to the version control benefits for community (OS) based work. As for the format I can see why some are afraid of XML as it can be bulky compared to simpler text formats. However XML does allow for powerful features like schema to reduce ambiguity and provide readable open standards. It also allows for easy transformation/conversion to other XML standards using standard XSL tools. Most programming languages also have good XML libraries opening the eagle files to an even wider audience for hacking and manipulation.

                           

                          As pointed out above by yzf600 size can be a real issue with larger netlists and projects, xml file editing with larger files is very difficult for a human. not so much on the parsing side for SAX based libraries. Probably the best way to alleviate size issue is to support and promote modularity. Modularity in Eagle allows sections of a design to be copied and pasted into new or existing projects, it also allows for repetitive reproduction of identical sections within a circuit/design. This can be represented in XML using XLinks pointing to accompanying XML modules. These modules get linked in during parsing. This is much more efficient as it reduces duplication of XML and allows for tree structures of a design. Such a standard could also allow for hardware licenses and or copyright/copyleft inclusion to make the entire sharing process much simpler. Modularity of this type would be a boon for community and opesource development within Eagle whilst simultaneously keeping file size under control. It also makes version control even easier particularly if the Xlink usage includes remote Git or subversion hrefs (via the standard url identity) allowing for remote version controlled modularity, something that the hardware opensource community would love to have.

                           

                          regards

                          Al

                          • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                            EvanFoss

                            I moved over to gEDA back in '03 and while it is cool to see vendors now distribute part footprints and symbols in Eagle format I really want a compatable option. While I am really happy to see CadSoft sticking there neck out on this  thing I have to wonder why I took so long? Why didn't it happen the same way gerber format did? I can see how no cad vendor wanted it because vendor lock in has a lot of appeal to them. Next we need a wikipedia like thing with all the various footprint files with written discriptions and compatability charts.

                             

                            I have been frequenctly irritated by the amount of open source  hardware that has it's schematics distributed in Eagle's binary format. I know the user interface for gEDA is an aquired tast and I am not trying to push it on everyone but I think the symbol file format which it text based is worth a group look. The footprint file format that PCB uses is nice but it lacks a lot of things I would really like. I hope they don't just try to build one giant format. I would rather see seperate XML formats for part footprint, mechanical 3d info, symbol, schematic and so on.

                            • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                              EvanFoss

                              What license are they going to put on the final specification? Since this is all about the open hardware someone had to ask.

                              • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                                gandrewstone

                                The great thing about an open format like this will be the ability to write small scripts to automatically generate parts.  I mean using a mouse to click on a graphical representation of packages has got to be the craziest, most error prone and time consuming way to assign numbers and names to pins.  And then linking the package pin to the part's pin really ought to be done in some editable manner, not some GUI dialog box!

                                 

                                And you don't want to get me started on footprints! :-)

                                • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                                  tz1

                                  I see this as only a good thing - especially for the libraries.  Too often there is a part that just has different labels or some other variation.  It is easier to edit XML (assuming they don't do something totally confusing) to change pin names or to type in the dimensions from the datasheet than to try to be precise in a GUI.

                                   

                                  Or automate the generation process - if there is some new fine pitch format, the polygons in each of the layers are typically copied out at fixed intervals.

                                   

                                  As to XML ambiguities - that is another beauty of XML - you can write verifiers and validators.

                                   

                                  I can see a lot of things becoming more compact or new utilities (same device, new package, if the description links to the package, just add in another link to the new package).

                                  • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                                    circuitpeople

                                    For me at least, it's difficult to argue with the "goodness" of XML.  The Devil, however, is in the details.  XML does not make the system "open" if the copyright to the schema is held close and no permissive license is granted other software developers to make use of it (that is to say, it could be possible to create applications that use/create XML from/from EAGLE yet still illegal to do so -- at least in the US.  If the XML is onerously difficult to implement (i.e. MS Office's implementation) then the only ones to do so will be professional software houses. You need only look into the Gerber format to understand that -- it's text, and published and yet still very poorly implemented in small projects. And PCB information is only a part of the problem.

                                     

                                    To capture a complete PCB design the XML will need to be fully elaborated (contain *everything* needed to reproduce the copper, ink, mask, placements, etc.) or have external dependencies. A superior approach would be to use something like the Open Packaging Conventions to embed (perhaps optionally) all the external resources needed to meet that need.  The existence of so many "how to prepare your design for manufacturing" tutorials is evidence enough that getting the right information from one party to another isn't trivial, and a long way from reliable.

                                     

                                    I love the idea of an open PCB CAD format that is easy to use and implement (XML or otherwise).  Any chance that the schema is available for public/peer review?  Getting it wrong on the first go creates some very difficult issues for developers; just look back at what Autodesk went through with the XML support in AutoCAD years ago.  It wasn't pretty.  If the software is anywhere near release, then the schema should be locked-down -- why not give the community a chance to comment on it?

                                    • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                                      hoeken

                                      Here is my $0.02 worth of suggestions:

                                       

                                      First: XML as a format = awesome!  go eagle!

                                       

                                      Second: It would be really cool if the footprint/schematic/layout drawings were  based on SVG which is also an XML format.  In one fell swoop, they could  start harnessing all sorts of libraries and tools for drawing, but also  allow people to use really nice tools for drawing things.  Obviously they would still want an in-editor drawing tool, but what if you had the option of using some really nice tool for drawing footprints, or pcb artwork, etc?  Making  drawings in a PCB editor has typically been such a nightmare and why  reinvent the wheel when software such as Inkscape, Illustrator, etc.  have all done such a nice job?

                                      • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                                        Drew Fustini

                                        Howdy,

                                         

                                        CadSoft has just updated their site with info on V6 including the new XML file format:

                                         

                                        http://www.cadsoftusa.com/eagle-pcb-design-software/new-in-v6

                                         

                                        Adafruit is excited:

                                         

                                        http://www.adafruit.com/blog/2011/09/13/whats-new-in-eagle-v6/

                                         

                                        Fire up your XML parsers!

                                         

                                        Cheers,

                                        Drew

                                            • Re: CadSoft EAGLE development team developing XML format for schematic, layout and parts libraries!
                                              blalor

                                              I'm really happy to see the EAGLE team adding XML support.  Any kind of native text support would be a big plus, but I've worked with XML a *lot* and there are some very powerful toolsets for working with it.

                                               

                                              That being said, a couple of things about this announcement worry me.  First is the use of the word "ASCII", which I'm just hoping is sloppy technical jargon for "text".  ASCII is a very limited character encoding that has been surpassed by UTF-8, which is able to represent characters like "é" and "€".  I'm also bothered by the lack of indenting in the screenshot above, as it doesn't show much attention to detail.  Neither of these are technical show-stoppers, however.

                                               

                                              What's most important is that a robust, stable, versioned XML schema is published, and that the tools that CadSoft provide for parsing and generating XML documents strictly adheres to it and provides meaningful errors when validation fails.  Do not fall into the trap of building a loosely-defined schema and then throwing errors in the program logic when the attribute you defined as an xsd:string isn't a valid floating-point number.  A good schema will allow you to use tools like xmllint to make sure the XML you're generating from gEDA or Kicad (or from the bash script you wrote for generating your schematics before GUI tools became cool) is valid and stands a solid chance of being usable by EAGLE.

                                               

                                              Thanks for listening!

                                              Brian