How to create successful Open Hardware projects — About White Rabbits and open fields

CERN's accelerator control group has embraced ''Open Hardware'' (OH) to facilitate peer review, avoid vendor lock-in and make support tasks scalable. A web-based tool for easing collaborative work was set up and the CERN OH Licence was created. New ADC, TDC, fine delay and carrier cards based on VITA and PCI-SIG standards were designed and drivers for Linux were written. Often industry was paid for developments, while quality and documentation was controlled by CERN. An innovative timing network was also developed with the OH paradigm. Industry now sells and supports these designs that find their way into new fields.


Why Open Hardware
The Controls Group of the Beams Department at CERN is responsible for the controls infrastructure for all CERN accelerators and the experimental areas.It provides services like general machine and beam synchronous timing distribution and signal observation systems, as well as support for drivers and higher-level software.
The hardware used in the control systems is diverse: analogue and digital I/O, level converters and repeaters, serial links and a range of timing modules.Overall around 120 module types are supported for systems such as beam instrumentation, cryogenics and power converters.About three quarters of the modules were developed and produced within the group.
We require that new modules can be modified by us, not only when the design is ours, but also when they were designed by others.At the same time we like industry to produce and support these designs.We have found a way to do this by using the concept of "Open Hardware", which is similar to the concept of "Open Software" that has proven its worth.
The main ideas behind the Open Hardware paradigm are: • All specifications and design files are published to benefit from peer review and to enable remote collaboration.
-1 - • All detailed production files for the hardware, including PCB production files, precise bill-ofmaterial and assembly instructions are published so that anyone can reproduce the hardware without requiring a commercial licence.
• Peer reviews are actively sought for to improve the designs and make them more easily re-usable.
Having Open Hardware allows us to freely select the companies we need based on their technical excellence, good support and price.It also creates opportunities for smaller companies that may be more flexible in supporting products that are not necessarily designed by them.
Finally, when a design is published as Open Hardware, it is likely to be reused by others who in turn may provide useful feedback or publish other designs.Every case of reuse is an example of dissemination of knowledge, which is one of CERN's key missions.

Open Hardware designs
We decided in 2009 that for ease of maintenance new designs were to be based on just two platforms: VME64x [1] and PICMG 1. 3 [2] that uses the PCI Express bus for its plug-in cards.
Most of the functionality that is required by the equipment groups is needed for both platforms.In order to increase flexibility and to reduce the development effort, we chose to adopt an architecture based on carrier cards containing basic functionalities and mezzanine modules implementing specific functionalities that can be freely exchanged between platforms.A carrier card typically contains the bus interface, an FPGA and a memory.The mezzanine modules usually contain only I/O logic such as buffers or an ADC, while front-panel space is available for external connectivity.
We selected the FPGA Mezzanine Card (FMC) standard [3] for this architecture.As an example, figure 1 shows a Time-to-digital converter (TDC) mezzanine module and a carrier card that uses a Xilinx Spartan 6 as FPGA.
Using this approach we have developed three carrier boards and four mezzanine boards.After having designed and debugged prototype cards, we made orders for a quantity between forty and ninety for each of the designs, always making clear that the company may produce and sell additional ones as the designs are 'open'.Other companies found it interesting too to produce some of the designs, even not requiring any order from CERN.This approach has resulted in the fact that -2 -every type of board is produced by up to three different companies (see table 1).Other users have subsequently ordered boards, with the SPEC carrier being the most popular one.Many users have found the switch designed for the White Rabbit project (see section 3) useful in their applications as well.
The fact that commercial engineering companies produce, stock and sell the cards allowed us to focus on new developments.

White Rabbit
One of the features that makes the CERN Open Hardware designs interesting is that the carrier boards are prepared to support the White Rabbit timing network [4].This innovative development, originating from CERN, implements a one gigabit per second optical Ethernet network in which all devices can be synchronized with an accuracy better than one nanosecond.Additionally it guarantees a deterministic low-latency and reliable delivery of control data.For many users [5] the open nature of the White Rabbit project has been a catalyser for the adoption of this technology.It has already been used in the CNGS experiment, while CERN and GSI are planning to use it as future accelerator timing networks.Also astronomic experiments, such as the Large High Altitude Air Shower Observatory (LHAASO) in China and the Cherenkov Telescope Array (CTA) project have tested advanced prototypes using White Rabbit.Other projects such as the neutrino telescope KM3NeT plan to use it to synchronise and read out the detector data.Finally, several metrology institutes are experimenting with White Rabbit to transfer the time from atomic clocks over distances up to 1000 km.
The White Rabbit protocol enhances the IEEE1588 Precision Time Protocol (PTP) used over Ethernet.It evaluates the link delay by exchanging packets and time-stamping their transmission and reception times.All devices in the network use the same reference clock signal that is encoded in the Ethernet carrier by the transmitter and that is recovered by the receiver.On recovery by the receiver of the slave device, this clock is again used to encode the data over the return channel, creating a loopback as depicted in figure 2. This aligning of the clock signals to the same frequency is called Layer 1 syntonization.As everywhere in the network the same clock frequency is used, with a phase measurement system between the clock used by the transmitter and the clock that is received on the return channel, the link delay can be measured with a precision in the order of picoseconds.
The combination of the PTP protocol and the Layer 1 syntonization provides all stations the International Atomic Time with an accuracy better than one nanosecond and a clock that has the same long term stability as the one connected to the system timing master in the network.In most cases this is an atomic clock or a GPS providing a 10 MHz and a pulse-per-second signal.
The central element in a White Rabbit network is the switch that implements the protocol and enables to interconnect and synchronize with sub-ns accuracy many devices.With the help of industry CERN has designed the hardware, gateware and software of a switch with eighteen ports.The full design is publicly available and it is produced and supported by industry.
-3 -  4 What makes Open Hardware work We consider an Open Hardware project successful when it has reached many users who share their experience and give feedback.At the same time industry should be producing the hardware and supporting the project.With four years of experience, we found that it is clearly not enough to just publish the design information to achieve this level.We found five points to be crucial.

Be open
Having a simple web site or mailing list will not enable the full power of cooperation in the design process.Therefore we set up a collaborative web tool that went public in 2009.This Open Hardware Repository site [6] contains for each project modules, such as a wiki, a file repository, a news page and notably an issues list to keep track of problems and changes.The site is truly open as anyone can access all information.We use the wiki to show the status of the project, including results of design reviews and information about the actual users of the board.We often use the associated mailing list to discuss design or usage issues.Finally we track problems and their possible solutions in the Issues module.For example, the SPEC carrier board has 73 issues documented.
-4 -There are still over 20 minor issues 'open' that may be resolved with future versions of the board or when a new version of the design is made.Having all problems documented also allows the site to be used as a teaching tool.
Even for Open Hardware it is necessary to have a licence that defines the conditions under which one is able to use or modify the licenced material.Although many licences exist for developing Free and Open Source Software, e.g.GNU GPL, none of these was appropriate to cover hardware developments.For this reason the Knowledge Transfer group, that promotes and supports knowledge and technology transfer from CERN to society, developed the CERN Open Hardware Licence [7].This CERN OHL shares the same principles as open-source software: anyone should be able to see the design documentation, study it, modify it and share it.In addition, if modifications are made and distributed, it must be under the same licence conditions to ensure that the whole community can benefit from the improvements.With these licence conditions it is immediately clear to anyone that the designs are truly open.

Make the design general enough
To make a design appealing to many users, it should be generally usable.The idea of having general purpose carrier boards and moving all specific logic to mezzanines makes this already easier.Also the mezzanine cards that we made contain little that is specific to CERN.For example the ADC card is very flexible in input range, while the TDC and Fine Delay cards use standard TTL logic levels.One has to keep in mind that if a user needs something specific, he can always adapt an existing design because it is open.This has already happened twice with the ADC card, where the input stage was adapted to accept high voltage signals and to accept higher frequencies.
Certain functions that at first we believed to be rather specific still found applications.For example a design for the outdated WorldFIP fieldbus has been used in a renovation project of trains.Also the White Rabbit network, although it has an almost too high accuracy to be considered general purpose, has found its place in applications well beyond its original purpose of controlling particle accelerators.The low cost and ease of use has helped its adoption.

Use standards and contribute to them
By using relevant standards, more people will already have knowledge of a part of a design, facilitating understanding and adaptation.For example we have chosen PCI Express, a bus standard that can be found in any desktop personal computer.Also VME is a modular bus system that is common in the high-energy physics world.For the mezzanine card concept we used the FMC standard, while for the internals of the programmable FPGA we use the open Wishbone internal bus [8], OpenCores IP and plain VHDL code.Finally, as driver software we support the Linux operating system.
In the cases where no standard could fulfill our needs, we made our own developments that we subsequently contributed to existing standards.As an example, the Wishbone standard did not allow for fast block data transfers and after discussing with the authors of the specification the pipelined transfer mode that we proposed was included in the latest revision of the standard.We are also active in the IEEE1588 standardization working group where White Rabbit solutions are evaluated for inclusion in the next revision of the standard.Similar things happened with a new -5 -Linux driver structure called FMC bus that is included in the official Linux kernel since version 3.11 and the ZIO Linux framework for data acquisition and control hardware that may be included in future Linux releases.

Be complete: from design to production test and drivers
To make an open design easily re-usable or reproducible one should publish more than the schematics and PCB documentation.Documentation may also include the quality norms required for production, such as IPC and J-STD norms.In our case, we created and published a comprehensive go/no-go test system that will verify all solder connections on a board just after it has been assembled.With this system the production startup cost is reduced, lowering the threshold to produce the designs.When CERN requested companies to build cards, we made the hardware of the production test system available to them.Also firmware and driver software is needed, as for most potential users the bare hardware is not usable.Even if a user can write this code, it is good to have working examples available.CERN did not provide user applications, but we see that companies selling the hardware start providing these in the form of starter kits.
Although it seems a considerable amount of work to provide more than just the hardware, time can be saved by reusing parts of other projects.With eight complete projects available, document templates, IP cores and portions of designs can often be taken from existing projects.Tools like hdlmake [9] that generates makefiles for FPGA projects and wbgen [10] that generates VHDL, Verilog, C code as well as documentation describing Wishbone registers, also simplify the design process.These tools and others used for driver developments such as ZIO, SDB and FMC bus [9,10] form an infrastructure that allows shorter development times than we had for earlier, oneoff designs.

Work intensively with industry
Industry has helped us greatly with different steps in the design process.The hardware design of some cards had been outsourced either to speed up projects or to get parts designed where we were lacking knowledge.In all cases we kept the end control of the schematics and PCB layout by reviewing them and having a final verification of the PCB layout by CERN's design office.Also the development of VHDL code and Linux drivers has been outsourced for the same reasons.In all cases the outsourced projects were done by small enterprises.
The actual production of the electronics has been outsourced to engineering companies.The advantage of using these, as opposed to using PCB production companies and assembly companies, is that the engineering companies understand the designs and can give feedback to improve them.At the same time these companies are set up to sell and to support the devices.We can now direct other interested parties to these companies who deliver the cards as off-the-shelf products.As a matter of fact, having engineering companies that produce our cards allows us to have more users and receive more feedback without having the burden of ordering components, assembling, testing and shipping cards.
-6 -of a project.Furthermore it helps to attract users by having designs that may fit many purposes while still being relatively simple.The White Rabbit timing and control network, although it looked at first rather specific, found a place in applications that we had not thought about before.Using standards for the bus type, mezzanine type and for the internals of the firmware help us to collaborate and makes modules that are easier to reuse.We contributed novel work to extend existing standards to include requirements that were not yet available.
Industry plays a vital role in our projects as it allows us to progress faster in hardware and software developments and allows us to receive specific knowledge.The fact that industry produces cards and can sell the boards to other users helps us to focus on new developments.This same industry is starting to support the products by adding firmware and modifying the hardware for customer specific purposes.
While running a project in an open fashion has many advantages, it may be perceived that it creates extra work.Motivation to overcome this concern can come from the understanding that it results in higher quality designs, better documentation and from seeing that other engineers are successfully using your work to solve problems in completely different fields.

Figure 1 .
Figure 1.Open hardware TDC FMC mezzanine and PCI Express FMC mezzanine carrier.

Figure 2 .
Figure 2. White Rabbit propagates the reference clock over the full network.

Table 1 .
Eight CERN Open Hardware designs found producers and users (Sept.2013).