Documentation: Project's Motivation

This document is part of the eXtensible Binary Universal Protocol project documentation. Provides reasons behind project birth and other motivation for development and future practical usage of project.

The main reason for establishing of this project was dissatisfaction with the current state of art. Terrible fractional-ism of used solutions without a unified standard makes the work of programmers hard, and slows the evolution of information technologies. At the same time, the formats are not able to cover the wide requirements for performance and interoperability. Nothing also indicates that the situation is going to improve in the foreseeable future. With the increasing complexity and scale of software products it also seems increasingly appropriate to use more formal approaches, such as using abstraction, verification of definitions, or to separate data protocols to the data and presentation layer.

Practical application of contemporary computer formats is burdened by a number of problems. Applications must handle many different formats, opening files of older versions and perform conversions, moreover, support of the formats is mostly static and built into the applications or limited for a specific platform. Errors or design flaws usually cause shortening the lifetime of the format, and lead to a much higher cost of later fixes.

The modern binary solutions are mostly designed for limited use. Many specific solutions are considered as protected company property or formats are used as a technique in competitive fight (vendor lock-in). Some other details follow:

• Optimization for a specific platform / purpose - Formats are often specialized for use on a particular family of computers, or exact purpose only. Issues like integer length, order of integer parts (endianity), and real numbers encoding, or other types respectively are about to be considered. It is then usually necessary to convert such specific solutions to other forms
• Limited value ranges - Used encodings are usually limited to static length of value ranges, and even in the case that the value might potentially overflow that range
• No or limited extensibility - In most cases, it's not possible to add blocks from other formats, adding or expanding of the existing blocks for the later fixes is also often impossible
• No or inaccurate specifications - The specification is often not available, or the description is incomplete, or allows multiple interpretations
• Disunity - The same data are encoded differently in different formats
• Limited support for compatibility - The support for formats is built into the applications, and therefore it's not possible to update it

These projects can be considered as examples of better designed binary formats: IFF / RIFF (PNG, AVI), Matroska, or ASN.1.

An example of another solution is to use the text formats with static marks for the definition of structures, such as the so-called plain text or XML. Markup languages are widely used on the Web and their popularity keeps on increasing. The advantage is an easy expand-ability, good readability, the theoretical limitlessness of the scope of numerical and string values, support multiple languages, and many more. Text form also forces companies to publish documents in the same form as they are being developed, and making it more difficult to develop undocumented formats. Thanks to the unified internal structure and existence of implementations in the most widely used programming languages, these formats are widely supported and are very good inspiration for future solution. This also proves that today there is no longer necessary to care about optimatization so much, that is memory usage and computational efficiency, and so it's more important to focus on abstraction and metadata representation. The disadvantages are:

• Higher performance consumption - Text and tags processing is performed on bigger data chunks therefore require more caching and searching for marks
• Bigger demands on space - For example, the text representation of numerical values is highly inefficient in its transmitted form
• Dependence on the language - A text file can be read only if we know the used language
• Limited lifetime - Due to the language change over time, these formats become obsolete (usually in tens of years)
• Problematic encapsulation of binary data - It's difficult to include binary data in the text, therefore text is losing on universality. Base64 encoding technique doesn't seem to be an appropriate solution.

Other objections can be found for example in the documents related to the Binary XML initiative.

This project aims to resolve as many current problems as possible and to allow better information technology evolution. The project should seek the best compromise to the essential requirements in every step. The resulting product should bring a benefit to both ordinary users and developers. To get as near as possible to the perfect solution, formal methods should be applied on particular steps of the project development.

The project seek to achieve such a situation for users that work with documents will be trouble-free. Users will be able to smoothly open and work with old documents, which would be, if needed, automatically transferred to the new version transparently without user intervention. Documents will be interconnected together and, if necessary, additional required libraries or the conversion scripts will be downloaded from the Internet. Documents won't be tied to any specific language, so it won't be necessary to learn foreign languages in order to work with them even at a deeper architecture levels. There should be tools available as part of the operating system such as for authentication, encryption, and possible repairing of damaged documents, as well as for editing and conversions. Document should be portable to different locales, including a transparent conversions of measurement units, currencies, and national specifics. Overall, the project should allow ordinary users to use new and old features of programs for long time period without any struggles. It should provide a choice between a number of competitive products. Users should be ensured that they can rely on wide usability of their documents that other people with computers and programs from different companies should be also able to use and even in the distant future.

From the programmer perspective, project actually doesn't bring any major innovation nor disallow the use of existing technologies. However, the project wants to bring benefits mainly due to a uniform access to data and metadata with the long-term stable structure based on solid mathematical foundations. This should help programmers to handle documents in this protocol, to create and implement new data formats with the possibility of a wide re-usability in other formats, and to have the overview of the existing specifications. With help of available tools, it should be possible to display the structure of a document in human acceptably readable, and understandable form with the possibility of adding the meaning of data. The usage of the protocol also should not lead to more excessive demands on computing power or memory capacity. It should also be easy to implement scripts for compatibility and for multiple formats interconnection, or to use formats for platform-independent interfaces, and to allow the creation of more complex applications. The final protocol should lead to easier communication between computers and allow better organization of data on the data media and become the basis for further more advanced projects, such as paradigmatic programming language with optional syntax for more abstraction, high performance compilation, and verification.

Use cases are largely taken from the documentation for the W3C XML Binary Characterization Use Cases relating to the binary XML. Even it's related to binarized variants of XML, it can be appropriately applied to this project.

If we want to include additional background information into multimedia broadcasting streams, is critical to save space. Using XML for that would not be too effective, like for example for television or radio broadcasting and information on the running program.

Transmission capacity savings are essential here, also processing efficiency, single-way communication and backward compatibility.

Especially in energy and information systems for the management of other networks, as well as for real estate cadastre, it seems to be appropriate to deploy the standard for data exchange. Emphasis is aimed on effective representation of real numbers and performance in the processing of large-scale maps and a large amount of data.

Another possible use is to deploy the project to create a uniform standard for the exchange of 3D graphics (real time). This is similar to the previous example, primarily performance and size is a critical again.

Another area with a critical emphasis on size and performance are small devices such as mobile phones, PDAs and the like. They are limited to the available capacity and performance of its processing units. In addition, there is a variety of architectures.

Another possibility is to use protocol for complex representation of formats which would combine a wide range of different types of data, for example multimedia and text documents with graphical layout and others. An unified access to such data should be the main benefit here.

The structure of the protocol should provide long-term work-ability for documents because of the uniform structure and conversion functions, as well as precise document definition.

The list of resources, literature and relevant links:

XBC-Use Cases - XML Binary Characterization Use Cases http://www.w3.org/TR/xbc-use-cases/
Binary XML - XML Binary Characterization Working Group http://www.w3.org/XML/Binary