Metastorage Frequently Asked Questions

Metastorage generates data object access classes. Currently it generates code in PHP. The data objects are stored in SQL based relational databases. In the future, other languages and storage containers will be supported.

Metastorage takes a description of the information of the entities of your application data model defined in a XML based format named CPML (Component Persistence Markup Language).

The data model information consists of the definition of a set of classes of objects with variables, validation rules, relationships and functions that are needed to access the data to be stored and retrieved.

Metastorage generates classes of objects in your programming language of choice. The generated classes make an API that you can use immediately to build your application.

Metastorage also generates classes that can install or upgrade your database, as well classes to create, validate and process forms that a provide Web based interface to access your data objects, ready to use in your applications.
Metastorage can also generate class diagrams using UML notation for the generated classes. The generated diagrams can be rendered in many image formats. You can use the UML diagram images in the documentation of your project. The diagrams also help to explain the implementation of your project to everybody involved in it, from the software developers to the clients or sponsors the project.

There are three main advantages of using Metastorage over traditional development methods that are less automated:

1. Development speed - Reduce development time drastically

Metastorage is capable of generating code from your component model definition in just a few seconds. It would take days or weeks to write, test and debug equivalent code manually by a human developer.

This is very important to reduce the time to market your software products, especially if they are medium or large size projects.

2. Software quality - Produce optimized bug free code

Metastorage analyzes all the hints that are provided in your component class definitions and makes conceptual optimizations to generate code that is fast and compact.

The generated code is as bug free as Metastorage itself. Since Metastorage has been thoroughly tested and it will be even more tested over time, this means that the generated code is virtually bug free.

Since it is programmed generator, Metastorage will generate bug free code consistently every day. It is reliable because it is not susceptible to moods or distractions that affect human programmers and make them produce code with bugs inadvertently.

3. Flexibility - Persistence container and language independence

The definition of the data model used by Metastorage is independent of the type of container that is used to store the data persistently.

This means that Metastorage can generate code that stores and retrieves the data objects from different types of storage, like SQL based database, XML files, flat file databases, LDAP servers, etc..

The generated API to access the different types containers is the same, making it possible to switch the storage of part or all data object classes, without the need to change your application.

This also means that the generated API is independent of the database type that is used to store the persistent objects when a SQL database container is used.

Absolutely not! Metastorage is meant to automate the tedious task of generating code to implement a design specification. The generated code follows well known design patterns. That leaves the software developer with more time to concentrate on software development tasks that require an high level of knowledge and creativity, which are proper of human beings.

It would be hard, if possible at all, to replicate with any advantage the human intelligence. That is definitely not the focus of Metastorage.

With a tool like Metastorage, the software developer will be more concentrated on designing the software models and implementing the business rules that address the requirements of the projects.

Metastorage only takes care of generating code that implements an API based on a specification defined by the software developer. This way, the software developer can be more productive.

In reality, Metastorage is just a wrapper script around the MetaL compiler engine. It takes the parameters passed in script command line and forwards them to the MetaL compiler. The parameters are used by specific MetaL compiler modules that take care of generating code and data files based on the specified component definition.

Most of the work done by Metastorage is performed by the MetaL compiler persistence module. The whole Metastorage build process can be divided in two stages: generation and compilation.

1. Generation

In the first stage, the MetaL persistence module parses the component definition file to build the definition data structure in memory. Then it validates it by verifying the consistency of the interdependent parts.

Finally, it generates all the code of the classes the MetaL language source files. In this stage it also generates database schema definition files, UML schema diagram graph definition files, and eventually, output template files when automatic layout forms are requested.

2. Compilation

In the second stage, the class source files generated earlier in the MetaL language are compiled into source files of the chosen target language.

In this stage, other modules of the MetaL compiler engine are used, such as the modules for generation of object oriented programming classes, database access code, forms generation and validation, output templates processing, execution flow, variable and expression processing, etc..

During the actual compilation, these modules implicitly validate the MetaL language source code files that was generated in the first stage. If there is a bug that leads to the production of code with incorrect syntax during the generation stage, the whole Metastorage build process is halted throwing an error.

This circumstance has been helping the meta-meta-programming driven code generation process to self-validate very early during the development of Metastorage. This fact contributes to generate highly reliable code that can be trusted.

The code generated by Metastorage can be as good as code written by hand by a human developer because the generator was also hand written by a human developer.

The generator is not based on static templates, XSLT (XML Style sheet Transformations) or any other method that is not self-validating. The generator produces MetaL language source code that is validated and compiled before it is outputted in the target language. Therefore it reflects the best of the knowledge of the Metastorage developer to write code in the supported target languages.

Over time the generated code will be reviewed iteratively. This makes the developers evaluate the code over and over again to think about better strategies to implement the tasks meant to be executed by each fragment of generated code.

If a better strategy is found, it can be either be implemented permanently in a new version of Metastorage or be made available as alternative code generation option. In the former situation, the users of Metastorage can automatically benefit of the newer approach just by rebuilding the code generated by Metastorage. In the later situation, the previous generation option is made the default and the users of Metastorage may explicitly switch to the new generation alternative approach if they find it convenient.

The users of Metastorage are encouraged to question and propose new generation approaches for any parts of the code produced by Metastorage. This way, Metastorage may evolve faster leading to a greater level of satisfaction.

No, Metastorage performs extensive analysis to determine which segments of code are needed considering the types of operations that the developer that designed the component model wants the data object classes to perform.

This way, Metastorage does not generate code that is not necessary. The resulting classes are very compact.

Yes. Metastorage provides an Object Query Language (OQL) that lets the developer define filters that specify search conditions that can be associated to the actions that retrieve objects.

The OQL expression conditions are pre-compiled and embedded in the generated code to maximize the object queries execution performance.

Metastorage OQL is very similar to SQL. It just has a XML syntax. Developers that are familiar with SQL are able to tune their object queries in the same way they tune SQL queries.

Although Metastorage can generate code in any languages supported by MetaL, currently, Metastorage only supports the generation of code in PHP. The reason for this is that the code generated by the database and forms modules of the MetaL compiler relies on components that are only available in PHP.

There are plans to evolve the MetaL modules so they support other languages besides PHP. Adding support to generate code in other languages will happen when the relevant MetaL modules supports such languages. There are no short term plans to support other languages unless somebody contributes with the necessary code.

No. MetaL is just a means to generate code in a target language of your choice. The generated code can be used in your applications like any other code of the programming language that you use.

Metastorage uses a XML based format named CPML to define the components that it generates, but this format has nothing to do with normal programs in the MetaL language.

Yes. You should never need to change the code generated by Metastorage, though.

Metastorage lets you define customized functions with your handwritten PHP code that is integrated in the component definition. This way, every time Metastorage generates the code, the classes always include your custom functions.

Metastorage is available for download in the MetaL download page. It comes with documentation and examples of use. You also need to download and install the MetaL project archives to use Metastorage.

You are recommended to read the Metastorage tutorial to learn how to install it, design components, generate code and UML class diagrams and use the generated code in your applications.

You may find different examples of how to use the generated code in the Metastorage tutorial.

No, if you need something different than what the form handling classes generated by Metastorage can provide, you can write your own user interface code and still use the data object classes generated by Metastorage to retrieve, validate, store or delete your persistent objects.

The current support for generating Web forms to access your class objects is limited. It is meant to implement the most common types of user interfaces based in Web forms. So, it is not unusual to need to develop special user interfaces to address specific needs of your projects.

The current list of features of Metastorage is available in its documentation.

The progress of the Metastorage project is also recorded in a XML/RSS based feed available on the URL below. This feed includes not only the newly added features, but also the features that were added to the to do list.

http://www.meta-language.net/metastorage-progress.xml

If you use a browser capable of rendering XML documents with XSL, you can see a list of projects items sorted by the date when they were updated. The items that are marked as done as those that are already implemented.

The features that are planed for future versions are listed in the feed that makes available the progress of Metastorage. The items of the feed that are marked as todo as those that are planned to be implemented in the near future.

Apart from those that are listed, there are some major features that are more relevant because they constitute significant enhancements to current version. Here follow some of the planned major features:

• Object editing forms

Currently, form handling classes only support one type of forms that is just meant for creating new objects. Another important type to be supported will allow the creation of forms for editing and deleting previously created objects.

• Multi-page forms

The classes of objects may have as many variables as needed. For large classes it may not convenient to display all the form fields for all the variables at once. Splitting the variables in groups of fields that are displayed in different pages may be more convenient. In that case there will be additional buttons to allow the navigation between the different form pages.

• Commenting and documenting the generated code

Some developers only use code that they fully understand. In the future, each block of code that is generated will have inline comments that provide detailed explanations about the purpose of the relevant lines of code.

Currently, MetaL already has the capability of generating documentation in printable formats from the source of the classes. It only requires that some special purpose documentation sections be present in the declaration of variables, functions and arguments. Multiple documentation sections may be defined to generate documentation of the same class in different idioms.

In the future, Metastorage will generate classes already built with documentation sections, so MetaL can automatically generate documentation from the classes' source.

• Report generation classes

The object model is useful to provide an interface to manage data entities. However, when you need to generate reports or do a batch processing of large amounts of data that may eventually need to be combined from multiple classes of related data objects, the object oriented mappings are usually inefficient.

A better solution for this purpose would be to retrieve data in tabular format, consisting of multiple rows of data returned in a predefined number of columns, just like it is stored in database tables and query result sets.

In the future, Metastorage will generate classes that retrieve data in tabular format composed from objects of related classes just like in SQL queries. The retrieved data may be formatted to compose reports that are suitable for presenting to the users of your applications.

• Web services provider and consumer classes

The standardization of the protocols for providing access to remote services over the Internet, turned the Web services more appealing to share information or provide programmatic access and control of resources that are based on the manipulation of data distributed over remote servers.

Web services that use popular protocols like SOAP and XML-RPC work based on the exchange of request and response messages between the servers that provide the services and the clients that consume them.

In the future Metastorage will be able to generate classes that encapsulate the functionality that is necessary to process and handle Web service requests that manipulate data objects. Additionally, it will also generate classes that will be able to send the requests and collect the responses to implement Web service consumer applications.

Code generation is not a new technique. There are many other tools that resort to code generation to achieve goals similar to those of Metastorage.

One site where you may find a lot of information about code generation techniques and tools is the Code Generation Network. Here you will find a database of existing database generation tools for various purposes that support many different languages.

The approach that Metastorage takes of generating code from high level specification is known as meta-modeling. One site that is dedicated exclusively to this approach is Metamodel.

Metastorage was created and developed by Manuel Lemos. Other developers also contribute with additional tools.

Metastorage development started in the September 2002. However, a significant part of Metastorage is based on MetaL compiler modules that started to be developed around middle of 1999.

Metastorage is an Open Source project. It is being released with a BSD like software license that is the same license under which MetaL is made available.

Despite Metastorage is Open Source, it requires a great investment of time and skill to research and develop. Making it Open Source was just a means to make it subject of as much testing and criticism as possible. That would help to find any design or implementation flaws as soon as possible.

However, opening the source of Metastorage was not an absolutely necessary step as that was not the main motivation of its development. The main motivation to develop Metastorage was to have a tool that would drastically reduce the time to market of medium or large size applications.

While it is being developed, Metastorage generates code that is being used in the implementation of services that will be offered to the users of the PHP Classes site. Such services will constitute a package that will be part of the paid subscription offer that has been planned in 2002.

The current free services that the PHP Classes site provides to the users will always remain free. The paid subscription services will generate the necessary income to keep the site open, while more free services will be provided for the joy of its already large community.

The main challenge of implementing the services that were planned is that they require significant development effort. The idea of Metastorage is to help reducing drastically the development time, so the site can survive until it starts generating enough income to keep it open with many new developments that were planned but there was never time to make them happen.

This will also foster further development of Metastorage itself, incorporating more and better capabilities that will address the needs of more and more developers.

Metastorage is mostly a derivative part of the MetaL project. So, for now you can obtain support by placing your queries in the MetaL mailing list.

One of the advantages of opening the source of a project is that you provide opportunities that let others fix any bugs and improve the project features. Sending bug fixes or improvement patches is one of the ways to contribute to the Metastorage project.

If you do not have the time or you do not feel capable to contribute with any code, you will also be contributing if you just report any bugs you find or suggest any features. In any case, use the mailing list to start contributing to the project.

Another way to contribute is to write articles, tutorials or use case presentations. Then you could share them with other users, so they can learn faster about how to take advantage of the capabilities provided by Metastorage.

Metastorage has a lot of potential but to take advantage of such potential there are several types of complementary projects that could be implemented in parallel by independent developers.

Here is a list of suggestions of possible independent projects that can improve the potential of Metastorage:

• Metastorage designer GUI tool

A GUI based tool that could make the creation of CPML component files more intuitive and more productive would be a very interesting project for the Metastorage users. Since CPML files are written in XML, such tool could be a standalone GUI application written in any language, or a plug-in component for an existing IDE like for instance Eclipse.

• Port Metabase and forms packages to other languages

Metabase is the API that is currently used by MetaL database interface module to generate code that accesses databases using SQL. Metabase API provides a way to write database independent applications. It also provides a way to create and maintain database schemas without the need to write any SQL code by hand.

The forms generation and validation package is a class used to compose and validate HTML forms. It generates automatically Javascript code with the forms HTML output to perform several types of validation on the client side. The class itself can perform the same types of validation on the server side.

Currently, Metabase and the forms packages are only available in PHP. Porting these packages to other languages would enable Metastorage to support more target languages, making it useful to more developers.

• Reverse engineer database schemas

Some developers will have the need to migrate existing database applications to start using Metastorage and develop those applications with greater productivity.

For such developers it would be useful to provide a tool that analyzes the schema of the application database and performs reverse engineering to generate an equivalent CPML definition of classes and relationships from the correspondent database tables and keys.

Additionally, it could be provided a tool that converts from Metabase schema definition format to CPML. Metabase API also provides database schema management functions that make it possible to generate a schema definition from a database that may have not been created by Metabase schema installation functions. This way, the migration from an arbitrary database to CPML could be simplified.

• Conversion between UML standard formats to CPML Already done by Peter Howard

UML is a standard visual language that is often used to design object oriented software systems. There are many applications that provide a graphical user interface to model software systems using UML, like for instance Umbrello and Argo UML, which are Open Source programs.

UML modeling tools are usually able to export models in XMI format. This is a XML based format meant for interchanging model meta-information.

A conversion tool could extract class model data from XMI files and convert it to the CPML format. That would allow the users of Metastorage to design their classes with UML modeling tools.

Ideally, such tool would be able to convert new models from scratch or merge a newly edited model with a CPML file generated from merge previously converted model that was possible edited manually by a developer.

• Forms presentation themes

Metastorage can generate forms to provide a Web interface to manage objects of the generated data classes. The output of the forms is based on re-definable themes that specify HTML templates, graphics, frame skins, colors, fonts, styles, among other aspects of the forms presentation.

Presentation themes provide great flexibility to improve the forms usability and make them visually more appealing to the users of the applications developed with Metastorage.

The number of predefined themes distributed with Metastorage is limited. Each developer may prefer use other themes. Anybody can design new themes following the specification detailed in the Metastorage documentation. New themes may be contributed to Metastorage, so other developers can reuse them in their applications.