All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
All the issues raised by reviewers have been satisfactorily addressed.
Please consider in addition to the recommendations made by Reviewer 1 the issue previously raised by Reviewer 2 related to an evaluation of the tool in an educational setting, by mentioning in your paper, in accordance to Answer 2.9, the aim to make arrangements in the future for this experiment.
The paper is well written, the literature is relevant. Some references to existing tools have been added and the some improvements have been done to make the paper clearer since the first revision. I could have access the update site and could have installed the tool on Eclipse Mars. However, I could not have replicated the running examples nor find the examples relevant w.r.t. how rendering on the fly notation models. A tutorial would be interesting to provide.
All-in one, I think the paper is interesting and fits the PeeJ Computer science journal, modulo some updated comments (see General Comments for the Author).
The research question is relevant and well defined. It fits the scope of the journal.
The conclusion is appropriately connected to the research question. The authors clarified how the different points of the conclusion fall under already achieved work or perspectives.
This paper presents Collaboro, both a Domain Specific Modeling Language (DSML) and a tool for the collaborative definition of DSMLs by a community composed of developers and end-users. This paper states the importance of involving end-users in the development of DSMLs to receive their feedbacks early in the development process. Therefore, the proposed DSML introduces concepts for representing the collaboration by means of participations, votes and changes operated to the DSML definition. Changes can occur on either the abstract syntax (the metamodel) or the concrete syntax (the notation). To do so, both are represented using the Ecore language. The concrete syntax is represented by a notation metamodel defined by the authors. It allows for the definition of graphical, textual, and hybrid notations, which has not been addressed in the literature before, according to the authors. The authors also argue that the Collaboro DSML can also be applied for collaborative modeling (not only for collaborative DSML modeling), modulo some changes in the DSML metamodel definition. Finally, the paper also introduces a metric-based recommender which is used to evaluate the quality of the DSML regarding a set of applied metrics. This contribution is mostly presented for ensuring the quality of the notation, based on the Moody's nine principles.
*** UPDATE ***
The authors clarified most of the points of the initial review. I agree with the answer and the changes made on the paper about the notation metamodel (Answer R1.5). I still find the answers discussion on the rendering of graphical notations half-convincing (Answers R1.3 and R1.6). It is something I was unable to test in the tool, and no example in the Git repository seems to address the creation of graphical notations.
While I think that the paper can be accepted for publication in the PeerJ Computer Science journal, I really think that the authors could benefit from illustrating the paper with examples of both the textual and the graphical notations. As is, some claims about the rendering on-the-fly, the presence of auto-layout engines and so on cannot be validated or tested over the tool. I would find appropriate for the authors to add an illustration (maybe alongside Figure 12b) coming from their tool to represent what a graphical notation example of the Baggage Claim case study would be. It is maybe what is represented in Figure 13, but the pictures in the two upper areas (Abstract Syntax and Concrete Syntax Examples) are still very small and unreadable (even after zooming on the computer).
Apart from that, I think that the fourth feature claimed in the introduction (support for collaborative modeling) is a bit of over statement as, as mentioned in the paragraph below, it is an adaptation envisioned by the author (which has consequently not been implemented yet, so it cannot be considered as a feature). I would suggest the authors to remove it.
Please address the comments made by the reviewers as they are meant to improve the quality of your submitted paper and make it clearer its contribution with respect to other work.
The paper is well written, the literature is relevant. Some references to existing tools can be added and some improvements can be done to make the paper clearer (See general comments below). Figure 13 is too small. The tool was made available on Github, but I did not find the link to the update site zip file, nor how to install the tool from the sources without error on recent Eclipse versions.
The research question is relevant and well defined. It fits the scope of the journal.
The conclusion is appropriately connected to the research question. It is unclear if some points of the conclusion fall under already achieved work or perspectives (See general comments below).
This paper presents Collaboro, both a Domain Specific Modeling Language (DSML) and a tool for the collaborative definition of DSMLs by a community composed of developers and end-users. This paper states the importance of involving end-users in the development of DSMLs to receive their feedbacks early in the development process. Therefore, the proposed DSML introduces concepts for representing the collaboration by means of participations, votes and changes operated to the DSML definition. Changes can occur on either the abstract syntax (the metamodel) or the concrete syntax (the notation). To do so, both are represented using the Ecore language. The concrete syntax is represented by a notation metamodel defined by the authors. It allows for the definition of graphical, textual, and hybrid notations, which has not been addressed in the literature before, according to the authors. The authors also argue that the Collaboro DSML can also be applied for collaborative modeling (not only for collaborative DSML modeling), modulo some changes in the DSML metamodel definition. Finally, the paper also introduces a metric-based recommender which is used to evaluate the quality of the DSML regarding a set of applied metrics. This contribution is mostly presented for ensuring the quality of the notation, based on the Moody's nine principles.
The paper is well written, the structure is clear, and the running example helps understand the paper. While I found the paper a bit too similar with the paper published at the CAiSE conference in 2013, the authors clearly state at the end of the introduction the added contribution (metric-based recommender, web frontend, ...).
While I recommend this paper for being accepted, the following suggestions can help the authors improve the paper:
- In the introduction, I was confused about what Collaboro really is. The author should clearly state at the beginning that the name "Collaboro" designates both the DSML and the tool to support the collaborative DSML definition task.
- The paper proposes a renderer engine to render on-the-fly model examples that are used to enable collaborations between end-users and developers. While the authors claim that it is possible to render graphical models using SVG, it is unclear how it is done, the running example only showing the rendering of textual examples. In particular, it is not clear if specific automatic layout engines are used, nor if it is possible to render edges to connect shapes. Specially, the excerpt of the notation metamodel shown in Figure 4 does not contain a "connector" concept to connect two graphical elements. Please provide more details about the rendering on-the-fly of graphical models using the notation metamodel.
- The authors justify the proposition of a new notation metamodel as no generic metamodel covering both graphical and textual syntaxes has been proposed yet. While this claim may be true, I am not convinced about the need of re-creating a notation metamodel from scratch. Specially, some graphical notation metamodels already exist -- such as the Pictogram Metamodel of Graphiti (https://eclipse.org/graphiti/images/pictograms.pdf) -- and are more complete, extensible and also based on EMF / Ecore. I understand that the purpose of the notation metamodel coupled to the renderer engine is not to provide a full modeling environment, as stated by the authors. Nonetheless, some concepts, such as anchors and connectors are as important as the other proposed graphical concepts and should also be discussed during the collaboration phase between end-users and developers. Let's imagine for example a case study where the position of a port (N, W, E, S) conveys important information about the nature of that port. Such concepts seem to be missing from the proposed metamodel (or at least from the excerpt in Figure 4). It could have been possible to extend existing notation metamodels to add the missing notation concepts instead of re-creating an entire notation metamodel from scratch. Please strengthen the discussion about the choice that has been made regarding existing notation metamodels.
- On Figure 4, only the textual part of the hybrid notation seems to be connected to the abstract syntax by means of EReferences and EAttributes. Could it be possible to connect all NotationElements to the abstract syntax ? Figure 4 suggests only textual element can refer to the abstract syntax while Figure 6 suggests that all notation elements can be connected to the abstract syntax (through the "maps" relationship). Would it be possible to elaborate that ? It is very common that a value of a graphical attribute is not simply defined by a primitive value such as an Integer or a String, but instead is computed regarding the properties of the model element. For example, let us consider a traffic light where the value of the "fillColor" attribute of each circle (green, orange, red) is computed regarding the state of the traffic light.
- I am not sure to understand how and where a model which conforms with the notation metamodel is created. Does Collaboro provide a specific wizard, e.g. an EMF tree-like plugin or a more advanced wizard to model and render the notation ? Is the rendering re-generated each time a change to the notation model occurs ? How can Collaboro be compared to existing tools such as Eclipse Sirius, that allows for the re-rendering of a diagram every time a change to the notation occurs ?
- The second paragraph of the conclusion section is really interesting, but I am not sure whether it is stated as a future work or if the automatic creation of Xtext or GMF configuration files has already been achieved. Whether it is already implemented or not, it could be a good idea to insert this discussion in the section "Making DSML development Collaborative" page 4.
Some minor comments:
- Figure 13 is too small ;
- Sections are unnumbered so several references to these sections are missing (e.g. at the end of the first paragraph of subsection "Renderer", page 7) ;
- The "GPL" acronym appears in the introduction and only defined page 11, while general-purpose modeling language was written out in full before the first use of GPL. Please use between GPL / GPML, DSL / DSML and define the acronyms before their first use.
Overall, the paper is nicely written and the used language is appropriate (sometimes it
has a little bit the flavor of spoken language) and clear. The introduction gives a good
overview on the context and adequate references to related work. The background
section called „Collaborative (Meta)Modelling“ gives a good overview on the setup of
the research. A running example of a baggage claim system on an airport is considered.
Then an imaginary collaboration scenario is discussed followed by a description of the
tool followed by some application scenarios.
The paper has huge overlaps with the paper „Enabling the Collaborative Definition of DSMLs. International Conference on Advanced Information Systems Engineering, Jun 2013, Valence, Spain. 2013“ by the same authors. This is understandable to a certain extend, because this
work aims to be an extension. However, is it really necessary to copy the abstract and
just append a few more words? Further, the lessons learned of this paper basically refer to the
lessons learned in the other paper (plus the insight that a web-based front end would be
useful). Also several figures are either reused or are very similar. The extensions in this work
cover mainly features like the support for graphical DSMLs, a web-based front-end
and a metric-based recommender. Especially the later is a very interesting extension
and offers a lot of potential - here I would like to see more on how the recommender
actually works - the paper mainly focuses mainly on the description of the several metrics
without giving a real comparison.
Some detailed comments:
It would better connect the first and second sentence of the abstract, e.g, Software
development processes are collaborative in nature, in particular the input of the
end-user is indispensable.
In general, most references need better integration into the text, e.g., Page 2, lines 49, 78, 93
Page 1, line 44: abbreviation GPL has not been introduced
Page 3: semantics … has been considered as part of future work -> will be considered
Figure 1: It is not entirely clear how example of abstract syntax relates to the example of
concrete example. For me the concrete syntax example looks like an instance of the model
shown in (a).
Some more details of the recommender system would be of interesting. For me it does not
come out clearly how much impact it really has on the development process. For the
recommender it should be feasible to setup interesting experiments accessing its quality
in an objective and reproducible manner.
The paper describes a tool and its implementation but not really an evaluation under an
experimental setup. If the paper would have such an evaluation (maybe in an educational
setting if no real-world project is at hand), the contribution would be much stronger.
Not empirically proven, but intuitively valid.
Basically, this kind of work is very important - collaboration is big challenge in modeling
and probably also in the development of DSLs. My main concern is that (1) it is too close to the
CAISE paper - here the additional scientific contributions have to be more explicit and (2)
there is no rigorous and reproducible evaluation of the approach. I know this is difficult
to evaluate if people are involved, but this is part of the scientific challenge. Intuitively, I
agree with most claims and observations made in this paper, but it would be important
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.