Call for Cases

What is this about?

Transformations of structured data such as relational data, abstract syntax trees, graphs and high-level software models are at the heart of a wide range of applications. Their success heavily depends on the availability of powerful and easy-to-use tools. There is an increasingly large number of transformation tools that follow many different approaches, and this creates challenges for the community at large. Users and tool experts may have missed a recent development in the area, and they may not use the best tool for the job. Tool developers may wish to compare their tool against others, but risk the threat to validity that they may not be using the other tools to their full extent.

The Transformation Tools Contest aims to help users, experts and tool developers to learn about the state-of-the-art through practical case studies. While some of these case studies may revisit well-known transformations, we are always looking for new case studies from the community that look at the bleeding edge in the field or challenge current tools in some way. If you have an interesting transformation problem in hand, or if you know about one, we would like to hear about it! We mention some specific areas that we consider relevant to TTC in our aims and scope page.

Two types of case modes are used in the contest (see examples of cases in the history of contests on the web page):

  • Offline case: for a reasonably complex problem that authors can solve over two months.
  • Live case: for a problem that can be solved from Monday to Thursday at STAF, considering that contestants are attending workshop/conference sessions.

In both modalities, please consider that the contest can lead to a thorough comparison of state-of-the-art transformation tool features for solving a particular problem and could thus lead to a publication. More information about what to consider in a case can be found below.

Expression of Interest

This year, we have an additional expression of interest deadline. This deadline is meant for people that have a challenging model transformation problem at hand but need assistance in turning this into a TTC case. If you let us know by 20th of January 2020 (via email), we can assist you setting up a benchmark.

Case submission

By February 24th 2020, please submit your case description in PDF format through EasyChair. A previous expression of interest is not a requirement to submit a case. The case description should include a URL to a source code repository (e.g. GitHub, Bitbucket, GitLab) that contains a reference solution and an evaluation methodology, and a basic issue tracker that solution authors may use to ask questions about the case study. For the evaluation methodology, you are welcome to draw from past case studies. If you have an idea for a case study but do not know where to start or which previous case to base it on, feel free to start a discussion with us at ttc20 AT easychair DOT org.

The case description should be in the ACM acmart LaTeX document class (see here and here), using the "sigconf" style and with the "review" option enabled, and not exceeding 10 pages (excluding references and appendices).

The description should answer these questions:

  • What is the context of the case?
  • What is the subject to be modeled?
  • What is the purpose of the subject?
  • What are the variation points in the case?
  • Are there any specific research questions you would like to answer through the participation of the community? This may be useful if you have a publication path in mind for the results of the contest.
  • What are the criteria for evaluating the submitted solutions to the case?
  • What should be the prizes for the case? Usually, there may be one or more "Best X" (where X is a desired attribute), and then a "Best Overall" solution with the best balance between all attributes.
  • Correctness test: which are the reference input/ouput documents and how should they be used? Ideally, a case description includes a test suite, as well as a test driver. The test driver can be an online web service, a local script that can be deployed in SHARE, or a Docker image on Docker Hub. You can reuse frameworks from past case studies - feel free to ask us!
  • Which transformation tool-related features are important and how can they be classified? (e.g., formal analysis of the transformation program, rule debugging support, ...)
  • What transformation language-related challenges are important and how can they be classified? (e.g., declarative bidirectionality, declarative change propagation, declarative subgraph copying, cyclic graph support, typing issues, ...)
  • How to measure the quality of submitted solutions, at the design level? (e.g., measure the number of rules, the conciseness of rules, ...)
  • How can the solutions be evaluated (ranked) systematically?

Subsequent Phases

Following the selection of a subset of the submitted cases by our programme committee, the following phases will be conducted.

Phase 2: Case solution submission. All those who like to participate in the contest will be asked to choose one or more case studies, take their favorite transformation tool and submit their solutions. A separate call for solutions will be distributed, after the cases have been selected.

Phase 3: Open peer review. The solution reviewing before the workshop will be done by other solution submitters. All solution submitters have to review three other solutions to the case that they have addressed. These reviews will not be anonymous, since these reviewers ideally will also be the opponents at the workshop. The purpose of the peer reviewing is that the participants get as much insight into the competitor's solutions as possible and also to raise potential problems. Case submitters should be available at this stage to resolve conflicting interpretations (if any) about the case description.

Phase 4: Workshop and live contest. Besides the presentations of the submitted solutions, the workshop will comprise a live contest. The case submitters should have evaluated independently the various presented solutions according to the spreadsheet, and should be present during the break to discuss the final awards to be given. For more details (such as example cases and solutions from previous editions), please consult the other sections of the TTC website.