September 25, 2018, Madrid, Spain

Third International Workshop on
Dynamic Software Documentation (DySDoc3)

Hosted by the IEEE International Conference on Software Maintenance and Evolution (ICSME 2018)

The DySDoc3 workshop will host the First Software Documentation Generation Challenge (DocGen).

ABOUT

Documentation is an essential component of the software development process, yet efficiently writing and maintaining high-quality documentation is extremely challenging. Automatically-generated On-Demand Developer Documentation (OD3) has the potential to make information on software more accessible to developers and cheaper to produce.

The third version of the DySDoc workshop (following DySDoc1 and DySDoc2) will host the First Software Documentation Generation Challenge (DocGen), which aims to spur research on automation of software documentation generation. Participation in the challenge involves developing a prototype tool and entering it in a competition. The competition will feature a live, juried tool evaluation, presentations of the best approaches, and a poster session. The extended summaries describing the entries will be published in the ICSME proceedings. The best entries will be recognized with awards.

2018 DOCGEN CHALLENGE

Call for submissions (PDF)

The challenge is to build an automated system that can create, on-demand, reference documentation for a Java class. The documentation should address, as best as possible, the information needs of an experienced programmer who is considering using the class as part of a software development task. The documentation will provide information helpful to understand the design of the class and how to use it, among others. Examples of the types of information that the document could integrate, include: key methods, usage constraints, usage examples, the role of the class in applicable design patterns, known issues and limitations, etc.

An entry to the 2018 DocGen Challenge is a program that takes as input the fully qualified name of a class from the Apache POI project and generates an output that contains the reference documentation for the input class in html format.

The target system for this challenge is the Apache POI project, version 3.17-final (r31807825, commit 219dff00e6, on Sept. 8, 2017). The source code of the project can be found on Github. The project's main contributors are aware of the challenge and willing to provide information to participants if needed.

The document generator may rely on the following (and any other) sources of information:

We provide a preprocessed dataset of some the above information, which includes the following:

  • All the commits of the project.
  • All the issues of the project.
  • The call graph of project methods and classes.
  • Inheritance tree of the project classes.
  • Stack Overflow posts.

Contestants are free to use these preprocessed data sets, develop their own tools to process the original information found in the repositories, or use a combination of both. The solution can involve human-produced intermediate data and annotations. However, any manual input to the generation process must be clearly detailed. Entries that focus on one aspect of the reference documentation are explicitly encouraged, in particular if they are novel and innovative.

SUBMISSION

Interested teams of participants can enter the challenge in one of two categories: challenge or exhibition. The Challenge category will only accept entries that address the specific 2018 DocGen Challenge. The summaries of the challenge entries will be published in the ICSME proceedings. The Exhibition category will accept any demonstration of documentation generation technology, for any programming language, technology, or types of documents. Entries in the Exhibition category will be invited to present at the workshop, but will not be evaluated as part of the competition. The purpose of the exhibition category is to showcase state of the art in documentation generation tools, which may inspire the following edition of the DocGen Challenge.

Participation in the 2018 DocGen Challenge involves two phases: the qualification and the competition. All participants in both categories must qualify. All qualifying entries will be invited to the DySDoc workshop.

Qualification

To qualify for participation in the challenge or exhibition, participants must submit a 2-page description of the key idea(s) behind their document generator. The description should contain the following information:

  • Authors
  • Data sources used
  • Brief overview of technique(s) used
  • Preliminary results, including an HTML link to:
    • for the challenge entries only - the generated reference documentation for class XSSFWorkbook
    • for the exhibition entries only - samples of generated documentation produced by the tool

All submissions must be formatted according to the ICSME Formatting Instructions and must be submitted through EasyChair.

Each submission will be evaluated by the program committee.

Competition

On the day of the workshop, the qualified challenge entries will be evaluated live. A list of target class names will be provided to the competitors, who will use their tools to generate the respective reference documentation. For each input class, the reference documentation must be generated in less than 2 minutes. The generated documents will be evaluated by a panel of judges and by the workshop attendants based on:

  • Comprehesiveness: does the information provided cover different specific needs and variety of usage scenarios for the given class?
  • Correctness: is the information provided consistent with the source code and does not present contradictions?
  • Conciseness: is the information provided succint, relevant, and non-redundant?
  • Originality: does the information provided present new and original content?
  • Usefulness: does the information provided address common information needs and is critical for understanding the given class?
  • Presentation: is the information provided easy to read, understand, and navigate?

The best entries will receive awards in several categories, such as: best tool, best presentation, people's choice, etc.

Exhibition

On the day of the workshop, the qualified exhibition entries will be presented to the audience, not as part of the challenge. See details in the program.

Important dates

July 20, 2018 - Qualifier submissions

July 27, 2018 - Notifications of qualifiers

August 3, 2018 - Camera-ready version for accepted challenge entries

REGISTRATION

The workshop registration is free, generously provided by ICSME 2018. At least one author of each qualified entry must register, attend the workshop, and present the entry.

PROGRAM

Overview of the workshop program (details will follow after the qualifying round):

  • Opening
  • Live judging of the challenge entries
  • Awards
  • Presentations of the award-winning tools
  • Poster and live demonstrations of all exhibition and challenge entries

ORGANIZATION

General co-chairs

Image

Martin Robillard McGill University, Canada

Image

Andrian Marcus The University of Texas at Dallas, USA

Image

Christoph Treude University of Adelaide, Australia

Image

Michele Lanza Università della Svizzera italiana, Switzerland

Program co-chairs

Image

Neil Ernst University of Victoria, Canada

Image

Marco Gerosa Northern Arizona University, USA

Challenge co-hosts

Image

Laura Moreno Colorado State University, USA

Image

David Shepherd ABB Corporate Research, USA

Data masters

Image

Sarah Nadi University of Alberta, Canada

Image

Shinpei Hayashi Tokyo Institute of Technology, Japan

Image

Hideaki Hata Nara Institute of Science and Technology, Japan

Image

Oscar Chaparro The University of Texas at Dallas, USA

Image

James Clause University of Delaware, USA

Web and publicity

Image

Takashi Kobayashi Tokyo Institute of Technology, Japan

Image

Neil Ernst University of Victoria, Canada

Image

Oscar Chaparro The University of Texas at Dallas, USA

Program committee

  • Christoph Treude
  • David Shepherd
  • Martin Robillard
  • Sarah Nadi
  • Laura Moreno
  • Andrian Marcus
  • Michele Lanza
  • Takashi Kobayashi
  • Shinpei Hayashi
  • Hideaki Hata
  • Marco Gerosa
  • Neil Ernst
  • James Clause
  • Oscar Chaparro

Panel of judges

TBA

VENUE

DySDoc3 will take place at the Universidad Rey Juan Carlos, in the building at Plaza Manuel Becerra.

Address: Plaza de Manuel Becerra, 14, 28028 Madrid, Spain