From Swsc-WIKI
Jump to: navigation, search


This workshop has been sponsored by Semantic Technology Institute (STI) International.

Sixth International Workshop on Evaluation of Ontology-based tools and the Semantic Web Service Challenge (EON & SWS-Challenge 2008)

Located at the 5th European Semantic Web Conference ESWC 2008

June 1st-2nd, 2008 (Tutorial and Workshop days)

Sheraton La Caleta, Tenerife, Spain

Here you will find the previous EON workshops: EON2007, EON2006, EON2004, EON2003, and EON2002.

Here you will find the previous SWS-Challenges: SWS-Challenge


In the successful series of EON workshops we intend to bring together researchers and practitioners from the research areas Ontologies and Semantic Web. Former EON workshops aimed at evaluating ontology-based tools (2002, 2003, and 2004), at evaluating ontologies themselves (2006), and at evaluating both (2007). Whereas this has provided a meeting point and a forum for researchers, as ontologies and Semantic Web technologies have gone beyond the research limits it is required that these evaluations are available to a broader range of users and developers, both in the research and in the industrial worlds. Thus the main goal of this workshop is to lay the foundations for sharing and reusing methods and tools for Semantic Web technology evaluation (ontology development tools, ontology merging and alignment tools, ontology-based annotators, semantic web service technology, etc.).

The large visibility of the Semantic Web already attracts industrial partners and Semantic Web technology companies are now present in the industrial world. But the evaluation of the Semantic Web technology and of the applications that use it is difficult and expensive. Practitioners want to evaluate their tools and to have benchmark suites available for doing that in order to minimize the evaluation cost. Also, there are some benchmark suites that are widely being used by the community (such as LUBM). The problems are that users and developers don't know where to find these benchmark suites, what these benchmark suites are intended for, or how to correctly use them.

A well-understood notion of Semantic Web technology evaluation will foster a consistent level of quality and thus acceptance by industry and the web community. But to achieve this it is necessary to enable reusing results and lessons learnt from others.

As described below, the Semantic Web Services Challenge (SWSC) also evaluates semantic technologies, in this case those used to mediate, discover, and compose web services. Like EON, the SWSC also provides an infrastructure for the development of reusable technologies. In order to leverage these two initiatives, the sixth EON workshop includes the sixth edition of the well-known Semantic Web Service Challenge. This year, we aim to make a step forward in the evaluation of the Semantic Web technology by making Semantic Web practitioners face real problems aimed at real use cases, and the best way to achieve this is by proposing a technology challenge such as is posed by the SWSC. EON-SWSC2008 will be a two-day workshop: The first day will be dedicated to ontology and semantic Web evaluation in general (in the spirit of EON) whereas the second day will focus on SWS technology evaluation (in the spirit of the SWS-Challenge).

The Semantic Web Service Challenge

Semantic web services (SWS in the following) have received a significant amount of attention and research spending since their beginnings roughly six years ago and the number of frameworks and algorithms targeted to (semi-) automate central web service tasks like discovery, matchmaking or composition is continuously growing. However, there is no scientific method of comparing the actual functionalities claimed and to comparatively evaluate the strengths and weaknesses of the various approaches in an objective way. Furthermore, there is no standard test bed to assess the robustness and applicability of proposed technologies to real world problem scenarios. Progress in scientific development and in industrial adoption is thereby critically hindered.

The Semantic Web Service Challenge is the major initiative dedicated to work on this problem. The Challenge consists of a set of problem scenarios to be solved by the participants. The scenarios are organized in sets of increasingly difficult levels, each designed to focus on a particular problem aspect. Solutions to the problem scenarios are presented, discussed and peer reviewed at the SWS-Challenge workshops and the functional coverage of the solution with regard to the problem levels is certified at the SWS-Challenge website.

Based on the different solutions to a common application, the SWS-Challenge workshops provide a forum for participants to develop a thorough understanding of each other’s technologies that greatly exceeds the level of insight that can be reached through the study of academic papers alone. During the course of the last five SWS-Challenge workshops, a high level of mutual understanding for the participating technologies has been developed and the level of communication and cooperation among participating teams has greatly increased.

The SWS-Challenge is an ongoing and continuous experiment in developing a methodology for evaluating the functionality of SWS technologies. Thus, apart from discussing and evaluating solutions to the Challenge scenarios, the SWS-Challenge workshops provide the forum to advance the Challenge itself. In this aspect the Challenge seeks proposals of new scenarios to be incorporated into the official SWS-Challenge test bed and fosters discussion on the Challenge’s evaluation methodology as such. Insights gained during the workshops will provide the necessary input to the W3C SWS Testbed Incubator Group which aims at standardizing an evaluation methodology for semantic web services technology.

Topics of Interest

The main track of the workshop is focused on the evaluation of Semantic Web tools and applications, including but not limited to:

  • Semantic Web technology evaluation methods
  • Methods to describe evaluation aspects and parameters
  • Identification of Resource consumption characteristics
  • Models for describing and measuring in evaluation
  • Techniques and formulas for evaluation
  • Infrastructures and tools for evaluation
  • Certification and interoperability of tools
  • Performance and scalability evaluations and benchmarks
  • Integration of evaluation tools into frameworks
  • Web scale distribution of Semantic Web

For the SWS-Challenge the workshop seeks novel papers

  • describing implemented solutions to the SWS-Challenge problem scenarios (SWS-Solution Papers, see below),
  • describing other benchmarks targeted at the evaluation of SWS technology,
  • dealing with SWS technology evaluation in theory, and
  • describing application of SWS technology in industry or on real world use cases.

Papers describing solutions to SWS Challenge problems must demonstrate completion of at least one problem level in order to qualify for the workshop. In order to be successfully certified, a solution has to send the correct sequence of SOAP-messages to the SWS Challenge test bed. The solution code must be demonstrated at the workshop and will be evaluated by a group of workshop organizers and participants. Please don't hesitate to contact Charles Petrie or any other organizer in case of questions.



We plan a two day workshop. To ensure a creative atmosphere during the workshop, the participants will be selected based on their submitted papers and demonstrations, posing important issues to be presented and discussed at the workshop. The first day of the workshop will consist of short presentations followed by discussions about the goals of the workshop with regards to the needs of the Semantic Web and outlining a plan for future research. The second day will be devoted to the practical session where the demonstrations will be given and discussed. In order to obtain an intensive exchange of ideas between the participants, extensive time will be left for discussion.

The first day of the workshop, papers are presented. On this day, or shortly before, testing participants are given a surprise problem similar to one they may have already solved. On the 2nd day of the workshop, the code of the testing participants is evaluated by the workshop and solutions to the surprise problem also evaluated.

Results of the tests are posted on the SWS Challenge website and the technologies are certified as able to solve particular problems to the extent agreed upon in the 2nd day of the workshop.

The previous EON workshops proposed a series of experiments for evaluating different aspects of ontology tools, e.g. their expressiveness and interoperability capabilities, and different aspects of ontologies. The aim of the EON series is to attract attention to a number of evaluation topics since we believe this to be a highly relevant issue for the adaptation of Semantic Web technologies by partners outside the Semantic Web community. As we mentioned before, this year we will tackle evaluation on the Semantic Web Service technology front.

Taking into account the success of the former workshops in this series, we will not only expect regular contributions on any of the topics addressed in this CFP, but we will also look forward to receiving submissions based on experimental results, surveys and tools for Semantic Web Technology Evaluation. The EON2008 workshop is intended to be a platform to discuss results and further steps with interested parties. Along with these experimental contributions, we explicitly encourage people to make demos of their tools. We will reserve time slots for demos in the workshop.

A final panel will discuss the research agenda for the coming years, based on the presentations and results from the demos and discussions.

Detailed program

June 1st:

  • 9:00 - 9:30 Introduction, existing challenge methodology and scenarios
  • 9:30 - 10:00 Heiner Stuckenschmidt: Debugging OWL Ontologies - A Reality Check (Research paper)
  • 10:00 - 10:30 Maciej Zaremba, Maximilian Herold, Raluca Zaharia and Tomas Vitvar: Data and Process Mediation Support for B2B Integration (SWS-Challenge solution paper)
  • 10:30 - 11:00 Coffee Break
  • 11:00 - 11:30 Nils Barnickel, Ralf Weinand and Matthias Fluegge: Semantic System Integration – Incorporating Rule-based Semantic Bridges into BPEL processes (SWS-Challenge solution paper)
  • 11:30 - 12:00 Stanislav Pokraev, Dick Quartel and Teduh Dirgahayu: Model-driven, Semantic Service Integration using COSMO framework (SWS-Challenge solution paper)
  • 12:00 - 12:30 Tiziana Margaria, Marco Bakera, Harald Raffelt and Bernhard Steffen. Synthesizing the Mediator with jABC/ABC (SWS-Challenge solution paper)
  • 12:30 - 13:00 Tiziana Margaria, Christian kubczak, Matthias Kaiser, Jens Lemcke and Björn Knuth. On-the-Fly Synthesis of the Mediator Scenario with jABC and Prolog (SWS-Challenge solution paper)
  • 13:00 - 14:00 Lunch Break
  • 14:00 - 14:30 Alessio Carenini, Dario Cerizza, Marco Comerio, Emanuele Della Valle, Flavio De Paoli, Andrea Maurino, Matteo Palmonari, Matteo Sassi and Andrea Turati. Semantic Web Service Discovery and Selection: a Test Bed Scenario (Research paper / scenario proposal)
  • 14:30 - 15:00 Discussion of new scenario proposal and on further scenarios etc.
  • 15:00 - 16:00 Panel discussion on SWS (evaluation methodology, industrial adoption, ... details to be released)
  • 16:00 - 16:30 Coffee Break
  • 16:30 - Start of code review and certification of solutions
  • 19:30 - adjourn for the evening

June 2nd:

  • 09:00 - Continuation of code review and certification of solutions, surprise scenario evaluation (details will be announced later)
  • 12:00 - End of Workshop

Certification Results

The table below shows the results of the certification of the solutions presented at this workshop. Please note that the table reflects only the solutions or solution aspects certified at this workshop. For a complete overview of the certifications provided within the SWS-Challenge, please refer to the tables on the main page of this wiki. For detailed information about the individual solutions and to access the available technical content, please visit the solution overview and documentation page or use the links in the table header below.

Evaluation Results
Problem Level DERI AT & DERI IE
(Solution Details)
University of Dortmund & University of Potsdam (Synthesis)
(Solution Details)
University of Dortmund & University of Potsdam & SAP Research
(Solution Details)
(Solution Details)
0: Static mediation 1 (√2)
1a: Changes data mediation 1
1b: Changes process mediation 1
1c: Mediation/integration for payment authorization

1 Solutions presented at previous workshops

(2)Partially implemented - The synthesis algorithm does not generate loops, thus in the current solution only a fixed number of line items is added to an order in Moon's OMS

Intended audience

The intended audience is the union of the EON and SWS-challenge audiences.

  • EON addresses researchers and practitioners working in evaluation and benchmarking of semantic web technologies. The previously held EON workshops were a great success, each time around 30 participants attended and discussed lively and actively.
  • The SWS-Challenge specifically addresses researchers and practitioners working on semantic web service related research. However, previous workshops have attracted not only participants from the semantic web community but also from the formal methods and software engineering community and from industrial research. The Challenge has thus succeeded to bridge communities working on different aspects in the field.

Workshop Organising Committee

EON Workshop founder: York Sure

Program Committee

  • John Domingue, Knowledge Media Institute, The Open University (UK)
  • Tiziana Margaria, Computer Science Dept., U. Potsdam (DE)
  • Axel Polleres, DERI, NUIG (IE)
  • Richard Waldinger, SRI (USA)
  • Peter Emmel, SAP (DE)
  • Tomas Vitvar, STI Innsbruck (AT)
  • Christoph Bussler, Merced Systems, Inc. (USA)
  • Jos de Bruij, KRDB, U. Bozen-Bolzano (IT)
  • Alexander Wahler STI International (AT)
  • Thorsten Liebig, Ulm University (DE)
  • Oscar Corcho, Universidad Politecnica de Madrid (ES)
  • David Martin, SRI (USA)
  • Baoshi Yan, Bosch (USA)

Submission and Proceedings

Submissions are to be formatted according to the guidelines of the ESWC2008 conference (Springer LNCS style). The page limit for full papers is 10 pages, for SWS-Challenge solution papers two additional pages are accepted.

Papers describing SWS-Challenge solutions should describe the technology employed for the solution and focus on how that technology was actually used to solve the challenge problems. The reader should be able to understand a particular solution in theory and practice by reading a solution paper. Additional submission of the code and documentation of the solution is strongly encouraged.

Note that ESWC requires workshop attendees to also register for the main conference. So try to register both for the conference and the workshop before the ESWC2008 early registration deadline.

Since the ESWC does not include workshop proceedings, we intend to publish the workshop proceedings online at CEUR-WS again. If the number of high-quality submissions is high enough, we will pursue publishing the proceedings as a journal special issue or as printed proceedings.

Guidelines for Paper Submission

Papers are to be submitted online via the EONSWSC 2008 EasyChair submission system.

In case of any further questions about submission process, contact Omair Shafiq (omair.shafiq@sti2.at).

Guidelines for Code Submission (only for SWS Challenge solution papers)

Interested participants should directly contact Srdjan Komazec (srdjan.komazec@sti2.at) to get an account for the SWS-Challenge FTP server (ftp://sws-challenge.org).

Code must be submitted 3 days before the actual date of the workshop, i.e. by 28 May 2008. After that date, access privileges of the participant logins will be changed to read-only (no further chance to upload or change a submission). During the workshop, participant accounts will receive write-access again to be able to upload necessary revisions at the workshop.


The proceedings of this workshop are available at CEUR-WS

Important Dates

Please note: the dates posted here previously were incorrect and did not correspond to those published in the CFP. The currently posted dates are the accurate ones from the CFP

Submission deadline extended to March 27th!

Abstract submission for Research Papers
March 23, 2008
Full Research Papers submission
March 27, 2008
Short SWSC-Solution Papers
March 27, 2008
Acceptance Notification
April 11, 2008
Research Paper Camera Ready
April 30, 2008
Full SWSC-Solution Papers
May 18, 2008
Workshop dates
June 1-2, 2008

For SWSC-Solution papers we require a short paper (<= 4 pages) describing the planned solution by March 23th and a full paper describing the *implemented solution* by May 18th. SWSC-Solution full papers will be included in the proceedings, if the claims made in the paper are verified at the workshop.

Please do not hesitate to contact Charles Petrie for any questions you have!