SWS Challenge Levels
From SWS Challenge Wiki
Please note that this page reflects the levels and evaluation criteria used until the Athens Workshop in November 2006. The current set of problem levels is displayed in the evaluation table on the main page of this wiki. Currently we do not use evaluation success levels anymore.
The general idea is that the workshop organizers provide
a set of challenge problems. These build upon the initial mediation
problem, which we call level 0. On top of this we add various levels,
each corresponding to a general kind of problem, and each with
sublevels of complexity. Level 2 adds discovery to the mediation
problem of level 1, and level 3 adds a different kind of discovery
as well as some composition with the system of level 2. Subsequent
levels will extend the supply chain, mixing mediation and discovery.
For now, we allow systems to avoid solving the mediation problem,
entering at level 2.
| Level | Description |
|---|---|
| 0 | Mediation Scenario (static) |
| 1 | Mediation Scenario (adopting to changes in systems) |
| 1a | Data Mediation |
| 1b | Process Mediation |
| 2 | Simple Discovery (single invocation based on service description) |
| 2a | Shipment coverage (countries, cities semantics) |
| 2b | Shipment price and weight calculations (arithmetic) |
| 2c | Shipment constraints on pick-up time and delivery (temporal semantics) |
| 2d | Shipment unit conversion (semantics of measures) |
| 3 | Composite Discovery (multiple invocations required for complete discovery) |
| 3a | discovery 2 including request for quote and a order operation |
| 3b | discovery 3a including a request for multiple packages that has to be split |
| 3c | discovery 3b including a dynamic currency conversion |
The organizers provide a web services and problem sandbox as an open source method of building out the problem set. Holger Lausen will be the "Linus Torvalds" of this initiative.
Evaluation Criteria
There are five(5) possible levels of success that we evaluate in transitioning from one problem (sub)level to another. The first, Evaluation Success Level 0 is minimal and is automatically determined by the system. The next three levels are determined by peer review. A higher evaluation success level indicates a better solution to the problem level transition.
- Evaluation Success Level 0:The evaluation will have a minimal criterion that the participating system adequately invokes the requisite web services, measured by the legality of the messages exchanged. Some of the tests will involve error handling: details to be disclosed later.
Participants will self-declare what part of their solution is executable code and what part is "data": declarations that describe the semantics of the problem and can be easily changed to adapt to new problem levels.
- Evaluation Success Level 1: Success Level 1 will be achieved whether the code had to be changed (the compiler or interpreter would execute different instructions.)
- Evaluation Success Level 2: Success Level 2 will be achieved if only data had to be changed: no execution code had to be changed.
- Evaluation Success Level 3: Success Level 3 will be achieved if there were no change to the system at all.
"Cheating" is encouraged: participants are encouraged to make at least the data/declarations part of their systems public in the challenge sandbox. Participants are then invited to build on top of anything useful provided by anyone else.
