Criteria for evaluation and ranking of experiments
Proposals can only be submitted by:
- Parties eligible for participation in the EC Framework Programme 7 (FP7) http://cordis.europa.eu/fp7/participate_en.html
- SMEs according to the definition used by the EC
- Single parties (no consortia are allowed)
- Multiple proposals may be submitted by the same party, however in case multiple proposals are submitted reference should be made to each submitted proposal and clear indication should be given on the complementarity of the proposals.
- Parties having been selected in previous Fed4FIRE+ Open Calls are not eligible to participate again
- Proposals submitted by parties who did not complete the on-line survey are not eligible.
Evaluation and ranking will be carried out by an external jury, which is only judging this category of SME proposals. The intention is to keep the same jury over the different calls if possible. Selection will mainly be based upon:
Proposals for experiments have to demonstrate:
Criteria I.
A degree of industrial and/or scientific innovation including a motivation for the experiment. (Section B of the Proposal Template)
The score given here should reflect the degree of innovation: if an experiment is pushing the boundaries of its domain, then it should get a higher score here then experiments testing trivial things. In order to demonstrate these criteria, the proposer may opt to indicate the State of the Art in the appropriate field.
Criteria II.
A degree of industrial relevance (Section B of the Proposal Template)
This score should reflect the industrial relevance including the expected and projected impact on the SME through product development.
Criteria III.
Clarity and methodology (Section B of the Proposal Template)
The experiment should be scientifically and/or technically sound. There should be a clear problem statement, a solid experiment design, a good methodology, etc.
Criteria IV.
An appropriate scale and complexity of experiment in respect to its implementation and execution in the scope of Fed4FIRE+ and defined time frame (Section B of the Proposal Template)
Use of only a single testbed is acceptable, but multi-testbed experiments are preferred. No distinction is made between achieving this by running the same experiment in sequence on multiple testbeds (e.g. to evaluate different wireless environments), or by running a single experiment that relies on resources from different testbed at the same time. If however proposals have made their design artificially more complex than needed just in order to use multiple testbeds, then the score will be lower. Similarly, if proposals have made their designs too trivial while you can easily identify opportunities for involving other testbeds that would have made the experiment stronger, then the score will also be lower. In order to optimise the design of the experiment, the proposer should seek information on the available testbeds.
Criteria V.
Criteria V. Relevance for Fed4FIRE+ framework in terms of planned facility and tools utilization and potential feedback to the project on their usage (Section C of the Proposal Template). (Advanced) network-level experiments will get a higher score.
The Fed4FIRE+ consortium is seeking feedback regarding the available tools, procedures and testbeds. Proposals which can indicate that more information and feedback on the use of these tools and procedures will be provided will get a higher score. So the more of the Fed4FIRE+ tools and APIs that an experiment will use, the better. Ideally, an experiment will select and provision its resources through an SFA client such as the portal, jFed or omni; it will control its experiment using a FRCP client (currently OMF and NEPI are available); and it will collect the results using OML. If they need to use additional non-Fed4FIRE+ tools, that is not a problem as long as they clearly indicate the added value of these additional tools.
Criteria VI.
Indication on possible future follow-up experiments and how this can support the sustainability of the federated testbed facilities. (Section F of the Proposal Template).
The proposer may indicate possible follow-up projects and experiments which can contribute to the sustainability of the Fed4FIRE+ facilities. The quality, the size and the expected feasibility to carry out these future experiments will be reflected by the score in this criterium.
Criteria VII.
The proposer should exhibit technological expertise and quality. This information must be included in Section E of the Proposal Template.
Amongst all above listed criteria, Criteria I, II and V will be weighted higher.
The proposed experiment must be executed on the available Fed4FIRE+ testbeds. This competitive call allows for both experiments using multiple testbeds (in parallel and/or in sequence) and experiments using a single testbed Information about the current Fed4FIRE+ testbeds is available at the dedicated pages. The proposed experiment must use the experimentation tools provided by Fed4FIRE+ in order to provide feedback to the project about their usefulness and maturity in a final report. In justified cases additional external tools may be used.
