Measurement-based research: methodology, experiments and tools
This full-day workshop aims at gathering all the FIRE projects under the measurement methodology, experiments, and tools thematic. Participants are invited to present their current needs and developments on measurements and associated tools in the context of experimental research. Measurements and measurement tools will be key elements in the operation and management of future network infrastructures, at the equipment and network performance monitoring level but also in support of higher-level control functionality such as on-line analysis and diagnostic. These tools also play a fundamental role in measurement-based experimental research relying on the experimental evaluation and benchmarking of project outcomes including protocols, systems, etc. by means of reliable and verifiable tools.
This workshop consists of 11 talks from 9 different FIRE European projects and a final round table. Speakers will present their current needs and developments on measurements and associated tools in the context of experimental research. Four talks present and discuss specific "measurement and tools" needs in FIRE projects. Presentations discuss tools needs in wireless/sensor networks, informatic-centric networking, large scale experimental facilities, and openflow-enabled networks. Six talks present tools that are currently used and/or under development and try to bridge gaps (how reusable some of these tools are, can they be extended/enhanced, what are their properties and capabilities with respect to other similar tools, etc.). The important point here is more about the potential of the tool(s) than their kinetic. Presentations discuss available tools to measure heterogeneous metrics in wireless/sensor networks, large scale experimental facilities, openflow-enabled networks, and Internet.
We are pleased to announce the publication of the revised papers as a post-proceedings in "Measurement methodology and tools (First European Workshop, FP7 FIRE/EULER Project, May 9, 2012, Aalborg, Denmark, Revised and extended papers)", editors Lluís Fŕbrega, Pere Vilŕ, Davide Careglio, Dimitri Papadimitriou, Lecture Notes in Computer Science (LNCS), vol. 7586, 2013, Springer-Verlag Berlin Heidelberg, ISSN 0302-9743, ISBN 978-3-642-41295-0, DOI 10.1007/978-3-642-41296-7 (online).
| Dates | May 9, 2012 |
| Co-located with | Future Internet week |
| Location | Aalborg, Denmark |
| Direction | Aalborg Congress & Culture Center |
| FIRE Thematic workshops | Agenda (FIRE Wiki) |
| Minutes | Minutes |
How to participate
A registration to the workshop is required by local organizers to reserve the rooms of the conference center. The registration process is available at this link.
http://aalborg-convention.dk/reg/reg.php?conf=FIA®step=start>
To participate to the Workshop, please mark the following items
- Registration fee for Future Internet Week DKK 375 / EUR 50EULER09:00 - 12:30 (morning)EULER13:30 - 17:00 (afternoon)
The fee for this Workshop is 50 euros to cover the cost of lunch and coffee breaks.
This fee includes the participation to the Workshop, to the hands-on FIRE demo session (Thursday 10) and to the events scheduled on May 7 and 8. It does not include the participation to the events scheduled on May 10 and 11.
Structure
The workshop will consist of four different
- Session 1: Introduction and motivation of the workshop thematic.
- Session 2: Measurement and tools needs in FIRE.
- Session 3: Presentation of tools developed in FIRE.
- Session 4: Round table discussion (how to bridge the gap).
Expected outcome(s):
- Identify what can be performed/reached by means of cooperation between projects from a directory of tools accessible to the FIRE community at large up to the joint development of tools, under which conditions, etc.
- Determine needs and document best practices in tools development for measurement-based experimental research
Agenda
9.20-9.30: Welcome
9.30-10.00: Session 1: Introduction talk: Measurement-based experimental methodology: matching measurements objectives, properties, and criteria with measurement tools
This introductory talk approaches the questions of what is meant by measurement in the experimental research context, what are the expected properties of measurements and associated tools, how to realize scientific and technical objectives by means of measurement-based experimentation (up to which extend a measurement tool can be designed in particular if the measurement model is not completely known, etc.).
Speaker: Dimitri Papadimitriou, dimitri.papadimitriou@alcatel-lucent.com
10.00-12.30: Morning session: Tools needs and tools developed in FIRE
Session chair: Josep L. Marzo, joseluis.marzo@udg.edu
- HOBNET: Experimental performance evaluation of sensor-based networking for energy efficiency in smart buildings
Speaker: Sotiris Nikoletseas, nikole@cti.gr
Project: http://www.hobnet-project.eu
This talk presents experimentation methodologies, test-beds and tools developed in the context of the FIRE/HOBNET project for smart/green buildings via IPv6 sensor networking. The problems addressed relate to key distributed computing primitives (such as data propagation, tracking) as well as specialized application commissioning (such as garden watering, CO2 monitoring). Towards better convergence and integration of collected measurements a REST architecture is adopted. Also, for experimental research, repositories of key networking components (reference topologies, traffic patterns, energy models, mobility profiles, protocol families and performance metrics and trade-offs) are developed.
10.30-11.00: Coffee break
- CREW: Metrics and measurement tools for assessing the channel condition in a wireless experimentation environment
Speaker: Ingrid Moerman, ingrid.moerman@intec.UGent.be
Project: http://www.crew-project.eu/
Novel wireless solutions, in particular new cognitive radio and networking concepts, require a rigorous experimental validation prior to uptake in wireless standards and commercial products. Wireless experimentation is very challenging, as experiments may be impaired by unwanted/uncontrolled interfering signals in the wireless environment. Within CREW a benchmarking methodology is developed to support the experimenter in getting more reliable and comparable results when executing their experiments. In this talk we will focus on a method for measuring the channel occupation during a wireless experiment. First we will explain how we can measure and score the channel condition in a wireless experimentation environment. Next we will show that the metrics for the channel condition can be used at the same time (1) to validate the performance of a cognitive wireless solutions in ISM bands and (2) to improve the efficiency of wireless experiments. - LAWA: Methods and tools for temporal Web analytics
Speaker: Marc Spaniol, mspaniol@mpi-inf.mpg.de
Project: http://www.lawa-project.eu/
The LAWA project develops methods and tools for temporal Web analytics. The focus of developments are semantic and structural analytics of time-versioned textual Web contents. In particular, we are developing methods that enable entity detection and tracking along the time axis as well as temporal studies of large (Web) graphs. To this end, we also prepare a reference data set and will provide analytics services. - CONVERGENCE: Metrics and measurement tools needs in Information-Centric Networking and CONVERGENCE in particular
Speaker: Nicola Blefari Melazzi, blefari@uniroma2.it
Project: http://www.ict-convergence.eu/
This talk first introduces Information-Centric Networking (ICN), a new paradigm in which the network layer provides users with contents, instead of providing communication channels between hosts, and is aware of content identifiers. Then it presents the needs on measurements required by this approach in general and by the project CONVERGENCE in particular. ICN is significantly different with respect to the current networking architecture, and poses several new requirements to measurements, which have to be performed both in the current network (to understand some of its aspects useful for the design of ICN) and possibly in the new one.
12.30-13.30: Lunch break
13.30-16.20: Afternoon session: Tools needs and tools developed in FIRE
Session chair: Davide Careglio, careglio@ac.upc.edu
- OFELIA: metrics and measurement tools needs in openflow and OFELIA in particular
Speaker: Hagen Woesner, hagen.woesner@eict.de
Project: http://www.fp7-ofelia.eu/
The core idea of OpenFlow and in a wider sense the new paradigm of 'software defined networks' is the separation of control, forwarding and processing of data. Performance evaluation of such networks has to look into details of switch internals that were not visible before, but also requires different tools for measurement. This talk introduces the available tools in the OFELIA testbed and demonstrates live access to a traffic generator/testing environment available inside the federation of OpenFlow-enabled campus islands that OFELIA is growing. - OpenLab: Delivers control and experimental plane middleware to facilitate early use of its large scale shared experimental facility by researchers
Speaker: Anastasius Gavras, gavras@eurescom.eu
Project: http://www.ict-openlab.eu/
OpenLab delivers control and experimental plane middleware to facilitate early use of its large scale shared experimental facility by future Internet researchers. Interoperability of testbeds at several levels is the main challenge of the technical work of the project. A key objective is to provide experimenters the testbed environments they need, including the tools to manage their experiments and measurement results. OpenLab has conducted an open call for experiments and is currently integrating the accepted experiments according to the capabilities of the testbeds and is ensuring the availability of the tools to run the experiments and assess the results. This presentation will focus in parts on the different requirements for measurements and measurement tools to support the researchers in their work. - EULER: Metrics and measurement tools for distributed and adaptive routing algorithms
Speaker: Dimitri Papadimitriou, dimitri.papadimitriou@alcatel-lucent.com
Additional talks/demos: Simulating Routing Models on Large-Scale Topologies , Aurélien Lancin, Aurelien.Lancin@inria.fr;Impact of power-law topology on IP-level routing dynamics: simulation results , Amélie Medem Kuatse, medem.amelie@gmail.com; Distributed UDP Ping , Elie Rotenberg, elie.rotenberg@gmail.com
Project: http://www.euler-fire-project.eu/
In the ICT domain, measurement is classically oriented to the verification process of the designed function set and their expectedly met performance objectives by the experimental corpus being an engineered artifact, a prototype, etc. When the scale of the real environment becomes difficult to reproduce (the Internet being the best examples) and the number of states together with their dynamics are themselves difficult to model ("how to model and quantify uncertainty"), a new approach shall be considered: simplify the experimental corpus (introduce abstraction) while maintaining its functionality and reproduce significant environmental phenomena derived from measurements of the actual environment. The EULER project in its investigation of adaptive and distributed routing models - alternative to policy-based path-vector routing - is confronted to this situation: there is no Internet scale facility where routing schemes can be experimented before being deployed. In this context, this talk will detail the proposed methodology, the measurement tools being developed in order to derive representative environmental conditions, and the verification means that have been elaborated to enable wide-scale experiments.
Additional materials: DRMsim-leaflet , DRMsim-doc
14.50-15.20: Coffee break
- OpenLab: Measurements and measurement tools in OpenLab
Speaker: Javier Aracil, javier.aracil@uam.es
Project: http://www.ict-openlab.eu/
This talk provides an overview of the measurement tools developed within openlab, with spaciel emphasis in the integration of several heterogeneous data formats within a common user interface. The talk summarizes the main concerns when dealing with many different measurement tools for the different testbeds present in OpenLab. As a possible solution, the use of ontologies provides an umbrella under which federation of the measurement tools becomes feasible. In this regard, the talk presents the measurements ontology developed within MOMENT and OpenLab, along with the standardization efforts. - NOVI: NOVI's experience in monitoring tools and measurements
Speaker: Józsi Stéger, stegerjozsef@caesar.elte.hu
Project: http://www.fp7-novi.eu/
The FP7 STREP project NOVI develops control, management and monitoring planes in order to allow the federation of various test beds, consisting of heterogeneous virtual resources. Monitoring and measurements of federated testbeds is of primary importance not only to allow other service components to make intelligent decisions, but it is also very valuable for the end-users. This talk focuses on the monitoring service component of NOVI, and it will be shown that with the use of monitoring ontology a wide range of monitoring tools, metrics and databases can be easily integrated in the NOVI monitoring framework. - CONECT: NITOS - Methods and measurement tools for experimentation on wireless testbeds
Speaker: Thanasis Korakis, nasoskor@gmail.com
Project: http://www.conect-ict.eu/
NITOS is a publicly accessible wireless testbed featuring WiFi-enabled nodes, USRP boards, cameras and various types of sensors, while it is currently also being extended with WiMAX, LTE and 3G femtocell components. In the EU project CONECT, NITOS is widely used for the implementation and experimentation evaluation of packet level cooperative approaches, from the MAC to the application layer. To cope with the heterogeneous set of measurements in these experimentation efforts, we use the OML measurement framework adopted from the EU project Openlab. OML is based on customizable measurement points inside applications running on the resources and provides a well-structured solution for capturing, processing, storing and visualizing measurements. Experimentation on cooperative networks, taking place in CONECT, exploits this framework, in order to benchmark different architectures tested on NITOS. The existence of USRP boards with spectrum sensing capabilities offers the capability of capturing the state of the wireless medium during an experiment and is extremely important for understanding the dynamics of cooperative networks implementation.
16.20-17.00: Session 4: Round table (40min)
A round table discussion about what can be performed/ reached by the projects together (from a directory of tools accessible to the FIRE community until joint tools developments), under which conditions, etc. (introductory questions)
Speakers: Javier Aracil, Thanasis Korakis, Dimitri Papadimitriou, Davide Careglio, Nicola Blefari Melazzi
17.00-17.15: Concluding session (15min)
Wrapping-up results of discussions and define possible action points and future activities
For additional information see FIRE wiki at http://bit.ly/HbAxLW and the workshop leaflet .