AiiDA
Automated Interactive Infrastructure and Database for Computational Science


Journal article  G. Pizzi et al. Comp. Mat. Sci. 111, 218-230 (2016)
Open access preprint  arXiv:1504.0116 [physics.comp-ph]

Posts

Tutorial on writing reproducible workflows for computational materials science

👤 🕔 January 19, 2019 Comments Off on Tutorial on writing reproducible workflows for computational materials science

Coordinates

When May 21st, 2019 (9:00) until May 24th, 2019 (13:00)

Where EPFL, Station 9, CH-1015 Lausanne, Switzerland

Registration Registration link
(as of April 15th 2019, the workshop is fully booked. You may still register if you would like to be added to the waiting list)

Materials Hands-on instructions

Schedule

Talks marked by (R) are planned to be recorded

About

This 3.5-day tutorial is designed to get Master students, PhD students and Postdocs from the field of computational materials science started with writing reproducible workflows. Participants will be introduced to the state of the art in workflow management and high-throughput computations by experts in the field, and gain in-depth hands-on experience using a tool that they can directly apply to their own research.

Our tool of choice is the AiiDA framework for workflow management and provenance tracking, which is backed by a significant community of users and developers, and has interfaces to more than 20 materials science codes (see plugin registry), including to the ab initio codes Quantum ESPRESSO, VASP, cp2k, Castep, Siesta, Fleur, Crystal, NWChem, Wannier90, and Yambo. AiiDA’s permissive open source license (MIT) enables participants to use it both in academic and commercial settings. By virtue of its general design and flexible plugin system, AiiDA is easily extended to new codes and new use cases.

  • The first day will serve as an introduction to AiiDA 1.0. In order to avoid losing time on installation issues, all participants will have the option to connect to a personal virtual machine preconfigured with AiiDA (or to come with AiiDA 1.0 already installed on their laptop).
  • The second day will cover workflows in AiiDA 1.0. Following input from the workshop questionnaire, participants will first learn the concepts using aiida-quantumespresso, and then have the option to get to know other plugins (such as aiida-vasp) in the following days.
  • In the remaining 1.5 days, participants will be able to start working on their own projects, choosing among three topics:
    • Developing a new workflow (S. Huber, G. Pizzi): 
      • includes: AiiDA workflow engine under the hood
    • Writing a plugin for your code (L. Talirz, O. Arbelaez): 
      • includes: How to write an AiiDA plugin
    • Wrapping a workflow in a web application (S. Yakutovich, C. Andersen): 
      • includes: Getting started with the AiiDA lab
  • Throughout the tutorial, lectures by the organizers and core developers of AiiDA are complemented by talks by the invited speakers highlighting the use of the workflows in computational materials science.
  • Discussion sessions, as well as a poster session (NEW Poster abstracts) and an open-mic session are scheduled to foster interaction between the participants, to discuss and answer questions.

While this tutorial is deliberately open to computational materials scientists at many levels, we do require the following  for participation:

  • familiarity with a Unix operating system and the bash shell (or equivalent)
  • familiarity with python
  • familiarity with a job scheduler (e.g. slurm, torque, sge, …)
  • full participation for (at least) the last 2.5 days

Speakers & organisers

The tutorial is organised by Leopold Talirz (EPFL, CH), Sebastiaan Huber (EPFL, CH), Espen Flage-Larsen (SINTEF, NO), Alberto García (ICMAB, ES) and Andrea Ferretti (CNR, IT) who bring expertise in a diverse set of ab initio codes and their applications in different work environments (universities, research institutes and companies).

The following invited speakers will illustrate the use workflows in their research using a recent example, highlighting both the challenges encountered and possible solutions.

  • Stefaan Cottenier (UGent, BE)
  • Marco Govoni (ANL, US)
  • Fawzi Mohamed (FHI, GE)
  • Guido Petretto (UC Louvain, BE)

Tutorial lectures and assistance during hands-on session will be provided by a team of core AiiDA developers and active plugin developers:

  • Espen Flage-Larsen (SINTEF, No)
  • Andrea Ferretti (CNR, Istituto Nanoscienze, IT)
  • Alberto Garcia (ICMAB-CSIC, ES)
  • Sebastiaan Huber (EPFL, CH)
  • Leopold Talirz (EPFL, CH)
  • Giovanni Pizzi (EPFL, CH)
  • Aliaksandr Yakutovich (EPFL, CH)
  • Casper Andersen (EPFL, CH)
  • Oscar Arbelaez (EPFL, CH)
  • Daniele Tomerini (EPFL, CH)

For generic information concerning the tutorial, contact aiida.workflows2019@gmail.com .

How to get there

Registration will take place in the 2nd floor of the MED building.

For general directions on how to arrive to EPFL, see here.

Funding

The tutorial will be offered free of charge, with roughly 40 participants. By default, participants are expected to cover their costs for travel and lodging, but we aim to reimburse the cost of lodging for participants who require financial assistance (which they need to motivate during registration).

We are very grateful to our sponsors for helping to make this event possible:

AiiDA plugins migration workshop – EPFL, 25-29 March 2019

👤 🕔 January 11, 2019 Comments Off on AiiDA plugins migration workshop – EPFL, 25-29 March 2019

An AiiDA plugin migration workshop will be held at EPFL Lausanne, Switzerland, aiming at collecting about 20 participants.

The workshop will start on Monday 25th March at 2PM and end on Friday 29th March at 1PM.

We are in the process of preparing the AiiDA code towards the 1.0 release, in which we have recently introduced python 2 + python 3 support (already available in the 1.0 alpha releases). While we strive to always maintain back-compatibility, we have realised that there were a few needed improvements to the API and we felt that the 1.0 release was the right moment to introduce them. (Note: while existing “codebases” will need (often very straightforward) migration-which is
what this workshop is for-existing “databases” will be fully compatible and be automatically migrated when users upgrade to AiiDA 1.0).

The aim of this workshop is to directly support AiiDA plugin developers in the migration of their plugins to support both python 2 and python 3, as well as the changes introduced in AiiDA 1.0.

This workshop is focused on migrating existing plugins (a full list of the plugins supporting almost 60 different codes can be found on the AiiDA plugin registry page). Another tutorial for new developers of plugins and workflows will be held in Lausanne in the week 20-24 May 2019 and will be advertised soon (stay tuned!).

We will also reserve some time for discussions on plugin interfaces and their homogenisation and common APIs, exploiting this unique occasion bringing together many plugin developers.

Main topics

  • explanation of the changes on 1.0;
  • hands-on workshop on porting plugins to py2+3 and new aiida 1.0;
  • discussions on common interfaces to different plugins for common functionalities (e.g. crystal structure relaxation, band structure, …);
  • discussion of automated plugin testing against different versions of AiiDA, python, …

Program
Every day there will be an informal coffee break from 11:00 – 11:15 and 16:00 – 16:15

Monday 25th
2:00 – 2:15: Registration
2:15 – 2:20: Introductory talk
2:20 – 3:20: What’s new in v1.0.0 + backwards incompatible changes (Sebastiaan)
3:20 – 4:00: Talk on python2/python3 compatibility
4:00 – 4:30: Coffee break
4:30 – 4:45: Plugin migration – step by step instructions
4:45 – 6:00: Hands-on plugin migration workshop

Tuesday 26th
09:00 – 09:45: Tutorial on plugin versioning and git-flow model of releasing
09:45 – 11:00: Hands-on plugin migration workshop
13:00 – 14:00: Lunch
14:00 – 18:00: Hands-on plugin migration workshop

Wednesday 27th
09:00 – 13:00: Hands-on plugin migration workshop
13:00 – 14:00: Lunch
14:00 – 14:45: Talk on new workflow and engine features
14:45 – 18:00: Hands-on plugin migration workshop
19:00 – 22:00: Social dinner (place to be announced)

Thursday 28th
09:00 – 13:00: Open discussion on plugin common guidelines and common workflow interfaces
13:00 – 14:00: Lunch
14:00 – 18:00: Hands-on plugin migration workshop

Friday 29th
09:00 – 13:00: Hands-on plugin migration workshop

Support

This workshop is supported by the NCCR MARVEL funded by the Swiss National Science Foundation and by the European H2020 MaX Centre of Excellence.

Report on the AiiDA tutorial held at CINECA, 30 May-1 June 2018

👤 🕔 June 4, 2018 Comments Off on Report on the AiiDA tutorial held at CINECA, 30 May-1 June 2018

Another AiiDA tutorial, sponsored by PRACE and the H2020 MaX Centre of Excellence, was organized in collaboration with the Italian CINECA supercomputer centre, who hosted the event.
The tutorial took place between May 29 and June 1 2018, at CINECA Bologna (Italy).
More than twenty participants, with a strong female representation, joined in this introductory tutorial to the AiiDA framework.

Group photo from the AiiDA tutorial in May 2018

On the first day, an introductory talk was delivered by Spyros Zoupanos about the general architecture and design of AiiDA, after which the participants started with the hands-on exercises to get to know with AiiDA’s interface.
A more detailed presentation on the workflow system was given by Martin Uhrin.

Introduction by the organisers of the AiiDA tutorial (May 2018)

On the second and third day, Giovanni Pizzi and Leonid Kahle gave an exposé on their research projects that actually implemented and used the AiiDA platform to discover new two-dimensional materials and solid state ionic conductors in a high-throughput way, respectively.
This gave a good idea to the tutorial participants of the potential of AiiDA in a broader sense.

Room with participants at the AiiDA tutorial

The opportunity was also used to introduce participants to the concept of open source collaboration – in particular, Sebastiaan Huber gave a crash course on the use of Git and GitHub to contribute to open source projects (branches, forks and pull requests).

During the tutorial, a poster session was also organized, where the participants had the chance to present their own work and get to know that of their colleagues, in an informal setting with drinks and fingerfood.

Poster session at the AiiDA tutorial

In between these extra events, the main focus lay on introducing the participants to AiiDA through a hands-on tutorial and exercises, with all the instructors available to provide support and explanation in person.
The participants were very committed during the tutorial and already provided comments to improve the tutorial during the event.

Instructors helping out participants at the tutorial

At the end of the tutorial, all participants were asked to give feedback by completing an evaluation questionnaire, which will be used to further improve the tutorial and AiiDA itself.
The feedback was overall very positive, with the tutorial graded with an average vote of 9.0 out of 10, and all votes between 8 and 10.
All participants stated that the well-prepared and helpful instructors helped the participants to get really well acquainted with AiiDA and that it may very well help them with their own work and research.
The scientific talks that gave examples of AiiDA in real-world scientific projects were also well received as they gave a more intuitive example of the power and potential of AiiDA.

You can find more photos of the event on our Facebook page.

We acknowledge support from PRACE and from there H2020 MaX Centre of Excellence.

 

Report on AiiDA tutorial: 24-25 January 2017 at ICTP in Trieste, Italy.

👤 🕔 January 27, 2017 Comments Off on Report on AiiDA tutorial: 24-25 January 2017 at ICTP in Trieste, Italy.

The end of January of 2017 saw the eighth hands-on tutorial for AiiDA within the “Advanced Workshop on High-Performance & High-Throughput Materials Simulations using Quantum ESPRESSO” organised at and by ICTP in Trieste. The tutorial was kicked off by an introductory lecture by Giovanni Pizzi (EPFL), explaining the philosophy and architecture behind AiiDA, and how the ADES model facilitates and simplifies performing and sharing calculations, under the umbrella of complete data provenance. After the scene was set, the approximately 75 participants went to their workstations to start with the first of the hands-on sessions. A completely configured virtual machine was provided for each individual participant, to which they only had to login and they were on their way. The first session served to familiarize the users with Verdi, a command line interface which acts as the user’s tool to interact with AiiDA. With the tools in hand to put AiiDA to good use, the participants learned by example how a Quantum Espresso calculation can be easily launched as AiiDA takes care of a lot of the repetitive work behind the scenes.

The virtual environment in which the participants were working came with a database filled with pre-computed calculations, which allowed the aptitude of AiiDA in working with big data sets to be displayed effortlessly. The QueryBuilder, the tool provided by AiiDA to efficiently query a user’s collection of calculations, was introduced and a step-by-step tutorial showed the participants how it could be used to efficiently analyze a large collection of calculations on perovskites. The final session detailed how the users could easily create such a big collection of calculations themselves, in a high-throughput manner, through the concept of Workflows.

Throughout the tutorial, five members of the AiiDA team were present to answer any questions of the participants and give introductory talks at the beginning of a new hands-on session. After the hands-on sessions, a good portion of the participants gave their feedback on the quality of the tutorial and the impact on their skills in using AiiDA. The results are very promising while at the same time they provide the AiiDA team with valuable information on how to improve both the tutorial and AiiDA itself.

The feedback on the quality of the tutorial itself, in terms of content and planning, were overall very positive

 

 

AiiDA tutorial Jan 2017 feedback 2

Participant feedback on the quality of the tutorial

The results of the feedback also show that the tutorial really helped improve the skill and knowledge of the participants regarding the use of AiiDA

 

AiiDA tutorial Jan 2017 feedback 1

The skill level of the participants before and after the tutorial

The AiiDA team thanks the organisers of the event for making this tutorial possible and all the participants for their enthusiasm. Until the next AiiDA event!

Report of the AiiDA tutorial of last year (June 2016) in Lausanne

👤 🕔 January 19, 2017 Comments Off on Report of the AiiDA tutorial of last year (June 2016) in Lausanne

Report from the AiiDA coding week – Dec 2016

👤 🕔 December 13, 2016 Comments Off on Report from the AiiDA coding week – Dec 2016

Last week (5-9 Dec 2016) we had a coding week for AiiDA, to which 15 people participated (roughly half of them core developers from the AiiDA, and the other half were developers who were interested in contributing to the platform).

The coding week has taken place in Leysin, in the Swiss Alps. A few discussion or work topics had been identified before the start by the AiiDA team, and have been addressed during the week. In particular, the following topics have been discussed and the result has been summarised in a draft document for later implementation:

  • New more efficient implementation of the AiiDA daemon
  • How third-parties can contribute to Materials Cloud sections
  • Redesign of the ORM classes (in particular Node) to improve the way multiple backends are supported

A few discussions have also led to new branches of development, that will be soon merged into the develop branch (and appear in a future release):

  • AiiDA plugin interface integrated with pip
  • Various improvements and bug fixes to the SQLAlchemy backend implementation

Finally, the remaining work topics have been taking up a large part of the coding sessions, and most of the code has already been merged into the develop branch:

  • Redesign of the AiiDA documentation, with a more clear structure, removal of duplicates, a new and more clear installation section.
  • Simplified procedure to install AiiDA.
  • New testing infrastructure to test in parallel any database backend (currently, both Django and SQLAlchemy)
  • Removal of the DbPath (transitive closure) table and triggers, and replacement with a dynamically created query (using either PostgreSQL recursive queries or PostgreSQL functions). The performance of these queries has been optimised.

Beside programming, most participants have taken place to an outdoor hiking activity on Wednesday morning to Berneuse, close to Tour d’Aï, that was a great team building opportunity. Participants have also enjoyed after-dinner sessions with board and card games.

The outcome of the feedback, summarised below, has been very positive, both in terms of participation, organisation, and improvement to the code. The results indicate also that similar events should be organised again in the future, with a frequency of about twice a year.

Some photos of the event can be found on our Facebook page.

— Giovanni Pizzi

Summary of the results of the feedback form

Results of the feedback form for the AiiDA Coding week (Dec 2016)