AiiDA
Automated Interactive Infrastructure and Database for Computational Science


Journal article  G. Pizzi et al. Comp. Mat. Sci. 111, 218-230 (2016)
Open access preprint  arXiv:1504.0116 [physics.comp-ph]

Posts

AiiDA project & Google Season of Docs

👤 🕔 April 22, 2019 Comments Off on AiiDA project & Google Season of Docs


The AiiDA project is happy to announce its application to participate in this year’s Google Season of Docs (GSoD). GSoD tries to bring open source projects and technical writers together: To give technical writers experience in contributing to open source projects and help opens source projects to improve their documentation and interact with the technical writing community.

What happens next

See also the full timeline:

  • April 30, 2019 Google publishes the list of accepted mentoring organizations
  • May 29, 2019 Start of technical writer application period. More details about the application process can be found in the technical writer guide, where also the application form will be published
  • June 28, 2019 End of technical writer application period

Prospective technical writers

Are you a technical writer with a background in (or affinity to) computational materials science, physics or chemistry? Would you be interested in boosting a tool that lets researchers manage thousands of simultaneous calculations on supercomputers and helps them work in an open-science environment, making their research reproducible and FAIR? Then please read on!

If you would like to discuss your application or project ideas, please feel free to contact Kevin or any of the other coordinators and prospective mentors:

What is AiiDA?

Simulations and high-throughput computational screenings have emerged as indispensable tools in materials’ science. The important impact of quantum simulations on science in general is underscored by the fact that 12 of the 100 most cited papers of the entire scientific literature deal with simulations based on density functional-theory .

Computational materials science involves screening thousands to hundreds of thousands of materials for key figures of merit. Running thousands of calculations is a nontrivial task in itself and calls for an infrastructure to help orchestrate the calculations and organize the output data, while also ensuring that others can easily reproduce the work. To this aim, there are two important requirements for such an infrastructure: First, it must be flexible enough to support a large variety of simulation workflows; second, it should require minimal overhead compared to the ‘conventional’ way of running simulations.

AiiDA is a python package that is built to make this possible: while automating simulations on supercomputers, it stores the full provenance of data and calculations in a directed graph (see Figure 1c) that can be easily queried using a high-level interface. Moreover, it provides a workflow engine that make it possible to automate and run complex simulations with simple, small snippets of code by using predefined ‘turn-key’ workflows (see Figure 1). Currently, 34 plugins provide support different codes and provide over 50 workflows to compute basic and advanced materials properties.

AiiDA workflows
Figure 1: Complex computational workflow for the Aluminum band structure (a) started using a turnkey workflow as shown in (b). The resulting computational graph (c) that is automatically stored in the AiiDA database. Source: Pizzi, G., Togo, A., & Kozinsky, B. (2018). Provenance, workflows, and crystallographic tools in materials science: AiiDA, spglib, and seekpath. MRS Bulletin, 43(9), 696-702. doi:10.1557/mrs.2018.203

Project ideas

AiiDA is documented on ReadTheDocs and also offers a variety of tutorial material from university classes, schools and workshops (https://aiidateam.github.io/aiida-tutorials/ , https://github.com/aiidateam/aiida_demos, https://github.com/aiidateam/aiida-max-examples, https://aiida-core.readthedocs.io/en/stable/tutorial/index.html) most of which were written for AiiDA 0.x.

Following several alpha and beta versions, 2019 will see the release of AiiDA version 1.0. AiiDA 1.0 brings a number of new features, a redesign of the provenance model as well as of many AiiDA internals. The projects for the 2019 GSoD will focus on updating the available documentation and tutorial material to reflect the changes which were introduced in AiiDA 1.0.

Note: These are project ideas. We highly welcome your suggestions on how to make the documentation more effective as well as how to best accommodate your talents as a technical writer.

AiiDA Tutorials: Teaching AiiDA

Recently, a new major release of AiiDA changed several parts of the API wherefore existing tutorials (https://aiidateam.github.io/aiida-tutorials/ , https://github.com/aiidateam/aiida_demos, https://github.com/aiidateam/aiida-max-examples, https://aiida-core.readthedocs.io/en/stable/tutorial/index.html) need to be updated to reflect these changes.

Moreover, from the perspective of a new user, it would be probably easier to find all existing tutorials in a consistent and updated form in one place (e.g. a with tag with the versions on which the tutorials were tested, see also http://matgenb.materialsvirtuallab.org/). On top of that, several codes that AiIDA supports are not represented with a tutorial — it would be great to add new jupyter notebooks, ideally on MyBinder, showcasing how to use AiiDA with those plugins.
We can suggest plugins which are well developed and contain advanced features (and for which we know the authors) and can help the technical writers in prioritizing.
The technical writer could then test examples from the plugin repositories and create a tutorial, maybe even in form of a screencast (some videos can already be found on the Materials Cloud), explaining a common workflow using AiiDA and one of the many AiiDA plugins.
Existing videos and screencasts could be updated, or at least annotated, to reflect the changes with the new major release. An ambitious technical writer might even set out to develop a online course teaching the basics of AiiDA (writing workflows, querying the database) in the form of a MOOC. An bit less ambitious goal would be to bring existing tutorials on our youtube channel into a consistent format.

AiiDA docs

The AiiDA documentation, written in Sphinx and hosted on ReadTheDocs, is our main effort to document the usage and development of AiiDA. A technical writer could contribute in several ways to this documentation: First, the complete documentation needs to be checked for compatibility with the AiiDA 1.0 release. Second, the documentation can still be extended and refactored to improve the user experience. Sometimes, we encounter issues that are not documented or are not clear enough, or we realise that the organisation of the documentation does not make it easy to find the section that is being searched for. Hence, it would be great to check for frequent issues on the mailing list and include those in the documentation.

AiiDA web site

We feel that our web page could to be updated to better highlight the main features of AiiDA. Currently, the feature section is hidden and should be updated to reflect the feature set of the most recent release. A technical writer could help developing this section and writing a ‘why use AiiDA’ section, ideally supported by appealing visualizations.
We would welcome interactive feature demonstrations or a carousel with examples in which AiiDA was used. This could include links to the databases on the Materials Cloud archive or even interviews with researchers, discussing how they used AiiDA in their research and publication process.

Tutorial on writing reproducible workflows for computational materials science

👤 🕔 January 19, 2019 Comments Off on Tutorial on writing reproducible workflows for computational materials science

Coordinates

When May 21st, 2019 (9:00) until May 24th, 2019 (13:00)

Where EPFL, Station 9, CH-1015 Lausanne, Switzerland

Registration Registration link
(as of April 15th 2019, the workshop is fully booked. You may still register if you would like to be added to the waiting list)

Materials Hands-on instructions

Schedule

Talks marked by (R) are planned to be recorded

About

This 3.5-day tutorial is designed to get Master students, PhD students and Postdocs from the field of computational materials science started with writing reproducible workflows. Participants will be introduced to the state of the art in workflow management and high-throughput computations by experts in the field, and gain in-depth hands-on experience using a tool that they can directly apply to their own research.

Our tool of choice is the AiiDA framework for workflow management and provenance tracking, which is backed by a significant community of users and developers, and has interfaces to more than 20 materials science codes (see plugin registry), including to the ab initio codes Quantum ESPRESSO, VASP, cp2k, Castep, Siesta, Fleur, Crystal, NWChem, Wannier90, and Yambo. AiiDA’s permissive open source license (MIT) enables participants to use it both in academic and commercial settings. By virtue of its general design and flexible plugin system, AiiDA is easily extended to new codes and new use cases.

  • The first day will serve as an introduction to AiiDA 1.0. In order to avoid losing time on installation issues, all participants will have the option to connect to a personal virtual machine preconfigured with AiiDA (or to come with AiiDA 1.0 already installed on their laptop).
  • The second day will cover workflows in AiiDA 1.0. Following input from the workshop questionnaire, participants will first learn the concepts using aiida-quantumespresso, and then have the option to get to know other plugins (such as aiida-vasp) in the following days.
  • In the remaining 1.5 days, participants will be able to start working on their own projects, choosing among three topics:
    • Developing a new workflow (S. Huber, G. Pizzi): 
      • includes: AiiDA workflow engine under the hood
    • Writing a plugin for your code (L. Talirz, O. Arbelaez): 
      • includes: How to write an AiiDA plugin
    • Wrapping a workflow in a web application (S. Yakutovich, C. Andersen): 
      • includes: Getting started with the AiiDA lab
  • Throughout the tutorial, lectures by the organizers and core developers of AiiDA are complemented by talks by the invited speakers highlighting the use of the workflows in computational materials science.

  • Discussion sessions, as well as a poster session (NEW Poster abstracts) and an open-mic session are scheduled to foster interaction between the participants, to discuss and answer questions.

While this tutorial is deliberately open to computational materials scientists at many levels, we do require the following  for participation:

  • familiarity with a Unix operating system and the bash shell (or equivalent)
  • familiarity with python
  • familiarity with a job scheduler (e.g. slurm, torque, sge, …)
  • full participation for (at least) the last 2.5 days

Speakers & organisers

The tutorial is organised by Leopold Talirz (EPFL, CH), Sebastiaan Huber (EPFL, CH), Espen Flage-Larsen (SINTEF, NO), Alberto García (ICMAB, ES) and Andrea Ferretti (CNR, IT) who bring expertise in a diverse set of ab initio codes and their applications in different work environments (universities, research institutes and companies).

The following invited speakers will illustrate the use workflows in their research using a recent example, highlighting both the challenges encountered and possible solutions.

  • Stefaan Cottenier (UGent, BE)
  • Marco Govoni (ANL, US)
  • Fawzi Mohamed (FHI, GE)
  • Guido Petretto (UC Louvain, BE)

Tutorial lectures and assistance during hands-on session will be provided by a team of core AiiDA developers and active plugin developers:

  • Espen Flage-Larsen (SINTEF, No)
  • Andrea Ferretti (CNR, Istituto Nanoscienze, IT)
  • Alberto Garcia (ICMAB-CSIC, ES)
  • Sebastiaan Huber (EPFL, CH)
  • Leopold Talirz (EPFL, CH)
  • Giovanni Pizzi (EPFL, CH)
  • Aliaksandr Yakutovich (EPFL, CH)
  • Casper Andersen (EPFL, CH)
  • Oscar Arbelaez (EPFL, CH)
  • Daniele Tomerini (EPFL, CH)

For generic information concerning the tutorial, contact aiida.workflows2019@gmail.com .

How to get there

Registration will take place in the 2nd floor of the MED building.

For general directions on how to arrive to EPFL, see here.

Funding

The tutorial will be offered free of charge, with roughly 40 participants. By default, participants are expected to cover their costs for travel and lodging, but we aim to reimburse the cost of lodging for participants who require financial assistance (which they need to motivate during registration).

We are very grateful to our sponsors for helping to make this event possible:

AiiDA plugins migration workshop – EPFL, 25-29 March 2019

👤 🕔 January 11, 2019 Comments Off on AiiDA plugins migration workshop – EPFL, 25-29 March 2019

An AiiDA plugin migration workshop will be held at EPFL Lausanne, Switzerland, aiming at collecting about 20 participants.

The workshop will start on Monday 25th March at 2PM and end on Friday 29th March at 1PM.

We are in the process of preparing the AiiDA code towards the 1.0 release, in which we have recently introduced python 2 + python 3 support (already available in the 1.0 alpha releases). While we strive to always maintain back-compatibility, we have realised that there were a few needed improvements to the API and we felt that the 1.0 release was the right moment to introduce them. (Note: while existing “codebases” will need (often very straightforward) migration-which is
what this workshop is for-existing “databases” will be fully compatible and be automatically migrated when users upgrade to AiiDA 1.0).

The aim of this workshop is to directly support AiiDA plugin developers in the migration of their plugins to support both python 2 and python 3, as well as the changes introduced in AiiDA 1.0.

This workshop is focused on migrating existing plugins (a full list of the plugins supporting almost 60 different codes can be found on the AiiDA plugin registry page). Another tutorial for new developers of plugins and workflows will be held in Lausanne in the week 20-24 May 2019 and will be advertised soon (stay tuned!).

We will also reserve some time for discussions on plugin interfaces and their homogenisation and common APIs, exploiting this unique occasion bringing together many plugin developers.

Main topics

  • explanation of the changes on 1.0;
  • hands-on workshop on porting plugins to py2+3 and new aiida 1.0;
  • discussions on common interfaces to different plugins for common functionalities (e.g. crystal structure relaxation, band structure, …);
  • discussion of automated plugin testing against different versions of AiiDA, python, …

Program
Every day there will be an informal coffee break from 11:00 – 11:15 and 16:00 – 16:15

Monday 25th
2:00 – 2:15: Registration
2:15 – 2:20: Introductory talk
2:20 – 3:20: What’s new in v1.0.0 + backwards incompatible changes (Sebastiaan)
3:20 – 4:00: Talk on python2/python3 compatibility
4:00 – 4:30: Coffee break
4:30 – 4:45: Plugin migration – step by step instructions
4:45 – 6:00: Hands-on plugin migration workshop

Tuesday 26th
09:00 – 09:45: Tutorial on plugin versioning and git-flow model of releasing
09:45 – 11:00: Hands-on plugin migration workshop
13:00 – 14:00: Lunch
14:00 – 18:00: Hands-on plugin migration workshop

Wednesday 27th
09:00 – 13:00: Hands-on plugin migration workshop
13:00 – 14:00: Lunch
14:00 – 14:45: Talk on new workflow and engine features
14:45 – 18:00: Hands-on plugin migration workshop
19:00 – 22:00: Social dinner (place to be announced)

Thursday 28th
09:00 – 13:00: Open discussion on plugin common guidelines and common workflow interfaces
13:00 – 14:00: Lunch
14:00 – 18:00: Hands-on plugin migration workshop

Friday 29th
09:00 – 13:00: Hands-on plugin migration workshop

Support

This workshop is supported by the NCCR MARVEL funded by the Swiss National Science Foundation and by the European H2020 MaX Centre of Excellence.

Report on the AiiDA tutorial held at CINECA, 30 May-1 June 2018

👤 🕔 June 4, 2018 Comments Off on Report on the AiiDA tutorial held at CINECA, 30 May-1 June 2018

Another AiiDA tutorial, sponsored by PRACE and the H2020 MaX Centre of Excellence, was organized in collaboration with the Italian CINECA supercomputer centre, who hosted the event.
The tutorial took place between May 29 and June 1 2018, at CINECA Bologna (Italy).
More than twenty participants, with a strong female representation, joined in this introductory tutorial to the AiiDA framework.

Group photo from the AiiDA tutorial in May 2018

On the first day, an introductory talk was delivered by Spyros Zoupanos about the general architecture and design of AiiDA, after which the participants started with the hands-on exercises to get to know with AiiDA’s interface.
A more detailed presentation on the workflow system was given by Martin Uhrin.

Introduction by the organisers of the AiiDA tutorial (May 2018)

On the second and third day, Giovanni Pizzi and Leonid Kahle gave an exposé on their research projects that actually implemented and used the AiiDA platform to discover new two-dimensional materials and solid state ionic conductors in a high-throughput way, respectively.
This gave a good idea to the tutorial participants of the potential of AiiDA in a broader sense.

Room with participants at the AiiDA tutorial

The opportunity was also used to introduce participants to the concept of open source collaboration – in particular, Sebastiaan Huber gave a crash course on the use of Git and GitHub to contribute to open source projects (branches, forks and pull requests).

During the tutorial, a poster session was also organized, where the participants had the chance to present their own work and get to know that of their colleagues, in an informal setting with drinks and fingerfood.

Poster session at the AiiDA tutorial

In between these extra events, the main focus lay on introducing the participants to AiiDA through a hands-on tutorial and exercises, with all the instructors available to provide support and explanation in person.
The participants were very committed during the tutorial and already provided comments to improve the tutorial during the event.

Instructors helping out participants at the tutorial

At the end of the tutorial, all participants were asked to give feedback by completing an evaluation questionnaire, which will be used to further improve the tutorial and AiiDA itself.
The feedback was overall very positive, with the tutorial graded with an average vote of 9.0 out of 10, and all votes between 8 and 10.
All participants stated that the well-prepared and helpful instructors helped the participants to get really well acquainted with AiiDA and that it may very well help them with their own work and research.
The scientific talks that gave examples of AiiDA in real-world scientific projects were also well received as they gave a more intuitive example of the power and potential of AiiDA.

You can find more photos of the event on our Facebook page.

We acknowledge support from PRACE and from there H2020 MaX Centre of Excellence.

 

Report on AiiDA tutorial: 24-25 January 2017 at ICTP in Trieste, Italy.

👤 🕔 January 27, 2017 Comments Off on Report on AiiDA tutorial: 24-25 January 2017 at ICTP in Trieste, Italy.

The end of January of 2017 saw the eighth hands-on tutorial for AiiDA within the “Advanced Workshop on High-Performance & High-Throughput Materials Simulations using Quantum ESPRESSO” organised at and by ICTP in Trieste. The tutorial was kicked off by an introductory lecture by Giovanni Pizzi (EPFL), explaining the philosophy and architecture behind AiiDA, and how the ADES model facilitates and simplifies performing and sharing calculations, under the umbrella of complete data provenance. After the scene was set, the approximately 75 participants went to their workstations to start with the first of the hands-on sessions. A completely configured virtual machine was provided for each individual participant, to which they only had to login and they were on their way. The first session served to familiarize the users with Verdi, a command line interface which acts as the user’s tool to interact with AiiDA. With the tools in hand to put AiiDA to good use, the participants learned by example how a Quantum Espresso calculation can be easily launched as AiiDA takes care of a lot of the repetitive work behind the scenes.

The virtual environment in which the participants were working came with a database filled with pre-computed calculations, which allowed the aptitude of AiiDA in working with big data sets to be displayed effortlessly. The QueryBuilder, the tool provided by AiiDA to efficiently query a user’s collection of calculations, was introduced and a step-by-step tutorial showed the participants how it could be used to efficiently analyze a large collection of calculations on perovskites. The final session detailed how the users could easily create such a big collection of calculations themselves, in a high-throughput manner, through the concept of Workflows.

Throughout the tutorial, five members of the AiiDA team were present to answer any questions of the participants and give introductory talks at the beginning of a new hands-on session. After the hands-on sessions, a good portion of the participants gave their feedback on the quality of the tutorial and the impact on their skills in using AiiDA. The results are very promising while at the same time they provide the AiiDA team with valuable information on how to improve both the tutorial and AiiDA itself.

The feedback on the quality of the tutorial itself, in terms of content and planning, were overall very positive

 

 

AiiDA tutorial Jan 2017 feedback 2

Participant feedback on the quality of the tutorial

The results of the feedback also show that the tutorial really helped improve the skill and knowledge of the participants regarding the use of AiiDA

 

AiiDA tutorial Jan 2017 feedback 1

The skill level of the participants before and after the tutorial

The AiiDA team thanks the organisers of the event for making this tutorial possible and all the participants for their enthusiasm. Until the next AiiDA event!

Report of the AiiDA tutorial of last year (June 2016) in Lausanne

👤 🕔 January 19, 2017 Comments Off on Report of the AiiDA tutorial of last year (June 2016) in Lausanne

Report from the AiiDA coding week – Dec 2016

👤 🕔 December 13, 2016 Comments Off on Report from the AiiDA coding week – Dec 2016

Last week (5-9 Dec 2016) we had a coding week for AiiDA, to which 15 people participated (roughly half of them core developers from the AiiDA, and the other half were developers who were interested in contributing to the platform).

The coding week has taken place in Leysin, in the Swiss Alps. A few discussion or work topics had been identified before the start by the AiiDA team, and have been addressed during the week. In particular, the following topics have been discussed and the result has been summarised in a draft document for later implementation:

  • New more efficient implementation of the AiiDA daemon
  • How third-parties can contribute to Materials Cloud sections
  • Redesign of the ORM classes (in particular Node) to improve the way multiple backends are supported

A few discussions have also led to new branches of development, that will be soon merged into the develop branch (and appear in a future release):

  • AiiDA plugin interface integrated with pip
  • Various improvements and bug fixes to the SQLAlchemy backend implementation

Finally, the remaining work topics have been taking up a large part of the coding sessions, and most of the code has already been merged into the develop branch:

  • Redesign of the AiiDA documentation, with a more clear structure, removal of duplicates, a new and more clear installation section.
  • Simplified procedure to install AiiDA.
  • New testing infrastructure to test in parallel any database backend (currently, both Django and SQLAlchemy)
  • Removal of the DbPath (transitive closure) table and triggers, and replacement with a dynamically created query (using either PostgreSQL recursive queries or PostgreSQL functions). The performance of these queries has been optimised.

Beside programming, most participants have taken place to an outdoor hiking activity on Wednesday morning to Berneuse, close to Tour d’Aï, that was a great team building opportunity. Participants have also enjoyed after-dinner sessions with board and card games.

The outcome of the feedback, summarised below, has been very positive, both in terms of participation, organisation, and improvement to the code. The results indicate also that similar events should be organised again in the future, with a frequency of about twice a year.

Some photos of the event can be found on our Facebook page.

— Giovanni Pizzi

Summary of the results of the feedback form

Results of the feedback form for the AiiDA Coding week (Dec 2016)