AiiDA – a bird’s eye view
This page provides answers to questions aimed at helping academic and industrial consortia to evaluate the AiiDA platform from a high-level perspective.
Note: For questions regarding installation or usage of AiiDA, please see AiiDA’s documentation.
Design goals – What is the scope of AiiDA and what isn’t?
- The mission of AiiDA is to help researchers in computational materials science manage and automate complex workflows and track the provenance to make them fully reproducible.
- The mission of the AiiDA lab is to provide access to key computational materials science capabilities through an intuitive user interface that does not require expertise in the underlying scientific codes or programming languages. In particular, AiiDA provides a powerful and robust workflow engine, encouraging the creating of an open ecosystem of modular plugins and workflows (many of which are community-contributed) to address a broad range of materials-science challenges.
- The mission of the Materials Cloud is to accelerate the transition to open computational materials science by providing storage, computational resources and technologies that enable FAIR (+R for reproducible) data management.
Design decisions – Why was AiiDA designed the way it is?
- AiiDA can run directly on the user’s machine (no worries for companies about a requirement to put data in the cloud: data lives on the computer of the user). It can run on the cloud as well (AiiDAlab).
- Tailored in particular for long-running HPC simulations on remote resources (supercomputers/clusters), with possibility to checkpoint and stop/start workflows without losing the progress
- AiiDA interacts with remote compute resources through SSH access (and it’s extensible to other protocols via plugins) – no installation needed on the supercomputer side
- AiiDA is highly extensible by design (plugin interface from the very beginning)
- AiiDA includes a powerful workflow engine with automatic provenance tracking (provenance graph stored in a relational DB)
- AiiDA works best for simulations codes that are stable, for which it is worth investing time to develop robust workflows. It is useful for high-throughput calculations and convergence tests.
AiiDA is less suited for method development, where the focus lies on adding new functionality to the simulation code (and long-term preservation of provenance in the long term may be less relevant).
Infrastructure requirements – What hardware do you need to run AiiDA?
- AiiDA: laptop or workstation
- for non-HTC use, a standard laptop works; for intensive HTC (~50’000 processes/hour) a modern server is preferable (e.g. >24GB RAM, SSD disk)
- on premises: 1VM can support up to ~50 users; if you have access to a kubernetes cluster, then seamlessly scalable up to the size of your cluster
- our installations: 2 OpenStack VMs at CSCS; 2 kubernetes clusters at CSCS and at the EOSC Hub
Interfaces to external tools/frameworks
- RabbitMQ + pika + asyncio for workflow engine/daemon
- PostgreSQL (SQL) + Django or SQLAlchemy (ORM python libraries)
- SSH+SFTP for data transfer
- slurm/PBSpro/torque/SGE/… [via plugins] as job schedulers
- Circus (daemon manager)
- flask for rest API
- ASE/pymatgen for atomic structures
- ipython widgets
Scalability – Can AiiDA handle big data?
- Sustained rate 50k processes/hour (bottleneck becomes the supercomputer scheduler; but AiiDA provides ways to avoid overloading)
- Queryable data in the DB: no problem to scale to 1TB/project or more (our large projects: ~50-100GB, with 10M nodes and billions of queryable attributes)
- Data that does not need querying: limited by disk space/inodes
Reliability – Can AiiDA operate 24/7?
- AiiDA relies on industry-standard technologies (PostgreSQL for transactional DB, RabbitMQ message broker)
- Automatic recovery from SSH/network/connection failures via exponential back-off mechanism
- Failures mainly come from crashes/bugs of HPC machines or simulation codes – workflows are meant to recover from these
Extensibility – What functionality can a plugin extend?
- add new calculations (support for generating input files for different codes), parsers (of code outputs) and workflows (from low-level error handling to turn-key solutions for specific materials properties)
- add new data types
- add new schedulers
- add new “transport” modes (e.g. communicate with REST API of HPC or other protocols other than SSH/SFTP)
- extend the command line interface
- (future) extend AiiDA REST API
Limitations – What is outside AiiDA’s scope?
AiiDA was not designed for:
1. running millions of HTC simulations on remote compute resources that take <1s each, and where provenance of each single calculation is not desired
2. workloads where calculation inputs may be terabytes of data whose provenance should be stored in the long-term
While 1. is generally a tough problem, we note that 2. has not been tackled mainly because this is not a typical use case in computational materials science.
Difficulties – What difficulties have the developers encountered?
- data format longevity – writing migrations for a data format to keep support for prior versions is hard; especially for querying, and since we support data formats defined by external developers via plugins
- the implementation of some features (e.g. current workflow engine) took much longer than what originally scheduled
- It is difficult (but possible) to find people who both have a background in our field of research and at the same time proficient coders who care about software quality and good development practices. For working on an exciting project they will need to accept having less publications than purely academic positions as well as a salary lower than in industry in CH
- AiiDA comes with a carrot (workflow management) and a stick (provenance tracking). While AiiDA tries to make provenance tracking automatic, it does not come with zero cost: people need to learn to learn basic concepts and how to use AiiDA. At the same time, people often underestimate the benefit of workflow management and the time they spend writing and fixing their homemade scripts
Regrets – What would we do differently, if we had to rewrite AiiDA from scratch?
- Extending the code to multiple database backends (Django and SQLAlchemy) was a good decision, but took longer than expected and, after a few years, some of the original reasons for the alternatives were no longer relevant
- Starting with an abstract concept of the file repository would have been useful
- Long period without stable releases (during the transition between the 0.x to the 1.x release series) resulted in lots of unreleased codes, binding resources and hiding progress from active users
- Stricter standards on coding practises and test coverage from the beginning (but now we are in a quite good shape)
- Workflow managers: fireworks/atomate in our field, more general Pegasus
- Data repositories for inputs and outputs of simulations: NOMAD
- Repositories of materials properties: Materials Project, AFLOWlib, OQMD, …
- Repository of data to reproduce a paper: QResp
Roadmap – What lies ahead for AiiDA development?
See the AiiDA release roadmap
Funding sources – Who pays for AiiDA? Is it enough?
[Data updated on March 2021]
- SNSF MARVEL NCCR: 4 FTEs for the Open Science platform (phase 1: 2014-2018; phase 2: 2018-2022; potential to renew until 2026)
- H2020 MaX Centre of Excellence: 3 FTEs for convergence of HPC, high-throughput and high-performance data analytics via AiiDA (phase 1: 2015-2018; phase 2: 2018-2021)
- H2020 MarketPlace: 2 FTEs for materials simulation services & data (2018-22)
- H2020 Intersect: 2 FTEs for automated modelling of complex devices using AiiDA (2019-21)
- Battery2030+ BIG-MAP project: 2FTEs to create the infrastructure to handshake experimental and simulation data, towards autonomous materials discovery of battery materials, based (among other technologies) on AiiDA and Materials Cloud (2021-2024)
- Private collaboration with major European company: 2 FTEs for AiiDA-powered materials discovery for Li-ion batteries (2019-21)
- Swissuniversities P5 “Materials Cloud”: 3 FTEs for transitioning Materials Cloud to self-sustaining service (2019-21)
- EPFL Open Science Fund “OSSCAR”: 1 FTE for creating educational hub for research and training (2019-21)
- H2020 Dome 4.0 project (starting 2021)
- H2020 OpenModel project (starting 2021)
- H2020 NEP project (starting 2021)
In total 15+ FTEs at EPFL for the next years
Team – Who develops/maintains AiiDA and how do we recruit?
User base (Who uses AiiDA?)
We don’t require users to register, but here are some resources to get an idea of who uses AiiDA:
Copyright and license
AiiDA-core is MIT-licensed, as are almost all plugins that we are aware of.
Help desk, training courses and consulting
User meetings, events and promotion
Relationships with other organizations (institutes, universities, companies)
- Code initially developed and co-licensed with Bosch
- Collaborations with universities on development of aiida-core and plugins (EPFL CH, ETHZ CH, University of Zurich CH, SISSA IT, Unimore IT, Ghent BE, Imperial College UK, Cambridge UK, Vilnius University LT, Trinity college Dublin IE, Kyoto University JP)
- Collaborations with research institutes on developing plugins or (FZ Jülich DE, SINTEF NO, ICN2 ES , CEA Grenoble FR, ICAMS DE, Empa CH, LLNL US)
- various companies that have supported with projects/funding (Constellium, Solvay, Bosch, IBM, Varinor, …)
See also the Materials Cloud project partners.
Main achievements for funders
- Open science platform with FAIR sharing of data
- State-of-the-art provenance tracking for full reproducibility of research (e.g. metadata starts to be used for machine learning by independent groups)
- Open source tool to manage workflows (AiiDA) supporting over 35 codes
- Turn-key workflows developed for important materials properties (phonons, band structure, STM images, automatic convergence of parametres, …)
- Archive: Open, moderated research data repository for computational materials science (recommended by Nature Scientific Data, indexed by FAIRsharing and re3data, …)
- Data management plan templates for using Materials Cloud (with & without AiiDA)
- Model for publishing online tools together with a paper
- Open source Quantum Mobile virtual machine for computational materials science (for learning/teaching and discovering of simulation tools)
- In our experience, it is often the senior researchers (group leaders, senior scientists) who care about provenance since they are familiar with the problems that arise when it is lacking. Postdocs who’ve already converged to a way of working that “gets the job done” have the highest barrier to cross, while younger students will find it easier to get started with a new technology.
- Developers of the platform should have an incentive to use it themselves. Most of the AiiDA (core & plugin) developers are users of AiiDA themselves, which helps us focus on issues that actually matter to day-to-day problems users encounter. Having a few “software architects” makes sense but we’ve been in contact with other platform projects that employ many software engineers and tend to find it more difficult to connect with their users.
- Scientists need flexibility, which is provided in AiiDA through the powerful plugin system. A flexible system is more difficult to manage than a constrained one but we believe it is necessary to do science.
Testimonials – How can AiiDA help my research?
See user testimonials.
Open issues – What is missing?
- Still a significant learning curve for new users – many users are happy to invest the time, but some aren’t
- For developers, AiiDA development requires time that is not spent doing research (applies mainly to aiida-core developers)
See also the roadmap.
Demos – Where to look for demos of what AiiDA can do?