Automated Interactive Infrastructure and Database for Computational Science

If you use AiiDA please cite:
AiiDA 1.x
Main paper: S.P. Huber et al., Scientific Data 7, 300 (2020)
AiiDA engine: M. Uhrin et al., Comp. Mat. Sci. 187 (2021)

AiiDA 0.x
First paper, ADES model: G. Pizzi et al. Comp. Mat. Sci. 111, 218-230 (2016) (open access version)


New positions in the THEOS group, EPFL

👤 🕔 May 18, 2021 Comments Off on New positions in the THEOS group, EPFL

There are three new openings at EPFL in the group of Prof. Marzari for a data architect/software scientist and two post-doctoral/PhD on first-principles operando spectroscopies. These positions are renewable yearly up to 4 years.

Outstanding candidates are sought with strong programming abilities, preferably with a background in physics/chemistry/materials or engineering/computer science. Candidates should be comfortable with project management and a diverse collaboration environment with teams across Europe.
The position will contribute to the development of the data and simulation infrastructure of BIG-MAP, and to the capabilities for automated simulations and data analytics. Core objectives will be automated data archiving (through the Materials Cloud ARCHIVE) and integration and dissemination of these data capabilities with simulation capabilities (through AiiDAlab, deployed on local workstations/Materials Cloud WORK/Kubernetes clusters/European Open Science Cloud/and commercial providers such as AWS or Azure). The position will be embedded in the AiiDA and Materials Cloud teams.

Outstanding and driven candidates are sought with a background in the physical or engineering sciences (physics, chemistry, materials, …), with passion and commitment to the field, and with a commensurate academic record. Expertise in the development and application of first-principles techniques is required.
Depending on skills and desires, each position could be focused more on the
development of novel capabilities, or in the deployment and application of current ones. Broadly, the two positions will contribute to these 3 goals:

  1. Quantum simulations in electrochemical environments: development and/or dissemination of electronic-structure simulations using the open-source Quantum ESPRESSO distribution and the ENVIRON library for embedding quantum simulations in the appropriate electrochemical environment.
  2. First-principles operando spectroscopies: development and application of established or novel capabilities for first-principles operando spectroscopies – case studies could be IR/Raman, XAS, ARPES, or NMR.
  3. Open science: integration of these capabilities into automated solutions that can be deployed locally (e.g. via the use of the Quantum Mobile) or directly on the web and cloud resources through AiiDAlab, for open-access use and dissemination.

All details and instructions on how to apply here:

Interested in a summer project working on AiiDA?

👤 🕔 March 31, 2021 Comments Off on Interested in a summer project working on AiiDA?

After a successful Google Summer of Code (GSOC) 2020 under the NumFocus umbrella, the AiiDA team plans to participate in GSOC 2021 as well. Last year’s GSOC student @unkcpz set the bar pretty high, but don’t be shy – there is lots more to do!

Google Summer of Code is a way for you to get a summer job getting to know an open source project in depth and moving it forward. Different from previous years, GSOC 2021 projects will be 175 hours during the period from June 7th until August 16th 2021. This makes it possible to participate part-time (e.g. 18h/week for 10 weeks) or in a shorter full-time stint.

Just as a reminder, here is an excerpt from the GSOC 2021 timeline:

  • Student Application Period March 29, 2021 – April 13, 2021
    Students can register and submit their applications to mentor organizations.
  • Student Projects Announced May 17, 2021
    Accepted students are paired with a mentor and start planning their projects and milestones.

If you’re interested in working on aiida-core as a student, please say hi in the dedicated discussion thread and ask any questions you may have.

See also the NumFOCUS guide for GSOC students.

The GSOC 2021 mentors
Leopold (@ltalirz) and Chris (@chrisjsewell)

AiiDA v1.6.0 released

👤 🕔 March 18, 2021 Comments Off on AiiDA v1.6.0 released

We are happy to inform you that we have just released v1.6.0!
As per SemVer versioning, this release is fully backwards-compatible.

The full changelog can be found at:—2021-03-15, but here are some highlights:

Migration to asyncio ♻️

This release marks the “under-the-hood” migration from the `tornado` package to the Python built-in module `asyncio`, for handling asynchronous processing within the AiiDA engine. Thanks to Jason Eu, a member of last year’s Google Summer of Code project!
The migration removes a number of blocking dependency version clashes with other tools, in particular with the newest Jupyter shell and notebook environments.
A substantial effort has been made to test and debug the new implementation, and ensure it performs at least equivalent to the previous code (or improves it!), but please let us know if you uncover any additional issues.

New calculation features ✨

The `additional_retrieve_list` metadata option has been added to `CalcJob`.
This new option allows one to specify additional files to be retrieved on a per-instance basis, in addition to the files that are already defined by the plugin to be retrieved.

A new namespace `stash` has been added to the `metadata.options` input namespace of the `CalcJob` process.
This option namespace allows a user to specify certain files that are created by the calculation job to be stashed somewhere on the remote.
This can be useful if those files need to be stored for a longer time than the scratch space (where the job was run) is available for, but need to be kept on the remote machine and not retrieved.
Examples are files that are necessary to restart a calculation but are too big to be retrieved and stored permanently in the local file repository.

See Stashing files on the remote for more details.

The new `TransferCalcjob` plugin allows the user to copy files between a remote machine and the local machine running AiiDA.
More specifically, it can do any of the following:

  • Take any number of files from any number of `RemoteData` folders in a remote machine and copy them in the local repository of a single newly created `FolderData` node.
  • Take any number of files from any number of `FolderData` nodes in the local machine and copy them in a single newly created `RemoteData` folder in a given remote machine.

See the Transferring data how-to for more details.

Profile configuration improvements 👌

The way the global/profile configuration is accessed has undergone a number of improvements:

  • When loaded, the config.json (found in the .aiida folder) is now validated against a JSON Schema that can be found in aiida/manage/configuration/schema.
  • The schema includes a number of new global/profile options, including: transport.task_retry_initial_interval, transport.task_maximum_attempts, rmq.task_timeout and logging.aiopika_loglevel.
  • The cache_config.yml has now also been deprecated and merged into the config.json, as part of the profile options. This merge will be handled automatically, upon first load of the `config.json` using the new AiiDA version.

In-line with these changes, the verdi config command has been refactored into separate commands, including verdi config list, verdi config set, verdi config unset and verdi config caching.
See the Configuring profile options and Configuring caching how-tos for more details.

Command-line additions and improvements 👌

In addition to verdi config, numerous other new commands and options have been added to verdi, including:

  • Added verdi group delete --delete-nodes, to also delete the nodes in a group during its removal.
  • Added verdi database summary,  which prints a summary of the count of each entity and (optionally) the list of unique identifiers for some entities.
  • Deprecated verdi export and verdi import commands (replaced by new verdi archive)

New REST API Query endpoint ✨

The /querybuilder endpoint is the first POST method available for AiiDA’s RESTful API.
The POST endpoint returns what the QueryBuilder would return, when providing it with a proper queryhelp dictionary (see the documentation here).

See AiiDA REST API documentation for more details.

Happy computing!

The AiiDA team

AiiDAlab paper published

👤 🕔 December 16, 2020 Comments Off on AiiDAlab paper published

The paper providing comprehensive information on the architecture and design philosophy of the AiiDAlab web platform was published:

A. V. Yakutovich et al., AiiDAlab – an ecosystem for developing, executing, and sharing scientific workflows, Comp. Mat. Sci. 188, 110165 (2021). DOI:10.1016/j.commatsci.2020.110165

The paper is of interest to anyone wanting to learn more about how AiiDAlab works and how to most effectively use the platform for their own research. A free preprint version of the paper is accessible here: arXiv:2010.02731

All users of AiiDAlab are kindly asked to cite the publication in their own work.

Paper describing AiiDA’s workflow engine published

👤 🕔 November 17, 2020 Comments Off on Paper describing AiiDA’s workflow engine published

The paper describing the details of AiiDA’s workflow engine has been published today:
M. Uhrin et al, Workflows in AiiDA: Engineering a high-throughput, event-based engine for robust and modular computational workflows, Comp. Mat. Sci. 187, 110086 (2021)

In it you will find useful details on how the AiiDA process engine (managing calculation and workflows) works in the AiiDA 1.x series, bringing a huge speed-up with respect to earlier versions also thanks to the event-based design, leveraging the RabbitMQ message broker.

We would be grateful it users running simulations with AiiDA 1.x could cite it along with the main reference paper S.P. Huber et al., Scientific Data 7, 300 (2020).

AiiDA v1.5.0 released

👤 🕔 November 13, 2020 Comments Off on AiiDA v1.5.0 released

This is the fifth minor version of the v1 series and comes with a couple of new features, as well as some relevant bug fixes. Support for Python 3.5 is dropped and support for Python 3.9 is added.

Process functions are now submittable

Since the changes introduced in v1.0, a process function creates a Process instance in the background, just as any other process. This means that, in principle, it could also be serialized and deserialized by a daemon worker from that point onwards. Here we remove the final limitations that prevented process functions from being fully submittable.

It is important to note that the usual rules for making processes submittable apply in this case as well: to be importable by the daemon, it will need to either be registered through an entry point or have its source module available through the PYTHONPATH environment variable. For more information on these requirements, you can check the specific section on launching processes in the AiiDA documentation.

New REST API features

This minor version introduces two new features to the REST API interface of AiiDA. The first one is that the base URL ( in a local deployment, for example) will now return a list of the available endpoints. The second one is the addition of a new endpoint full_types_count: this returns the same branched structure that the already existing full_types, but with the addition of a counter key which contains the count of all available node instances of the type indicated.

Archive Refactor

This is part of a series of modifications that aim at making several improvements in the performance and versatility of the process of exporting and importing databases. These changes here should minimally affect the existing interface and might already have some performance boosts, but are an important stepping stone for more significant improvements in the upcoming versions.

For more details on this release, see the change log.

Happy computing!

The AiiDA team

Two 4-year postdoctoral positions (Marzari@EPFL)

👤 🕔 October 26, 2020 Comments Off on Two 4-year postdoctoral positions (Marzari@EPFL)

There are two positions for computational materials scientist/software scientists immediately available in our laboratory for Theory and Simulation of Materials at EPFL. The two positions, for up to 4 years each, are dedicated to data analytics, data integration, and simulation/machine-learning workflows, and are supported by the new H2020 projects DOME4.0 and OpenModel.

The DOME4.0 position is centered around the successful execution of research case studies in combination with the existing Materials Cloud platform (https:/, with a primary focus on the development of machine-learning-based data analytics workflows and auxiliary data services. Successful candidates will develop tools and services required to support their own research aims and those of project partners. Potential scientific projects include the efficient identification of next-generation Li-Ion batteries, machine learning of materials properties from structure and electronic structure, or similar suitable projects.

The OpenModel position is centered around the development of software services to create and execute computational materials science workflows, with a primary focus on the integration of semantic universal data structures and ontology development. Successful candidates would develop tools and services required to support their own research aims and those of project partners. Potential scientific projects include the development and integration of domain ontologies, machine-learning-driven decision-making on models and methods for workflow builders, or similar suitable projects.

Close synergies are envisioned with the Swiss National Centre for Computational Design and Discovery of Novel Materials (NCCR MARVEL), the H2020 MaX Centre of Excellence for Materials Design at the Exascale, the H2020 Materials Modelling MarketPlace, H2020 Intersect and H2020 BIG-MAP Battery2030+, and with the AiiDA and Materials Cloud teams onsite.

Interested candidates should apply by Nov 6, following the instructions at for the two separate openings; the website has also more information on the group. The positions will remain open until filled (the ads will then be removed from the link above). Each position is fully funded for 4 years; the contract is initially for 1 year (as required by EPFL), and renewable yearly upon mutual satisfaction. Gross salaries for a freshly minted Ph.D. start at 83’600 CHF/year (~77’500 EUR/year); social benefits (unemployment, pension) are also provided.

AiiDA v1.4.0 released

👤 🕔 September 24, 2020 Comments Off on AiiDA v1.4.0 released

This is the fourth minor version of the v1 series and comes with some important performance improvements and features, but also various bug fixes. This is the last version to support Python 3.5.

RabbitMQ server can now be hosted on a different machine

In previous versions, AiiDA only supported connecting to a RabbitMQ instance running on localhost. With v1.4, one can now configure AiiDA to connect to a RabbitMQ instance over TCP/IP with optional SSL enabled. Existing profiles will be automatically migrated to add the standard configuration. To see how a new profile can be configured to connect to a remote RabbitMQ instance, please refer to the documentation.

Group extras

Groups now support setting “extras”: just like nodes, you can now store arbitrary metadata on group instances, as long as it is JSON serializable. The interface to setting, getting and deleting extras on a group is exactly the same as that for nodes. Note that, just as they are for nodes, extras are mutable. It is therefore not advised to use them to store information that is important for data provenance.

Scheduler output parsing

This version comes with a new method for the scheduler base class Scheduler.parse_output that can be implemented by scheduler plugins to parse the output that is generated by the scheduler when running a calculation job. The output consists of the detailed job info (if implemented by the scheduler plugin) and the output written to the stdout and stderr file descriptors. This information can hold important information about how the job terminated. It can be used to determine common problems, such as out-of-memory issues or out-of-walltime errors. The parse_output can return a corresponding exit code that is defined on the CalcJob class. You can take a look at the SlurmScheduler.parse_output implementation as an example of how it parses the OOM and OOW errors. The scheduler output parsing, if implemented, will be called by the engine before the output Parser is called, if any. If the scheduler parsing returns an exit code, the engine will already set it on the node but the output parser will still be called. This allows the output parser to still try to parse some data from the retrieved files, and if desired, override the exit code of the scheduler with a more specific exit code. For details on this process and the interaction between exit codes of scheduler and output parsers, see this section of the documentation.

For more details on this release, see the change log.

Happy computing!

The AiiDA team

AiiDA and Materials Cloud papers published in Nature’s Scientific Data

👤 🕔 September 10, 2020 Comments Off on AiiDA and Materials Cloud papers published in Nature’s Scientific Data

The new paper on AiiDA (describing all the work happened in the past 4+ years) and the paper describing the Materials Cloud (an Open Science Platform designed to enable the seamless sharing of resources in computational materials science) have been published on the Nature’s journal Scientific Data.

You can read the full press story on the NCCR MARVEL website, and you can access them (open access) here:

AiiDA virtual tutorial report

👤 🕔 July 27, 2020 Comments Off on AiiDA virtual tutorial report

A few weeks ago we organised the first virtual edition of our AiiDA tutorial week. Arranging a tutorial for 13 different time zones was challenging, but it was a great experience and we received a lot of positive feedback on the virtual format by the participants. You can read the full report here:

In case you missed the tutorial, you can find all the presentations and hands-on material on the webpage of the event:

Thanks again to the participants for attending! We look forward to organising more virtual events in the future.