Symposium on

Scaling AI Assessments

Tools, Ecosystems and Business Models

September 30th & October 1st, 2024 | Design Offices Dominium, Cologne

Motivation

Trustworthy AI is considered a key prerequisite for Artificial Intelligence (AI) applications. Especially against the background of European AI regulation, AI conformity assessment procedures are of particular importance, both for specific use cases and for general-purpose models. But also in non-regulated domains, the quality of AI systems is a decisive factor as unintended behavior can lead to serious financial and reputation damage. As a result, there is a great need for AI audits and assessments and in fact, it can also be observed that a corresponding market is forming. At the same time, there are still (technical and legal) challenges in conducting the required assessments and a lack of extensive practical experience in evaluating different AI systems. Overall, the emergence of the first marketable/commercial AI assessment offerings is just in the process and a definitive, distinct procedure for AI quality assurance has not yet been established.

 

AI assessments require further operationalization both at level of governance and related processes and at the system/product level. Empirical research is pending that tests/evaluates governance frameworks, assessment criteria, AI quality KPIs and methodologies in practice for different AI use cases.

Conducting AI assessments in practice requires a testing ecosystem and tool support, as many quality KPIs cannot be calculated without tool support. At the same time automation of such assessments is a prerequisite to make the corresponding business model scale.

Target group

This symposium is aimed particularly at practitioners from the TIC sector, tech start-ups offering solutions to the above-mentioned challenges and researchers from the field of trustworthy AI. A further objective of the symposium being to clarify the legal framework of AI particularly with regard to the recently adopted AI Act of the European Union and its operationalization, it is also aimed at legal experts in the field of AI. The conference places particular emphasis on the commitment of young researchers along more experienced participants in the conference.

Attending & Registration

Participation is possible in three different ways. First, we offer a conventional academic track that accepts full papers. Second, we offer a practitioner track that is particularly suitable for industry or start-up representatives. Third, it is also possible to attend the conference without submitting own work. This provides the opportunity to get up to date with latest research insights, present and discuss practical challenges, and identify possible ways to test promising approaches in practice.

 

  • Academic track: This track is suitable for scientific papers that present or discuss novel ideas on concepts, technology or methodologies related to the topics listed above. Submissions should include a clear description of the problem statement / research gap that is addressed, and the contribution of the research presented.
  • Practitioner track: This track is suitable to apply for a presentation slot during the conference. As application for a presentation slot, a brief abstract of the topic to be presented must be submitted in order to review that the presentation fits the conference theme. It is possible to submit extracts from whitepapers or position papers that have been published already.
  • Attendance of the conference: Besides one of the above tracks, it is also possible to register for sole attendance of the conference. This involves active participation in the exchange formats. To this end, participants should select a suitable field of interest (either “Operationalization of market-ready AI assessment” or “Testing tools and implementation methods for trustworthy AI products”).

Highlights

Keynote speakers

  • 30th September: To be announced
  • 1st October: To be announced

Panel with legal experts

30th September, Afternoon:

  • What requirements are imposed by the AI Act on generative AI and how can compliance be ensured? Drawing from a specific case study, what distinct challenges do high-risk AI systems pose and how does the AI Act address them? How does the AI Act intertwine with other complementary legislative frameworks such as the GDPR?
  • These questions will be subject to a series of presentations given by selected legal experts followed by a discussion.

Activities with all participants

  • 30th September: Conference Dinner
  • 1st October: Roundtables

Instructions for authors

We invite members of the international trustworthy AI community – from researchers to practitioners – to submit and present their work in either the academic track or the practitioner track. Submissions will be subject to a double-blind review by the programme committee based on the following criteria: relevance and significance, originality, soundness, clarity and quality of presentation. The symposium is planned as an in-person event and at least one author of each accepted paper must register for the conference and give an oral presentation of the submitted work.

 

Academic track:

  • Length: We expect papers with a length of around 4.800 – 6000 words, excluding references.
  • Submission Instructions: Papers must be written in English and the submission can be made using either the OASIcs LaTeX Template or the (initially provided) Springer LNCS template (LaTeX or Word). Please note that the proceedings will use the OASIcs template and the camera-ready version of all accepted papers (by September 16) shall therefore be formatted using the OASIcs template, see section ‘templates’ below.
  • The review will be double-blind, thus make sure you use the „anonymous article” template for submission (i.e., anonymous is added as argument of the \documentclass of the OASIcs template) or anonymize your submission in another suitable way.
  • Submission Deadline: 05.08.2024 (23:59 AoE)
  • We accept submissions of previously unpublished material as well as previously published papers that contain at least 30% new content.
  • To submit your paper, please use the button below.

Practitioner track:

  • Length: We expect a brief abstract with a length of around 200 – 1000 words, excluding optional references.
  • Submission Instructions: Practitioner abstracts must be written in English and the submission can be made using either the OASIcs Template or the (initially provided) Springer LNC Template. Please note that the proceedings will use the OASIcs template.
  • A single-blind review is intended, but fully anonymized contributions are also accepted.
  • Submission Deadline: 05.08.2024 (23:59 AoE)
  • We accept submissions of previously unpublished material as well as new compilations of previously published work, e.g. white papers.
  • To submit your paper, please use the button below. Alternatively, the summary can be sent by e-mail to the contact e-mail address zki-symposium@iais.fraunhofer.de.

Templates:

Proceedings:

  • All accepted papers (both tracks) will be given the option to be included in the conference proceedings, which will be published in the OASIcs series. For this purpose, the camera-ready version of the submission must fulfill the requirements of the OASIcs series (e.g. style guide, LaTeX template, author agreements). The LaTeX template can be downloaded below.
  • In addition to the LaTeX template, which can already be used for the submission of conference papers, authors will be sent further information on the publication process of the conference proceedings with the final notification regarding acceptance of the conference papers.
  • Note regarding paper length: If authors want to pursue this option, their submitted paper should be formatted via the OASIcs template with a length of about 2-6 pages (practitioner track) and a length of about 10 to 15 pages (academic track) until 16th September. Excluded from this page limit are the bibliography, the title page(s) (authors, affiliation, keywords, abstract, …) and a short appendix (up to 5 pages). Especially this is also possible after the initial submission. This means that the initial submission by August 5 (brief abstract, short paper or regular paper) can then be extended to the required amount of pages by September 16 (if this length is not already reached with the initial submission), if a contribution in the proceedings is desired. 

Note: Of course, authors also have the option of deciding against integration into the proceedings. In this case, the papers do not have to be adapted to the requirements of the proceedings.

Call for Papers

>>> Submission deadline extended until August 5th <<<

This symposium aims to advance marketable AI assessments and audits for trustworthy AI. Specifically, papers and presentations both from an operationalization perspective (including governance and business perspectives) and from an ecosystem & tools perspective (covering approaches from computer science) are encouraged. Topics include but are not limited to:

Perspective:
Operationalization of market-ready AI assessment

  • Standardizing AI assessments: How can basic concepts of AI assessments such as the target-of-evaluation, the operational environment and the operational design domain (ODD) be specified in a standardized way? How must assessment criteria be formulated and which AI quality KPIs are suitable to make AI quality and trustworthiness measurable? How can compatibility with existing assessment frameworks for other domains (e.g. safety, data protection) be guaranteed? How to deal with third party components, in particular general-purpose AI models, that are difficult to access during an assessment?” 
  • Risk and vulnerability evaluation: What methodologies can be employed to effectively characterize and evaluate potential risks and vulnerabilities, considering both the technical aspects and broader implications? How must AI governance frameworks look like to mitigate those risks efficiently?
  • Implementing regulatory requirements: How can compliance with the AI Act and upcoming regulations be implemented into AI software and AI systems during operationalization, particularly in specific use cases, and what steps are required for achieving and maintaining compliance? In other words, how does a trustworthy AIOps framework look like?
  • Business models based on AI assessments: What are business models based on AI assessments and what are key success factors for them? How need AI quality seals be designed and how do they influence consumers’ decisions?

Perspective:
Testing tools and implementation methods for trustworthy AI products

  • Infrastructure and automation: What infrastructure and ecosystem setup is necessary for effective AI assessment and certification, including considerations for data and model access, protection of sensitive information, and interoperability of assessment tools? Which approaches are there to automate the assessment (process) as much as possible?
  • Safeguarding and assessment methods: What strategies or methods can developers employ to select suitable testing or risk mitigation measures tailored to the specific characteristics of their AI systems? What are novel techniques, tools or approaches for quality assurance? How can generative AI be used as part of novel assessment tools (e.g., for generating test cases)?
  • Systematic testing: How can systematic tests be performed and what guarantees can these tests give? In particular, how can diverse test examples be generated, including corner cases and synthetic data, to enhance the robustness and quality of AI products?

Program commitee

Bertrand Braunschweig

Bertrand Braunschweig

Confiance.ai

Lucie Flek

Lucie Flek

University of Bonn, Lamarr Institute for AI and ML

Antoine Gautier

Antoine Gautier

QuantPi

Manoj Kahdan

Manoj Kahdan

RWTH Aachen

Foutse Khomh

Foutse Khomh

Polytechnique Montreal

Julia Krämer

Julia Krämer

Erasmus School of Law in Rotterdam

Qinghua Lu

Qinghua Lu

CSIRO

Jakob Rehof

Jakob Rehof

TU Dortmund, Lamarr Institute for AI and ML

Franziska Weindauer

Franziska Weindauer

TÜV AI.Lab

Stefan Wrobel

Stefan Wrobel

University of Bonn, Fraunhofer IAIS

Jan Zawadzki

Jan Zawadzki

Certif.AI

Important dates

 

7th July 2024
Start of submissions via easychair

22nd July 2024 Submission deadline extended until August 5th
Paper submission deadline (academic track and practitioner track)

19th August 2024
Author notification (Practitioner track)

26th August 2024
Author notification (Academic track)

9th September 2024 16th September 2024
Camera-ready version deadline (academic track and practitioner track) and registration deadline (all tracks). Optional: Camera-ready version of the practitioner abstract to prepare it for possible integration into the Proceedings.

30th September & 1st October 2024
Conference (all tracks)
including key notes, paper presentations, exchange formats, panels and a social event

Registration

Note: Please be aware that the event is planned as an on-site event and that the event language is English. There will be no possibility to participate virtually.

Required field
Bitte füllen Sie das Pflichtfeld aus.
Required field
Required field
Bitte füllen Sie das Pflichtfeld aus.
Bitte füllen Sie das Pflichtfeld aus.
Required field
Required field
Bitte füllen Sie das Pflichtfeld aus.
Bitte füllen Sie das Pflichtfeld aus.
Required field
Bitte füllen Sie das Pflichtfeld aus.
Required field
* Required

Location

Design Offices Dominium

Cologne

Tunisstraße 19-23
50667 Cologne
Germany

Contact

Organization Committee

Rebekka Görge, Elena Haedecke, Fabian Malms, Maximilian Poretschkin, Anna Schmitz

Email: zki-symposium@iais.fraunhofer.de

Contact for the Legal Perspective

Mathias Schuh, Erik Weiss, Maxine Folschweiller, Lena Zeit-Altpeter

Email: mathias.schuh@uni-koeln.de