Symposium on
Scaling AI Assessments
Tools, Ecosystems and Business Models
September 26th & 27th, 2024 | Design Offices Dominium, Cologne
Motivation
Trustworthy AI is considered a key prerequisite for Artificial Intelligence (AI) applications. Especially against the background of European AI regulation, AI conformity assessment procedures are of particular importance, both for specific use cases and for general-purpose models. But also in non-regulated domains, the quality of AI systems is a decisive factor as unintended behavior can lead to serious financial and reputation damage. As a result, there is a great need for AI audits and assessments and in fact, it can also be observed that a corresponding market is forming. At the same time, there are still (technical and legal) challenges in conducting the required assessments and a lack of extensive practical experience in evaluating different AI systems. Overall, the emergence of the first marketable/commercial AI assessment offerings is just in the process and a definitive, distinct procedure for AI quality assurance has not yet been established.
AI assessments require further operationalization both at level of governance and related processes and at the system/product level. Empirical research is pending that tests/evaluates governance frameworks, assessment criteria, AI quality KPIs and methodologies in practice for different AI use cases.
Conducting AI assessments in practice requires a testing ecosystem and tool support, as many quality KPIs cannot be calculated without tool support. At the same time automation of such assessments is a prerequisite to make the corresponding business model scale.
Target group
This symposium is aimed particularly at practitioners from the TIC sector, tech start-ups offering solutions to the above-mentioned challenges and researchers from the field of trustworthy AI. A further objective of the symposium being to clarify the legal framework of AI particularly with regard to the recently adopted AI Act of the European Union and its operationalization, it is also aimed at legal experts in the field of AI. The conference places particular emphasis on the commitment of young researchers along more experienced participants in the conference.
Participation
Participation is possible in three different ways. First, we offer a conventional academic track that accepts full papers. Second, we offer a practitioner track that is particularly suitable for industry or start-up representatives. Third, it is also possible to attend the conference without submitting own work. This provides the opportunity to get up to date with latest research insights, present and discuss practical challenges, and identify possible ways to test promising approaches in practice.
- Academic track: This track is suitable for scientific papers that present or discuss novel ideas on concepts, technology or methodologies related to the topics listed above. Submissions should include a clear description of the problem statement / research gap that is addressed, and the contribution of the research presented.
- Details on submission will be announced soon.
- Practitioner track: This track is suitable to apply for a presentation slot during the conference. As application for a presentation slot, a brief abstract of the topic (up to 2 pages) to be presented must be submitted and this will be reviewed. It is possible to submit extracts from whitepapers or position papers that have been published already.
- Details on submission will be announced soon.
- Attendance of the conference: Besides one of the above tracks, it is also possible to register for sole attendance of the conference. This involves active participation in the exchange formats. To this end, participants should select a suitable field of interest (either “Operationalization of market-ready AI assessment” or “Testing tools and implementation methods for trustworthy AI products”).
- Details on registration will be announced soon.
Highlights
Keynote speakers
- 26th September: To be announced
- 27th September: To be announced
Panel with legal experts
26th September, Afternoon:
- What requirements are imposed by the AI Act on generative AI and how can compliance be ensured? Drawing from a specific case study, what distinct challenges do high-risk AI systems pose and how does the AI Act address them? How does the AI Act intertwine with other complementary legislative frameworks such as the GDPR?
- These questions will be subject to a series of presentations given by selected legal experts followed by a discussion.
Activities with all participants
- 26th September: Conference Dinner
- 27th September: Roundtables
Submission
More infos coming soon.
Program commitee
More infos coming soon.
Topics of interest
This symposium aims to advance marketable AI assessments and audits for trustworthy AI. Specifically, papers and presentations both from an operationalization perspective (including governance and business perspectives) and from an ecosystem & tools perspective (covering approaches from computer science) are encouraged. Topics include but are not limited to:
Perspective:
Operationalization of market-ready AI assessment
- Standardizing AI assessments: How can basic concepts of AI assessments such as the target-of-evaluation, the operational environment and the operational design domain (ODD) be specified in a standardized way? How must assessment criteria be formulated and which AI quality KPIs are suitable to make AI quality and trustworthiness measurable? How can compatibility with existing assessment frameworks for other domains (e.g. safety, data protection) be guaranteed? How to deal with
- Risk and vulnerability evaluation: What methodologies can be employed to effectively characterize and evaluate potential risks and vulnerabilities, considering both the technical aspects and broader implications? How must AI governance frameworks look like to mitigate those risks efficiently?
- Implementing regulatory requirements: How can compliance with the AI Act and upcoming regulations be implemented into AI software and AI systems during operationalization, particularly in specific use cases, and what steps are required for achieving and maintaining compliance? In other words, how does a trustworthy AIOps framework look like?
- Business models based on AI assessments: What are business models based on AI assessments and what are key success factors for them? How need AI quality seals be designed and how do they influence consumers’ decisions?
Perspective:
Testing tools and implementation methods for trustworthy AI products
- Infrastructure and automation: What infrastructure and ecosystem setup is necessary for effective AI assessment and certification, including considerations for data and model access, protection of sensitive information, and interoperability of assessment tools? Which approaches are there to automate the assessment (process) as much as possible?
- Safeguarding and assessment methods: What strategies or methods can developers employ to select suitable testing or risk mitigation measures tailored to the specific characteristics of their AI systems? What are novel techniques, tools or approaches for quality assurance? How can generative AI be used as part of novel assessment tools (e.g., for generating test cases)?
- Systematic testing: How can systematic tests be performed and what guarantees can these tests give? In particular, how can diverse test examples be generated, including corner cases and synthetic data, to enhance the robustness and quality of AI products?
Important dates
22nd July 2024
Paper submission deadline (academic track and practitioner track)
19th August 2024
Author notification (academic track and practitioner track)
9th September 2024
Camera-ready version deadline (academic track and practitioner track) and registration deadline (all tracks)
26-27th September 2024
Conference (all tracks)
including key notes, paper presentations, exchange formats, panels and a social event
Location
Design Offices Dominium
Cologne
Tunisstraße 19-23
50667 Cologne
Germany
Contact
Organization Committee
Rebekka Görge, Elena Haedecke, Fabian Malms, Maximilian Poretschkin, Anna Schmitz
Contact for the Legal Perspective:
Mathias Schuh, Erik Weiss, Maxine Folschweiller, Lena Zeit-Altpeter
E-Mail: mathias.schuh(at)uni-koeln.de