Symposium on

Scaling AI Assessments

Tools, Ecosystems, and Business Models

26. & 27. September 2024 | Design Offices, Cologne Dominium

Motivation

Trustworthy AI is considered a key prerequisite for Artificial Intelligence (AI) applications. Especially against the background of European AI regulation, AI conformity assessment procedures are of particular importance, both for specific use cases and for general-purpose models. But also in non-regulated domains, the quality of AI systems is a decisive factor as unintended behavior can lead to serious financial and reputation damage. As a result, there is a great need for AI audits and assessments and in fact, it can also be observed that a corresponding market is forming. At the same time, there are still (technical) challenges in conducting the required assessments and a lack of extensive practical experience in evaluating different AI systems. Overall, the emergence of the first marketable/commercial AI assessment offerings is just in the process and a definitive, distinct procedure for AI quality assurance has not yet been established.

 

AI assessments require further operationalization both at level of governance and related processes and at the system/product level. Empirical research is pending that tests/evaluates governance frameworks, assessment criteria, AI quality KPIs and methodologies in practice for different AI use cases.

Conducting AI assessments in practice requires a testing ecosystem and tool support, as many quality KPIs cannot be calculated without tool support. At the same time automation of such assessments is a prerequisite to make the corresponding business model scale.

Target group

This symposium is aimed particularly, but not limited to, at practitioners from the TIC sector, tech start-ups offering solutions to the above-mentioned challenges and researchers from the field of trustworthy AI. The conference places particular emphasis on the commitment of young researchers along more experienced participants in the conference.

Participation

Participation is possible in three different ways. First, we offer a conventional academic track that accepts full papers. Second, we offer a practitioner track that is particularly suitable for industry or start-up representatives. Third, it is also possible to attend the conference without submitting own work. This provides the opportunity to get up to date with latest research insights, present and discuss practical challenges, and identify possible ways to test promising approaches in practice.

  • Academic track: This track is suitable for scientific papers that present or discuss novel ideas on concepts, technology or methodologies related to the topics listed above. Submissions should include a clear description of the problem statement / research gap that is addressed, and the contribution of the research presented.
    • Details on submission will be announced soon.
  • Practitioner Track: This track is suitable to apply for a presentation slot during the conference. As application for a presentation slot, a brief abstract of the topic (up to 2 pages) to be presented must be submitted and this will be reviewed. It is possible to submit extracts from whitepapers or position papers that have been published already.
    • Details on submission will be announced soon.
  • Attendance of the conference: Besides one of the above tracks, it is also possible to register for sole attendance of the conference. This involves active participation in the exchange formats. To this end, participants should select a suitable field of interest (either “Operationalization of market-ready AI assessment” or “Testing tools and implementation methods for trustworthy AI products”).
    • Further information will be announced soon.

 

Submission

More infos coming soon.

Registration

More infos coming soon.

Program commitee

More infos coming soon.

Topics of interest

This symposium aims to advance marketable AI assessments and audits for trustworthy AI. Specifically, papers and presentations both from an operationalization perspective (including governance and business perspectives) and from an ecosystem & tools perspective (covering approaches from computer science) are encouraged. Topics include but are not limited to:

Perspective: 
Operationalization of market-ready AI assessment

  • Standardizing AI Assessments: How can basic concepts of AI assessments such as the target-of-evaluation, the operational environment and the operational design domain (ODD) be specified in a standardized way? How must assessment criteria be formulated and which AI quality KPIs are suitable to make AI quality and trustworthiness measurable? How can compatibility with existing assessment frameworks for other domains (e.g. safety, data protection) be guaranteed? How to deal with
  • Risk and Vulnerability Evaluation: What methodologies can be employed to effectively characterize and evaluate potential risks and vulnerabilities, considering both the technical aspects and broader implications? How must AI governance frameworks look like to mitigate those risks efficiently?
  • Conformity with Regulations: How can compliance with the AI Act and upcoming regulations be implemented into AI software and AI systems during operationalization, particularly in specific use cases, and what steps are required for achieving and maintaining compliance? In other words, how does a trustworthy AIOps framework look like?
  • Business models based on AI assessments: What are business models based on AI assessments and what are key success factors for them? How need AI quality seals be designed and how do they influence consumers’ decisions?

    Perspective:
    Testing tools and implementation methods for trustworthy AI products

    • Infrastructure and automation: What infrastructure and ecosystem setup is necessary for effective AI assessment and certification, including considerations for data and model access, protection of sensitive information, and interoperability of assessment tools? Which approaches are there to automate the assessment (process) as much as possible?
    • Safeguarding and assessment methods: What strategies or methods can developers employ to select suitable testing or risk mitigation measures tailored to the specific characteristics of their AI systems? What are novel techniques, tools or approaches for quality assurance? How can generative AI be used as part of novel assessment tools (e.g., for generating test cases)?
    • Systematic testing: How can systematic tests be performed and what guarantees can these tests give? In particular, how can diverse test examples be generated, including corner cases and synthetic data, to enhance the robustness and quality of AI products?

    Important dates

     

    22nd July 2024
    Paper submission deadline (academic track and practitioner track)

    19th August 2024
    Author notification (academic track and practitioner track)

    9th September 2024
    Camera-ready version deadline (academic track and practitioner track) and registration deadline (all tracks)

    26-27th September 2024
    Conference (all tracks)
    including key notes, paper presentations, exchange formats, panels and a social event

    Location

    Design Offices

    Cologne Dominium

    Tunisstraße 19-23
    50667 Cologne
    Germany

    Contact

    Organization Committee

    Rebekka Görge, Elena Haedecke, Fabian Malms, Maximilian Poretschkin, Anna Schmitz

    Email: zki-symposium@iais.fraunhofer.de