FINOS LLM Exploration

Summary of Previous Calls and Meetings: 

• 29th of May: FINOS kicked off the FINOS LLM Exploration Project, looking to identify areas of collaboration for Financial Services Institutions to build and fund collaborative research and infrastructure. The call was well-attended with a high level of engagement. 
• 12th of June: the feedback received during the kick-off call - “Don’t try to solve the world!” - was embedded in the presentation. In the second half of the call, the following “prior art” was presented: 
  • FinBen (Financial Benchmarking), by Yanglet (Columbia University) 
  • OS-Climate Data Commons, by Cara Delia (RedHat) 
  • Zenith Open Data Repository, by Keith O'Donnell (Feynic Technology)

Use Case Identification - Workshop #1

• 25th of June: during the workshop, the following topics were discussed:  

1. Potential Outcomes of the FINOS LLM Exploration Project

We started by recapping how use cases fit within the overall vision of FINOS LLM Exploration Project and more broadly within FINOS AI Strategic Initiative. Then focused on the potential outcomes of the project. Here below are possible outputs: 

1.1 Reference Architectures: find specific Financial Services Institutions (FSI) use cases and build demo applications stitching together existing open source solutions to:

1. 1. Enable organizations to replicate (from design to deployment). 
   2. Fill the gap: highlight where open source solutions in the necessary architecture / MLOps do not currently exist, which might inform the future roadmap. 
   3. Bring clarity: help solve the problem where people treat LLMs just as the model, instead of considering them as a Service / SaaS / Architecture, which can help solve some of the problems with the models themselves. 
      • Example: Google Vertex avoids inadvertently adding copyrighted code to your codebase by adding a "citation engine" in their architecture which checks outputs against known copyright. Google trusts this enough to offer a liability shield to their users - this is an architecture solution to a non-deterministic model problem and we think similar patterns could solve other common issues/pain points/use cases.

1.2 “Model Creation & Evaluation Engine'': we are working on two main streams: 

1. 1. Evaluating the Open Source Climate Data Commons (contributed by RedHat) as 'prior art' for developing a public/private data store for licensed public/private data - this will allow FSIs to benefit by having a central target for bringing private/on-prem data, vendor data, and public/open datasets together. We think this will help create standardization for training, fine-tuning, model creation, and benchmarking between the data providers and the model providers (whether external vendors or self hosted). 
   2. Building on the work done by the AI4Finance team (FinGPT, FinBen paper on financial benchmarks) as 'prior art' for developing a suite of benchmarking tools.

1.3  MLOps: Any tools for model deployment, data pipelines, orchestration, gateways, architecture, etc. As mentioned above, we believe the reference architectures will help highlight missing puzzle pieces, but our industry partners have already identified some places to start looking. This also relates to the above point of focusing on the wider Service as opposed to just the model.

2. Use Case Generation and Validation

We briefly recapped which types of use cases are in scope (i.e. only pre-competitive and no competitive use cases). Refer to the deck for more details. 

A use case validation matrix was used to validate the use cases generated. The use cases proposed and validated by the participants are summarized in the canvas below (the ones outside the quadrant are TBD, they were not in the “final canvas”, but could/should be reconsidered).

2.1 Low Feasibility / High Impact: These use cases can be moved to Zenith SIG, Universities, and other labs.

2.2 Low Feasibility / Low Impact: These use cases can be “outsourced to tech providers”, meaning that given their low impact, it would be difficult to find public or public-like sponsors.

2.3 High Feasibility / Low Impact: Engage with the wider community.

2.4 High Feasibility / High Impact: The right target use cases for the FINOS LLM Exploration project.

Calls to Action 

The use cases in the top right corner (High feasibility / High Impact) will be developed further: Reach out to luca.borella@finos.org if you are interested in contributing to: 

1. USE CASE 1: CDM documentation interpretation/generation, a prerequisite to potentially expand CDM to include more products, asset classes, events, etc, as well as using CDM as a basis for digital reporting. 

2. USE CASE 2: Attribution, synthetic data generation, data federation and mapping -> (TBcontinued)

Other Relevant Links and Material

• Here are some pictures of the workshop
• Here is the actual post-it
• AI Readiness Special Interest Group in-person meeting minutes: This meeting was held in person right after the FINOS LLM Exploration Use Case Identification Workshop.
  • The AI Readiness (SIG) has been making significant strides in developing a simple yet effective governance framework for AI systems in financial services. Focusing on a Large Language Model with Retrieval Augmented Generation (RAG) use case, the group has identified key threats such as data leakage, hallucination risk, and unauthorized access.
  • To mitigate these risks, the SIG is exploring a range of controls, from data filtering and system observability to user/application firewalling and robust acceptance testing.
  • This foundational work, targeted for completion by September 2024, serves as a stepping stone towards a more comprehensive, enterprise-grade governance framework.
  • FINOS Members are encouraged to follow the group's progress and contribute to this important initiative.
    

Authors: Luca Borella, AI Strategic Initiative Program Manager, FINOS and Karl Moll, Technical Project Advocate, FINOS.