In order to begin this work, CDAO is calling on subject matter experts to provide input on priorities and gaps for AI T&E in both industry and the government, as well as existing products and solutions (particularly open-source software) that the Department can leverage for the development of this capability. Information gained from this RFI will be used to inform the requirements and directions of work for JATIC contracts in FY23.

The focus of this RFI is the T&E of AI/ML models for Computer Vision (CV) classification and object detection problems. All questions below refer exclusively to T&E of AI models for CV classification and object detection. Other areas of T&E, such as systems integration, human-machine, and operational T&E, as well as T&E of other AI modalities, such as autonomous agents and natural language processing, are pressing and complex issues, but are out of scope for this RFI.

If more information is needed, CDAO T&E will follow-up to specific responses with further requests.

General Question:

What is your business type and size, to include specific classification (e.g., SDVSOB, etc.) if a small business?

CDAO Questions:

The JATIC effort has identified the following five priority dimensions for AI T&E:

1. Performance Measures: given a labeled test set, compute standard measures of performance, including:
1. Accuracy, precision, recall and other CV metrics to assess how the model performs on the prediction task
2. Expected calibration error, reliability diagrams and other probability calibration metrics to assess the reliability of the model's measure of predictive uncertainty
3. Model throughput, resource usage and other similar metrics to assess the efficiency and computational needs
2. Robustness to Natural Shifts in Data: assess how performance changes as data in the original test set is corrupted using natural perturbations, including:
1. Pre-sensor, environmental or physical corruptions (e.g., fog, snow, rain, changes in target shape or dimensions)
2. Sensory corruptions (e.g., out-of-focus, glare, blur)
3. Post-sensor, in-silico corruptions (e.g., Gaussian noise, digital compression)
3. Robustness to Adversarial Attacks: assess how performance changes as data in the original test set is corrupted using adversarial strategies, with characteristics of the attack described by varying dimensions, such as:
1. White-box vs. black-box attack
2. Pre-defined vs. adaptive attacks
3. Lp norm-constrained vs. physically-realizable perturbations
4. Empirical vs. certified attack
4. Model Analysis: facilitate deeper insight into model performance, such as:
1. Reporting performance on known partitions of the dataset given available metadata
2. Automated clustering of data inputs that result in similar outputs, to understand potential sources of error or high performing regions
5. Dataset Analysis: evaluate the quality of a dataset, including:
1. Testing for class imbalance or biases in the dataset
2. Quantifying the similarity between two datasets
3. Assessing the sufficiency (e.g., number of samples and variation) of a dataset
4. Detecting outliers, anomalies, label errors, or data poisoning

The above five AI T&E dimensions are by no means the only dimensions of AI T&E. We strongly encourage you to add up to three other AI T&E dimensions in which you believe there is a critical capability gap. For any added AI T&E dimensions, please provide a short description of the dimension.

For each of the above dimensions on which your company possess expertise (including any added dimensions), please provide the following information:

1. Please briefly describe your understanding of the maturity of research in this dimension.
2. What are existing software products and capabilities in this dimension? Which, if any, are open-source capabilities?
3. Do gaps exist in currently available capabilities? If so, what are the gaps?
4. Where or how could value be provided to the existing state-of-practice in this dimension?
5. If you have a product or expertise that could provide a solution that falls within this dimension, please describe it.

Any statement submitted in response to this Call should be no more than eight (8) pages (not including a cover sheet), single-spaced, using Times New Roman 12-point font.