Tuesday, April 18, 2023

Creating A Pilot Program For ChatGPT And Other AI Tools

 
Like others, I've been working with ChatGPT from a developer and engineering perspective over the past few months, and as developers, we can't shy away from these tools. We have to know and understand them, just like other tools, albeit with the knowledge and understanding that there's a difference in what these new AI tools are offering both from a development standpoint, and a human standpoint.

Lately I've been thinking about how to start a pilot program for developers and engineers for tools like ChatGPT where it's not overly prescriptive, but also has guardrails if being used and paid for like any other software, and how to gather research on that use.

This post outlines some of the pieces to get up and running, and to use as a template for further development of a pilot program specific to your team/department including:

  1. Goal Of The Initiative
  2. Outcomes And Objectives
  3. Pilot Guidelines/Standards
  4. Pilot Project Milestones
  5. Use Case Template


Goal Of The Initiative

To explore the use of ChatGPT and GitHub Copilot/X in the development process and create general guidelines, standards, and feedback on the effective/efficient use and acknowledgment of AI tools.

Research has shown that ChatGPT and GitHub Copilot can help developers write code more efficiently and accurately. For example, ChatGPT can assist with natural language processing tasks, such as summarizing text, answering questions, and generating text. GitHub Copilot, on the other hand, uses machine learning models to suggest code snippets and even whole functions based on the context of the code being written.

However, there are also potential risks and limitations associated with using these tools. For instance, ChatGPT can generate biased or inappropriate text if it is trained on biased or inappropriate data. GitHub Copilot may suggest code that is not optimized or secure, which can lead to software bugs or vulnerabilities.

To address these concerns, it is important for users to understand the capabilities and limitations of the tools. This can include training on how to evaluate the quality of the suggestions provided by the tools, how to review the code generated by the tools, and how to incorporate these tools into the software development process

Outcomes And Objectives

The main objectives and outcomes are the following:

  • Capture general use of the tools during the pilot by developers/engineers/consultants

  • Capture capabilities of ChatGPT and GitHub Copilot specific to organizational domain.
    • Capture and Define use cases where most appropriate with notes.
    • Capture general poll/data on use, effectiveness, etc.

  • Understand metrics around effectiveness.
     
  • Capture and Define limitations and potential risks of use (PHI/Security/Specific Organizational IP)

  • Define some standards/ethics/guidelines around the use of the tools

Pilot Guidelines/Standards

Some specific standards/rules at the outset of the pilot to be adhered to:

  • ChatGPT/AI Tools are still just programs and tools. While tools should help make some tasks and coding easier and to take less time, it is not a solution for everything and the developer/engineer/user, still needs to code, test, understand, and do the work, even if that involves working with AI tools. These are still programs and tools. It is like a camera with AI to help make the image better. You still need the photographer.

  • At the end of the day, the code is yours. While AI tools can help development they are only part of the solution, and like any other tool, is guided by the user. You are responsible for the code, including the understanding of that code (vs just copy/paste).

  • Do not send in any PHI information.

  • Use judgement on any code/data/information being used to create a solution, document, etc.
     
  • Efficiency in code should only be a part of the equation in assigning and taking on more work.

  • The use of AI tools should help us be better developers, programmers, and humans.


Pilot Project Milestones

  • Select the users of the tools for the pilot program

  • Create and define use cases and templates for user input

    • Create an initial set of use cases for ChatGPT/Other OpenAI technologies, and GitHub Copilot. These can be both general and unique to specific technology domains.

    • Create a template for each user to be able to add in their own use case.
       
    • Set schedule/guidelines for when these are updated.

  • Incorporation of the tools into the software development process.
    • This will include setting up paid accounts for each of the developers

  • Initial meeting to walk through different scenarios for use at the beginning of the project

  • One month check-in

  • Two month final check-in

  • Sharing and summary of results internally

  • Standardizing guidelines, rules of use, documentation, and any trained data/models where appropriate

  • Sharing and summary of results to the rest of the IS.

  • Follow up conversations to other tools/phases


Use Case Template

Below is a general list of use cases and a sample table to store information in for each user. Rows could be deleted as needed, and more specific examples could be documented by each user in the pilot program (e.g. "Had bug in code and fixed it").

Use Case Category Was This Used For A Specific Project/Ticket? Did It Save Time? Can you give a percentage (only)
Ask To Create Code You Know Testing


Ask To Create Code You Don't Know Testing


Creating Unit Tests Testing


Code Review Testing


Creating CSS/HTML Frontend Development


Creating Javascript and HTML code together Frontend Development


Creating C# Code Backend Development


Creating Python Code Backend Development


Creating SQL Code Database Development


Creating Excel Function, DAX, Macro Data Analysis


Make Script/Code Run More Efficiently Performance Optimization


Creating Templates For X Code Development Workflow


Explain Code To Me General



Other Information For Consideration 

 

at
Labels: , , ,