Portfolio 1: Human-AI Decision Support

Part 1: Choosing Your Task

Objective:

Select a task where you possess specific expertise to be the subject of this portfolio. You will perform this task in the real world while strictly following instructions generated by an AI. To achieve a high evaluation, your task must meet the following four requirements.

1. The Expertise Requirement

You must choose a task that requires specialized knowledge. Among a random group of 10 students, only 1 or 2 should be able to complete this task correctly without instructions.

• Acceptable: Rebuilding a carburetor, tuning a guitar, sewing a specific garment pattern, soldering a circuit board.
• Unacceptable: Making a sandwich, changing a lightbulb, assembling basic furniture, boiling pasta.

2. The Physicality Requirement

The task must take place in the physical world and involve the manipulation of real objects, tools, or materials.

• Constraint: No screen-based tasks are permitted. You must use your hands on physical items.
• Excluded: Things like coding, Excel, video games, or software configuration.

3. The Duration Requirement

The task must require a minimum of 30 minutes of active work to complete when performed by you at your normal pace.

• Constraint: This must be active manipulation. Passive waiting time (e.g., waiting for glue to set) does not count toward the 30 minutes.

4. The Complexity Requirement

The task cannot be a linear list of steps. It must require you to make judgment calls based on what you see, hear, or feel.

• Constraint: The task must include conditional decisions (“If X happens, do Y; if Z happens, do Q”).
• Example: “Check the tension. If it is too loose, tighten the screw; if too tight, loosen it.”

The Task Abstract

In the written document you build a the end you must identify:

1. The Goal: What will be achieved or built?
2. The Tools: What physical objects are involved?
3. The Expertise: Why do you have expertise? Why would the other 8/10 people (or more) fail at this task without support (AI or guide)?
4. The written document you build at the end must provide a Hierarchical Task Analysis (HTA) of the task you choose written IN ADVANCE of doing Part 2.

Part 2: The Task

Objective:

You will role-play as a “Naive User.” You must convince the AI that you have absolutely no expertise in this domain. This forces the AI to carry the entire cognitive load. In this assignment do not provide any photos to the AI: Your interaction with the large language model must be completely textual.

1. Your Persona: The Novice

You will explicitly tell the AI that you are a complete beginner. Because you are role-playing a person with zero knowledge, you effectively cannot help the AI.

• The Logic: A naive user doesn’t know that step 3 is wrong. A naive user doesn’t know that the part is backwards. Therefore, you cannot correct these things. You must simply do what you are told. For example: If the AI uses a technical term you know but a novice wouldn’t, ask the AI what that word means. Have it describe a tool’s appearance rather than using its name.

2. The Conversation

Open a new conversation with the Large Language Model (Microsoft Copilot, ChatGPT, etc.) and enter a prompt that establishes this persona.

• Required Prompt: “I need to perform [Task Name]. I have never done this before and I have no idea how to do it. Please guide me through this process step-by-step. Do not give me a list. Give me one instruction at a time and wait for me to finish it.”

3. Handling Failures with “Naive” Responses

When the AI gives you a bad instruction that results in a poor outcome (e.g., parts don’t fit, the result is ugly), you must act like a confused beginner.

• Do not say: “You forgot to account for the thread pitch.” (Expert response).
• Do say: “I did what you said, but it doesn’t fit.” (Naive response).
• The Goal: You are forcing the AI to figure out what is wrong.

4. No Permanent Damage

Even a naive user knows not to walk off a cliff.

• Rule: If an instruction seems like it will cause you physical injury or break your equipment, etc, stop. No damage or danger should be a part of this assignment.
• Report: Tell the AI something nonspecific (no expertise) like, “I am afraid to do this because it looks like it will break. What should I do now?”

5. The 90-Minute Time Cap

If the “30 min”task exceeds 90 minutes of active effort, you are permitted to stop.

• The Logic: We are measuring fragility, not your physical endurance. If the model cannot guide you to completion in this time, the task has failed. The transcript of the failure is sufficient data for your analysis.

6. Creating the Transcript

In the written document you build a the end you must identify you must preserve the entire conversation for your analysis. It will be included in the final document. You also must identify your start time and end time for this portion of the project, full date stamp.

• Cross-Device Sync: You will likely use your mobile phone during the physical task. Microsoft Copilot automatically syncs this conversation to your account.
• The Desktop Protocol: Even if you used your phone for the task, you must go to a desktop computer to save the file.

1. Log into Microsoft Copilot on a desktop browser.
2. Locate your specific session in the history.
3. Use the Export function or Print to PDF (Ctrl+P / Cmd+P) to save the full text.
4. Constraint: Ensure the PDF is legible and searchable. Do not submit screenshots.

Part 3: The Analysis

Objective: Using the transcript of your interaction as data, analyze the implications of this technology for yourself, your future users, and the engineering process itself.

1. The Task Analysis

In the written document you build a the end you must provide a Hierarchical Task Analysis (HTA) of the task as performed by a novice with AI support.

1. Make sure you are using the Hierarchical Task Analysis (HTA) framework
2. Make sure you are not committing the happy path fallacy.
3. Make sure to capture not just your ininital task, but instead A novice user performing [TASK] with the assistance of Microsoft Copilot

Advice: “Your first HTA (Appendix A) was likely a clean tree. Your second HTA (Appendix C) should contain more loops for the ‘Prompting,’ ‘Waiting,’ ‘Clarifying,’ and ‘Error Correction’ that the AI forced you to perform.”

2.The User Context

Analyze the risks and benefits of this system for two different groups. Cite specific moments from your transcript to support your argument.

• Relative to You (The Expert): Did the AI help you or hinder you? Did the experience increase or decrease your trust in the tool?
• Relative to Your Users: Imagine integrating this system for a specific user group (either company employees you manage or customers using a product you designed). What happens when these userswho likely lack your expertiseencounter the same errors or hallucinations you did?

3. The Impact on Task Analysis

How does the inclusion of a Large Language Model change the specific engineering practice of Task Analysis?

• The Shift: Does the AI actually remove steps from the human’s task list, or does it simply trade physical work for cognitive work (auditing/verifying)?
• The Complexity: When an AI is inserted into the loop, does the task hierarchy become simpler, or does it become more complex due to the need for “prompt engineering” and error checking?

4. Engineering Recommendations (Heuristics)

Based on your findings, act as a professional Industrial Engineer. Create a set of Actionable Heuristics (rules of thumb) for users of Large Language Model systems as task decision support in task with they they are unfamiliar.

• The Goal: These should be practical warnings or protocols that mitigate the risks you identified.
• The Format: Use clear, directive language.

• Example: “Never execute a command involving torque without visually verifying the thread alignment first.”
• Example: “If the model’s output is instantaneous, treat it as a hallucination until verified by a secondary source.”

Your Final Deliverable: A Human-AI Decision Support Report

Submit a formal engineering document centered around the question, “what are the Human engineering considerations around Human-AI Decision Support?

This document addresses all of the Requirements for the written document listed above, and then uses them as evidence to answer the question, before finally providing Engineering Recommendations. Appendix A should be your initial task analysis (From Part 1). Appendix B should be your transcript with Microsoft Copilot (From Part 2). Appendix C should be your revised Novice used with Copilot task analysis (From Part 3). Appendix D is described below. you should reference these appendices in your report at the top of the document as such: (Appendix A, pg. 5).

• Format: Format for this portfolio is a single PDF with cover page. Your filename should be [YOURSURNAME]_Portfolio_1.pdf.
• This document should be appropriate as a deliverable for a professional industrial engineer in an industry or government setting.
• Evidence: You must quote specific lines from your transcript (Appendix ) to validate your claims.
• Length: There is no required length. The document should should be succinct, readable, and demonstrate attention to detail and considerable thought. It is expected that you have spent approximately nine hours in total on your portfolio assignment; a document that is evaluated well will reflect this work. A document like this should be able to stand as a portfolio item when you apply for a professional industrial engineering job.
  
• Due date: Friday, February 13th at 9 a.m.
• Use of Generative AI: Generative AI may be used on this assignment. The ideas and arguments should reflect your own, and you’re responsible for full understanding of all content. Provide a brief statement of how you used Generative AI technologies, and their contribution of the work you produce. Also provide an “Appendix D” with your entire conversation with Generative AI, if you used it in this fashion.

WRITE MY PAPER