Internal Service Portal Responsive Design.
Machine Learning/AI
A machine learning project, Diagnostic Engine, aims to revolutionize the property management industry by collecting data to automate touch points, standardize pricing, and create predictive work orders.
Role
User Experience Architect
Tools
Miro
InVision
Sketch
Deliverables
Mockups/Prototypes
User testing
Digital Analytics
Presentation
Team
Karn Kapoor (PM)
Jojo Dupuy (UID)
Problem #1
How might we collect structure data in our proposals and invoices to feed our NLP model?
First, there is a need to start collecting structured data by capturing commonly used industry services codes and statistically pricing data for plumbing, electrical and HVAC services through our existing proposals and invoices. The first attempt of implementing proposals which collected structured data came back with much frustration on the part of our affiliate user. Affiliates were frustrated with how much longer it took for them to complete proposals now with the new structured data collection version of the proposal amongst other changes that caused confusion.
Problem #2
How might we build a forward facing interface that allows the Call Service Reps to know that our model is working to produce an accurate prescriptive work order diagnosis with minimal user touches?
Next, a Prescription is a set of one or more predicted resolution codes that can range from very high-level to very specific. Depending on the confidence level of the model, the Prescription can include multiple alternative specific resolution codes which are equally likely for a given work order. We need to integrate our AI/ NLP prescription model & diagnostic (elimination) questions services bundle to One Queue for pilot group and link with services codes for a feedback loop. This requires a heavy front-end change to the One Queue service portal that will affect how Call Service Reps (internal user) interact with the platform.
Solution
Solution 1
Structuring Proposals and Invoices in a way that leads us away from the free form text box formatting of legacy and introducing a structured drop down approach to choosing service codes in order to feed our machine learning model. From our research, we also learned that users like to see everything all on one screen at one time. Data showed having more information above the fold of the screen for users made for a more pleasurable experience. Improving the user experience in a way that allows for users to effortlessly input task and fill out the proposal quickly, so they can proceed to other proposals.
Solution 2
The purpose of launching a diagnostic engine that generates Prescriptions allows us to:
→ Set a dynamic Not to Exceed (NTE) on work orders (WOs) that leads to cost efficiencies for customers.
→ Dispatch the work order to the appropriate Affiliate (trade, skill, price) which leads to higher customer satisfaction.
→ Automate Proposal and Invoice processes for Affiliates leading to higher retention of Affiliates.
→ The logic behind how the Prescriptions model will work is captured in the diagram to the right.
Impact
This is a one of a kind project that has the potential to revolutionize the property management industry. The potential impact of a project varies, but some immediate results would be:
→ Capture 95% of commonly used Resolution Codes for plumbing and HVAC, and less than 70% for all other trades by 2022 Q4.
→ For each res code, capture pricing (fixed or labor/material) from over 100 unique affiliates proportionately across each national region and set pricing thresholds.
→ Capture structured res codes on more than 75% of work orders completed and keep custom Task Not Listed date less than 30%.
→ Establish baseline accuracy of Description to Prescription ML Model (using elimination diagnostic questions) by comparing prescriptions to the actual resolution.
→ Reduce time take by internal SMS’ers (Call Service Reps) to create work orders by 15% in pilot group.
→ A machine learning data model that collects thousands of work orders, proposals and invoices to eventually become self efficient and able to predict the resolution to the work order received, the appropriate price and the best vendor to do the job.
Key Features & Screens
Structured Proposals and Invoices
Affiliates will need to select from a drop down of options to populate their incurred cost, task and additional fees. They will also need to provide labor hours, cost and materials in a structured formatted way to ensure our machine learning model has all the attributes needed to provide predictive pricing and work orders. Users have also stated having all pertinent information and tasks on one screen all at once is a desired outcome.
Prescription Codes
The aim of this launch plan is to lay out how the Diagnostic Engine Prescriptions will be piloted – beginning with operationalization of the model technically and then preparing it for production- and identifying what outcomes will be analyzed to drive next steps, with the goal of productionizing the AI model mid-2023.
The plan will focus on steel-thread of Plumbing i.e. generate accurate specific prescriptions for the Plumbing category in Residential, while generic and item level prescriptions will suffice for other Trades. This will require a major overhaul of the current forward facing UI and potentially a learning curve for our Call Service Reps.
Prescription Codes User Flows in Phases
Phase 1
UI/UX (front-end): Description Box on Queue > Service Combo selection > Troubleshooting Q > Prescription
Thus, a Call Services Rep (CSR) will continue to create WOs the way they currently do, with no change to the Queue UI.
Job-to-be-done: Validate feasibility of machine learning framework and architecture used validate the hypothesis/questions to be answered, including baselining the confidence level for prescriptions establish platform for real-time iterations and calibration of models
Phase 2
UI/UX (front-end): Description Box on Queue > Service Combo Selection > Troubleshooting Q’s + Diagnostic Q’s > Prescription codes
Queue will surface some new questions (diagnostic questions) on the front-end troubleshooting section.
Job-to-be-done: Validate viability of diagnostic questions establish an error tolerance level for prescription predictions.
Phase 3
UI/UX (front-end): Description Box > Description Completeness (static elements/ DQ) > Service Combo selection > TSQ > Prescription
Thus, a CSR will continue to create WOs the way they currently do, but the UI may be modified to guide CSR to input critical information. The guidance will not be dynamic or prompted by the AI model.
Job-to-be-done: Validate the hypothesis/questions to be answered Identify any changes to AI model features (input data) required.
Phase 4
UI/UX (front-end): Description Box > Description completeness (dynamic elements/ DQ) > Service combo selection > TSQ > Prescription
CSR will use a slightly modified interface that guides them in real-time as they create a WO on how to input relevant information as needed by our model. They will still complete all the other steps that were required by the original Queue process. The Diagnostic Questions will include relevant legacy troubleshooting questions, and the troubleshooting section may only retain key questions.
Job-to-be-done: Validate feasibility of ML framework and architecture used Validate the hypothesis/questions to be answered, including baselining the confidence level for prescriptions establish platform for real-time iterations and calibration of models.
Phase 5
UI/UX (front-end): Description Box > Description completeness (dynamic elements TSQ+DQ) > Prescription > Service combo determination
At this stage, we will likely have the prescription model predicting the prescription which will trigger the related service combination for legacy system compatibility. The dynamic troubleshooting and DQs will be triggered based on responses of CSR that will feed into the specific prescription model to derive a high probability prescription using the validation/backward induction method.
Job-to-be-done: Determine the feasibility and accuracy of the Diagnostic Engine prescription model with the goal of productionizing this solution in its current form (model v5.0).
Phase 6
This phase will include the deployment of a production-grade AI model and downstream implementation of prescriptions in business processes (NTE, dispatch, automatic SC update etc.)
On the technical infrastructure level, re-training and feature engineering will be automated/dynamic.
Testing
Jobs to be done questions for proposal usability update:
Emotional Job-to-be-done:
→ What’s your mind frame when you are completing multiple proposals?
→ React to a time when you found it difficult not having all components presented at once. What the first emotion that comes to mind?
→ React to the review section and how it made you feel?
Functional Job-to-be-done:
→ Does having “add a task” front & center a priority?
→ What issues do you have finding res codes today?
→ How do you utilize the right-hand rail if you do at all?
Takeaways
→ Although this project was terminated due to budget concerns; cross - scrum team collaboration was crucial for this project. Changing the front end of the One Queue service portal had a ton of down stream affects that would change the way Call Service Reps interacted with our other products that were the byproducts of the work order creation such as proposals, invoices, dispatching etc. Bringing in all members of the team early and often was vital in order to keep this project moving in a timely manner.
→ There was resistance from the users to change how they input their data from a free form format to a structured format. Affiliates complained that it took much more time for them to complete everyday tasks. User testing gave us insights on how we can possibly remedy this issue, but we also learned that overtime Affiliates began to adopt to this new way of data collection.
→ Stakeholder management was the biggest learning curve for me during this project. Having the ability to allow everyone to be heard, but also not allow for momentum to stop was an obstacle I had to overcome with this project. My approach during all meetings with stakeholders was as follows:
Thank stakeholders for their feedback and repeat back their concerns. I would consider the what, why and how of this project and reference back to this as often as possible because with a project that is on the cutting edge of technology there were times stakeholders and the teams focus would wander.
Identify the problem and define the solution that solves the problem that way we can stay on task and stakeholders can see a clear connection from problem to resolution.
Empathize with the users, which were mainly internal ‘SMSers’, and then bring those insights to stakeholders while appealing to the business and how these insights would help us achieve our goals.
Lastly, locking in agreement from the team making sure all team members were clear on their next steps and confirming acceptance with stakeholders about the proposed course of action.