EV Myth 3 – My Vendor has a Solution with High Accuracy so I’m Set

Many vendors today claim 90% or 95% accuracy, which sounds like a solid track record. But here again, it’s time to turn to the science. 

What do we need to be measuring? How do we define what success looks like in a data science classification algorithm? How do we know if a data science algorithm is good? Or bad? Or somewhere in between?

Let’s put this in the context of a television meteorologist. Every day that person makes a prediction as to whether it will rain tomorrow. And TV viewers usually assume he or she will be right most of the time. 

But what does it look like if we actually plot the meteorologist’s predictions over time vs. the actual weather? A true positive occurs if the meteorologist predicted rain, and it did in fact rain. A false positive occurs if the meteorologist predicted rain, and no rain fell. A false negative occurs when the meteorologist predicted sunny skies, and instead we got caught in a downpour. And finally, if the forecast is for no rain, and no rain falls, we have a true negative. 

We can think about EVs in this same way. Do I predict the customer has an EV? Or do I predict they don’t have an EV? And then do they actually have an EV? Or do they not have an EV? 

The simplest measure of accuracy quantifies the percentage of true positives and true negatives over the total number of predictions. The higher the percentage the better.

But there are other important metrics as well. 

Precision, for example, quantifies the number of times a meteorologist predicts rain and it rains, or we predict a customer has an EV and there is an EV in the household.  And again, we always want to maximize this result. 

Then there is recall. Here we want to understand out of all the EVs in my territory, how many am I actually able to detect. Recall is the measure of the number of your true positives over both your true positives and false negatives combined. 

And finally, there’s the false positive rate. 

Let’s look at this in practice. 

EVs are growing in popularity, but currently most communities in the U.S. have an average of 1-2% penetration. This changes how we view success. I can easily create an algorithm that finds no one has an EV. In a territory with two percent EV adoption, I would actually be able to claim 98% accuracy for that algorithm. On the other hand, if we use a generic approach such as the one I described when we were examining Myth 2, we would have quite a few false positives, so the accuracy measure would be absolutely terrible – under 30%. 

Taking it a step further, it’s important to play with the back-and-forth  between false negatives and false positives. What is better for the outcome? Is it better to minimize false positives? Which means we won’t be telling customers they have an EV if they don’t, but we might also be missing a lot of customers who do have EVs. Alternatively, we might decide it’s really important to maximize the number of detections, by identifying everyone who has an EV, even if we end up suggesting to many people that they have an EV when they don’t, and increasing our false positives. (In most cases, our recommendation is to avoid false positives, because they erode consumer trust in the utility.) 

The key takeaway is that to determine EV detection success requires a bigger picture understanding of what goes into an accuracy claim, as well as how measures of precision, recall and false positives figure into the equation that determines how accurate an algorithm is. A simple claim of general accuracy doesn’t equate to EV detection accuracy.

EV Myth Four – What Works for One Use Case Will Work for All

We’ve described above how EV intelligence is critical input when managing EV growth. And that is true across the board, for both customer and grid use cases, including targeted EV program marketing, EV owner engagement, EV program planning, infrastructure planning, EV growth analysis, and EV forecasting. 

Given that there are such a wide range of potential EV intelligence applications, the question that we must analyze is whether the same data science approach can be employed for all. 

The truth is that while it is tempting to look for a single data silver bullet, different use cases have unique EV intelligence requirements.

For example, in the context of targeted program marketing, a lower level of accuracy might be acceptable, but establishing EV customer engagement requires a much higher level of accuracy. In the first instance, we don’t need to know a great deal about the customer — only that they own an EV. Promotional language that uses more generic phrasing is acceptable – such as to recommend a new charger purchase or invite a customer to join a managed charging program.  

By contrast, customer engagement requires personalized messaging, and in order for that personalized messaging to be effective, it must be accurate and highly relevant. For instance, if we tell a customer that, based on their current charging time of use, they could save $X if they were to switch to a time of use rate, or send them an alert that they’ve charged during a peak period, what we say must be true. Similarly, billing estimates have to be highly accurate. False positives, or incorrect hyper-personalized content, create a negative experience for the customer, and an increased likelihood that they will lose faith in your solution. 

Identifying the precise level of accuracy required for each unique EV use case is key to successfully employing EV intelligence.

EV Myth 5 – It’s Too Hard to Really Leverage This Data 

Maybe you don’t have in-house analysts, or the data team you do have is already overburdened. In either case, it’s tempting to believe that you don’t have the resources to leverage behind-the-meter EV intelligence. 

At one point in time, this was likely an accurate notion. In a territory with a million customers, each one is generating ~35,000 AMI data points annually. Once we add the estimation from every single device in the home, suddenly we’re faced with crunching billions of data points. With legacy tools, analysis at this scale would require days or weeks of work.

But today, Bidgely’s UtilityAI Analytics Workbench tool is helping utilities affordably and efficiently distill those billions of data points into actionable EV intelligence, regardless of their in-house analytics capacity. Analysis that historically would have taken multiple highly skilled analysts several weeks to complete manually can now be accomplished within hours even by non-analysts.

If I am a program manager, and I’m really interested in understanding the utility’s EV load shape, a single click applies the filter I need to reveal where my EV owners are located geographically and what their load shape is. It’s also possible to map substations, feeders and transformers to EV customers in order to pinpoint where electric vehicles fall on the network, and the load shape that is occurring on each grid asset. 

I invite you to visit our demo portal (contact me at [email protected] to request access)  to see for yourself. Teams across the utility – including those with limited technological sophistication – can use Analytics Workbench to distill EV intelligence from AMI data and put it to work to inform and improve the complete range of EV-related use cases. It is easy for analysts and program managers alike to use the tool and its outputs to make data-driven decisions.

EV Myths busted. Now what?

Learning to leverage data science to inform utility operations is a process that takes time. Given the anticipated rapid pace of transportation electrification, now is the time to inform EV programs with behind-the-meter intelligence so that you have  sufficient time to design, refine, and scale.

If you are curious about how other utilities are managing their data transformation, check out the case studies, video interviews and more featured on our EV solutions web page and reach out to our team to schedule time to learn more. 

And don’t forget to WATCH my webinar where I dispel these EV myths, and provide an overview of UtilityAI Analytics Workbench: EV Intelligence.