Polynomial or Linear Regression? (Part 2)
On working with data science - cont.
Welcome back to the ‘how to work with data scientists’ series. This is the second and last post in this short series.
In the first post (here) I introduced some of the basic concepts in data science so the whole topic will feel more approachable to you and you can have professional discussions with the DS team.
In this post I’m gonna focus on the actual interaction with the DS team, what you should keep in mind and what is your role in such interactions.
But first - let’s recap on the challenges we’re trying to tackle:
The challenges
When product management meets data science the following challenges may surface:
The data science team is unwilling to work with a product manager. They may claim that their daily routine involves a lot of research and their work is hard to quantify. They may not embrace any agile methodology and refuse to give commitments nor accept deadlines.
The product manager (you) may lack the confidence to interact with the team given the deep academic & engineering concepts.
You are both willing to work together, but it just doesn’t ‘click’. The PRDs (specs) you compose, which were perfect for the engineering team, are not well understood nor fit how the DS team works. Something just doesn’t work with the ongoing interaction.
You are working together, the vibe is good, it seems like you are on the same page - but time flies by and the DS team fails to deliver a true business impact. The models are not converging, the training takes forever, or whatever. They have perfect explanations of why this is fine… but the business suffers.
Your role
Now that we’ve covered the high level challenges, let’s discuss your role when it comes to interacting with data science teams.
First, from a high level perspective - the division of responsibilities between you and the DS team needs to be exactly the same as with any other engineering team.
You are responsible for the 3 w’ - the why, the what and the when.
They are responsible for the ‘how’.
It means that you should determine what they should work on. You must be able to explain why it needs to be worked on and when.
The DS team, on the other hand, will decide how they would like to tackle the challenge, which models to use and provide you with EE of how much time they believe it’s going to take.
The differences between the engineering team and the DS team are becoming apparent when switching from a ‘high level perspective’ to the day-2-day work and planning. But we’ll touch it in a bit.
Your second role as a product manager who interacts with the DS team is to make sure that everything they do is aligned with the business goals and making an impact in a timely manner.
Being responsible for the 3 w’ somewhat encapsulates this statement already, but I wanted to emphasize the ‘timely manner’ aspect because when it comes to the DS teams - there is a true risk of divergence. Your role is therefore to make sure it doesn’t happen, or at least reduce the risks of this happening to the minimum.
Why is there a risk of divergence?
As you’ve learned in the previous post - the data science work involves developing models, training them and iterating over this until the results are satisfying.
Unlike engineering problems, when the team comes up with a design which is reviewed and then put into work, with high chances of solving the problem/building the feature successfully - the data science problems are different in their nature.
Whether it’s a prediction problem, an optimization problem, classification, anomaly detection or any other data science problem category - it’s usually challenging to know in advance which type of approach (model, training set, features selection, etc..) will deliver the best results. Therefore, if the DS has made the wrong choices when considering the right approach - a true business impact may take time to show up… if at all.
You can help mitigate this, or at least reduce the chances of this happening. We’ll soon see how.
Working with data science teams
Well - after all these introductions and background information we finally reached the essence of this post. Let’s dive in.
Step 1 - setting up expectations
If you joined a company which already has a data science team, and you interact with this team on an ongoing basis then the first step would be setting up expectations.
The team may be working for a while with or without a product manager involved. You need to understand the current setting first.
If they worked with a product manager before you came on board - how was it for both sides? What worked well? What didn’t work well? Are you going to be the only PM interacting with this team or other PMs are interacting with it as well.
As a guideline - I am a strong believer in a single point of contact for each team (unless it’s a team which provides services to other teams like Devops, but this is not the case for DS teams). Therefore, if there are several PMs interacting directly with the DS team - you should make an effort to change this. There should be only one PM serving as a gateway to this team. If it’s not you - then you may not need all the advice I’ll provide below, though it’s good to know anyway.
If the DS team didn’t work with a product manager before on a constant basis - then this needs to be changed. You need to look into why that is. If it’s because no PMs in the company felt confident enough - then great - you can be the pioneer on that front.
However, if the DS team didn’t work with any PM so far because they don’t think they need one - then proper education is due. Working through this resistance may pose a challenge, so you first need to understand the reasons for such a resistance. The main reasons I personally encountered were:
They worked with a PM before but this person wasn’t qualified to work with data scientists and hence their experience was poor.
They believe they know what is expected of them on the long run, and they believe that you intervening will just slow them down
(the worst option) They prefer to take their time and not be held accountable for not meeting deadlines. Working with you just means a ‘more stressful’ working environment as they see it.
There could be other reasons, of course, but whatever it is - you’ll need to work through this. You need to open a direct channel to their team leader or head of data science - and convince him/her to give you a chance. You should highlight the benefits of such a cooperation:
More direct business impact
More timely deliveries
Greatly reducing the chances of them spending the time on projects that will later be scraped.
At the end of the day, I personally believe that most people want to make a positive impact in their workplace. Data scientists are no different. Unless a person got the wrong impression that they he/she were hired merely to play with data in a sandbox, they should hear you out.
However, in order to convince them - you’ll need to prove to them that you know what you’re doing.
Therefore, before talking to anyone, ask yourself why you believe the DS team in your company can make a big positive impact and what projects they should be focused on.
Until you have concrete projects that you strongly believe should be part of the roadmap - don’t engage with the team.
Once you have an initial plan, or ideas you strongly believe in - make your move and build a relationship with their leader (and later with each team member).
Step 2 - choosing the projects that move the needle
As I noted before - you are the one responsible for the why, the what and the when. If you completed the previous step successfully then this authority is already established and agreed upon.
This authority brings a lot of responsibility.
The DS team is probably the most expensive team in the company if you measure it by the average cost per team member.
If you instruct them to work on the wrong projects - you’re going to burn a lot of money to your company without bringing meaningful business results.
But no pressure 🙂
Don’t worry, though. Things are generally not that dramatic. The DS team was hired and formed for a reason, and it usually has a lot to do with the core of the business.
The CEO, when he/she goes raising funds they usually brag about their ‘advanced AI’ for taking care of the problem they are trying to solve. You just need to make sure that what they brag about - is actually happening..
Hence, you should have a sense of direction of where the focus of the team should be on and the team probably knows it as well. It’s therefore a matter of fine tuning the efforts of the team.
For example - if the claim to fame of the company is a web service that given an image returns a list of all the ‘named entities’ within the image (the name of any object or person which/who are featured in the image) - then the DS team is probably constantly trying to improve the accuracy of this entity extraction process by experimenting with various models.
You, as a product manager, can help them create a solid roadmap of what would be possible, when and with what accuracy.
For example, you may decide that being able to successfully identify famous people and VIPs is in higher priority than identifying famous geographic locations or monuments (like the Golden Gate bridge in San Francisco). And then, when zooming in on VIPs, you may decide that identifying politicians is in higher priority than identifying other celebrities.
You are also the person who needs to define when to stop working in a direction that doesn’t yield the desired results, or on the contrary - is already good enough.
The example I provided above is quite simple, of course. Most companies in reality have bigger projects and more possibilities to choose from, so let’s take another example:
Imagine a company which sells surveillance software with the promise to automatically alert about various crimes happening in real time using an advanced computer vision AI. Naturally, as a product manager, you’ll have to do quite a lot of work for defining all the possible ‘crimes’ you’d like the AI to be able to identify and in what level of accuracy. The DS team will then need to align their efforts according to your roadmap.
Things can get much more complicated depending on what the company does. Imagine a company with AI technology for forecasting diseases on people based on a person's profile. Imagine a company with AI which identifies malfunctions in production lines. Imagine a company with AI that optimizes marketing campaigns and so forth.
On one hand - everybody knows what the DS team should focus on, but on the other hand - when you drill down to the details - there are a lot of options to choose from, a lot of KPIs to track, a lot of constraints to obey and performance goals for each.
This is where you should step in and build a solid roadmap for the team, based on the product strategy - and then ‘spec’ it out (see below).
Needless to say, you are not working in a vacuum. When working on such a roadmap you need to be in constant communication with the DS team leader. This person will help you understand what’s possible and in what time frame. Working closely with this person will create an alignment between the business expectations and reality.
From time to time, though, and depending on the business, you may identify a brand new project that the team may work on. It could be that someone else in the company ‘tipped you off’ or came up with this idea, but you were convinced that this side project, that no one originally thought about - is very aligned with the company’s vision and it’s a perfect fit for the DS team to work on.
For example - let’s take the company mentioned before which sells surveillance software for identifying crimes. And let’s assume that the company becomes very successful with selling its software to airports. You, as a product manager, may discover, while talking to your customers, that a major pain point of the airport security officers is to find the owners of ‘forgotten baggage’ in a timely manner. The customers are willing to pay quite a bit if you could upgrade your software to find owners of forgotten baggage in less than 5 minutes (I’m making this all up of course, I have no idea if anyone really cares about this).
This could be a very interesting project for the DS team to pursue. Your next step is therefore to check the feasibility of this with the team lead, and if feasible to see where it fits into the roadmap.
Without you - most likely no one would have promoted this business opportunity, and it probably would have been missed.
Step 3 - ‘Spec-ing’ out each project
Back then I wrote a couple of posts on how to write an awesome spec (see part 1 and part 2). Sadly, when it comes to writing specs for the DS team - some of the things mentioned in these posts don’t apply here.
You see, the specs you write for the DS team need to be structured a bit differently than the specs you compose for the engineering team.
The specs you write for the data science team should start the same - by elaborating on the ‘why’. Mainly, you should provide an overview of the problem, the goals and the success criteria and KPIs of how to measure success. You may also add a ‘definitions’ section just to make sure everyone is using the same terminology across the board. You can find all the small details (and an example) in the links above.
However, the sections of the spec which deals with the ‘what’ (user stories + functional requirements) need to be stripped down to the bare minimum as most of the time the ‘what’ is some type of model which you are probably not qualified to describe or even to choose. Personally, I rarely use ‘user stories’ in a spec for data scientists and the functional requirements will only include:
Performance requirements - for example: ‘the input must be processed within 10ms’.
Constraints - for example: ‘the model must not recommend an item that was recommended within the last hour’
Reporting & analytics - for example: ‘it must be possible to answer the following product related queries about the outputs….’
Alerting - for example: ‘In case [something happens] a message must be sent to [Some Slack channel] with this description’
Reliability - for example: ‘The service must provide uptime availability of 99.9%’
Interfaces & APIs - for example: ‘The model will pull [some data points] from [a specific storage or API which is the single source of truth for this information]
I guess you can add some other sections, but overall it should continue the same trend - you are only framing the product that needs to be built and let the DS figure out the rest.
Also, I assume this is obvious, but I’ll mention it anyway - most of the time - specs aimed for the DS team don’t have a section devoted to UX/UI, as their projects are mostly purely backend.
By providing them with the KPIs of how they measure whether the model is doing well or not, and providing the framing of the product - they have everything they need to start working.
Step 4 - The day-2-day
Once you have the roadmap (and following that the quarterly plan) and all the projects are spec-ed out you need to make sure the team is progressing effectively with their commitments.
I mentioned it before and I’ll repeat it again - the data science team, due to their more ‘research’ nature (of both their projects and the people involved) - have a bigger tendency to diverge than the engineering team.
Officially, it’s the responsibility of their team/group lead to keep them in line. However, I’ve observed that on many occasions, the team lead of the DS team is more of a technical & academic authority rather than a true manager, in terms of project management at least.
It’s therefore up to you to make sure everything progresses as planned.
Here are tips that should help you with that:
“Scrum it” - do exactly what you do with the engineering team: dailys, sprint planning and proper sprint commitments. You may embrace other ceremonies, but those are the crucial ones.
Be very involved in the sprint planning. Many times you’ll hear that a team member refuses to commit to a delivery because the project requires some research time and the results are unknown. That’s fair. Work to minimize the risks. Can you cap the effort required for understanding whether this is leading nowhere or it’s actually promising? Never agree to let the team spend time on research without any capping on the maximum amount of time to put into this. Merely having the discussions about it will send the message that endless research is non-acceptable. Also, question every experiment, but not in an annoying way. Just ask the right questions: “what will we know at the end of this experiment?”, “how does it support the quarterly plan?”, “what’s the business value of this?”. Again, by merely asking those you’ll see that you’ll be saving tons of time & money to your company by avoiding running unnecessary experiments that are probably leading nowhere.
Involve the team with the various planning processes. The team must be part of the quarterly planning and the sprint planning. Each team member must provide explicit commitments which are measurable. An example of a bad commitment: “I’m going to work on the linear regression experiment”. This is not a commitment. This is a statement about what the person is going to work on, and from a project management perspective it’s almost meaningless. A proper commitment would be something like “I’m going to work on the linear regression experiment and by the end of the sprint we’ll know what is the features set we should focus on”. Ok. Now we have clear goals. If the team misses their goals quite often, then have some retros with the team, and understand why. Explain to everyone what is the (negative) business impact of these delays.
Celebrate the team’s achievements. The team made a positive business impact? Even if it's a small one - communicate it to them and celebrate it. This will help connect the team to the business and help motivate everyone to focus on projects that move the needle.
Make sure there is a ‘control’ for all relevant projects - this is one of the most powerful tips I can give you. I wasn’t sure whether to put this tip on the previous step about ‘specs’ or here. But since I recommend making it a design pattern for almost every project - I put it here. Anyway, in the previous post I explained the concept of having a control group (or ‘control’ in short). Almost any project the team works on needs to have a ‘control’. It means that some of the traffic/inputs will be going through a ‘dummy’ model that probably just randomizes outputs. This allows you and the team to constantly measure their performance and contribution to the world, as the ‘control’ represents what the performance would be if the data science team didn’t exist. It also implicitly sends the message that the team is accountable for their deliveries and their impact on the business is being monitored. You can even use the ‘control’ for defining the quarterly team goals in terms of business impact (‘we commit to deliver an uplift of 5% over the control’ for example).
By applying the tips above the team will understand that everything they work on must be directly correlated to business outcomes. Their time must be devoted to something that at least in theory moves the needle in a meaningful way.
And when you loop back their positive impact - it will keep the team motivated and in line with the business.
Summary
Data scientists are highly intelligent individuals with vast academic knowledge and are on the top-tier of the salaries in the hi-tech industry. Working with data science teams as a product manager can therefore be intimidating. I get it.
However, we are all humans at the end of the day, and we all have strengths and weaknesses. Data scientists are not unique in that aspect. Their weakness, from what I’ve observed, is that they may stir away and focus their efforts and time on projects that do not necessarily align with the business goals, or simply don’t converge.
If your product is dependent on this team then it’s therefore your duty to get involved with them by learning their language, building solid relationships, setting expectations, making them part of the planning process and getting them to work according to agile methodologies.
It’s also your duty to take ownership over the roadmap and make sure each minute of their time is devoted to projects that are moving the needle.
That’s it for today.
If you found this post/series useful - feel free to ‘like’ it. If you think others can benefit from it - feel free to share it with them.
Thank you, and until next time :-)