Two questions we often receive are:
- How do you get your data?
- How accurate is your data for predicting trends and market shares?
By now, our regular readers know that we acquire our data browsing public sources throughout the web. For question #2, the answer is more complex. This post gives you more insight knowledge on how we decided to do it.
When we find a product used by an institution, we try to add as many dates as possible. We gather different types of dates in our database: RFP, purchase, and implementation. Even though RFP dates and purchase dates can be helpful to understand the whole project and the process of implementing a new product, implementation dates are the crux of our database. This explains why we aim to get as many implementation dates as possible for our products. The implementation dates help us understand trends and market share. But, as we often say: our database does not have all implementation dates. We currently have between 70% and 80% of the implementation dates for the products we have listed in our portal. This means that when you see 100 implementations for a specific product category in a said year, our database has 20% more data for this segment.
We also know that we don’t have all the data. When we compare our data with some of our clients, we often only have about 80% of their customers. Multiple reasons can explain this reality; some of which being:
- the implemented system is behind a firewall or hosted on a secure portal (so we can’t find it);
- no publicly available sources mention the product with the institution;
- the implemented system uses a branded product name linked to the institution.
We often see systems listed in contracts that don’t make it to our database. Many contracts will sell products in a bundle. If we have doubt that one product of the bundle is not used, we remain cautious, and we will not list it. We only add a product if we have enough proof of usage.
How to Define “Proof of Usage”?
If we don’t see an actual « front door » URL (ex.: moodle.university.com), we try to find multiple links to prove it. This is where the manual detective work of our employees comes in. A good example of proof of usage is an employee guide + a system maintenance message on the status website of the university or a Twitter message from a student talking about the product + an invoice. This is not a perfect method, but the more we find proof, the more we are confident in product usage.
We can also find proof of usage on an institution’s current IT project page, board approvals or student announcements. These sources of information also help us identify products that have been replaced if, within two years, a product is decommissioned.
No Implementation Date?
As mentioned earlier in this post, we prefer to gather all dates (RFP, purchase, implementation), but any date will help us position the product on a timeline. If we can’t find the actual implementation date for a specific product, we use the purchase date instead. If we don’t have the purchase date, we use the RFP date.
Our Implementation dates can also change. An example is the implementation of an ERP. Imagine an institution saying that they have just purchased a new system and that it will be implemented in July two years from now. We add this to our system. The following year, we find that the date moved… we adjust our date.
We often get questions from vendors saying that they had the best sales year in 2010, but our data is not showing it. Usually, this is because implementations are done a few months or even years after the purchase date. Another question our clients and edtech companies ask: why does our data trail so much? We have always said that our data is trailing a few months. At the end of any given year, we have between 60 to 65% of all implementations for this year.
A few reasons for this are:
- Our web scrapers are not (and will never be) very fast. Would you like to be blacklisted?
- We find data on product purchase or implementation but not the actual dates. We will most likely find this information in the contract the following year.
- The system is live, but we don’t see it or have not found a document (student FAQ, staff training) to prove it.
We often get offers from companies to share new customers lists. This is something that we refuse in order to remain impartial and not overrepresent one product in a specific category. We might ask for general numbers to help us better understand if our work is on track with reality, but this will be the furthest we will go.
Do you want to know something else about LISTedTECH and how we work? Reach out on social media @LISTedTECH or by email at marketing@listedtech.com. We will be pleased to answer your questions in an upcoming post.
