If it looks like a quadrant and sounds like a quadrant then is it magic? Not always. A cornerstone of many analyst firms research repertoire’s are vendor evaluation methodologies.
What are evaluative research reports for B2B technology?
Starsight defines evaluative reports as research methodologies that screen and include B2B technology solutions and/or service providers in a given category and compare their offerings against set criteria to help technology buyers understand the market landscape and select a vendor in an RFI. They almost always provide a visual representation of the players in a market as and often detailed feature or use case comparisons, which can be used as a starting point by tech buyer teams in the B2B buying process to build long lists, refine short list and even make a final purchase decision. They are not just listings and unidimensional rankings.
The evaluative methodologies of the FIGs are the most well known as the Gartner Magic Quadrant, the Forrester Wave and the IDC Marketscape, but there is a vibrant ecosystem of other reports out there. Vendors in dynamic (or noisy) market categories can expect to have RFIs every month from a number of different firms attempting to make sense of your market. However, each of those requests comes with a bandwidth requirement as vendors are expected to provide questionnaire responses, vendor briefings, product demos and sometimes customer references as part of the research process. Therefore, as analyst relations professionals (AR pros), it’s imperative to understand which ones are relevant for your business, which ones are being read by your buyers and which ones you can ignore.
The Gartner Magic Quadrant.
The ubiquitous Magic Quadrant, or MQ in industry terms, is used and abused by tech-buyers everywhere and thus, it has become notorious among technology providers. With 125 Gartner Magic Quadrants covering as many technology markets, it’s a goal of many companies to feature in one of these reports. The Gartner MQ is a 2×2 grid with vendors plotted, based on their scores for completeness of vision and ability to execute, into 4 groups: leaders, challengers, visionaries and niche players. Its companion methodology is the Gartner Critical Capabilities, evaluating vendors according to use cases and published using the same RFI as the MQ.
Gartner’s signature research method has been perfected over many years. The MQ has a robust methodology. A participating vendor can expect to spend over 150-hours working on one, spread across getting on the analyst’s radar, filling in the 200+ question RFI and preparing for product demos. The end report is based on the factual information provided by vendors themselves and analyst opinion formulated from end-user analyst inquiries, industry expertise and customer reviews on Gartner Peer Insights. Where there is any question of validity, Gartner has an Ombudsman to investigate and take corrective action as needed.
The Forrester Wave.
Forrester are the trend spotters and as such, its Wave typically covers software and digital transformation categories. The Wave graphic plots vendors according to current offering and strategy (x and y axis) and as of 2024 the third dimension (dot size) will be customer feedback, whereas it was previously market presence (market share). The Wave now categorises vendors into three buckets: leaders, strong performers and contenders, prior to this year there was also a challenger category. It also now offers an interactive vendor selection tool.
Unlike other evaluations, the Forrester Wave provides a granular, weighted scorecard for each vendor. This allows both buyers and vendors to understand the exact areas where a company excels or lags, delivering transparency in scoring and weighting by breaking down vendor strengths and weaknesses in detail. For analyst relations professionals, this methodology is particularly valuable because it presents clear, actionable insights on how vendors can improve their positioning in future assessments, while offering clients a straightforward view of where each vendor stands within a competitive market. However, those weightings are not provided during the RFI process, where vendors are required to fill in a questionnaire, provide a product demo and vendor briefing and supply customer references to the analysts.
The IDC MarketScape.
IDC are the number crunchers and their landscape vendor evaluation, the MarketScape, reflects this. The report evaluates technology vendors based on a combination of both current capabilities and future strategies (x and y axis) and displays a vendors relative size in market (dot size) based on IDC market share data. IDC also now has ProductScapes which is an AI-based methodology that lists vendor offering functionalities, though this is not an evaluation per se.
What sets the IDC MarketScape apart is its emphasis on a 360-degree view of the vendor landscape. It offers a global perspective and thorough analysis of each vendor’s ability to execute and innovate in their respective market segments. The MarketScape evaluates not only product or service offerings but also the vendor’s overall business strategy, including innovation, customer service, and market leadership. Based on this, the bubble-shaped graphic of the IDC MarketScape groups vendors as leaders, major players, contenders or participants in the given category.
Beyond the FIGs there are many more vendor evaluations.
Vendors should adopt a portfolio approach to vendor evaluations – don’t fall into the trap of only focusing on the FIGs. The long tail of analyst firms also have evaluative research methodologies that are read by technology buyers. This is especially true for those firms covering niche markets where they are trusted advisors to many companies in their given category. Below is a non-exhaustive list of the other vendor evaluations out there beyond FIGs with descriptions from the analyst firms themselves explaining what makes their methodology unique. Ignore these players at your peril!
BARC Score.
What makes it unique: “Many ranking models do not accurately reflect the market development of a vendor as often a small change is enough to move from one to an opposite category. BARC Score was designed to overcome this. Moreover, through showing vendors scores on dimensions “Portfolio Capabilities” and “Market Execution” the reader receives a clearer picture of software’s capabilities vs. vendors market execution.”
Dresner Advisory Services Wisdom of Crowds.
What makes it unique: “Wisdom of Crowds® Market Reports offer in-depth research and reporting on key industry and technology topics, including user trends, perceptions, intentions and other drivers. Each report includes a section with objective and inclusive vendor ratings. Our research is independent, objective, and 100% data driven.”
Everest PEAK Matrix.
What makes it unique: “Everest Group’s PEAK Matrix provides an objective, data-driven comparative assessment of service and technology providers based on their overall capability and market success across different global and regional markets.”
Fosway 9-Grid.
What makes it unique: “The Fosway 9-Grid provides in-depth independent analysis of the main HR, talent and learning solutions for European and International enterprise organisations, captured in 5 dimensions: potential, performance, presence, TCO and trajectory. All underpinned by expert analysis, detailed functional profiling, independent customer research and reference analysis, and detailed vendor and product assessment.”
Frost & Sullivan Frost Radar.
What makes it unique: “The Frost Radar uniquely tracks companies’ momentum in growth and innovation, offering a dynamic visual benchmark that captures their future potential. Unlike static reports, it highlights transformative shifts in the industry, providing actionable insights. Each company is carefully selected by analysts for its leadership in navigating rapid market evolution and disruption.”
GigaOm Key Criteria and Radar.
What makes it unique: GigaOm Key Criteria and Radar Reports are written by engineers, architects and hands-on business practitioners that have operational, implementation and management expertise. Our reports reflect how an end user buyer evaluates technology and how the technology enables the business.
ISG Buyers Guide.
What makes it unique: “The ISG Buyers Guide is an assessment of software providers’ and product offerings meet enterprise requirements using a RFI-based methodology. Instead of assessing vendors’ vision or execution, we examine the product and customer experience that enterprises need for successful deployment. We do more than a Quadrant, but rank software providers across 7 evaluation categories, PX, CX and overall.”
Kaleido Intelligence Vendor Hub.
What makes it unique: “Kaleido’s Vendor Hub evaluative research provides an impartial analysis of roaming and connectivity vendors, emphasising product innovation and market alignment. Through a rigorous multi-method approach, including interviews, demos, and feedback, vendors are assessed on strengths and future potential, ensuring a balanced, data-driven evaluation for informed decision-making.”
KuppingerCole Leadership Compass.
What makes it unique: “The KuppingerCole Leadership Compass covers a wide range of segments in IAM and cybersecurity with a high degree of granularity and a multi-dimensional approach. It stands out in the depth of information provided. It does not exclude small vendors, but provides full market coverage. Reports are updated regularly.”
Nucleus Research Value Matrix.
What makes it unique: “In the Nucleus Research Value Matrix vendors are positioned according to the relative usability and functionality of their solutions, as well as the value that customers realized from each product’s capabilities, presented as a snapshot of the market rather than an empirical ranking of vendors. Positioning is informed by conversations with end-users, along with the most recently released capabilities and areas of vendor investment.”
Omdia Universe.
What makes it unique: “Omdia Universe is a robust yet highly digestible guide for technology buyers. We cover only topics where Omdia holds significant market data and insights, meaning the thorough vendor evaluation builds from a fact-based foundation. Through Informa the Universe has unparalleled exposure to buyer audiences from the enterprise and channels (Canalys).”
PAC RADAR.
What makes it unique: “The PAC RADAR is positioned as a tool for ICT decision-makers to compare providers’ strategies, competencies, and market strength. Our RADARs cover the European market as a whole or go even deeper at country level, highlighting vendor capabilities and market strength in France, Germany and the UK.”
Quadrant Knowledge Solutions SPARK Matrix.
What makes it unique: “QKS Group’s SPARK Matrix excels through a comprehensive research approach, combining proprietary databases, extensive secondary sources, and detailed primary research, including client interviews and market estimation. The unique SPARK Matrix visually ranks market participants, offering strategic insights that provide a more reliable and actionable analysis than competitor reports.”
Research in Action Vendor Selection Matrix.
What makes it unique: “The Vendor Selection Matrix™ is a primarily survey-based methodology for vendor evaluation where 63% of the evaluation is based on a survey of enterprise business decision makers and 37% on the analyst’s opinion. For each report we interview 1,000 marketing and business managers with budget responsibility in enterprises globally.”
If you are an analyst firm with a vendor evaluation report not listed here, please contact us to get added to the list.
Check out the latest transmissions.
- Is there magic beyond the quadrant? The top 15 alternative methodologies to the Gartner Magic Quadrant.
- Analysts are VIPs – Very Influential Persons.
- 5 benefits and challenges for vendors participating in Gartner’s Emerging Market Quadrant.
Ludovic, you forgot the Nelson Hall NEAT 🙂
Thanks Rahul, we’ve asked all of them and waiting for a reply 🙂
Also, if you consider the analyst sides of Avasant and ISG then IPLs and RadarViews
Hi, very interesting, thanks.
I’m surprised not to see ISG or Avasant mentioned.
In France Markess, though on a different scale, is also producing rankings.
Thanks for your comment PY. We’ve asked them and will be adding them. Markess publishes rankings which aren’t a vendor evaluations according to our definition.