ResearchWiseAI‘s answers to ESOMAR‘s 20 Questions to Help Buyers of AI-Based Services for Market Research and Insights

A. Company Profile

What experience and know-how does your company have in providing AI-based solutions for research?

ResearchWiseAI began prototyping and developing solutions leveraging the modern wave of generative AI to solve problems in market research in mid-2023. Our founding team has over a decade of experience delivering production solutions prioritizing data security with a background in market research, social research, medical research, and medical services. Additionally, we have over 45 years of experience solving market research problems.

Where do you think AI-based services can have a positive impact for research? What features and benefits does AI bring, and what problems does it address?

We believe that modern AI will have a similar impact on the market research industry as the vast majority of automation has had on many industries throughout history. AI will greatly reduce the effort required for researchers to do the most routine and repetitive elements of their work. At AI’s current abilities, even when focused on the simplest parts of a market research project, we should still have a human-in-the-loop, to protect quality. That is why at ResearchWiseAI we are focusing on developing a platform that produces draft documents and presentation assets for researchers to adapt and use.

What practical problems and issues have you encountered in the use and deployment of AI? What has worked well and how, and what has worked less well and why?

Firstly, like all data analysis, the quality of the input data is an integral part of producing high-quality outputs, AI is no different. Ensuring users know this and submit high-quality data is key to a successful experience.

Generative AI, is vulnerable to hallucinations and has a high failure rate when compared to more traditional AI and deterministic algorithms; therefore, it is key to place guardrails around the automated analysis work to reject invalid responses from AI. We have found building a layered system of protections, including static algorithms, other AIs, and a human-in-the-loop works well to protect against these issues.

Finally, we have seen some challenges around handling the significantly different lengths of responses provided by market research respondents and automated sentiment analysis. Yet again, we have found building multiple layers of AI helps to mitigate this.

B. Is the AI capability/service explainable and fit for purpose?

Can you explain the role of AI in your service offer in simple, non-technical terms in a way that can be easily understood by researchers and stakeholders? What are the key functionalities?

ResearchWiseAI uses AI providers to automate the process of generating an initial analysis of a relatively simple market research dataset. A user uploads a dataset using an Excel sheet or CSV file, and the system then asks the user one or more questions about the project. Once the question(s) are answered, the AI will analyze each column, writing summaries of the data collected, assigning determining and assigning themes to open-text responses where possible, performing sentiment analysis on open-text responses, and finally generating an overall summary of the project in prose. The user is then able to talk to the AI via a chat interface to ask follow-up questions of the AI regarding the dataset.

What is the AI model used? Are your company’s AI solutions primarily developed internally or do they integrate an existing AI system and/or involve a third party and if so, which?

We leverage existing AI tools and systems, sometimes with additional fine-tuning performed by ResearchWiseAI. Fine-tuning and all other training are never done using client data. Our two AI providers are currently AWS and OpenAI. We use AWS for sentiment analysis via their Comprehend product. We use OpenAI’s GPT-3.5 Instruct, GPT-3.5, GPT-4, and GPT-4o models to provide generative AI and classification services.

How do the algorithms deployed deliver the desired results? Can you summarise the underlying data and the way in which it interacts with the model to train your AI service?

We do not train models using client data. All training is performed using datasets generated by simulations created by ResearchWiseAI.

C. Is the AI capability/service trustworthy, ethical and transparent?

What are the processes to verify and validate the output for accuracy, and are they documented? How do you measure and assess validity? Is there a process to identify and handle cases where the system yields unreliable, skewed or biased results? Do you use any specific techniques to fine-tune the output? How do you ensure that the results generated are ‘fit for purpose’?

Throughout the development of ResearchWiseAI, we do extensive testing comparing the results generated by our AI-powered automation and manually created by experienced market research analysts. Additionally, we invite clients to perform this exercise themselves using existing results from an earlier analysis. All outputs are designed to be the first draft used, with the ability for the end-user to edit any asset within their chosen word processing, presentation, and spreadsheet packages.

What are the limitations of your AI models and how do you mitigate them?

Although modern AI models are becoming more accurate and more powerful all the time, there are still limitations to their abilities to reason and generate reliable outputs consistently. Our mitigations to these issues are as discussed before, by employing checks with traditional programming and additional AI layers. A second challenge encountered when using Large Language Models to analyze large datasets is that they have limits on the number of tokens that can be analyzed at one time. We can mitigate the token limit issue by sampling the dataset, running parallel analyses, and/or merging the resulting analysis.

What considerations, if any, have you taken into account, to design your service with a duty of care to humans in mind?

As stated previously, our route is based on keeping the human-in-the-loop. ResearchWiseAI conducts initial of analysis to provide researchers with an accelerated start, it does not replace the human and the human remains responsible for whatever is delivered to the end user.

D. How do you provide Human Oversight of your AI system?

Transparency: How do you ensure that it is clear when AI technologies are being used in any part of the service?

Our entire product centers around using AI to automate initial analysis, and we make this clear in all of our branding and marketing. AI is mentioned clearly on our website's homepage.

Do you have ethical principles explicitly defined for your AI-driven solution, and how in practice does that help to determine the AI's behaviour? How do you ensure that human-defined ethical principles are the governing force behind AI-driven solutions?

ResearchWiseAI is used to conduct initial analysis of the data. The ethical use is the responsibility of the human-in-the-loop.

Responsible Innovation: How does your AI solution integrate human oversight to ensure ethical compliance?

All users are told that the outputs generated by ResearchWiseAI should be considered the first draft to be used in a report or presentation written by a person. We include a slide in the presentation export recommending that the user double-check the generated results.

E. What are the Data Governance protocols?

Data quality: How do you assess if the training data used for AI models is accurate, complete, and relevant to the research objectives in the interests of reliable results and as required by some data privacy laws?

We generate our own training datasets using simulations that are tailored to fine-tune the AI models we use. The results of the fine-tuning process are then analyzed to evaluate the effectiveness of the new model. No user data is used in the process.

Data lineage: Do you document the origin and processing of training or input data, and are these sources made available?

All data used to perform AI training by ResearchWiseAI have been generated by us, not using any other data, and are proprietary.

Please provide the link to your privacy notice (sometimes referred to as a privacy policy). If your company uses different privacy notices for different products or services, please provide an example relevant to the products or services covered in your response to this question.

Here is our Privacy Policy.

What steps do you take to comply with data protection laws and implement measures to protect the privacy of research participants? Have you evaluated any risks to the individual as required by privacy legislation and ensured you have obtained consent for data processing where necessary or have another legal basis?

We comply with the data protection law of the United Kingdom and GDPR by publishing this information in our Privacy Policy and guarantee to keep the information up to date.

What steps do you follow to ensure AI systems are resilient to adversarial attacks, noise and other potential disruptions? Which information security frameworks and standards do you use?

All of our AI providers have created and implemented their guardrails to ensure resilience. Additionally, we ensure that the libraries we use to parse AI responses are scanned for vulnerabilities and are kept up to date.

Data ownership: Do you clearly define and communicate the ownership of data, including intellectual property rights and usage permissions?

We comply with the data protection law of the United Kingdom and GDPR by publishing this information in our Privacy Policy and guarantee to keep the information up to date.

Data sovereignty: Do you restrict what can be done with the data?

ResearchWiseAI’s data center is located in the Republic of Ireland, which is where we store data and do most of our processing. Currently, the only processing we do out of Ireland is when we process data with OpenAI’s whose servers are located in the United States. When data is transferred to the US, it is encrypted in transit using TLS and it is only held transiently by OpenAI.

Ownership: Are you clear about who owns the output?

Yes, outputs generated by ResearchWiseAI are owned by the organization that owns the dataset submitted, i.e. the client.