Prerpints.org logo
Preprint
Article

Chef Dalle: Transforming Cooking with Multimodal AI

Altmetrics

Downloads

288

Views

286

Comments

0

Submitted:

03 April 2024

Posted:

04 April 2024

You are already at the latest version

Alerts
Abstract
In an era where dietary habits significantly impact health, technological interventions can offer personalized and accessible food choices. This paper introduces Chef Dalle, a recipe recommen-dation system that leverages multimodal human-computer interaction (HCI) techniques to pro-vide personalized cooking guidance. The application integrates voice-to-text conversion via Whisper, ingredient image recognition through GPT-Vision, and employs TF-IDF vectorization alongside cosine similarity for personalized recipe recommendations. These methods enable users to interact with the system using voice, text, or images, accommodating various dietary re-strictions and preferences. Furthermore, the utilization of DALL-E 3 for generating recipe images enhances user engagement. User feedback mechanisms allow for the refinement of future rec-ommendations, demonstrating the system's adaptability. Chef Dalle showcases potential appli-cations ranging from home kitchens to grocery stores and restaurant menu customization, ad-dresses accessibility, promoting healthier eating habits. This paper underscores the significance of multimodal HCI in enhancing culinary experiences, setting a precedent for future developments in the field.
Keywords: 
Subject: Computer Science and Mathematics  -   Artificial Intelligence and Machine Learning

1. Introduction

The contemporary intersection of technology and daily living has revolutionized how individuals interact with their environments, particularly in personal health and dietary habits. With the global rise in dietary-related health issues, such as obesity and nutritional deficiencies [1], there’s a pressing need for personalized solutions that cater to individual dietary preferences and restrictions. Chef Dalle, a recipe recommendation system, represents a pioneering step in this direction, utilizing multimodal human-computer interaction (HCI) to revolutionize the culinary experience.
Designed to meet a broad spectrum of dietary needs, Chef Dalle integrates advanced technologies through OpenAI API [2], including voice-to-text conversion, image recognition, and machine learning algorithms, to offer a user-centric platform that enhances accessibility and user engagement. Utilizing Whisper [3] for voice-to-text conversion, GPT-Vision [4] for ingredient image recognition, and DALL-E 3 [5] for generating recipe images, Chef Dalle’s recommendation engine employs Term Frequency-Inverse Document Frequency (TF-IDF) vectorization and cosine similarity measures. This sophisticated model dynamically refines its recommendations based on user feedback, ensuring continuous adaptation to users’ evolving preferences.
Chef Dalle’s utility extends beyond home kitchens, with potential applications in grocery stores, restaurants, and even challenging dietary scenarios, such as onboard airplanes or within low-income families. By simplifying meal discovery and preparation, Chef Dalle not only fosters healthier eating habits but also introduces a level of convenience and personalization previously unattainable. Moreover, its multimodal interaction capabilities significantly enhance accessibility, making it an invaluable tool for individuals with diverse abilities and preferences.
This paper delves into the development and application of Chef Dalle, elucidating its impact on the HCI field and its implications for dietary planning and health. Through an exhaustive analysis of its features and functionalities, we aim to showcase how Chef Dalle capitalizes on the latest advancements in artificial intelligence and machine learning to provide a personalized cooking assistant that is adaptable, accessible, and in tune with users’ dietary requirements.
RQ1: How does the integration of multimodal AI technologies in Chef Dalle enhance user experience in recipe recommendations.
RQ2: How do the various modalities of the OpenAI API (Whisper for voice recognition, Dalle-3 for image generation, and GPT-4 Vision for ingredient identification) in Chef Dalle enhance accessibility to users?
RQ3: In what ways does Chef Dalle contribute to promoting healthier eating habits and making home cooking more accessible to diverse user populations?

2. Related Works

The fusion of artificial intelligence (AI) with culinary applications has significantly broadened the scope of recipe recommendation systems, introducing a new era of personalized cooking experiences. This evolution is evident in applications like SuperCook [6], Yummly [7], and BigOven [8], which have redefined meal planning by enabling users to get recipes recommended based on the ingredients they have on hand, using machine learning algorithms and looking through large datasets. These platforms demonstrate the potential of intelligent systems to transform the culinary landscape by making recipe discovery more intuitive and tailored to individual preferences.
The concept of multimodality in these applications extends beyond mere recipe suggestions. It encompasses various forms of user interaction, such as textual inputs, image recognition, and voice commands, thereby making cooking more accessible and engaging for a diverse audience. The integration of multimodal Large Language Models (LLMs) like ChatGPT-4[9], Google Gemini [10], Anthropic Claude3[11], and others have propelled a variety of applications, such as AssureAIDoctor [12], An AI Doctor Assistant which utilizes Dalle and GPT-4, improving the accessibility to medical advice to disadvantaged individuals. Additionally, the Multilingual Eyes Multimodal Traveler’s App (MEMTA) [13], a travel assistant for tourists as well as the visually impaired and multiple other uses, utilizes GPT-4, GPT-Vision, and YOLO v8 Object detection. The Growth Mindset Emojifier (GMSE) [14] App is a feedback tool for educators to utilize emojis in student assignments’ comments, GPT-4 and GPT-Vision. These multimodal tools transformed not only healthcare and educational tools but also revolutionized the culinary world.
Recently platforms have been ditching recipe datasets to generate custom recipes from an AI model. Apps such as PlantJammer [15], and Let’s Foodie [16], focus on reducing waste with the leftover ingredients on hand, allowing users to edit the recipe to create their perfect custom recipe. Alternatively, DishGen [17] and ChefGPT [18] have leveraged AI to generate recipes tailored to users’ inputs, such as ingredients on hand or description of an item. These culinary platforms share similarities with multimodal AI applications in other fields, such as Meta’s Smartglasses [19] and Samsung’s Galaxy S24[20] which leverage AI to interpret the physical and digital world innovatively, from object recognition to intuitive photo editing. Such multimodal applications demonstrate the versatility of AI in understanding and responding to a wide array of user inputs and preferences, thus making technology more accessible and personalized.
Various studies have deeply explored the integration of Artificial Intelligence (AI) into culinary and health-focused applications. Hwang et al. delve into AI’s role in fostering culinary creativity and sustainable cooking practices in their work “Recipe 2.0” [21]. Similarly, Kansaksiri, et al. investigate generative recipes and ChatGPT-powered nutrition assistance in “Smart Cuisine” [22]. Degerli and Tatlisus [23] showed how AI tools can be used for recipe correction, recipe adaptation, recipe detailing, time management, and presentation.
Research on the reliability of dietary advice generated by AI, focusing on individuals with food allergies, is provided by Niszczota and Rybicka [24], while AI’s capability in supporting renal diets is explored by Qarajeh et al. [25]. emphasizing the importance of accuracy and reliability in AI-generated health and dietary recommendations
Moreover, the broader implications of AI in transforming food systems towards enhanced sustainability, efficiency, and consumer experiences across the global food industry are comprehensively reviewed by Pravin and Sundarapandiyan [26], highlighting the technology’s potential to revolutionize food production, nutrition, and culinary experiences globally. From precision agriculture to personalized nutrition and food safety assurance, AI-driven solutions are increasingly adopted, pointing to a future where food systems are more resilient, equitable, and sustainable.
This integration of AI into culinary technologies and multimodal applications not only enriches personal cooking experiences but also heralds a new era of technology-driven solutions across sectors, emphasizing the significant potential of AI to foster innovation, sustainability, and personalization in daily life.

3. Methodology

3.1. System Architecture

Chef Dalle intricately combines AI-driven and web technologies to craft a user-centric culinary platform. Users interact with the system through a web interface, entering ingredients, dietary preferences, and allergies into a form. This input can be enriched by image uploads, processed by the GPT-vision API for ingredient recognition, and voice recordings transcribed by OpenAI’s Whisper API. The system’s backend, developed with Flask and SQLAlchemy, manages user interactions, stores data, and handles complex queries with Flask-Migrate for database migrations.
The core recommendation engine operates on a repository of recipes stored in a CSV file. It employs a TF-IDF vectorizer and cosine similarity to match user inputs against this repository, enhancing recommendations based on user feedback. For visual engagement, recipe images are generated on-demand using DALL-E 3, stored as Base64 encoded strings in the database for efficiency and quick retrieval.

3.1.1. User Interface (UI)

The UI is meticulously designed, focusing on ease of use, aesthetic appeal, and accessibility. Utilizing HTML5, CSS3, and JavaScript, the interface adapts to various screen sizes and devices, ensuring a seamless user experience. AJAX is implemented for asynchronous data fetching, allowing real-time updates (e.g., recipe suggestions) without page reloads, significantly improving the interaction speed. Figure 1 displays Chef Dalle’s User Interface, including the home page, a recommendation, and the profile page.

3.1.2. Flask Application

Acting as the middleware, Flask orchestrates the system’s core functionalities, including request handling, session management, and communications between the front end, AI models, and the database. Its lightweight nature ensures efficient processing, while Flask’s extensive libraries support complex functionalities like user authentication and image processing. The Architecture of Chef Dalle and the role of Flask as a middleware is shown in Figure 2.

3.1.3. Recommendation Engine

This engine uses TF-IDF to extract meaningful features from recipes and cosine similarity to identify recipes that closely match user inputs. It’s designed to refine recommendations based on user feedback, utilizing a dynamic learning approach to improve suggestion accuracy continually. The logic of this code is shown in Algorithm 1.
 Algorithm 1: Recipe Recommendation System
 Input: user_diet_preferences: list of user’s dietary preferences, allergies: list of user’s allergies, recipes_df: data frame containing recipe data, tfidf_matrix: TF‐IDF matrix of recipe content, tfidf_vectorizer: TF‐IDF vectorizer, user_input: list of user’s input or a single string, liked_recipes: list of IDs of recipes the user likes (optional), disliked_recipes: list of IDs of recipes the user dislikes (optional)
 Output: The AI‐generated response containing top recommended recipes that match user preferences and do not contain allergens.
1.  
Initialize user_input_vector by transforming user_input using tfidf_vectorizer
2.  
Compute cosine similarities between user_input_vector and tfidf_matrix
3.  
If liked_recipes is not empty:
4.  
 For each liked_recipe_id in liked_recipes:
5.  
 Find index in recipes_df matching liked_recipe_id
6.  
 If index is found:
7.  
  Compute boost factor using cosine similarity.
8.  
  Apply boost to the corresponding cosine similarity score.
9.  
If disliked_recipes is not empty:
10. 
 For each disliked_recipe_id in disliked_recipes:
11. 
 Find index in recipes_df matching disliked_recipe_id
12. 
 If index is found:
13. 
  Compute reduction factor using cosine similarity.
14. 
  Apply reduction to the corresponding cosine similarity score.
15. 
Apply clipping to ensure cosine_sim values are between 0 and 1
16. 
Sort recipe indices based on cosine similarity scores in descending order.
17. 
Initialize recommended_recipes as an empty array.
18. 
For each index in sorted_indices:
19. 
 Retrieve recipe at index from recipes_df
20. 
 If recipe meets dietary preferences and does not contain allergens:
21. 
 Add recipe to recommended_recipes
22. 
 If the number of recommended_recipes reaches 5:
23. 
 Break loop
24. 
Return DataFrame containing recommended_recipes.

3.1.4. Database

Using SQLAlchemy for efficient ORM, the system maintains a robust database. This includes secure user credential storage, tracking of dietary preferences and allergies, and recording feedback on recipes. Base64 encoded images of recipes are also managed for efficient retrieval, using Flask-Migrate for maintaining database integrity across schema changes.

3.2. AI Models

3.2.1. Whisper for Voice-to-Text Conversion

The integration of Whisper for voice recognition allows users to verbally input their cooking preferences, ingredients at hand, or dietary restrictions. The system captures these voice inputs, converting them into text that is then analyzed to understand user requests accurately. This feature significantly enhances the accessibility and user-friendliness of Chef Dalle, making recipe discovery a hands-free experience that accommodates busy kitchen environments or users with mobility limitations.

3.2.2. GPT-Vision for Ingredient Recognition

GPT-Vision plays a pivotal role in identifying ingredients from user-uploaded images. When a user takes a picture of available ingredients, GPT-Vision processes this image to recognize and list the ingredients. This sophisticated image recognition capability simplifies the input process, allowing users to easily add ingredients to their profile without manually typing them. The identified ingredients are automatically populated into the ingredient form field, streamlining the recipe search and recommendation process.

3.2.3. Dalle-3 for Image Generation

DALL-E 3 addresses the need for visual content within the Chef Dalle platform, particularly when displaying recipe suggestions. If a requested recipe lacks a corresponding image in the database, DALL-E 3 generates a high-quality image representative of the final dish. This not only enriches the user experience by providing a visual expectation of recommended recipes but also enhances the content’s appeal and engagement. The ability to generate images on demand ensures that every recipe, regardless of its source, is visually represented, making the exploration of culinary options more engaging and informative.

3.3. Data Collection and Preparation

The recipe dataset is sourced from Kaggle’s “food-com-recipes-and-user-interactions” by shuyangli94. This dataset provides a comprehensive collection of recipes along with user interactions, but we only utilized the RAW_recipes.csv file which contains 230186 recipes.

3.3.1. Data Cleaning and Preprocessing

The dataset was processed in a Google Colab notebook using the Pandas library to focus on key features relevant to our recommendation system. Here’s an outline of the cleaning and preprocessing steps:
  • Deduplication: Removal of any duplicate entries to ensure dataset uniqueness.
  • Column Selection: Isolates relevant data such as recipe names, cooking times, tags, nutritional content, and ingredients.
  • Textual Cleaning and Normalization: Ensures consistency in the data by removing special characters and standardizing text to lowercase.
  • Handling Missing Values: Drops entries with incomplete data to ensure integrity.
  • Nutrition Data Transformation: Expands and categorizes nutritional information for detailed analysis.
  • Feature Aggregation: Combines relevant attributes into a single feature to improve the matching algorithm’s efficacy.
  • RecipeID Assignment: Each recipe is assigned a unique identifier.
  • Data Integration: The CSV file is loaded into a panda Data frame and ensures the recipeID is treated as strings.
  • Text Vectorization: Utilizes TfidfVectorizer to convert recipe text to feature vectors, while ignoring common stop words.
  • Model Training: Applies TF-IDF transformation to the ‘combined features’ column of the recipe dataset, learning the vocabulary and preparing the model for similarity comparisons.

3.3.2. Database Structure

User Model: Stores information about the user including login, password, dietary preferences, allergies, saved recipes and the user’s feedback.
User Feedback Model: Collection of recipe IDs, user ID and whether the user liked or disliked the recipe. The feedback is used to continually improve the recommendation engine’s accuracy.
Recipe Image Model: Manages the storage and retrieval of base64 encoded recipe images since Dalle-3 returns a URL which expires after an hour. If DALLE 3 generates a new image it is stored in the database for future reference with the matching recipe ID from the CSV file.

3.4. Workflow for Recommendations

1)
The user accesses Chef Dalle’s main page, which displays a form for inputting ingredients, allergies, and dietary preferences. Each field is equipped with a record button, allowing for voice input.
2)
When a voice input is recorded, Chef Dalle sends an API request to OpenAI, utilizing the Whisper model to convert the spoken words into text. The converted text is then automatically filled into the corresponding form field.
3)
If the user opts to use an image instead, they can upload a picture of their ingredients. Upon image upload, Chef Dalle makes an API call to OpenAI, leveraging the GPT-Vision model to identify the ingredients depicted in the image. The recognized ingredients are listed and inserted into the ingredients input field on the form.
4)
After the user completes the input fields either through text, voice, or image, they can submit the form by clicking the “Get Recommendations” button.
5)
The recommendation function is triggered, which collects data from the user inputs on diet, allergies, and ingredients, along with the user’s previous feedback on liked or disliked recipes if they are logged in.
6)
The user input is transformed into a TF-IDF vector, which is then used to calculate cosine similarity scores against the TF-IDF matrix generated from the recipe dataset.
7)
Any user feedback on recipes is applied to adjust the cosine similarity scores—increasing scores for liked recipes and decreasing them for disliked recipes to improve personalization.
8)
The recipes are sorted based on their adjusted cosine similarity scores. The system then filters out recipes that do not align with the user’s dietary restrictions or contain allergens.
9)
The database is checked for Images of the recommended recipes, if no image exists, Chef Dalle submits a request to OpenAI’s DALL-E 3 model to generate a visual representation of the dish, enhancing the user’s browsing experience.
10)
The top recommendations that match the user’s preferences and meet the dietary criteria are then presented to the user.
11)
Finally, the user can provide feedback on the recommended recipes, which Chef Dalle stores in the database. This feedback is utilized to further refine future recommendations, creating a dynamic and adaptive system that evolves with the user’s tastes and preferences.

3.5. Test Cases

To rigorously evaluate Chef Dalle’s effectiveness and versatility, a structured methodology was implemented, focusing on simulating a range of user interactions. This included testing for basic functionality, accessibility features, dietary preferences handling, and user feedback adaptation. Each test case was designed to reflect potential real-world usage scenarios, ensuring comprehensive coverage of Chef Dalle’s capabilities. The methodology also accounted for system responsiveness on various devices and the accuracy of AI-driven functionalities like voice-to-text conversion and image recognition. In the Table 1 below ✓ stands for ‘As expected’.
These test cases illustrate Chef Dalle’s broad functionality spectrum, from accommodating specific dietary needs to leveraging AI for enhanced user interaction. Through systematic testing, Chef Dalle demonstrates a high degree of accuracy and reliability across different functionalities, underscoring its potential to significantly improve the culinary experience for users with diverse needs and preferences.

4. Results

4.1. Enhanced User Experience through Multimodal AI Integration

Chef Dalle introduces a novel approach to recipe discovery by employing AI-driven modalities that cater to a variety of user interactions. The integration of voice-to-text, image recognition, and textual input methods offers a versatile and user-friendly platform. The high accuracy of multimodal inputs is expected to significantly reduce the complexity of finding recipes that align with users’ dietary preferences and restrictions. Although direct user feedback is not available, the system’s design suggests a streamlined process that likely enhances overall user satisfaction and engagement.

4.2. Accessibility

Chef Dalle’s advanced features, including Whisper for voice recognition and GPT-Vision for ingredient identification, play a crucial role in making cooking and recipe discovery accessible to all users, including those with disabilities. For instance, voice recognition enables users with visual impairments to interact with the system hands-free, while image recognition assists users who may find manual ingredient entry challenging as shown in Figure 3. This design philosophy underscores Chef Dalle’s commitment to inclusivity, potentially facilitating a wider adoption across diverse user demographics.

4.3. Promotion of Healthier Eating Habits

By providing personalized recipe recommendations based on individual dietary needs and preferences, Chef Dalle is positioned to influence users’ eating habits positively. The system’s ability to filter recipes according to dietary restrictions, allergies, or specific health-related goals suggests a direct impact on promoting healthier eating patterns. While empirical data from user engagement would provide concrete evidence, the theoretical basis indicates that Chef Dalle could serve as an invaluable tool in guiding users toward more nutritious and balanced meals.

4.4. Scope of Application

The scope of application for Chef Dalle is vast, touching various aspects of daily life and offering transformative solutions to cooking and meal planning challenges across different settings. At its core, Chef Dalle is designed not just as a recipe recommendation system but as a multifaceted tool that bridges the gap between dietary needs, culinary creativity, and accessibility.
In the Kitchen: Chef Dalle stands as an invaluable assistant in home kitchens, where its tailor recipes to the ingredients available, considers dietary restrictions and preferences, and even offers visual guides through DALL-E 3-generated images. This support is crucial for encouraging home cooking, exploring new cuisines, and ensuring nutritious meals are accessible and enjoyable to prepare.
Grocery Stores: Envision a scenario where shoppers at grocery stores can use Chef Dalle to scan items they intend to purchase and receive instant recipe recommendations based on those ingredients. This application not only enhances the shopping experience but also aids in meal planning and reducing food waste by suggesting recipes that use all the purchased ingredients.
Restaurants: For culinary professionals, Chef Dalle could serve as a source of inspiration, helping chefs to devise new dishes or adapt existing ones to meet contemporary dietary trends or ingredient availability. In restaurant settings, it could also offer personalized menu recommendations to patrons based on their dietary preferences, creating a customized dining experience.
Airplane Kitchens: The constraints of airplane kitchens, where space is limited and ingredient storage is challenging, make Chef Dalle a potential game-changer. Providing recipes that can be executed within these constraints can enhance the quality of in-flight meals, offering passengers healthier, tastier, and more varied dining options.
Low-Income Families: One of the most impactful applications of Chef Dalle lies in its potential to assist low-income families. By generating recipes that are not only cost-effective but also nutritionally balanced and tailored to the limited ingredients they might have on hand, Chef Dalle can play a pivotal role in improving dietary habits and food security among economically disadvantaged groups.

5. Discussion

Chef Dalle’s deployment of multimodal AI technologies—voice recognition, image processing, and textual analysis—significantly elevates the culinary experience. This integration reflects a growing trend in HCI, where user convenience and accessibility are paramount. The seamless interaction facilitated by these modalities not only simplifies the recipe discovery process but also embodies the shift towards more intuitive, user-centered design in digital platforms.
Chef Dalle’s emphasis on accessibility highlights the potential of AI to democratize cooking, making it more approachable for individuals with varying abilities and dietary needs. By breaking down barriers to entry, such as the complexity of recipe discovery and adaptation to specific dietary requirements, Chef Dalle aligns with broader societal goals of inclusivity. Continued advancements in AI could further enhance these capabilities, offering even more personalized and accessible culinary aids. The discussion around accessibility also raises questions about digital divide issues, where technology availability and digital literacy may limit access to such innovative solutions.
The potential of Chef Dalle to promote healthier eating habits invites a broader discussion on the role of technology in public health. By providing tailored recipe recommendations, the system has the potential to influence dietary choices directly, encouraging home cooking and consumption of nutritious foods. This aspect of Chef Dalle’s functionality underscores the importance of integrating nutritional science with AI development, ensuring that recommendations not only cater to user preferences but also align with health guidelines.
Chef Dalle’s innovative approach to recipe recommendations has the potential to play a significant role in promoting sustainability, particularly in the context of reducing food waste. By offering personalized recipe suggestions based on the specific ingredients users already have at their disposal, Chef Dalle encourages the efficient use of food items that might otherwise be overlooked or discarded. This not only helps in minimizing food waste but also aids individuals in discovering new and creative ways to prepare meals from their existing pantry items. Such a system can be particularly beneficial in households where leftovers and odd ingredients accumulate without a clear plan for use. By intelligently suggesting recipes that utilize these ingredients, Chef Dalle contributes to a more sustainable kitchen practice, ensuring that food is consumed more mindful and waste conscious. This aligns with broader environmental goals of reducing the global food waste footprint, as reducing waste at the consumer level is critical in addressing the overall challenge of food sustainability.
Exploring Chef Dalle’s potential exposes the need for robust security measures and ethical considerations, particularly concerning data privacy and model integrity. Addressing these concerns becomes critical to maintaining user trust as AI systems become increasingly integrated into daily life. Discussions about model poisoning and privacy breaches underscore the importance of developing AI with security and ethics at the forefront, an area ripe for further research and development.

6. Conclusions and Future Research

Chef Dalle has embarked on an innovative journey to transform the culinary landscape through multimodal AI integration, redefining the recipe discovery and meal preparation process. By adeptly combining voice-to-text, image recognition, and textual input, Chef Dalle has not only made cooking more accessible but has also significantly streamlined the recipe search process. This seamless fusion of technologies facilitates a user experience that is both engaging and highly intuitive, aligning perfectly with modern expectations of digital interaction.
The system’s dedication to accessibility and inclusivity stands as a testament to the transformative potential of AI in democratizing cooking, making it accessible to individuals regardless of their abilities or dietary constraints. Chef Dalle’s approach goes beyond mere convenience, embodying a deep commitment to fostering healthier eating habits and sustainable living. By leveraging AI to provide personalized recipe suggestions, Chef Dalle encourages the efficient use of ingredients, potentially reducing food waste and guiding users toward more nutritious meal choices.
Looking forward, the scope of Chef Dalle’s application is vast, potentially impacting various sectors, including home kitchens, grocery stores, restaurants, and beyond. Its capacity to adapt to constrained environments like airplane kitchens and support low-income families highlights its versatility and alignment with broader societal and environmental goals. The anticipated integration with technologies such as Sora for generating instructional cooking videos promises to enhance the culinary experience further, making sophisticated recipes accessible to all skill levels.
However, the journey ahead is not without its challenges. Robust security measures and ethical considerations, particularly data privacy and model integrity, are paramount to ensuring the system’s continued trustworthiness and effectiveness. Addressing these concerns will be crucial as Chef Dalle becomes increasingly integrated into daily life.
In conclusion, Chef Dalle represents a significant leap forward in leveraging AI to enhance human-computer interaction within the culinary domain. Its innovative approach not only improves the user experience but also promotes healthier eating habits, accessibility, and sustainability. As we look to the future, continuous improvement and research will be vital to unlocking Chef Dalle’s full potential, making it an indispensable tool in kitchens worldwide and a leading example of how AI can be harnessed to improve our quality of life.
Chef Dalle’s potential integration with upcoming technologies such as Sora for instructional video generation presents exciting possibilities for enhancing the cooking experience. This perspective feature could revolutionize how users learn to cook, making complex recipes accessible through step-by-step video guides. There are also plans to create a feature that integrates an AI assistant to help take a user through each step of the cooking process for recipes, including setting timers to remind the user it’s time for the next step.
Additionally, future research could explore the specific impact of each modality on user engagement and satisfaction, providing insights into how similar approaches could be optimized for other applications. Further studies could investigate the long-term impact of such systems on dietary behavior and health outcomes.

Author Contributions

Conceptualization, B.H.; methodology, B.H.; software, B.H.; validation, Y.K. and J.J.L.; formal analysis, B.H.; investigation, B.H.; resources, Y.K.; data curation, P.M.; writing—original draft preparation, B.H.; writing—review and editing, Y.K, J.J.L. and P.M.; visualization, B.H.; supervision, P.M.; project administration, Y.K.; funding acquisition, Y.K. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the NSF, grants awards 1834620 and 2137791, and Kean University’s Students Partnering with Faculty 2023 Summer Research Program (SPF).

Data Availability Statement

Data available on request due to privacy restrictions (personal nature of human-chatbot communication).

Conflicts of Interest

The authors declare no conflicts of interest.

References

  1. Facts Sheet – Malnutrition. Available online: https://www.who.int/news-room/fact-sheets/detail/malnutrition (accessed on 15 March 2024).
  2. API Reference – OpenAI API. Available online: https://platform.openai.com/docs/api-reference (accessed on 15 March 2024).
  3. Speech to text – OpenAI API. Available online: https://platform.openai.com/docs/guides/speech-to-text/quickstart (accessed on 15 March 2024).
  4. Vision – OpenAI API. Available online: https://platform.openai.com/docs/guides/vision (accessed on 15 March 2024).
  5. Image generation – OpenAI API. Available online: https://platform.openai.com/docs/guides/images (accessed on 15 March 2024).
  6. SuperCook – Zero Waste Recipe Generator. Available online: https://www.supercook.com/#/desktop (accessed on 15 March 2024).
  7. Yummly: Personalized Recipe Recommendation and Search. Available online: https://www.yummly.com/ (accessed on 24 March 2024).
  8. 1,000,000+ Recipes, Meal Planner and Grocery List | BigOven. Available online: https://www.bigoven.com/ (accessed on 15 March 2024).
  9. ChatGPT. Available online: https://chat.openai.com/ (accessed on 24 March 2024).
  10. Gemini. Available online: https://gemini.google.com/app (accessed on 24 March 2024).
  11. Claude. Available online: https://claude.ai/chats (accessed on 24 March 2024).
  12. Y. Kumar et al., “AssureAIDoctor- A Bias-Free AI Bot,” 2023 International Symposium on Networks, Computers and Communications (ISNCC), Doha, Qatar, 2023, pp. 1-6. [CrossRef]
  13. W. Villalobos, Y. W. Villalobos, Y. Kumar, J. Jenny Li, et al. (2024) The Multilingual Eyes Multimodal Traveler’s App. 7th International Conference on Information and Computer Technologies (ICICT 2024).
  14. Kupershtein, Y. Kumar, A. Manikandan, et al. (2023) ChatGPT: A Game-Changer for Embedding Emojis in Faculty Feedback. In proceeding of the 19th International Conference on Frontiers in Education: Computer Science & Computer Engineering (FECS 2023). [Google Scholar]
  15. Plant Jammer – Inspiration. Available online: https://www.plantjammer.com/empty-your-fridge/inspiration (accessed on 15 March 2024).
  16. AI Recipe Generator (100% Free) | Lets Foodie. Available online: https://letsfoodie.com/ai-recipe-generator/ (accessed on 15 March 2024).
  17. DishGen | AI Recipes | AI Recipe Generator. Available online: https://www.dishgen.com/ (accessed on 15 March 2024).
  18. ChefGPT - Your AI-Powered Personal Chef . Available online: https://www.chefgpt.xyz/ (accessed on 15 March 2024).
  19. Ray-Ban Meta Smart Glasses | Meta Store | Meta Store. Available online: https://www.meta.com/smart-glasses/ (accessed on 24 March 2024).
  20. Galaxy AI | Mobile AI on Galaxy S24 Ultra | Samsung, US. Available online:. Available online: https://www.samsung.com/us/smartphones/galaxy-s24-ultra/galaxy-ai/ (accessed on 24 March 2024).
  21. Angel Hwang, Samy Badreddine, Frederick Gifford and Tarek R. Besold, “Recipe 2.0: Information Presentation for AI-Supported Culinary Idea Generation”, Proceedings of the 14th International Conference on Computational Creativity (ICCC), 2023.
  22. Kansaksiri, P. Panomkhet, P., & Tantisuwichwong, N. Smart cuisine: Generative recipe & chatgpt powered nutrition assistance for sustainable cooking. Procedia Computer Science 2023, 225, 2028–2036. [Google Scholar] [CrossRef]
  23. DEĞERLİ, A. H. TATLISU, N. B. Cooking with ChatGPT and Bard: A Study on Competencies of AI Tools on Recipe Correction, Adaption, Time Management and Presentation. Journal of Tourism & Gastronomy Studies 2024, 11, 2658–2673. [Google Scholar] [CrossRef]
  24. Niszczota, P. , & Rybicka, I. The credibility of dietary advice formulated by CHATGPT: Robo-diets for people with food allergies. Nutrition 2023, 112, 112076. [Google Scholar] [CrossRef]
  25. Qarajeh, A. Tangpanithandee, S., Thongprayoon, C., Suppadungsuk, S., Krisanapan, P., Aiumtrakul, N., Garcia Valencia, O. A., Miao, J., Qureshi, F., & Cheungpasitporn, W. AI-powered renal diet support: Performance of CHATGPT, Bard Ai, and Bing Chat. Clinics and Practice 2023, 13, 1160–1172. [Google Scholar] [CrossRef]
  26. Pravin, M. Sundarapandiyan, R. Integrating Artificial Intelligence in food systems: Future trends, innovations, and prospects for sustainable development and enhanced culinary experiences. International Journal for Multidimensional Research Perspectives 2024, 2, 2584–0029. [Google Scholar]
Preprints 103077 g001
Figure 1. Chef Dalle app. (a) Home page; (b) Recipe Recommendation page; (c) Profile Page.
Figure 1. Chef Dalle app. (a) Home page; (b) Recipe Recommendation page; (c) Profile Page.
Preprints 103077 g001
Preprints 103077 g002
Figure 2. Chef Dalle System Architecture.
Figure 2. Chef Dalle System Architecture.
Preprints 103077 g002
Preprints 103077 g003
Figure 3. App GUI snapshot: (a) Output from Image Identification; (b) Image uploaded.
Figure 3. App GUI snapshot: (a) Output from Image Identification; (b) Image uploaded.
Preprints 103077 g003
Table 1. Test cases for Chef Dalle.
Table 1. Test cases for Chef Dalle.
Case # Description Input Data Expected Output Test result
1 Basic recipe recommendation using single ingredient User inputs “pasta” System recommends pasta recipes ✓
2 Allergy filtering User inputs “pasta” as ingredients and “peanut” as an allergy System excludes recipes containing peanuts ✓
3 Voice to text conversion accuracy multiple items: ingredients User says “beans, onions, tomatoes, cilantro” into the microphone System accurately converts spoken words to “tomatoes, onions” comma separated in the input field ✓
4 Voice to text conversion accuracy single item: allergy User says “vegan” into microphone System accurately converts spoken words to “vegan” ingredient in the input field ✓
5 Dietary preference accommodation User inputs “vegan” as diet and inputs “pasta” as “ingredients” System recommends vegan pasta recipes ✓
6 Image recognition for ingredient identification User uploads an image of “flour, cinnamon sticks, butter, baking powder, sugar, brown sugar, eggs, milk” System identifies “flour, cinnamon sticks, butter, baking powder, sugar, brown sugar, eggs, milk” and adds to ingredients input field ✓
7 Recipe image generation with DALL-E 3 User requests a recipe without an existing image System generates and displays a new recipe image. Some images have a camera in the photo. ✓
8 User dislike feedback adaptation User dislikes a “beans, rice, cilantro” recipe, then searches for “beans, rice, cilantro” System doesn’t show disliked recipe and returns different set of recommendations ✓
9 User like feedback adaptation User likes a pasta recipe, then searches for pasta recipes again System prioritizes similar pasta recipes in recommendations ✓
10 Usability on mobile devices Accessing Chef Dalle on a smartphone System is fully functional with responsive design ✓
11 Multi-ingredient recipe suggestions User inputs “chicken, lemon, garlic” System suggests recipes that use all three ingredients ✓
12 Handling rare ingredients User inputs “kohlrabi” System suggests appropriate recipes or alternatives if direct matches aren’t found ✓
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Copyright: This open access article is published under a Creative Commons CC BY 4.0 license, which permit the free download, distribution, and reuse, provided that the author and preprint are cited in any reuse.
Prerpints.org logo

Preprints.org is a free preprint server supported by MDPI in Basel, Switzerland.

facebook logotwitter logolinkedin logo
MDPI Initiatives
Important Links
Subscribe

Disclaimer

Terms of Use

Privacy Policy

© 2024 MDPI (Basel, Switzerland) unless otherwise stated