End to end deep learning-based Mobile app.
COOK SMART
Hi There, Today in this post I am gonna share my experience building my dream application, i.e Cook smart.
We will discuss what is the problem statement, the idea, and execution step by step.
Please download the app and wait for downloading the model too.
WHY?
The Problem Statement
This was the problem that I faced whenever I thought of cooking, and it's not just me. I have heard the same from many of my friends. They have a similar problem i.e. We have ingredients at our home but still lack the knowledge of preparing a different kind of food. While preparing a meal or snack item, people tend to prepare the same old item which they have been preparing for years. The list hardly consists of 5 to 10 recipes which are regulated often. Hence, I came up with an idea of making different delicacies with the same ingredients we had earlier. An idea to make different food every time gathering the homely ingredients
HOW?
The Concept
So the idea goes like.. before making any meal we have to scan the ingredients by opening the cook smart app. what we have with us. Once the items are successfully recognized it will search for recipes using the ingredients and shows the results in a list format.
How to use this app ?
Here is a gif of the app. and its workflow.
WHAT?
We will learn
We will go through so many different practices for completing this app. Those are Image classification with Tensorflow, Native android application development, Tensorflow lite with Native android app development, Python Scripting, Python web scraping, Python Flask-API development, Serverless Function (GCP functions or lambda in AWS) with python, Mongo DB, Firebase Realtime Database, And for designing all the UI/UX done with Adobe Xd. Let’s see how we can make an awesome app using all this. and note as all this will be live, we will see how to deploy everything. So Keep on going.
The Complete system design for the application. We will go through each block as we follow through.
Ingredients image classification
Here comes the part of Machine learning usage in this application. For doing this again we need to divide this task into smaller chunks. i.e,
- collecting data for ingredients.
- Train a deep neural network model for this.
- Once it will acquire satisfactory accuracy we can save into Firebase.
Collecting Data
No one can download each ingredient image from the web, and at least we need 500 images for each ingredient. So I used a script using google_image_download() lib for downloading images from google. Here is my script for collecting images for ingredients.
While working on this google_image_download() I found out that all images are not being downloaded from this lib, So I added a recursive function manage_download() to download all required no of images.
For this work, I collected little less than 12k images, at around 160 images as each ingredient, we have 74 ingredients (output labels). :)
Preprocessing and Train a Model using these images
The golden part to make a model that will classify ingredients. As our data is ready, we need to start training for the model. So there are two major ways of doing this.
- Transfer learning with MobileNetV2, inception v3
- own CNN model.
Transfer learning: In simple words, transfer learning is the way we will be using a pre-trained CNN model with our existing model, it will help us for feature extraction. but first, let’s download.
IMG_SHAPE = (IMAGE_SIZE, IMAGE_SIZE, 3)# Create the base model from the pre-trained model MobileNet V2
insp_base_model = tf.keras.applications.InceptionV3(input_shape=IMG_SHAPE,include_top=False,weights=’imagenet’)
here the base model has MobileNetV2 or we can take InceptionV3 or VGG16 also, here is a sample code on how we can use the above model.
Then we will compile the model and fit(train it). here is my code for compilation.
callback list maintains early_stoping(i.e if the model doesn't improving on accuracy stop it), checkpoint_save(i.e for every iteration check if we got any good accuracy then previous if true then save the model file.), learning_rate_reduction(i.e dynamic learning rate change) all these so that we will not miss any improved model for our project.
insp_model.compile(loss=’categorical_crossentropy’,optimizer=adam,metrics=[‘accuracy’])history = insp_model.fit_generator(train_generator, epochs=100, callbacks = callbacks_list, validation_data=val_generator)
Once the model got trained. we can save the model file into a model.tflite, here is the code for that.
What accuracy do I found?
I found out that the Vgg16 layer worked very well.Take a look here.
saved_model_dir = 'save/fine_tuning'
tf.saved_model.save(model, saved_model_dir)
converter=tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
with open('model.tflite', 'wb') as f:
f.write(tflite_model)and if you are using google colab like me, we can download a certain file with the below code. or save to
from google.colab import files
files.download(‘model.tflite’)
files.download(‘labels.txt’)2nd Way is to create your own model.
So for this way we need to use the CNN filter, max pooling, and dropout layer according to us, so here is the way I have managed to create the model.
model = tf.keras.Sequential([tf.keras.layers.Conv2D(filters = 32, kernel_size = 3, input_shape=input_shape_img, activation=’relu’, padding=’same’, ),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.5),tf.keras.layers.Conv2D(filters = 32, kernel_size = 3, activation= ‘relu’, padding=’same’, kernel_initializer = ‘he_uniform’),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.5),tf.keras.layers.Conv2D(filters = 64, kernel_size = 3, activation= ‘relu’, padding=’same’, kernel_initializer = ‘he_uniform’),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.5),tf.keras.layers.Conv2D(filters = 64, kernel_size = 3, activation= ‘relu’, padding=’same’, kernel_initializer = ‘he_uniform’),
tf.keras.layers.MaxPooling2D(2),
tf.keras.layers.BatchNormalization(),
tf.keras.layers.Dropout(0.5),tf.keras.layers.Flatten(),
tf.keras.layers.Dense(128, activation= ‘relu’),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(len(train_generator.class_indices), activation=’softmax’)
])model.compile(loss='categorical_crossentropy',optimizer=adam,metrics=['accuracy'])history = model.fit_generator(train_generator, epochs=100, callbacks = callbacks_list, validation_data=val_generator)#Loss plot
acc = history.history['accuracy']
val_acc = history.history['val_accuracy']
loss = history.history['loss']
val_loss = history.history['val_loss']
plt.figure(figsize=(8, 8))plt.subplot(2, 1, 1)
plt.plot(acc, label='Training Accuracy')
plt.plot(val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.ylabel('Accuracy')
plt.ylim([min(plt.ylim()),1.2])
plt.title('Training and Validation Accuracy')plt.subplot(2, 1, 2)
plt.plot(loss, label='Training Loss')
plt.plot(val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.ylabel('Cross Entropy')
plt.ylim([0,3.0])
plt.title('Training and Validation Loss')
plt.xlabel('epoch')
plt.show()
Save model into an executable format (in this case .tflite)
Once the model is ready we need to save the tflite formatted model somewhere where the android application gets access too. For this, I have used Firebase. Here is my script for uploading a file into Firebase realtime Db.
Making REST API Endpoints
For Making REST API Endpoint, I followed these steps. i.e
- Collecting Recipe Data.
- Cleaning recipes data and make these into recipes object.
- Saving into Mongo DB.
- Serverless functions to access these endpoints over the internet.
Collection (Web Scraping Recipe Data)
For my recipes, I scraped these data from http://allrecipes.com/. A big shout out to team allrecipes.com, If these guys won’t have the recipe data it could take me 100x more time to collect data.
Cleaning Class for recipes and making them into objects.
Here they have two different user interfaces for different recipes, I found out there are major 2 types of UI. and I got more than 1000 Indian recipes, fetching from two types of UI. With this, I had to differentiate the ingredient's name. i.e The data I got for ingredients is like “1 teaspoon crushed garlic”, and I had to convert them into amount:1, ingredient: garlic, method: crushed, so for doing this I used spacy lib.
Here is a snippet of my recipe object looks like.
Saving To Mongo
This is a very simple step where we just need to import pymongo. and using insert_one() we can add all recipes into our mongo DB.
Then you will find out data like the below images. Here I have used the compass for visualizing data in Mongo DB. To make it live I have uploaded the data into Mongo DB Atlas. atlas gives 500mb free mongo hosting, and I have only around 20mb of data, which is a perfect fit for me.
Serverless functions
For communicating with this DB, we need an endpoint where we can get these recipes by querying the ingredients. For this, we can either choose Flask Server-based API or Cloud serverless endpoints. I have two APIs. i.e
- Getting recipe list with ingredient names
- Getting Recipe detail with the recipe id to showcase the complete recipe to the user.
I choose serverless API as it supports 2 million calls for a month. What more anyone can ask for.
Android App
Now comes the part where we will combine all the chunks for making an android app that can really solve the problem. In order to do this we need to follow these steps:
- Integration of Tensorflow with android app.
- Using Model.tflite file and label.text file for making a prediction in-app.
- Calling REST APIs for recipes and recipe details.
For designing this application I have used Adobe XD for user experience design and User Interface design.
The first step became easy because on the Tensorflow web they have given a sample app using Tensor flow lib. Still, if someone finds it hard just add to the app.gradle file as below.
// Build off of nightly TensorFlow Lite
implementation 'org.tensorflow:tensorflow-lite:0.0.0-nightly'
implementation 'org.tensorflow:tensorflow-lite-gpu:0.0.0-nightly'
implementation 'org.tensorflow:tensorflow-lite-support:0.0.0-nightly'Then you can get access to using TensorFlow-lite into android apps. Change your model and label file in ClassifierFloatMobileNet.java
@Override
protected String getModelPath() {
return "model.tflite";
}
@Override
protected String getLabelPath() {
return "labels.txt";
}and it will reflect in CameraActivity.java This detects ingredients from the camera directly. Which looks like the below image.
- *image* — — Camera Activity in Android app
For calling an API with android I have used AsyncHttpClient which is an extension of okhttp. Here is a function with this we can fetch recipes in the android application.
private void getRecipesData(String ingredientStr) throws JSONException, UnsupportedEncodingException {
avi.show();
mTextProgress.setVisibility(View.VISIBLE);
recipesList = new ArrayList<>();
AsyncHttpClient client = new AsyncHttpClient();
// Log.v("ingr_pass", ingredientStr);
JSONObject jsonParams = new JSONObject();
jsonParams.put("ingredients", "" + ingredientStr); //{"ingredients":"chicken,onion"}
StringEntity entity = new StringEntity(jsonParams.toString());
client.setTimeout(15 * 1000); // 15 sec timeout
client.post(ListOfRecipeActivity.this, RECIPES_URL, entity, "application/json", new AsyncHttpResponseHandler() {
@Override
public void onSuccess(int statusCode, Header[] headers, byte[] responseBody) {
Gson gson = new Gson();
String jsonOutput = new String(responseBody);
if (!jsonOutput.trim().equals("[]")) {
mEditSearchRecipe.setVisibility(View.VISIBLE);
llNoData.setVisibility(View.GONE);
avi.hide();
mTextProgress.setVisibility(View.GONE);
Type listType = new TypeToken<ArrayList<Recipes>>() {
}.getType();
recipesList = gson.fromJson(jsonOutput, listType);
populateRecipes(recipesList);
System.out.println("recipes:" + jsonOutput);
ingrListStr = "";
} else {
llNoData.setVisibility(View.VISIBLE);
avi.hide();
mTextProgress.setVisibility(View.GONE);
ingrListStr = "";
mEditSearchRecipe.setVisibility(View.GONE);
}
}
@Override
public void onFailure(int statusCode, Header[] headers, byte[] responseBody, Throwable error) {
avi.hide();
mTextProgress.setVisibility(View.GONE);
ingrListStr = "";
}
});
}and for populating the data as a list i used recycler view and with help of an adapter.
private void populateRecipes(ArrayList<Recipes> recipesArrayList) {
adapter = new RecipesAdapter(this, recipesArrayList);
adapter.setClickListener(this);
mRecyclerRecipes.setAdapter(adapter);
mEditSearchRecipe.setVisibility(View.VISIBLE);
}adapter
public class RecipesAdapter extends RecyclerView.Adapter<RecipesAdapter.ViewHolder> {
private ArrayList<Recipes> mRecipeList;
private LayoutInflater mInflater;
private RecipeClickListener mClickListener;
private String mIngrList;
// data is passed into the constructor
public RecipesAdapter(Context context, ArrayList<Recipes> data) {
this.mInflater = LayoutInflater.from(context);
this.mRecipeList = data;
}
// inflates the cell layout from xml when needed
@Override
@NonNull
public ViewHolder onCreateViewHolder(@NonNull ViewGroup parent, int viewType) {
View view = mInflater.inflate(R.layout.each_recipe_design, parent, false);
return new ViewHolder(view);
}
// binds the data to the TextView in each cell
@Override
public void onBindViewHolder(@NonNull ViewHolder holder, int position) {
Recipes recipes = mRecipeList.get(position);
holder.mTextCount.setText(""+String.valueOf(position+1));
Random rnd = new Random();
int color = Color.argb(255, rnd.nextInt(256), rnd.nextInt(256), rnd.nextInt(256));
holder.mCardViewRecipe.setCardBackgroundColor(color);
holder.mTextRecipeName.setText(""+recipes.getRecipeName().substring(0, 1).toUpperCase() + recipes.getRecipeName().substring(1).toLowerCase());
// holder.mTextIngrList.setText(mIngrList);
holder.mTextIngr1.setText(""+recipes.getRecipeIngredient().get(1).getIngr());
holder.mTextIngr2.setText(""+recipes.getRecipeIngredient().get(2).getIngr());
holder.mTextIngr3.setText(""+recipes.getRecipeIngredient().get(3).getIngr());
mIngrList = "";
if (recipes.getRecipeServings() != null) {
holder.mTextServing.setText("Servings: "+String.valueOf(recipes.getRecipeServings()));
}else {
holder.mTextServing.setText("Data Not found!");
}
if (recipes.getRecipePreptime() != null) {
holder.mTextPrepTime.setText(recipes.getRecipePreptime());
}else {
holder.mTextPrepTime.setText("Data Not found!");
}
if (recipes.getRecipeCalories() != null) {
holder.mTextCalories.setText("Calories: "+String.valueOf(recipes.getRecipeCalories())+" cals");
}else {
holder.mTextCalories.setText("Data Not found!");
}
holder.mTextIngrFirstChr.setText(""+recipes.getRecipeName().substring(0, 1).toUpperCase());
}
public void filterList(ArrayList<Recipes> filterdNames) {
this.mRecipeList = filterdNames;
notifyDataSetChanged();
}
// total number of cells
@Override
public int getItemCount() {
return mRecipeList.size();
}
// stores and recycles views as they are scrolled off screen
public class ViewHolder extends RecyclerView.ViewHolder implements View.OnClickListener {
TextView mTextRecipeName, mTextPrepTime, mTextIngrFirstChr, mTextServing, mTextCalories, mTextCount;
TextView mTextIngr1,mTextIngr2,mTextIngr3;
CardView mCardViewRecipe;
ViewHolder(View itemView) {
super(itemView);
mTextRecipeName = itemView.findViewById(R.id.tv_recipe_name);
mTextIngrFirstChr = itemView.findViewById(R.id.tv_ingr_name);
// mTextIngrList = itemView.findViewById(R.id.tv_recipe_ingr_list);
mTextServing = itemView.findViewById(R.id.tv_recipe_ingr_serv);
mTextPrepTime = itemView.findViewById(R.id.tv_recipe_prep_time);
mTextCalories = itemView.findViewById(R.id.tv_recipe_calories);
mTextCount = itemView.findViewById(R.id.tv_count);
mTextIngr1 = itemView.findViewById(R.id.tv_ingr_1);
mTextIngr2 = itemView.findViewById(R.id.tv_ingr_2);
mTextIngr3 = itemView.findViewById(R.id.tv_ingr_3);
mCardViewRecipe = itemView.findViewById(R.id.card_view_recipe);
itemView.setOnClickListener(this);
}
@Override
public void onClick(View view) {
if (mClickListener != null) mClickListener.onRecipeClick(view, getAdapterPosition());
}
}
// convenience method for getting data at click position
Recipes getItem(int id) {
return mRecipeList.get(id);
}
// allows clicks events to be caught
public void setClickListener(RecipeClickListener itemClickListener) {
this.mClickListener = itemClickListener;
}
// parent activity will implement this method to respond to click events
public interface RecipeClickListener {
void onRecipeClick(View view, int position);
}
public void clear() {
int size = mRecipeList.size();
mRecipeList.clear();
notifyItemRangeRemoved(0, size);
}
}This is how we can populate the recipe list in the application.
This is a list of recipes I found when I fetched ingredient apple.
We can search for relevant recipes that we want from the search result.
If we aren’t getting the expected results, always we can use update ingredients to modify our search.
As this application has done developing we goona deploy it into the Android play store.
Deploying App into Playstore
For deploying the app first we have to create signed apk and rest you can follow from here is a youtube video.
What’s Next?
- To improve this application most certain way is to add more ingredients.
- To improve the accuracy of the model.
- Adding more recipes, as we have only Indian recipes in future I’ll add more recipes with
Please reach me for any queries.
Thanks for reading.
CREDITS & REFERENCES
- Vegetable ingredients classification.
https://www.ijraset.com/fileserve.php?FID=16219
2. A comprehensive approach.
3. For all recipes
4. Google Cloud for Serverless functions
