Different dropout in Tensorflow

Dropout is a regularization technique for reducing overfitting in neural networks by preventing complex co-adaptations on training data. It is a very efficient way of performing model averaging with neural networks.

neural-networks-in-the-wild-handwriting-recognition-29-638

Why Dropout: Dropout helps prevent weights from converging to identical positions. It does this by randomly turning nodes off when forward propagating.

In simple terms some of your neurons will not participate in the calculation.

In Tensorflow we have two dropout functions.

  • tf.nn.dropout
    • It has parameter  keep_prop which state the probability of neuron which will not be drop. If we dive keep_prop value to 0.6, it means 60% of neurons will remain and 40% will be drop.
  • tf.layers.dropout
    • It has two main parameters, rate, and training. Rate means the number of neurons which will be drop.If the rate is 0.6 then 60% neurons will be dropped and 40% will be used.
    • We can see keep_prop = 1- rate.
    • Training parameter is to differentiate if the network is running for training or to get the result.We need dropout only when we are training the neural network, not while we are testing(inferencing) it.

So what is the difference between these two functions?

tf.layers.dropout is a wrapper over the tf.nn.droput, which give us the demarcation of whether to return the output in training mode (apply dropout) or in inference mode (return the input untouched).”

 

What Yagni Is?

Often developer debates a lot about what YAGNI is actually. Writing any new piece of code bind developer’s hand in the name of YAGNI.

I have worked in both product base company and service based company.I personally feel that YAGNI is more appreciated in the service based company, as in product based any extra feature is helpful.

People back YAGNI by saying “You don’t wipe before you shit.”.So true.We should not increase the scope of the problem as it may open the gates of new bugs.So YAGNI tells us “don’t write any extra/new code unless it is actually required”.

I agree with the last quoted statement.But should we bind our hands in implementing the requirement in a fashion that it should always be open for extensibility, or to write any functionality in a more generic way so that it can be reused any time in near future?

So, what my definition of YAGNI is: – Always write the code in the scope of the current requirement, but don’t cry out YAGNI if you want to solve it in the generic or extensible way.The code should close for modification but open for extensibility.

Try to solve problems in generic ways, that can be used in future.

Difference Between Generative and Discriminative machine learning

To understand these two models we first have to see what is the difference between joint probability [P(x,y)] and conditional probability[P(x|y)].

Joint probability:  p(A and B).  The probability of event A and event B occurring.  It is the probability of the intersection of two or more events.  The probability of the intersection of A and B may be written p(A ∩ B). Example:  the probability that a card is a four and red =p(four and red) = 2/52=1/26.  (There are two red fours in a deck of 52, the 4 of hearts and the 4 of diamonds).

Conditional probability:  p(A|B) is the probability of event An occurring, given that event B occurs. Example:  given that you drew a red card, what’s the probability that it’s a four (p(four|red))=2/26=1/13.  So out of the 26 red cards (given a red card), there are two fours so 2/26=1/13.

For better understanding, click here for more on probability.

generative algorithm models how the data was generated in order to categorize a signal. It asks the question: based on my generation assumptions, which category is most likely to generate this signal? Let’s say you have input data x and you want to classify the data into labels y. A generative model learns the joint probability distribution p(x,y). A generative algorithm models how the data was “generated”, so you ask it “what’s the likelihood this or that class generated this instance?” and pick the one with the better probability.

discriminative algorithm does not care about how the data was generated, it simply categorizes a given signal. Discriminative model learns the conditional probability distribution p(y|x) – which you should read as the probability of y given x. A discriminative algorithm uses the data to create a decision boundary, so you ask it “what side of the decision boundary is this instance on?

The fundamental difference between discriminative models and generative models is:

  • Discriminative models learn the (hard or soft) boundary between classes
  • Generative models model the distribution of individual classes

Given input data point x, the aim is to predict continuous (regression) or discrete (classification) output. That is given x, we are interested in modeling p(y|x). There are three approaches to this:

1. Generative Models:
One way is to model p(x, y) directly. Once we do that, we can obtain p(y|x) by simply conditioning on x. And we can then use decision theory to determine class membership i.e. we can use loss matrix, etc. to determine which class the point belongs to (such an assignment would minimize the expected loss). For e.g. in Naive Bayes model, you can learn p(y), the prior class probabilities from the data. You can also learn p(x|y) from the data using said maximum likelihood estimation (or you can Bayes estimator if you will). Once you have p(y) and p(x|y), p(x, y) is not difficult to find out.

2. Discriminative Models:
Instead of modeling p(x, y), we can directly model p(y|x), for e.g. in logistic regression p(y|x) is assumed to be of the form 1 / (1 + exp(-sigma(wi. xi))). All we have to do in such a case is to learn weights that would minimize the squared loss.

Generative models often outperform discriminative models on smaller datasets because their generative assumptions place some structure on your model that prevent overfitting. For example, let’s consider Naive Bayes vs. Logistic Regression. The Naive Bayes assumption is of course rarely satisfied, so logistic regression will tend to outperform Naive Bayes as your dataset grows (since it can capture dependencies that Naive Bayes can’t). But when you only have a small data set, logistic regression might pick up on spurious patterns that don’t really exist, so the Naive Bayes acts as a kind of regularizer on your model that prevents overfitting. There’s a paper by Andrew Ng and Michael Jordan on discriminative vs. generative classifiers that talks about this more.

Whenever an algorithm involves assuming, calculating or estimating the distribution of Y, it is generative, or simply put, if the algorithm cares about the distribution of Y, it is generative, if not, then it is discriminative.

Now a Small story to tell your 12-year-old kid, so that they can also understand the difference between these two models

Let’s say you have two kids “Gen” and “Dis”, and since their birth, they never opened their eyes. Today is the first day they will open their eyes, and you want to celebrate this occasion by teaching them the difference between Cat and Dog. You take them to pet store nearby.

Before showing around, you tell Gen and Dis to pay special attention to color, size, eye color, fur size, their voice etc.(feature set) of the pets they are going to see. After the end of this visit, you want to check if they understood the difference between cat and dog.

Now you give two photos one of a cat and one of a dog to Dis and ask which one is which. Dis has meticulously written down several conditions like if the voice sounds like meow and eyes are blue or green and has stripes with color brown or black then the animal is a cat. Thanks to her relatively simple rules, she quickly detected which one is a cat and which one is a dog.

Now instead of giving two photos you gave Gen two pieces of blank paper and ask her to draw what a cat and a dog looks like.

Well now, given any photo Gen can also tell which one is cat and which one is dog based on the drawing she created. In most cases drawing of cat and dog was unnecessary and time consuming for the task of detection which one is a cat.

But if there were only a few dogs and cats to look for Gen and Dis (low training data). In such cases if you give a photo of a brown dog with stripes with blue eyes, there is a chance that Dis would mark it as a cat. While Gen has her drawing and she can better detect that this photo is of a dog.

If you ask Gen to pay attention to more things(features), it will create a better sketch. But, if you show more examples(data-set) of cat and dog, Dis would mostly be better than Gen.

Since Dis is very meticulous in her observations if you ask her to pay attention to more features it will create more complicated rules(overfitting) and the chance of wrongly identifying a cat and a dog will increase, but that would not happen easily with Gen.

What if before going to pet store I don’t tell them that there are only two types of animal(no labeled data). Dis would fail completely because she will not know what to look for while Gen would be able to draw the sketch anyway. This is a huge advantage sometimes(semi-supervised).

Now let me reveal the suspense which you might already know: Dis is for discriminative and Gen is for generative.

Continue reading

Puppet – Introduction for Beginners

In our tech world, every day new tools and framework emerge to help us in our live.One of them is Puppet.A Configuration Management tool. This blog will be for who are trying to explore any configuration management tool or someone who is hearing a name puppet for the first time.Lets us explore what Puppet is actually. Why we need it, and how it works.

What is Configuration Management Tool?

Configuration management (CM) is a systems engineering process for establishing and maintaining consistency of a product’s performance, functional, and physical attributes with its requirements, design, and operational information throughout its life.

In simple term, if you have 100s of the production server and you have to upgrade the OS version in all the server or you have to install new software on all systems.Any configuration change required. Basically, your system admin helps you with this.

For more on CM click here.

Puppet Intro

Puppet is a pioneering configuration automation and deployment orchestration solution for distributed apps and infrastructure.

This open source configuration management solution is built with Ruby and offers custom Domain Specific Language (DSL) and Embedded Ruby (ERB) templates to create custom Puppet language files and offers a declarative paradigm programming approach. Puppet uses an agent/master architecture—Agents manage nodes and request relevant info from masters that control configuration info.

The Puppet Enterprise product offers the following capabilities:

  • Orchestration
  • Automated provisioning
  • Configuration Automation
  • Visualization and reporting
  • Code management
  • Node management
  • Role-based access control

Pros:

  • Strong compliance automation and reporting tools.
  • Active community support around development tools and cookbooks.
  • Intuitive web UI to take care of many tasks, including reporting and real-time node management.
  • Robust, native capability to work with shell-level constructs.
  • Initial setup is smooth and supports a variety of OSs.
  • Particularly useful, stable and mature solution for large enterprises with adequate DevOps skill resources to manage a heterogeneous infrastructure.

Cons:

  • Can be difficult for new users who must learn Puppet DSL or Ruby, as advanced tasks usually require input from CLI.
  • Installation process lacks adequate error reporting capabilities.
  • Not the best solution available to scale deployments. The DSL code can grow large and complicated at a higher scale.
  • Using multiple masters complicates the management process. Remote execution can become challenging.
  • Support is more focused toward Puppet DSL over pure Ruby versions.
  • Lacks push system, so no immediate action on changes. The pulling process follows a specified schedule for tasks.

How Puppet Works

[embedyt]https://www.youtube.com/watch?v=lxJQX2ipliY[/embedyt]

Puppet works on the master-slave relationship. There will be master which handle all the changes and logging.Every client/machine have the puppet agent which are slaves.

In general, there is two type of master-slave relationship.Pull based and push-based architecture.Below are the images which will explain this architecture.

In Push configuration, the centralized server will push the changes or any action to all the nodes, but in pull based configuration node will as for new changes, and then get the changes from the centralized server.

Puppet is written in Ruby. It is available in enterprise version also.From version 2.0 it is available under Apache License.

Master server contains the manifest file, where with the Puppet declarative language/Ruby  DSL we have to write the task we have have to perform.

There are resources[services, packages etc] in Puppet, we have to define these resources in the manifest .Group of the resource is called as the class to logically combined the resources.

We often have multiple modules to logically group the manifest file.

Puppet has multiple masters to handle the failure conditions.All the agents have to sign the certificate and all change of information is over SSH , so every exchange is fully protected and authenticated.

Below is the video from Edureka , which explain and demonstrate how it works.

Alternatives of Puppet

  • Chef – Push Based master-slave architecture
  • Ansible – Push Based master-slave architecture
  • Salt Stack – Pull Based master-slave architecture

MicroService – Brief Introduction

This post will focus on what Microservices are, why is it so famous these days, what are the positive and negative aspects of these services and what all area we will try to cover in future posts.

In addition, will share few youtube links, which are quite helpful to understand this concept. As a developer, I will say it is just one of the way how well we are packaging or modularizing code.

First of all, I would like to tell you, that I am inclined toward MicroServices, so most the things you find here will kind be in favor of these.But I will discuss all the challenges which you might face when you try to follow the awesome journey of MicroServices.

What is Micro Service?

Rather than going for a definition we will try to find the what are the common characteristics of Mirco Services.

In simple terms, you can think of a very small independent project capable of performing all the task.So If in a monolith you have various responsibilities, let each of this responsibility as a separate service.What responsibility does not have any concrete boundary. So this lets to the rise of the new question, how big or how small the micro services should be.Some people says it should be small enough to be handled by one developer, some say it should not be more than then few hundred line of code.To solve this, we will term service as a micro service if they have few of the properties/characteristics mention below.

Characteristics of MicroServices

  • Can be upgraded or rewrite independently.
  • Have fault tolerance and monitoring mechanism.
  • Each service is a complete Product.
  • Should have their own Data Management.
  • Should be easily replacable.
  • Should only expose the endpoint to dependendant services.

 

What a fuss is this, and Why is it becoming so popular these days?

As we all know that it is simple to solve number of small problems and then join all of them to solve the bigger problem.What we call is devide and conqure. Only problem in this approch is we need a very good merger technique to have a successful solution.With Devops of today(Docker,Kubernets,Mesos) it have been made possible to developer to manage large number of services and there deployment.Even it has helped to increase the resource utilization , which have further decreased the cost of maitaining multiple services.

Pros

  • It break the problem into smaller problem, helping developer to solve it more accuratley and in most optimized manner.
  • Partial Deployment and Partial Upgradation of Application is possible.
  • Help to reduce the develoment time.
  • We can easily rewite any service .
  • Services can be written in any programming languages , and we can easily try new progamming language.
  • Easier to find and fix bottleneck in the system.
  • Maintainance and Bug fix can be simper.
  • Test scope increases as we have to test smaller units independently.
  • CI/CD can be implemented easily.
  • Preserve modularity.

Cons

  • It increases the Devops activities.
  • Organization have to monotor and handle large number of sevices.
  • Service Discovery and trackng of request can be tidious task.
  • Need change in Oraganization culture.
  • Developer have to work more closely with devops , in order to make the development more stream line.
  • Need advance Devops.
  • Since it is kinda a new, not every one have clear idea what is a baoundary of MiroService.

Some Great Videos to give more insight.

 

Topics to be cover in upcoming blogs

  • Modules vs Microservices
  • How to share domain between different micro services
  • Should we have a common DAO[Database ] layer for every service or each service should have their own DAO layer?
  • All micro services in single Git Repo or they should have separate Git Repo
  • How to do monitoring of Micro Services.
  • How to track the flow of any particular request in real time between services.
  • Containerization of micro services.
  • Creating an environment of  Containerization service, which is self-deployed.
  • When to go for Micro Services.
  • How to shift from Monolith to Microservices.
  • How to perform Integration testing involving couple of Microservices as dependencies.
  • Distributed logging for Services.

Spring Boot + Docker + Bit Bucket + Automate Deployment

An idea behind this blog to walk through the auto build of docker image in docker hub for Spring boot application.

Note: – This method is not production level, so use it for staging or development phase only.

(Why it is not suitable for production ?)

[As we have to push the jar to the git repository, to reduce few steps.]

Technology Stack Used

  • Spring Boot [1.4.1 Release]
  • Java 8
  • Maven 2
  • Bit Bucket
  • Docker Hub

Prerequisite

  • Docker Hub Account
  • Bit Bucket Account
  • Minimum knowledge of Spring boot and Java
  • Link Bit Bucket to Docker Hub. CLICK HERE for the reference.

Steps

  1. Create Spring boot Application Capable of creating docker image
    • Create simple spring boot application with web module.[CLICK HERE to create using spring start.io or get it from BIT-BUCKET repo (master branch)].
    • Add Simple test controller to the Spring Project or check out to the branch controller-test.
    • Add Docker support to your current spring boot application or check out to branch docker-support. [To learn how to add docker support to spring boot application you check this blog.]
  2.  Create New Repository and link for auto creation of image
    1. Go to docker hub and click on create -> create automatic build
    2. Click on create Auto-build bit-bucket
    3. Select the desired bitbucket repository
    4. Fill required information and click on create.
    5. Go to build setting and provide the branch name from which image should be created and save changes.
    6. Click on build details, it will be empty right now
    7. Make Some changes in that particular branch and push it to the Origin.
    8. Now again check the build details tab, there will be new entry populated with status queued.
    9. Wait for some time its status will change to building than to success.
    10. Now your docker image is ready to use.
  3. Test created docker image
      1. Now pull the docker image
      2. Run the docker image with command [docker run –name docker.test -d -p 8088:8080 jainamit333/docker.demo:docker-support]
      3. Navigate to http://localhost:8088/docker/test.It will show the desired result.

Now each time user push to anything to branch, new docker image will be created.

Dockerize Spring Boot Application

This blog will help the audience to enable them to create docker image from a Spring boot Application.We have created a Spring Boot with maven.

We have already created the code repository. This repository contain 3 branches

1.Master: – Vanilla Spring Boot with Web Module

2.Controller-Test: – Master with test Rest repository.

3.Docker-Support :- Added docker support to Controller-Test branch

Prerequisite

  • Spring Boot Knowledge
  • How to create simple DockerFile

Here we will see what changes are required to add docker support to Spring Boot Application. It is the changes required to create Docker-Support branch from Controller-Test branch.

Changes Required in Pom

We need to add below mention plugin in the pom.

<plugin>
   <groupId>com.spotify</groupId>
   <artifactId>docker-maven-plugin</artifactId>
   <version>0.2.3</version>
   <configuration>
      <imageName>horizon/${project.artifactId}</imageName>
      <dockerDirectory>/</dockerDirectory>
      <resources>
         <resource>
            <targetPath>/</targetPath>
            <directory>/</directory>
            <include>${project.build.finalName}.jar</include>
         </resource>
      </resources>
   </configuration>
</plugin>

Add DockerFile in the project path

FROM frolvlad/alpine-oraclejdk8:slim
VOLUME /tmp
ADD target/docker-0.0.1.jar app.jar
RUN sh -c 'touch /app.jar'
ENTRYPOINT ["java","-Djava.security.egd=file:/dev/./urandom","-jar","/app.jar"]
EXPOSE 8080

There are two ways to create a docker image from the current source now.

1.Create docker image using maven plugin [can also be used using Jenkin worklow]

Run the command from project directory:- mvn clean package docker:build

2.Create docker image using docker build command

Run mvn install:- mvn clean install

Create docker image by executing the following command from project directory:

docker build  -t  docker.demo:latest  .