Getting Started with Apache Spark: Find maximum commits by an author in a git log file 10:20 PM

- Install sbt. (scala build tool)
- Install apache-spark.
- Go to the unzipped apache-spark directory and in command line run
sbt assembly

(this takes a while, one may have to increase the memory allocated to run this in config file)

- Clone some git project
  git clone https://github.com/apache/groovy

- Save the log into a text file
  git log > C:\\temp\\log.txt

- Launch spark terminal and execute :

scala> val file = sc.textFile("C:\\temp\\log.txt")
file: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[21] at textFile at <console>:27

scala> val authorLines = file.filter(line => line.contains("Author"))
authorLines: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[22] at filter at <console>:29

scala> var maxAuthorTuple = authorLines.countByValue().maxBy(_._2)
maxAuthorTuple: (String, Long) = (Author: Paul King <paulk@asert.com.au>,2991)

- Verify that maxAuthorTuple has the author who made maximum commits in that branch with the      number of commits.

Tweet

Basic Authentication using Groovy 2:48 AM

Here is a very minimal groovy based Basic Authentication Client which can come handy whilst talking to APIs.

def jsonBody = '{"day" : "11", "month" : "03", "year" : "2010", "hour" : "09", "min" : "08", "lat" : "18.9750", "lon" : "72.8258", "tzone" : "5.5", "gender" : "male"}'

new BasicAuthClient(
              userName:"xyzzy",
              password:"ba4d90e133ad76b103fcedcd00ab5681", 
              address:"https://api.vedicrishiastro.com/v1/basic_panchang/"
).doPost(jsonBody)

class BasicAuthClient{

    def userName, password, address, conn
    
    def getAuthHeader(){
        def authHeader = "$userName:$password".getBytes().encodeBase64().toString()
        println(authHeader)
        return authHeader
    }
    
    def getConnection(){
        this.conn = address.toURL().openConnection()
        conn.setDoOutput(true)
        conn.setRequestMethod("POST")
        conn.setRequestProperty("Authorization", "Basic ${getAuthHeader()}")
        return conn
    }
    
    def doPost(body){
        def out = getConnection().getOutputStream(); 
        out.write(body.getBytes()) 
        out.close();
        return getResponse()
    }
    
    def getResponse(){
        def responseCode = conn.getResponseCode();
        println(responseCode)
        def instream = responseCode < 207 ? conn.getInputStream() : conn.getErrorStream()        
        int i = instream.read()
        while (i != -1) {
            print((char)i)
            i = instream.read()
        } 
        instream.close()
        conn.disconnect()
    }
}      


    

Tweet

ConEmu for developer productivity 11:57 PM

For the last one and half years, I have been a loyal user of Console 2, an alternative to command prompt on windows. Now, I have moved on to ConEmu for better productivity at workplace. It comes with many settings for customization. What I really love in ConEmu is its split screen support. At times when one is even lazy to shift between tabs/windows. In conjunction with vim, it is a breeze.

Say, you have an executable jar that also produces a log file. To hell with notepad and other UI editors - You could simply split your ConEmu screen to open up two or more console windows and spare one to skim through the output log file (using vim) simultaneously as and when you run the executable through an other.

The only caveat that I see with conEmu is that it eats up a tad bit more of memory than Console2 does (around 40 MB). But I guess I can live with that considering the fact that my laptop is powered with 16 GB of RAM. :)

Tweet

VIM - first things first 2:49 AM

Install theme :

https://benaiah41.wordpress.com/2012/01/17/customizing-gvim-in-windows-7/

Append the following lines in ../.vimrc (Startup Settings)

set guifont=Consolas:h11:cDEFAULT
color wombat
set guioptions-=T
set nu
let g:netrw_liststyle=3

Tweet

Command Pattern in java/groovy 11:14 PM

Command pattern encapsulates a request. A command pattern maybe used whenever there is a Sender, Receiver who communicate via a request (let's call it command). Assume, we simulate the game of cricket via a Simulator class which sends commands  to the Batsman objects to play certain shots. Each shot is a command given by the simulator to a batsman.

interface Shot{
public void execute();
}

class CoverDrive implements Shot{

private Batsman batter;

public void execute(){
batter.playCoverDrive()
}
}

class straightDrive implements Shot{

private Batsman batter;

public void execute(){
batter.playStraightDrive();
}
}

class Batsman{

def playCoverDrive(){
println("Cover drive")
}

def playStraightDrive(){
println("Cover drive")
}
}

class BatSimulator{

Shot shot

def play(){
shot.execute()
}
}

// main - groovy script start

def sachin = new Batsman()
def command = new CoverDrive(batter : sachin)
def simulator = new BatSimulator(shot : command)
simulator.play()

Tweet

Write a custom event handler in groovy/java 6:31 AM

One may write a couple of classes and an interface to achieve event handling without having to implement the Observable/Observer interface. Assume, we have a computer that runs several processes. If it receives a shutdown command, we fire an event to let the running process handle its own clean up method to release resources and halt execution.

interface ShutdownListener{
    public void handle();
}

public class Computer{

    private def listeners = []

    def addShutdownListener(ShutdownListener l){
listeners.add(l)
    }

    def shutdown(){
for(ShutdownListener l : listeners)
l.handle()
    }

    def compute(){ // whatever  }

}

public class Process implements ShutdownListener{

    public void handle(){
// release resources of "this" Process 
// halt execution of self
println("Process ended.");
    }
}

//groovy script start 
def computer = new Computer()
def process = new Process()
computer.addShutdownListener(process)
computer.compute()
computer.shutdown()

Tweet

Simple way to implement find(select) and collect block closures in java 1:47 AM

Languages like ruby, groovy, smalltalk have language constructs for blocks and closures. Java 8 now has feature to evaluate lambda expressions. But here is one simple and straight way to implement the same:

In groovy :

[1, 2, 3, 10, 20, 30].findAll { num -> num > 5 }
=> yields a new list [10, 20, 30]

[ 1, 2, 3, 4 ].collect{ num -> num * num}
=> yields a new list [1, 4, 9, 16]

The above code for those more familiar with the smalltalk syntax will be:

#(1 2 3 10 20 30) select: [ :num | num > 5 ]
==> displays [10, 20, 30]

#( 1 2 3 4 ) collect: [:i | i * i ].
==> displays [1, 4, 9, 16]

In java, the same could be implemented with a set of abstract classes as a collection utility that mandate implementation of method which returns the boolean condition to add or transform the element under iteration :

Find.java

abstract class Find extends ArrayList{ 

    /** let the default constructor worry about iteration **/

    public Find(List list){    

        Iterator it = list.iterator();

        while(it.hasNext()){

            Object element = it.next();        

            if(all(element))           

                this.add(element);         

        }

    }

     

    /** This defines the boolean condition for find/select **/

    public abstract Boolean all(Object obj);

     

}

FindDemo.java

// find all elements in the list that are greater than 5

List list = Arrays.asList(1, 2, 3, 10, 20, 30);

selected = new Find(list) {

    public Boolean all(Object num) {               

        return (int) num > 5;

    }

};

// => yields a new list [10, 20, 30]

Collect.java

abstract class Collect extends ArrayList{  

    /** let the default constructor worry about iteration **/

    public Collect(List list){

        Iterator it = list.iterator();

        while(it.hasNext()){

            Object element = it.next();

            Object transformed = transform(element);

            this.add(transformed);

        }

    }

    /** This defines the transformation logic for collect **/

    public abstract Object transform(Object obj);

}

CollectDemo.java

// collect the squares of all elements in the list

List list = Arrays.asList(1, 2, 3, 4);

collected = new Collect(list) {    

    public Object transform(Object num) {

            return (int) num * (int) num;

    }

};

// => yields a new list [1, 4, 9, 16]

Tweet