JDBC Client Sample Code

import java.sql.SQLException;

import java.sql.Connection;

import java.sql.ResultSet;

import java.sql.Statement;

import java.sql.DriverManager;

 

public class HiveJdbcClient {

  private static String driverName = "org.apache.hadoop.hive.jdbc.HiveDriver";

 

  /**

 * @param args

 * @throws SQLException

   */

  public static void main(String[] args) throws SQLException {

      try {

      Class.forName(driverName);

    } catch (ClassNotFoundException e) {

      // TODO Auto-generated catch block

      e.printStackTrace();

      System.exit(1);

    }

    Connection con = DriverManager.getConnection("jdbc:hive://localhost:10000/default", "", "");

    Statement stmt = con.createStatement();

    String tableName = "testHiveDriverTable";

    stmt.executeQuery("drop table " + tableName);

    ResultSet res = stmt.executeQuery("create table " + tableName + " (key int, value string)");

    // show tables

    String sql = "show tables '" + tableName + "'";

    System.out.println("Running: " + sql);

    res = stmt.executeQuery(sql);

    if (res.next()) {

      System.out.println(res.getString(1));

    }

    // describe table

    sql = "describe " + tableName;

    System.out.println("Running: " + sql);

    res = stmt.executeQuery(sql);

    while (res.next()) {

      System.out.println(res.getString(1) + "\t" + res.getString(2));

    }

 

    // load data into table

    // NOTE: filepath has to be local to the hive server

    // NOTE: /tmp/a.txt is a ctrl-A separated file with two fields per line

    String filepath = "/tmp/a.txt";

    sql = "load data local inpath '" + filepath + "' into table " + tableName;

    System.out.println("Running: " + sql);

    res = stmt.executeQuery(sql);

 

    // select * query

    sql = "select * from " + tableName;

    System.out.println("Running: " + sql);

    res = stmt.executeQuery(sql);

    while (res.next()) {

      System.out.println(String.valueOf(res.getInt(1)) + "\t" + res.getString(2));

    }

 

    // regular hive query

    sql = "select count(1) from " + tableName;

    System.out.println("Running: " + sql);

    res = stmt.executeQuery(sql);

    while (res.next()) {

      System.out.println(res.getString(1));

    }

  }

}










https://cwiki.apache.org/confluence/display/Hive/HiveClient

View System configuration - Ubuntu

How to see your system configuration

1. Open Terminal.
2. Type this command : sudo lshw -html > lshw.html
3. Go to Home folder
4. Open lshw.html file in browser

Thats it.... :)


sudo lshw -html > lshw.html will create a html file on your home folder with all configuration details.

About the book "Instant Apache Sqoop"

Instant Apache Sqoop 

Apache Sqoop(TM) is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases. Sqoop successfully graduated from the Incubator in March of 2012 and is now a Top-Level Apache project.

Developers may need to import data from Sql to Hadoop HDFS, Hive, HBase. Sqoop is the best tool for it. “Instant Apache Sqoop” is describing how to use Sqoop. “Instant Apache Sqoop” the title is accurate and self describing. Introduction itself is in a good and informative one. Even the layman can use Sqoop by using this book, the author “Ankit Jain ” wrote this book as simple. This book covers almost every apsects of Sqoop, import to HDFS, Hive, HBase and the exports as well.

This book is well illustrated especially in “How it works” is added in each and every part. This helped me alot to understand the back-stage things of Sqoop. Actually i am blind about the various connectors supporting by the Sqoop. This book helps me to find out these.

• MySQL
• Oracle
• SQL Server
• PostGre
• DB2
• HSQLDB

The important thing is I learned “Incremental Import”. Incremental import means importing the new version of records or the latest inserted records from the RDBMS table into HDFS . I think that is a very good option in Sqoop.

I can't find a Sqoop client  (Sqoop-Java client) in this book. I expect that also. But as the name implies it is Instant.

So my friends if you need a quick start on Sqoop, “Instant Apache Sqoop” will help you.

All codes used in this book are available at http://www.PacktPub.com . 

You can buy this book from : 

 http://www.packtpub.com/apache-sqoop-transfer-data-between-rdbms-hadoop/book

HTML File input tag for specific file type

FILE UPLOAD

Sometimes we may need to specify the exact file type for upload/select.
This is one way to handle that.

Using File Extension(Here .csv is using. You can use any extension like .jar, .ppt etc.)
<input type="file" name="pic" id="pic" accept=".csv" />

Using File Type
<input type="file" name="pic" id="pic" accept="text/plain" />
You can use this link to find out the file types. http://www.iana.org/assignments/media-types

For selecting video files only
<input type="file" name="pic" id="pic" accept="video/*" />


For selecting audio files only
<input type="file" name="pic" id="pic" accept="audio/*" />

For   Excel Files 2010 (.xlsx) 
<input type="file" accept="application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" />

Here is the Working Example : http://jsfiddle.net/LzLcZ/371/

Tomcat – Java.Lang.OutOfMemoryError

Apache Tomcat server may spit this error. To solve this error you need to update catalina.sh file.

1. Locate catalina.sh in bin directory. (Eg : apache-tomcat-7.0.22/bin/catalina.sh).
2. Edit it and append this line,


JAVA_OPTS="-Djava.awt.headless=true -Dfile.encoding=UTF-8 
-server -Xms1536m -Xmx1536m
-XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m 
-XX:MaxPermSize=256m -XX:+DisableExplicitGC"

3. Save file and restart the server.

Thats it.

Partial example of the catalina.sh file

#   JSSE_HOME       (Optional) May point at your Java Secure Sockets Extension
#                   (JSSE) installation, whose JAR files will be added to the
#                   system class path used to start Tomcat.
#
#   CATALINA_PID    (Optional) Path of the file which should contains the pid
#                   of catalina startup java process, when start (fork) is used
#
# $Id: catalina.sh 609438 2008-01-06 22:14:28Z markt $
# ----------------------------------------------------------------------------
 
JAVA_OPTS="-Djava.awt.headless=true -Dfile.encoding=UTF-8 -server -Xms1536m 
-Xmx1536m -XX:NewSize=256m -XX:MaxNewSize=256m -XX:PermSize=256m 
-XX:MaxPermSize=256m -XX:+DisableExplicitGC"
 
 
# OS specific support.  $var _must_ be set to either true or false.
cygwin=false
os400=false
darwin=false
case "`uname`" in
CYGWIN*) cygwin=true;;
OS400*) os400=true;;
Darwin*) darwin=true;;
esac
 
# resolve links - $0 may be a softlink
PRG="$0"

String array Intersect

String array Intersect 

import java.util.Arrays;
import java.util.LinkedHashSet;
import java.util.Set;
/**
 * @author devan
 * @date 03-Oct-2013
 * @email devanms@am.amrita.edu
 */

public class ArrayIntersect {
 public static void main(String[] args) {

  String[] strings1 = { "a", "b", "c", "f" };
  String[] strings2 = { "b", "c", "d", "f" };
  Set set = new LinkedHashSet(Arrays.asList(strings1));
  set.retainAll(Arrays.asList(strings2));
  System.out.println(set.toString());
 /* 
  String[] stringIntersect = set.toArray(new String[0]); //For storing the result in another array 
  System.out.println(Arrays.toString(stringIntersect));
 */
 
 }
}

Output
======
[b, c, f]

Get json object data without knowing the keys :)

Dynamic Json Object Keyset Finding

import java.util.Iterator;
import java.util.Set;

import org.json.simple.JSONObject;

public class Json {
public static void main(String[] args) {
	JSONObject jsonObject = new JSONObject();
	jsonObject.put("a", "aaa");
	jsonObject.put("b", "bbb");
	
    Set keys = jsonObject.keySet();
    Iterator a = keys.iterator();
    while(a.hasNext()) {
    	String key = (String)a.next();
        // loop to get the dynamic key
        String value = (String)jsonObject.get(key);
        System.out.print("key : "+key);
        System.out.println(" value :"+value);
    }
}
}

Sqoop Java Client

Sqoop Java Client

Apache Sqoop

Apache Sqoop(TM) is a tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases.

Sqoop successfully graduated from the Incubator in March of 2012 and is now a Top-Level Apache project: More information

Latest stable release is 1.4.4 (download, documentation). Latest cut of Sqoop2 is 1.99.2 (download, documentation).

Here is the Java Client for Apache Sqoop import data from MySql to Hadoop hdfs :)

//Here I am using a table Persons, with columns PersonID and LastName
import org.apache.sqoop.client.SqoopClient;
import org.apache.sqoop.model.MConnection;
import org.apache.sqoop.model.MConnectionForms;
import org.apache.sqoop.model.MJob;
import org.apache.sqoop.model.MJobForms;
import org.apache.sqoop.model.MSubmission;
import org.apache.sqoop.validation.Status;

/**
 * @author  devan
 * @date 19-Sep-2013
 * @mail msdevanms@gmail.com
 */

public class SqoopImport {
 public static void main(String[] args) {
  
  
  String connectionString = "jdbc:mysql://YourMysqlIP:3306/test";
  String username = "YourMysqUserName";
  String password = "YourMysqlPassword";
  String schemaName = "YourMysqlDB";
  String tableName = "Persons";
  String columns = "PersonID,LastName"; //comma seperated column names
  String partitionColumn = "PersonID";
  String outputDirectory = "/output/Persons";
  String url = "http://YourSqoopIP:12000/sqoop/";

  
  SqoopClient client = new SqoopClient(url);
  //client.setServerUrl(newUrl);
  //Dummy connection object
  MConnection newCon = client.newConnection(1);

  //Get connection and framework forms. Set name for connection
  MConnectionForms conForms = newCon.getConnectorPart();
  MConnectionForms frameworkForms = newCon.getFrameworkPart();
  newCon.setName("MyConnection");

  //Set connection forms values
  conForms.getStringInput("connection.connectionString").setValue(connectionString);
  conForms.getStringInput("connection.jdbcDriver").setValue("com.mysql.jdbc.Driver");
  conForms.getStringInput("connection.username").setValue(username);
  conForms.getStringInput("connection.password").setValue(password);

  frameworkForms.getIntegerInput("security.maxConnections").setValue(0);

  Status status  = client.createConnection(newCon);
  if(status.canProceed()) {
   System.out.println("Created. New Connection ID : " +newCon.getPersistenceId());
  } else {
   System.out.println("Check for status and forms error ");
  }

  //Creating dummy job object
  MJob newjob = client.newJob(newCon.getPersistenceId(), org.apache.sqoop.model.MJob.Type.IMPORT);
  MJobForms connectorForm = newjob.getConnectorPart();
  MJobForms frameworkForm = newjob.getFrameworkPart();

  newjob.setName("ImportJob");
  //Database configuration
  connectorForm.getStringInput("table.schemaName").setValue(schemaName);
  //Input either table name or sql
  connectorForm.getStringInput("table.tableName").setValue(tableName);
  //connectorForm.getStringInput("table.sql").setValue("select id,name from table where ${CONDITIONS}");
  
  
  connectorForm.getStringInput("table.columns").setValue(columns);
  connectorForm.getStringInput("table.partitionColumn").setValue(partitionColumn);
  
  //Set boundary value only if required
  //connectorForm.getStringInput("table.boundaryQuery").setValue("");

  //Output configurations
  frameworkForm.getEnumInput("output.storageType").setValue("HDFS");
  frameworkForm.getEnumInput("output.outputFormat").setValue("TEXT_FILE");//Other option: SEQUENCE_FILE / TEXT_FILE
  frameworkForm.getStringInput("output.outputDirectory").setValue(outputDirectory);
  //Job resources
  frameworkForm.getIntegerInput("throttling.extractors").setValue(1);
  frameworkForm.getIntegerInput("throttling.loaders").setValue(1);

  status = client.createJob(newjob);
  if(status.canProceed()) {
   System.out.println("New Job ID: "+ newjob.getPersistenceId());
  } else {
   System.out.println("Check for status and forms error ");
  }
  //Now Submit the Job
  MSubmission submission = client.startSubmission(newjob.getPersistenceId());
  System.out.println("Status : " + submission.getStatus());
 
 }

 
}

Save a webpage using JAVA

Save a webpage using JAVA

import java.io.InputStream;
import java.net.URI;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;


public class DownloadWebpage {
public static void main(String[] args) {
    Path path = Paths.get("PATH TO SAVE WEB PAGE"); // Eg: /home/devan/google
       URI u = URI.create("https://www.google.co.in/");
       try (InputStream in = u.toURL().openStream()) {
           Files.copy(in, path);
   } catch (Exception e) {
    e.printStackTrace();
   }
}
}

/sbin/start-stop-daemon: unable to open pidfile '/var/run/hive/hive-metastore.pid' for writing (No such file or directory)

unable to open pidfile '/var/run/hive/hive-metastore.pid'

Hi friends,

This error eat my 3 hours. Finally i fixed it.

The problem in my side was  the post install scripts didn't create directories /var/run/hive and /var/lock/subsys. Or it may removed :).

So i just created these directories manually.

sudo mkdir /var/run/hive
sudo mkdir /var/lock/subsys

Problem solved :)




Courtesy : https://cwiki.apache.org/BIGTOP/running-various-bigtop-components.html