Thursday, December 17, 2015

Tachyon Cluster Configuration Setup Manual


Tachyon is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks, such as Spark and MapReduce. It achieves high performance by leveraging lineage information and using memory aggressively. Tachyon caches working set files in memory, thereby avoiding going to disk to load datasets that are frequently read. This enables different jobs/queries and frameworks to access cached files at memory speed.

Tachyon Cluster Configuration Setup Manual

In Master Node

In Slaves


(This script needs to be run on each node you wish to configure.It will configure your workers to use 2/3 of the total memory on each worker.)

In Master Node


HDFS as underFS (Tachyon can run with different underlayer storage systems)

By default, Tachyon is set to use HDFS version 1.0.4. You can use another Hadoop version by changing the hadoop.version tag in pom.xml in Tachyon and recompiling it. You can also set the hadoop version when compiling with maven:

  • $ mvn -Dhadoop.version=2.2.0 clean package

After completing this,

  • Edit tachyon-env.sh file. And set TACHYON_UNDERFS_ADDRESS

TACHYON_UNDERFS_ADDRESS=hdfs://HDFS_HOSTNAME:HDFS_PORT.

Thats all
=======================================================================

Possible Errors :



For more : http://tachyon-project.org/documentation/v0.7.1/Running-Tachyon-on-a-Cluster.html

No comments:

Post a Comment

Total Pageviews

Translate