Sunday, June 4, 2023

Free workspace for you - Powered by Google

 Hi Folks,


I am writing this new blog about Free Workspace powered by Google.

If you are a data engineer, data scientist, or big data enthusiast, this will help you.


I am talking about Google Colab



Colab, or "Colaboratory", allows you to write and execute Python in your browser, with

  • Zero configuration required
  • Access to GPUs free of charge
  • Easy sharing

Whether you're a student, a data scientist or an AI researcher, Colab can make your work easier. Watch Introduction to Colab to learn more


So happy collab https://colab.research.google.com/


Some samples : https://github.com/msdevanms/pyspark_training




Wednesday, July 8, 2020

AWS Lambda Python - Build from your local machine - Upload as zip file - boto3 - import modules - Linux - Ubuntu

Hi Folks,

Now everything is PaaSified :)

So today we are going to looking in to creating and deploying AWS Lambda - Python from ZERO.


Here we go...



Step1. Create a new AWS Lambda using AWS portal or AWS CLI

Step 2. Install Python and pip in your local machine

Step 3. Create a new folder in your local machine

Step4. create a new file YOUR_FILE_NAME.py in that folder

Step5. Do some coding in that file

Sample code
import json print('Loading function') def lambda_handler(event, context): print("This is from Lambda ") return "Lambda Executed Successfully !"

Step7. Save the file

Step8.  Install boto3 modules for making python function ready

pip install boto3 -t ./

Step9. Make a ZIP package with correct file permissions



cd YOUR-FOLDER-NAME
chmod -R 755 .
zip -r ../myDeploymentPackage.zip .

Step10. Upload myDeploymentPackage.zip in Lambda using AWS portal or AWS CLI


  1. Open the AWS Lambda console.
  2. Choose Functions on the navigation pane, and then open your function.
  3. In the Function code section, expand the Code entry type drop-down list, and then choose Upload a .ZIP file.
  4. Choose Upload, and then select your .zip file.
  5. Choose Save.
  6. Choose Test.
You can also upload your .zip file from the AWS Command Line Interface (AWS CLI) using the update-function-code command.


Step11.  Make sure that you have proper settings


Handler should set as YOUR_FILE_NAME.YOUR_HANDLER_METHOD_NAME

For example

If you saved your file as "lambda_function.py" and the handler method is  "lambda_handler(event, context)" you should set "lambda_function.lambda_handler" as Handler in settings. Otherwise you will get error.


Same like step8 you can install any other modules for your code.


You are all set... All the best ! Happy Coding...!!


Saturday, July 4, 2020

Eclipse taking lot of network and time to import maven projects ? Here is the simple trick

Hi Folks,

By default maven option will be on for downloading sources and docs. This will take more network and time to import maven projects.

So disable it.

Window > Preferences  > Maven
Disable Download Artifact Sources
Disable Download Artifact Javadoc

So the final settings of maven preferences will be like :



Thanks :)


Thursday, June 25, 2020

How to check a port is open or not in pod

In kubenetes cluster, sometimes we may force to debug. It may be related networking, checking open ports, API availability etc.


So here is the simplest way to do this,


1. Create another pod for debugging

kubectl run curl1 --image=radial/busyboxplus:curl -i --tty -n YOUR_NAMESPACE
This will create a new pod in the specified name space and it will enable a command line for typing any command supported by linux

If you don't see a command prompt, try pressing enter.
[ root@curl1-5955d6546b-p5pm2:/ ]$ 

telnet will be enabled by default 

So you can try to telnet to your service which is running in kubernetes cluster.


Wednesday, September 13, 2017

Enabling HPKP (Http Public Key Pinning) in Tomcat

The Public Key Pinning Extension for HTTP (HPKP) is a security feature that tells a web client to associate a specific cryptographic public key with a certain web server to decrease the risk of MITM attacks with forged certificates.

Unfortunately Apache Tomcat does not support HPKP filter. So we have to write our own 'Global filter', Valve.
First you need to extract the public key information from your certificate or key file and encode them using Base64.
The following commands will help you extract the Base64 encoded information from a key file, a certificate signing request, or a certificate.
openssl rsa -in my-rsa-key-file.key -outform der -pubout | openssl dgst -sha256 -binary | openssl enc -base64
openssl ec -in my-ecc-key-file.key -outform der -pubout | openssl dgst -sha256 -binary | openssl enc -base64
openssl req -in my-signing-request.csr -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
openssl x509 -in my-certificate.crt -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64
The following command will extract the Base64 encoded information for a website.
openssl s_client -servername www.example.com -connect www.example.com:443 | openssl x509 -pubkey -noout | openssl pkey -pubin -outform der | openssl dgst -sha256 -binary | openssl enc -base64

Example HPKP Header

Public-Key-Pins: 
  pin-sha256="cUPcTAZWKaASuYWhhneDttWpY3oBAkE3h2+soZS7sWs="; 
  pin-sha256="M8HztCzM3elUxkcjR2S5P4hhyBNf6lHkmjAHKhpGPWE="; 
  max-age=5184000; includeSubDomains; 
  report-uri="https://www.example.org/hpkp-report"
In this example, pin-sha256="cUPcTAZWKaASuYWhhneDttWpY3oBAkE3h2+soZS7sWs=" pins the server's public key used in production. The second pin declaration pin-sha256="M8HztCzM3elUxkcjR2S5P4hhyBNf6lHkmjAHKhpGPWE=" also pins the backup key. max-age=5184000 tells the client to store this information for two months, which is a reasonable time limit according to the IETF RFC. This key pinning is also valid for all subdomains, which is told by the includeSubDomains declaration. Finally, report-uri="https://www.example.net/hpkp-report" explains where to report pin validation failures.
Now create a Valve as described below.
  1. Create a Maven Java Application.
  2. Add the following dependency:
<!-- https://mvnrepository.com/artifact/org.apache.tomcat/catalina -->
<dependency>
<groupId>org.apache.tomcat</groupId>
<artifactId>catalina</artifactId>
<version>6.0.53</version>
<scope>provided</scope>
</dependency>
  1. Create your Java class and extend it from ValveBase.
  2. Implement the invoke(Request, Response) method.
  3. import org.apache.catalina.valves.ValveBase;
    
    public class GlobalFilterValve extends ValveBase {
    
        @Override
        public void invoke(Request request, Response response) throws IOException, ServletException {
     response.setHeader("Public-Key-Pins", "pin-sha256=\"cUPcTAZWKaASuYWhhneDttWpY3oBAkE3h2+soZS7sWs=\"; pin-sha256=\"M8HztCzM3elUxkcjR2S5P4hhyBNf6lHkmjAHKhpGPWE=\"; max-age=5184000; includeSubDomains");                getNext().invoke(request, response);
        }
    
    }
  4. Build your library (.jar) file
  5. Copy the jar file in the ${tomcat.home}/lib directory.
  6. Configure the server.xml to use your new valve. For example:
<valve className="org.devan.GlobalFilterValve"/>
  1. Start the server to see your new valve in action


Thank you !


Global custom filter in Apache Tomcat


We are aware about custom filters and built in filters available in Apache Tomcat.
If we are writing a custom java filter in our project, it will be look like follows,

@WebFilter("/*")
public class MyFilter implements Filter {

    @Override
    public void doFilter(ServletRequest request, ServletResponse response, 
                         FilterChain chain) throws IOException, ServletException {

        HttpServletResponse httpResponse = (HttpServletResponse) response;
        //your custom filter or set custom response header here
        chain.doFilter(request, response);
    }
}
This custom filter can help us to track and filter all requests coming to our webappliation, but it can not act like a global filter for all webapplications hosted in Apache Tomcat.

Apache Tomcat supports custom Valve elements which can help us to make this happen.

Valve element represents a component that will be inserted into the request processing pipeline for the associated Catalina container (EngineHost, or Context)

Following are the steps to implement a custom Valve in Apache Tomcat:



  1. Create a Maven Java Application.
  2. Add the following dependency:
<!-- https://mvnrepository.com/artifact/org.apache.tomcat/catalina --><dependency> <groupId>org.apache.tomcat</groupId> <artifactId>catalina</artifactId> <version>6.0.53</version> <scope>provided</scope></dependency>
  1. Create your Java class and extend it from ValveBase.
  2. Implement the invoke(Request, Response) method.
  3. import org.apache.catalina.valves.ValveBase;
    
    public class GlobalFilterValve extends ValveBase {
    
        @Override
        public void invoke(Request request, Response response) throws IOException, ServletException {
            //Do your filter/setting header in response here
            getNext().invoke(request, response);
        }
    
    }
  4. Build your library (.jar) file
  5. Copy the jar file in the ${tomcat.home}/lib directory.
  6. Configure the server.xml to use your new valve. For example:
<valve className="org.devan.MyValve"/>
  1. Start the server to see your new valve in action


Thank you !!!