AWS Lambda Layers for Pandas library


Written by:

what’s going on everyone and welcome to this video tutorial in this tutorial I will take you through on how to package panda’s library and deploy it as a lambda layer also we will install an additional package along with pandas library that is ideally required to read excel files so ideally I will do the subsequent video tutorial for reading and loading excel files within lambda function using s3 bucket program and of course lambda layers so here I am going to use ec2 instance to package panda’s library so I will jump to the ec2 management console and I will I will launch an instance with Ubuntu server I will select instance type as t2 micro within configure details I want to give permission for as three bucket to this ec2 instance so I will simply go ahead and create a rule and I will say create rule so here I will choose a service that is ec2 because we are creating this I enroll for éxito service within permission it’s going to be s3 plexus ideally I want the write permission next tags I’ll say name SADC to a three axis we will copy this next give it a roll name and say create role now once we’re done with creation of the road we’ll jump back to the launch console so we need to reload this role select ec2 as three access over here and then next add storage I’ll say 12gb tanks give it a name tag say Arnaz security group I will select an existing security group maybe floss with security and review and launch and launch here it is asking for keypad I have this keypad so it’s an on sentence so if you want to follow the tutorial on launching an instance then I have done that separately even are done two to three videos and lambda layers using ec2 instance so that’s the reason I took it a bit fast forward while launching an instance so if you want to refer that the links are in the description so Christians will take some time to know it’s running the copy and IP address terminal as a sketch right on I lost one dot p.m. Ubuntu has better than IP address again if you are not aware about performing and as as H within an ec2 instance then again I have did a tutorial on that you can find the link at the description below so since we are here we will get started with executing few commands so the first will be used to do I have to get update now one thing I want to highlight that we are packaging Hondas library for runtime Python three point seven and not three point six so ideally you went to 18.04 have Python three point six pre-installed so if I look at Python three – V so it says Python three point six point eight it is pre-installed in Ubuntu 18.04 correct so now we want to install Python three point seven or we want to upgrade from Python 3 point 6 to 3 points so well executed someone that is sudo I have got install Python 2.7 – why so now it’s installing Python 3.7 for us and it’s successfully installed now I want to check which one versions of Python that is exist within this instance so I will say LS user been pipe in – spring so as you can see we have Python 3.6 oh yeah right and python 3.7 so now we want to change the default version from 3.6 to Python 3 point 7 so here we will use update alternatives to update python 3.7 all right so let’s go ahead and run a command sudo update alternatives – – install so post install parameter it’s this part that is python 3 that is what we run that is Python 3 – me that that is gotta give us the Python version here right and then the Python actual path so that is going to be this one copy paste and then the final parameter is the priority so I will say 1 so what does it say install needs link me are we missing something sudo upgrade alternatives – – install user then Python 3 oh yes we are missing something so python 3 is followed by the Python three parameter so you can go ahead and look at the man pages of Ubuntu I’m done for about update alternatives but to give a high level definition so ideally update alternatives is used to create remove maintain and display information about symbolic links within the Debian alternative system right so that’s where here we are going to update the symbolic links from that are pointing to Python 3.6 and now what we want those symbolic links to point to python 3.7 right so that’s where we are using update alternatives in here so now we are done for the Python 3.6 now we have to perform same for Python 3.7 so I will simply change this parameter that is from 3.6 to 3.7 and here it will be 2 okay so now if I go ahead and check Python 3 – V that should be Python 3 point X right so whatever version is delayed it is so in case if you want to choose back for 3 point 6 point 1 for example then you have to execute a command that is sudo update alternations – – on three python 3 right so as you can see on my screen so we have a selection that we have the priority right so as you can see the selection is this one right now right and it’s asking to enter the current choice or type the selection number to update right but right now I don’t want to update back to python 3.6 i will give it as python 3.7 but this is how you can change back to Python 3.7 so we’ll put an enter and we are good to go now we are all sad I think so we will first create the directory structure so I will say mkdir you need to make sure that you don’t make a spelling mistake within this directory structure else it’s not going to work so make directory – P well Python followed by Lib – three point seven why python three point seven because we are packaging this pandas library for runtime Python 3 point 7 and not three point six so another thing I want to update you that for example if right now we have Python 3.6 version and we are building or packaging the pandas library for python 3.6 so I had read many comments where people say is that just change this path to to Python 3 point 7 and it will walk but ideal it’s not going to work for that you need python 3 point 7 version install Danone then only it’s going to work it’s not like manipulating the path will walk in both tarentines all right so that’s one so here – 3.7 followed by site packages and enter so here we have our belt introductory ok so now we want to install pandas and to deal with the excel file we will require excel already packaged along with pandas so we will execute a pity let’s check if v is installed or not so we don’t have Python package manager or p3 install so we will go ahead and install v 3 so that is pseudo after install Python 3 pip and it will install 3 4 this might take a while now once we are done with this installation we’ll go ahead and install Fonda’s and xlrd packages within the directory structure that we just created earlier alright so let’s wait and it’s good to go so now we help you frenched all let’s take so it’s that so clip 3 install pandas and Excel already for manipulating Excel files and then followed by – T parameter that is target parameter that we where we want to install so we want to install this in a directory that is build Python with Python 3.7 and site packings followed by – – system Herat and now it’s installing the packages within the site packages directory and we are good to go it’s a clear now let’s go to build directory and check whether it is installed or not so I’ll say CD – live – reference on inside packages and wellness so here we are so we have Excel already we have ponders and all the dependent cells that is required to implement or run find us correct so we are here now we are going to zip this package go to home directory then build and here we are going to the or to create the pandas and xlrd package so we will say this so let’s check whether it’s this is installed or not and it’s not so it says you do apt-get install accept yes and we help the punch tall now I will say sir – are and as underscore XLR T xlrd dots the followed by dot that means the route directed the current directory not route and we are good to go cool CLS and we have pandas and this flex allowed it on sir alright so now we are going to upload the zip file and they to the s3 bucket but for that we will require a double UCLA so for that we need to again install first database unit it is pseudo I get pinched all AWS CLI yes so if you remember then while launching an instance we have created an I enroll saying that you permission for read and write to the s3 bucket so this ec2 instance okay so now this instance have the permission have access to s3 bucket so once this is installed and it is we can directly run the come on and we’ll say a table is s3 copy what we want to copy that is pandas underscore lift not let me do Alice first yes so it’s hideous s3 copy pandas underscore excel or d dot sir followed by the s3 bucket Eura s3 colin to put forth slash so I have an s3 bucket that is a cognate so I am going to upload this package over that you can name on whichever bucket or wherever you want to upload this packet and I will say and so it’s uploaded successfully correct so now the next step is to go ahead and create a lambda layer so we’ll go to s3 bucket that is a cognate and check whether it is there or not so here we are that is a comment and what is the package that made it load this yeah here it is that is pandas and respond Excel already so now I will jump to lambda management console within that click on layers and say create layer name it that else I will said pandas hundreds not under school – xlrd description and I’m doing blank and he also like upload a file from s3 and copy the URL of our package file and I will put in here and choose the compatible runtime so it’s – 3.7 and I will say create now once this is created we will go ahead and create the lambda function so it’s created successfully now we’ll click on left panel click on functions and say create function and name it I will say finders – itself file trigger all right so that is what we are going to achieve something in the next tutorial and then select runtime as Python 3.7 choose or create an exist execution roll so I will say use an existing roll and I will select lambda on the school roll here and say create function so I’m not creating the role in here right so you have to go ahead and create a roll saying that give permission to s3 bucket right since we are going to set up a trigger so give the permission for s3 and I think lambda music execution alright so go ahead and create and bind it with the lambda function and we are here so now lick on layers click on add a layer select the layer and from your layers that is pandas xlrd select new version it’s going to be 100 and say add now once we are done with adding click on save and then we are going to teach to this function so we’ll jump back to and as excel file trigger and here I’m going to say import import and as speedy now so save it ideally this should walk totally okay so I’m not expecting an error over here so I said test create an event I will say Testament create and say test again and here we go so over pandas lair package is walking perfectly fine so that’s it for now in the next tutorial I will show you on how to read excel file in lambda function with s3 bucket trigger and as usual if you want me to do tutorial on any use case or service then please leave them below and I will try my best to come up with a tutorial as soon as possible and if you have any queries or comments then please leave them below and don’t forget to Like and subscribe to my channel and see you next time

2 Replies to “AWS Lambda Layers for Pandas library”

  1. Mohmadhaidar devjiyani says:

    Thanks Sir awesome video

  2. Rakesh Kumar says:

    Your clear explanation about Python installation which affects how pip will fetch binaries is perfect. Also, thanks for showing the entire process. I used SAM to create layers but that is irrelevant. Thanks.

Leave a Reply

Your email address will not be published. Required fields are marked *